One hundred tables, one config you can't guess

The source database had more than a hundred tables, and the job was to replicate all of them, continuously, into Iceberg. That sounds like it should be a setting. It is not. It is a hand-written configuration, one key at a time, and the only reliable way to get it right is to copy a working setup from someone else and change the names.

That is the honest shape of the Debezium-to-Iceberg setup most teams end up with: an incantation. Get every key right and it runs for months. Get one wrong and it fails in a way the logs do not explain. This post is what we learned running it, what to do if you are running it today, and how Fabric replaced it.

The part nobody warns you about

A working source connector has to get several things right at once: table.include.list, naming every table; topic.prefix, which decides the topic names and must never change once offsets exist; slot.name; publication.autocreate.mode; snapshot.mode; the ExtractNewRecordState unwrap transform that turns change envelopes into flat rows; and the offset and schema-history stores that remember where the connector stopped. Debezium's default is one Kafka topic per table, so topic.prefix plus the schema and table name is the whole naming scheme. And tasks.max does nothing for PostgreSQL: the connector always uses a single task, so you cannot scale one source by adding tasks.

Then there is the sink. We generated a community-maintained Kafka Connect Iceberg sink connector, and its routing was per table. Every table needed its own route key so the sink could map a topic to an Iceberg table:

{
  "topics": "app.public.orders,app.public.customers,app.public.invoices",
  "iceberg.tables": "raw.public.orders,raw.public.customers,raw.public.invoices",
  "iceberg.tables.route-field": "_topic",
  "iceberg.table.raw.public.orders.route-regex": ".*\.public\.orders",
  "iceberg.table.raw.public.customers.route-regex": ".*\.public\.customers",
  "iceberg.table.raw.public.invoices.route-regex": ".*\.public\.invoices",
  "transforms": "addTopicField",
  "transforms.addTopicField.type": "org.apache.kafka.connect.transforms.InsertField$Value",
  "transforms.addTopicField.topic.field": "_topic",
  "errors.tolerance": "all"
}

Three tables, three route keys, plus matching entries in topics and iceberg.tables. At a hundred and fifty tables it is a hundred and fifty route keys, a topic list that has to stay in sync with iceberg.tables, and an InsertField transform writing the Kafka topic into _topic so routing has something to match. Adding one table meant editing the connector and restarting it.

The failures were not subtle, but they were not self-explaining either.

• Routing. When a topic had no matching route, the sink rejected its records with is not present in the configured table list. Nothing in the message said which of the hundred keys was missing.
• Silent duplicates. We found duplicated rows only by counting. A union across four raw tables returned 611,229 rows against 157,381 distinct ids.
• Schema files. One unchecked boolean on the MySQL side, store.only.captured.tables.ddl, let the connector record schemas for tables we never captured. Thousands of files accumulated.
• Snapshots. A table committing every 30 seconds under a 7-day retention policy reaches roughly 20,000 snapshots; metadata.json grows about 768 bytes per snapshot to 9–14 MB, and reading it on every commit exhausted the Debezium JVM heap. We dropped the policy to two days.

The Debezium-side fix

If you are running this today, all of it is solvable. The tax is that you find each fix by failing first.

• Scope the publication. publication.autocreate.mode=filtered builds the CREATE/ALTER PUBLICATION ... FOR TABLE ... statements from your include lists. At 20 tables it is tidy; at 500 it is a wall of DDL, but the publication stays in step with the config.
• Adding a table. There is no way around editing table.include.list and restarting. If schema.history.internal.store.only.captured.tables.ddl=true, Debezium documents a longer sequence: stop the connector, delete the schema history so it rebuilds, then recover with snapshot.mode or an incremental snapshot through the signals table. Their write-up on adding a table calls it one of the most common questions from the community.
• Kafka-less deployments. Debezium Server still needs offset and schema-history storage, just not Kafka's. Point debezium.source.offset.storage and debezium.source.schema.history.internal at file, JDBC or Redis stores:

debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
debezium.source.topic.prefix=app
debezium.source.table.include.list=public.orders,public.customers
debezium.source.slot.name=debezium_app
debezium.source.publication.autocreate.mode=filtered
debezium.source.snapshot.mode=initial
debezium.source.transforms=unwrap
debezium.source.transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
debezium.source.offset.storage=org.apache.kafka.connect.storage.FileOffsetBackingStore
debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.schema.history.internal=io.debezium.storage.file.history.FileSchemaHistory
debezium.source.schema.history.internal.file.filename=data/schema-history.dat
debezium.sink.type=iceberg

And whichever store you land on, the community-maintained Iceberg sink needs scheduled compaction; without it, reads degrade as small files pile up.

And it is not only the config: one container set per source

Debezium Server runs exactly one connector per instance. One pipeline per source therefore means one JVM per source. Before that, the Kafka Connect version of the same pipeline carried a broker, a schema registry and Connect workers just to move bytes between two systems.

Then there is the sink itself. It is a community-maintained project, separate from Debezium, with a single configuration page, and the docs have drifted from the code. The project's own example still uses database.server.name beside topic.prefix, and its upsert-dedup-column default disagrees with the page that documents it.

The public numbers point the same way. The Debezium Server image is roughly 0.6–0.9 GB compressed, depending on version. Debezium's own FAQ says the default 256 MB heap in the Connect image covers "less than 10,000" total columns; a hundred tables of a hundred columns is already at that ceiling. Those are Debezium's numbers, not ours.

How Fabric handles it

We ran Debezium Server Iceberg ourselves, built the config generation for it, and then replaced it. With Fabric, the configuration stops being a document you maintain and becomes something the runner derives.

• Connect a source and pick tables. Fabric reads the real catalog — every non-system schema — and lists the tables. You select with checkboxes or select-all and set per-table defaults for the destination name, ID columns, namespace and snapshot chunk size. There is no config file anywhere.
• It starts the runner for you. Point Fabric at a source and a destination and it provisions the pipeline, then automatically starts a supervised runner that replicates the selected tables from source to destination. There is no config file to write, and nothing to keep in sync by hand.
• One supervised process per source. No Kafka, no Kafka Connect and no Schema Registry in the data path. The same process carries the heartbeat that keeps an idle slot's WAL reclaimable, so a quiet database does not fill a disk.

Most teams cannot work out connector generation, and the honest position is that they should not have to. That friction belongs to the tool, not to the customer. What a data team should be spending its time on is the analytics, not the incantation that feeds it.

Or skip the incantation entirely: sign up, pick your tables, and the pipeline is running.

Keep reading

• Small files, big bills: the silent tax on every CDC pipeline
• Why your idle Postgres database is quietly running out of disk
• Debezium PostgreSQL connector documentation
• Debezium Server documentation
• memiiso/debezium-server-iceberg (community-maintained)