Why one blank string can stop your CDC pipeline

It does not take a corrupt page, a failed failover or a network partition to stop a change-data-capture pipeline. For one of our customers, it took one empty string.

A legacy application had been writing '' — an empty string — instead of NULL for missing values, in columns that hold UUIDs. Postgres was fine with that. The pipeline was not: when a record carrying one of those strings reached the sink, the converter that turns values into UUIDs refused it, and the connector stopped.

One row out of millions. Their recovery was a full re-snapshot of the source. Here is why the crash happens where it happens, what to do if you are running Debezium today, and how Fabric turns a bad row into a quarantine entry instead of an outage.

The crash is downstream of Debezium

Debezium's Postgres connector does not validate UUIDs. Values come out of the replication stream as strings and are passed along as such. The failure happens in the component that consumes the column as a real UUID — in this case, the Iceberg sink's converter, which parses the string with UUID.fromString(...):

java.lang.IllegalArgumentException: Invalid UUID string:

Notice what follows the colon: nothing. The offending value is an empty string, so the message has no value to quote, no column name and no table. All it tells you is that something, somewhere, was not a UUID.

What happens next is what makes this expensive. The row never processes, so its offset never commits. On restart, the connector replays from the last position it did commit — and hits the same row again. A single unprocessable row becomes a permanent crash loop: in a Kafka Connect deployment the sink task is killed and restarted, with Debezium Server the engine stops and comes back to the same wall. And the whole time, the replication slot keeps holding WAL, so the database's disk usage climbs as well.

You cannot skip your way out of it, either. A replication slot can only serve changes that happened after it was created, and Debezium's documentation is blunt about what happens when a connector is pushed past changes it never read: silent data loss. The honest recovery is stop, repair the source rows, reset the pipeline, and take a fresh snapshot.

Finding the rows is most of the work

The error tells you nothing about location, so you have to go looking. Two tools answered it, and together they cover both ends of the problem: the log, and the schema.

First, the WAL itself. We wrote a small slot inspector that decodes the replication stream without consuming the slot and flags every event whose UUID columns carry an empty string. It ends with the sentence we needed:

RESULT: Found events with empty strings in uuid-typed columns.
        Fix the source rows (UPDATE col = NULL WHERE col = '')
        then do a pipeline reset.

Second, the catalog. A management command scans every synced table for text columns that store UUID values and counts the empty strings in them, then prints a per-column fix list. Two statements do the actual remediation:

-- empty strings cannot be cast to uuid; fix them first
UPDATE "public"."users"
   SET "company_id" = NULL
 WHERE "company_id" = '';

ALTER TABLE "public"."users"
  ALTER COLUMN "company_id" TYPE uuid
  USING "company_id"::uuid;

If you are running Debezium today

The crash happens downstream of the source connector, so a fix has to happen before the value reaches the converter, or inside the converter itself. Three things are worth knowing, in the order we would try them.

• Rewrite the record in flight. Our config generator learned to inject a Debezium scripting transform for every column flagged as needing it, rewriting the value before the sink sees it:

debezium.transforms.null_company_id.type=io.debezium.transforms.scripting.ScriptingTransformation
debezium.transforms.null_company_id.language=groovy
debezium.transforms.null_company_id.expression=value.get('company_id')?.trim() == '' ? value.put('company_id', null) : value

Three generated lines per affected column, ordered into the transform chain after the routers and before the unwrap. It works, and it is still a workaround: you are patching values as they stream because nothing downstream will accept them unpatched.

• Change the type mapping. Relational connectors support a converters SPI; a custom converter can register the column as a plain string instead of a UUID. If the destination does not need the strict type, this removes the failure mode entirely.
• Not a fix: config flags. It is tempting to think a flag covers this. Two look close, and neither is the fix: Kafka Connect's errors.tolerance governs record-level failures in the Connect framework, and Debezium's event-processing failure modes decide how Debezium reports problems it catches inside its own pipeline. The exception we hit fires in the converter that feeds the destination — past the point those modes watch.

We sent the empty-string guard upstream; it was closed without merging. Failing loudly on invalid data is a defensible connector default, and a pipeline dying on it is the problem.

How Fabric handles it

Fabric treats the value check as part of the write path, not as an exception to handle at the connector level. Before anything lands, every value is coerced against its column's canonical type. A blank UUID is not a crash; it is a named result:

coerce_value(UUID, "")     -> (None, "invalid_uuid")   # blank string
coerce_value(UUID, "   ")  -> (None, "invalid_uuid")   # whitespace counts
coerce_value(UUID, None)   -> (None, None)             # NULL is a value
coerce_value(UUID, "9c1e3b7a-…") -> (uuid.UUID, None)  # the real thing

From there, the rules are the ones a pipeline should have had all along:

• The bad row is diverted, not fatal. It goes to quarantine and the rest of the batch commits. The checkpoint moves with the rows that landed, so a restart resumes after them instead of replaying the same failure forever.
• An all-bad batch is still not fatal. No checkpoint is written and nothing raises; the next run re-reads the batch and re-quarantines it. That is idempotent retry, not a crash loop.
• The row is kept. Reason, column, offset and the full record payload stay in Fabric's metadata database, and the pipeline's status shows the count, so a quarantined row is a visible gap rather than a silent drop. You can review what was rejected and reprocess it once the mapping is fixed.
• The repair is a transform. Fix the mapping in the transform editor: a small Python function that runs per batch in a sandbox, with a preview in the UI, so future rows arrive already shaped correctly — blank turned into null, or the row dropped — without a source-wide re-snapshot.

That is the part of the work ETL tools are supposed to do for you. A value that does not fit its column is a data problem, and it belongs in the data layer — classified, kept, visible, fixable — not in your pipeline's uptime.

Fabric ships this as standard: sign up, add your source, and the next bad value is a quarantine row, not an outage.

Keep reading

• Why your idle Postgres database is quietly running out of disk
• One hundred tables, one config you can't guess
• Debezium PostgreSQL connector documentation
• debezium-server-iceberg #691 — the empty-string guard we sent upstream