Small files, big bills: the silent tax on every CDC pipeline

Your queries are slow, your storage bill is climbing, and nobody changed anything. The usual suspect: millions of tiny files quietly piling up in your data lake.

I have lived this one. Replicating Postgres through Debezium into MinIO, the disk filled up and queries crawled. Moving to S3 fixed the disk and swapped it for a bill, because every small write is a request you pay for. It turns out this is just what happens by default.

How the pile grows

• Streaming and micro-batches. Parquet files cannot be appended to, so every batch writes new files. Short intervals mean thousands of files a day, most only kilobytes. CDC is exactly this workload: a steady trickle of small changes.
• Over-partitioning. Partitioning by a high-cardinality column gives you a directory per value with one tiny file in each.
• Untouched defaults. Spark's 200 shuffle partitions turn a 500 MB write into 200 files of a couple of megabytes.

What it actually costs

• Slow queries. The engine opens every file and reads its footer before reading a row. In an AWS Athena benchmark cited in the original, 582K small files took 40 seconds. Compacted into 336 files, the same query took 9.7 seconds.
• More requests, higher bills. Object stores charge per request, so a million small files costs about a thousand times more in API calls than a thousand well-sized ones. The original cites a 10x cost reduction from compaction at Glovo.
• Metadata bloat. On HDFS every file also costs NameNode memory, and clusters have fallen over from it.

The rule of thumb: aim for roughly 128 MB to 1 GB per Parquet file on object storage.

The fix is boring: run maintenance

Every table format has a compaction answer. Delta has OPTIMIZE, Iceberg has rewrite_data_files, and Hudi steers new writes into existing small files. The catch is the same everywhere: somebody has to schedule it, and expire old snapshots and clean up orphan files too. Tables left alone can grow to millions of files without a single alert.

How Fabric handles it

With Fabric, that scheduling job is not yours. When you add a source, a maintenance policy is created for it and switched on:

• Iceberg: compaction of files below a size threshold (128 MB by default), snapshot expiry, and orphan-file cleanup, on a schedule.
• Delta Lake: compaction plus vacuum, with a 7-day default retention.
• Amazon S3 Tables: managed compaction is built into the service itself.

Maintenance runs alongside the live writer and retries when the two collide, so compaction does not pause replication. The defaults are tunable per source if your workload is unusual.

No extra containers, no cron job to babysit, and one fewer thing to put in your on-call runbook — hosted and self-hosted alike: start for free and add a source.