Delta Lake

Delta Lake is an append-only history sink with no catalog plane. A table is simply a directory containing Parquet files and a _delta_log transaction log side by side, under a warehouse location you own. Every change is a new row; nothing is collapsed or upserted.

Prerequisites

• A warehouse location that exists and is writable — scheme and trailing slash included, e.g. s3://bucket/prefix/. Fabric creates the table directories; it does not create the bucket.
• Credentials with read, write, and delete on that prefix. Delete is needed for vacuum and pipeline resets.
• Conditional-put support on the object store for S3-compatible backends. Delta commits are atomic conditional puts; a store without that support fails the commit rather than writing unsafely.
• A single writer per table. Do not point an external writer, such as a Spark job, at the same Delta table.

Connection fields

The rest of the form is storage-specific. S3/MinIO/R2/custom take an endpoint, access key, secret, region, and a path-style switch (real S3 derives its endpoint and uses virtual-hosted addressing). Azure takes a connection string, or account name and key. GCS takes a service-account credential.

How tables land

A sync's namespace and table name join onto the warehouse: <warehouse><namespace>/<table>/, with the namespace defaulting to raw. Listing the warehouse shows one directory per table.

• On the first non-empty batch, the table is created from data; an empty or heartbeat-only flush never creates a directory.
• Schema evolution is additive — a new column is added, an existing one is never retyped. A case-collision against an existing column is quarantined before the write.
• Maintenance runs compact (bin-pack small files) and vacuum (expire old files), both enabled by default.
• A pipeline reset deletes the whole table prefix, data and log together. This is S3-compatible only; Azure and GCS resets must be cleaned up manually.

Gotchas