Apache Iceberg is the default sink, and the one most deployments should start with. Tables are created through Iceberg itself — partition specs, identifier fields, and metadata cleanup are set at create time, not hand-rolled DDL. Data is versioned, schema evolves additively, and any Iceberg-aware engine can query it.
An Iceberg destination is two orthogonal choices: a catalog (the metadata plane) and an object store (the data plane). The object store holds the Parquet files; the catalog tracks which tables exist and which files belong to them.
Prerequisites
- A reachable catalog. A Postgres database for JDBC, a REST catalog such as Nessie, an AWS account and region for Glue, or a Hive Metastore Thrift endpoint.
- A warehouse location that already exists and is writable— scheme and trailing slash included, e.g.
s3://bucket/prefix/. Fabric creates tables inside it but does not create the bucket. - Object-store credentials valid for that location: an access key and secret for S3/MinIO/R2/custom; a connection string or account name and key for Azure; a service-account credential for GCS.
- Network egress from the runner to the catalog endpoint, the object-store endpoint, and — for Glue — the AWS APIs.
Choosing a catalog
| Catalog | Fields | Notes |
|---|---|---|
jdbc (default) | Catalog URI (jdbc:postgresql://host:5432/db), user, password | Iceberg metadata stored over PostgreSQL. |
rest | REST URI, plus a bearer token or client credential, and an optional prefix | e.g. Nessie at http://nessie:19120/iceberg. Token or credential is required. |
glue | Region, optional catalog ID, optional AWS access key and secret | Blank credentials use the default AWS chain (environment, IAM role, or profile). |
hive | Hive Metastore URI (thrift://host:9083) | A Hive Metastore Thrift endpoint. |
Choosing an object store
| Store | Fields | Notes |
|---|---|---|
| Amazon S3 | Access key, secret, region | The endpoint is derived from the region — you never type it. Virtual-hosted addressing. |
| MinIO | Endpoint, access key, secret, region | Defaults to http://minio:9000; path-style access is on. |
| Cloudflare R2 | Endpoint, access key, secret, region | Region defaults to auto; path-style access is on. |
| Custom S3-compatible | Endpoint, access key, secret, region | Reachable from the edit dialog; path-style access is on. |
| Azure Data Lake (ABFS) | Connection string, or account name and account key | ADLS Gen2. |
| Google Cloud Storage | Service-account credential, optional project ID and endpoint | Service-account JSON, or an HMAC key and secret. |
Common fields across the form are the destination Name and Catalog name (defaults to iceberg), plus the Warehouse location and the Default destination switch.
How tables land
- Convention is direct: a sync's configured table name maps to
<namespace>.<table>, and the namespace defaults toraw. There is no topic-prefix or schema-name munging. - Tables are created with Parquet data, Snappy compression, and a 128 MiB target file size; metadata count-based garbage collection is disabled so old metadata stays available.
- Schema evolution is additive: new source columns are added, existing columns are never retyped.
- Each stream commits on its commit interval (default 30 s) or when its buffer exceeds 100,000 rows; a commit conflict is retried automatically.
Gotchas
Watch out
- The warehouse location must include a scheme and end with
/— tables join as<warehouse><namespace>/<table>, so a missing slash would bad-join. Both the form and the API reject it. - Test connection writes and deletes a probe object under the warehouse, so a missing or read-only bucket fails there before any sync runs.
- For S3 the endpoint is derived from the region on purpose: the global endpoint redirects a regional bucket to an address that object-store clients then reject.
- Path-style access is required for MinIO, R2, and most self-hosted S3-compatible stores; without it, buckets can resolve to virtual-host URLs and fail once data files move.