Fabric is designed to be small to run. The streaming path has no external platform to provision — the only services you need are Fabric itself and the storage you're writing to.
Local development
The docker compose stack runs the full picture for development: the Fabric backend and frontend, a Postgres metadata database, Redis (Celery broker), object storage, a query engine, and synthetic source databases. You can develop against it without pointing Fabric at a real production database.
QA / Production
For QA and Prod the footprint is deliberately minimal. To begin streaming you need only:
- Fabric — the backend (Django / Celery) and frontend (React SPA, served by nginx)
- PostgreSQL (external, e.g. RDS) — Fabric's metadata database, and the Iceberg catalog store by default
- Object storage + a catalog — the data plane: S3, MinIO, Cloudflare R2, a custom S3-compatible endpoint, Azure Data Lake Storage (ABFS), or Google Cloud Storage; the Iceberg catalog is pluggable (
jdbc/rest/glue/hive)
Production hardening — authentication, TLS, monitoring, and backup — is handled by the deployment environment, not a fragile in-cluster streaming platform.
Migrating from docker compose
If you're running Fabric today on a docker-compose.prod.yml/.qa.yml-style stack on a single box, that path is retired — the Helm charts above are now the deployable artifacts for every environment, laptop through cluster. PostgreSQL was already external to that stack (an RDS instance or similar), so its data doesn't move at all: point the same database.* values at the install instead of the .env file. Object storage needs one extra step first — see below.
You have two ways to run those charts on the same single box:
- Install k3s — a real, single-binary Kubernetes distro that runs as a systemd service, so (unlike a throwaway dev cluster) it survives a reboot the same way your compose stack's
restart: unless-stoppeddid. Once it's running,cli/fabric-cli local updetects it automatically and installs both charts onto it — see get.k3s.io and the Quickstart. - Move to a managed cluster (EKS or similar) if you're ready to go beyond one box — same charts, same values, just a real control plane you don't operate yourself.
One thing is changing: production compose bundled its own MinIO container; neither chart bundles object storage at all — destination.* always points at a bucket you run, whether that's real S3 or a MinIO you deploy and operate separately. One thing stays the same either way: there's no bundled query engine. Point Trino, Athena, Snowflake, or any other Iceberg-compatible engine at the warehouse, same as before.
Kubernetes
The Helm charts (deploy/helm/fabric-web and deploy/helm/fabric-worker) are the deployable form of that footprint. They install as two releases: fabric-web is the shared control plane — one release per cluster — and fabric-worker is the execution layer, one release per tenant, so a single-tenant install is both releases in one namespace. Between them they create:
| Workload | Chart | Role |
|---|---|---|
fabric | fabric-web | Django API — migrates the schema, bootstraps the admin user, then serves gunicorn |
celery worker | fabric-worker | Lifecycle tasks — the workers scale |
beat | fabric-web | The periodic sweep — a singleton |
ui | fabric-web | nginx serving the built SPA and proxying /api |
redis | fabric-web | Celery broker and shared cache — bundled, optional |
postgres | fabric-web | Bundled metadata database — off by default; bring your own instead |
ingress | fabric-web | Optional; a NodePort or LoadBalancer Service needs no controller |
service account + role | fabric-worker | The identity Fabric manages runner Pods with — create / get / list / delete Pods and read their logs, in its own namespace only. No Docker socket, no cluster-wide access |
The bundled dependencies — both of them fabric-web's — are installs-for-you, not requirements: set redis.enabled or postgres.enabled to false and point the relevant database.* / broker settings at infrastructure you already run. fabric-worker bundles no Postgres or Redis of its own: it reads the broker coordinates instead, and it templates no Secret — it requires fabric.existingSecret, the web release's, so the runner's env decryption key matches the API's. Object storage was never either chart's: destination.* is always your bucket. Resource requests and limits, storage classes, and volume sizes are values too — a larger install is a values file, not a fork of the manifests.
The bundled Postgres and Redis in fabric-web each run as a StatefulSet with its own PVC. By default helm uninstall deletes that PVC along with everything else — set postgres.retainDataOnUninstall or redis.retainDataOnUninstall to true before uninstalling to keep the volume around for a later reinstall. Scaling a StatefulSet down never deletes its PVC, with either setting.
Credentials
fabric.existingSecret to the name of a Secret you create yourself (Vault, External Secrets Operator, Sealed Secrets) and fabric-web never holds your credentials in plaintext values. Leave it empty and fabric-web templates one from the values you pass at install time. fabric-worker names that same Secret and templates none of its own.Bring-your-own requirements
Pointing database.* or the destination at infrastructure you already run needs more than a reachable endpoint — the credentials need specific permissions:
| Dependency | Required permissions |
|---|---|
database.* (Postgres / RDS) | CONNECT to the target database plus CREATE/ALTER/DROP on its own schema — Django's migrations run DDL on every upgrade. A least-privilege role owning one database is enough; it never needs superuser or access to other databases on the instance. |
destination.s3* / bring-your-own object storage | An IAM policy (or bucket policy) granting s3:GetObject, s3:PutObject, s3:DeleteObject, and s3:ListBucket on the warehouse bucket and everything under its prefix — Iceberg commits write and rewrite data files and manifests, and expired snapshots are deleted. No account-wide S3 access is needed; scope the policy resource to that one bucket. |
destination.catalogUri (external jdbc catalog) | Same as database.* above, scoped to the catalog's own database — the JDBC catalog implementation issues DDL against its own tables the first time it runs. |
destination.catalogUri (external rest/glue/hive catalog) | Whatever that catalog's own API requires to create namespaces and commit table metadata — Fabric makes no calls beyond the standard Iceberg REST/Glue/Hive catalog contract, so the catalog's own read/write role for its warehouse is enough. |
The charts are the artifact; deploy/helm/fabric-web and deploy/helm/fabric-worker are self-contained directories, packaged and versioned like any other. What the repo also ships is a bootstrapper — uv run --project cli fabric local up installs the published images, into k3s (if it's already running) or a throwaway kind cluster otherwise, and installs these charts with their values-local.yaml files (deploy/helm/fabric-web/values-local.yaml and deploy/helm/fabric-worker/values-local.yaml). It exists so standing the charts up is one command; local up --build is the same command with both images built from the checkout instead of pulled, for working on the source. It is not a second way to deploy — on k3s it is a real single-box deployment of the exact same charts, and the values files it uses are not shipped inside the packaged charts.
Runners
Each source gets exactly one runner — a supervised process launched from the same image Fabric itself runs. The runner mounts no code or config from the host: it runs what the image was built with, so the control plane and the runner can never drift apart, and the image is all you need to ship to another host.
On a cluster the runner is a Pod in the fabric-worker release's namespace, created and deleted through the Kubernetes API — that is what the worker chart's ServiceAccount and Role are for. Nothing is mounted from a node, and the Pod carries no Kubernetes credentials of its own, so a runner can never reach the API server or the Docker socket. On the docker compose stack the same process runs as a sibling container instead; RUNNER_BACKEND is the switch, and each deployment sets it for you.
One runner per source is a hard rule, not a default. A runner is the single reader of its source's replication slot, so two runners on one source would fight over the stream position. More capacity means more sources running concurrently — never a replica count on a single source.
State lives outside the runner
Where to go next