Sinks

ClickHouse

Append-only history on MergeTree — database mapping, safe retries, and reading current state.

The ClickHouse sink lands the full append-only CDC history on a plain MergeTree table — one row per change event. Fabric never upserts and MergeTree never merges duplicate keys away, so this is the sink where nothing collapses rows. Current state is a read-side concern.

Prerequisites

  • A reachable ClickHouse server, self-managed or Cloud. It must support insert_deduplication_token on plain MergeTree, async_insert, and the column types Fabric maps to.
  • Network access on the port matching the chosen protocol: 8123 (HTTP) or 8443 (HTTP + TLS), 9000 (native) or 9440 (native + TLS).
  • A user with CREATE DATABASE, CREATE TABLE, ALTER TABLE, and INSERT, plus SELECT on system.columns:
CREATE USER fabric IDENTIFIED BY '<password>';
GRANT CREATE DATABASE, CREATE TABLE, ALTER TABLE, INSERT ON *.* TO fabric;
GRANT SELECT ON system.columns TO fabric;

A single node is expected: clustered and distributed writes are not implemented.

Connection fields

FieldRequiredDefaultNotes
NameYes—Destination label.
HostYes—
PortYes8123 (HTTP) / 9000 (native)Follows the protocol until you edit it.
ProtocolYeshttphttp or native.
UsernameYesdefault
PasswordNoemptyBlank on create means an empty password (the stock default user); blank on edit keeps the stored password.
DatabaseYesdefaultThe database the client initially connects to — not where syncs land.
SecureNooffUse TLS (HTTPS for the HTTP interface).
Cluster nameNo—Reserved for a future clustered writer; leave blank.

How tables land

A sync's namespace becomes a ClickHouse database, created on first write, and the table name sits directly inside it: <namespace>.<table>. There is no schema level in between.

  • Tables are plain MergeTree, ordered by the identifier columns and __source_ts_ms. The ordering is a locality index, not a uniqueness constraint.
  • Every INSERT carries a deterministic deduplication token derived from the batch, so a replayed block after a lost acknowledgement no-ops instead of duplicating history.
  • Schema evolution is additive: a new column lands as a Nullable column, and existing columns are never retyped.
  • Small blocks are written asynchronously; larger ones go synchronous. The commit gate is tuned to avoid ClickHouse's "too many parts" failure.

Reading current state

Because the table holds the raw history, resolve current state on read: group by the identifier columns and take the latest version with argMax, dropping rows where __deleted is true:

SELECT id, argMax(email, __source_ts_ms) AS email
FROM raw.customers
WHERE __deleted = 'false'
GROUP BY id;

An AggregatingMergeTree materialized view on top is the performant alternative for large tables. Fabric ships the history, not the current-state view.

Gotchas

Watch out

  • The Database field is not where syncs land — the namespace database is created for you.
  • Frequent small inserts can outrun background merges and hit "too many parts". The tuned commit gate, asynchronous small blocks, and default partitioning are the mitigations — do not lower them.
  • On a table created by an older writer, Fabric re-applies the deduplication settings; a user without ALTER silently loses retry dedup.
  • Test connection proves reachability and auth only. Verify CREATE DATABASE/CREATE TABLE/ALTER/INSERT with the first sync.