v0.35.0 — Citus: World-Class Distributed Source CDC

Full technical details: v0.35.0.md-full.md

Status: Planned | Scope: Large

The culmination of pg_trickle’s Citus integration. v0.35.0 delivers end-to-end incremental view maintenance over Citus distributed tables — per-worker change capture, distributed stream table storage, cross-node coordination, and a full Citus-specific test suite.

What is this?

With v0.32.0’s naming and frontier foundations in place, v0.35.0 implements the full Citus integration. There are three distributed table problems that need solving:

  1. Change capture — triggers don’t propagate to Citus workers, so pg_trickle can’t use its normal trigger-based CDC for distributed tables.
  2. Stream table storage — a distributed stream table can’t be updated with a standard MERGE statement; Citus imposes restrictions on cross-shard operations.
  3. Coordination — PostgreSQL advisory locks and LISTEN/NOTIFY are node-local; a cluster needs a different mechanism for cross-node mutual exclusion.

Change capture on distributed tables

For distributed tables (where data lives on Citus worker nodes), pg_trickle switches from trigger-based CDC to WAL-based CDC over per-worker logical replication slots. Each worker gets a publication and a replication slot for the source table. The pg_trickle scheduler on the coordinator polls each worker’s slot, decodes the change stream, and writes the changes into the coordinator’s local change buffer — exactly the same buffer used by trigger-based CDC.

This reuses the WAL decoder already in src/wal_decoder.rs; the only new code is a connection path to call pg_logical_slot_get_changes() on a remote node. Reference tables and local tables continue to use the existing trigger path unchanged.

Distributed stream table storage

When a stream table materializes a view over distributed sources, it can store its result in three ways:

  • Local (default for small outputs) — a regular coordinator-side table, same as today.
  • Reference — a Citus reference table, replicated to every worker. Ideal for dimension tables read by many queries.
  • Distributed — a Citus distributed table, sharded by the stream table’s row identity. Required for outputs too large to fit on the coordinator.

For distributed stream tables, the standard MERGE statement is replaced by a DELETE … WHERE row_id IN (…) followed by an INSERT … ON CONFLICT DO UPDATE — the pattern Citus supports everywhere. The result is identical; only the internal SQL changes.

Cross-node coordination

Advisory locks that guard refresh scheduling are node-local. v0.35.0 introduces a lightweight catalog-based lock table (pgtrickle.pgt_st_locks) that provides cross-node mutual exclusion without an external coordinator. The table uses INSERT … ON CONFLICT DO NOTHING for lock acquisition and timestamp-based lease expiry for failure recovery. Advisory locks remain the fast-path for the common single-node case.

Observability

A new pgtrickle.citus_status view surfaces the health of all distributed source connections: which workers are reachable, the replication lag per worker per source, slot positions, and any workers in a failed state. This integrates with the existing Prometheus metrics endpoint.

Test suite

v0.35.0 ships a dedicated Citus E2E test harness: tests/e2e_citus_tests.rs backed by a docker-compose environment (one coordinator + two workers) and tests/Dockerfile.e2e-citus. The test matrix covers 10 scenarios including local, reference, and distributed sources; mixed CDC backends; outbox integration; DDL propagation; and worker restarts.

The non-Citus benchmark suite shows zero regression — all Citus code paths are guarded by the detection check introduced in v0.32.0 and compile out entirely when Citus is absent.

Scope

v0.35.0 is a large release. Phase 3 (per-worker slot CDC) is the deepest new code; it reuses the WAL decoder infrastructure but adds remote connection management. Phase 6 (test suite) requires building and maintaining a multi-container Citus CI environment.

Previous: v0.32.0 — Citus: Stable Naming & Per-Source Frontier Foundation Next: v1.0.0 — Stable Release