v0.13.0 — Columnar Change Tracking, Shared Buffers, and TPC-H DIFFERENTIAL

Full technical details: v0.13.0.md-full.md

Status: ✅ Released | Scope: Large (~5–6 weeks)

Track exactly which columns changed (not just which rows), share change buffers across multiple stream tables reading the same source, full TPC-H DIFFERENTIAL pass at all 22 queries, and LIST and HASH partition type support.

What problem does this solve?

The differential engine was treating every row change the same way regardless of which columns were actually affected. If a row in the customers table had only its phone_number column updated, but a stream table only uses customer_name and customer_email, the stream table was still being refreshed unnecessarily. Shared change buffers became important as more users defined multiple stream tables on the same source table.

Columnar Change Tracking

The change-capture triggers now record which specific columns changed alongside the row change. The differential engine uses this information to skip stream table refreshes when none of the columns a stream table reads were actually modified.

In plain terms: if your stream table computes sales summaries and only reads amount and product_id, an UPDATE to a customer’s phone number will not trigger any refresh at all — because no relevant column changed.

This is especially impactful for tables with many columns where only a few are relevant to any given stream table.

Shared Change Buffers

Previously, each stream table maintained its own change buffer. If three stream tables all read from the same orders table, three separate change buffers were maintained with overlapping data.

Shared change buffers consolidate this: all stream tables reading from the same source table share a single change buffer. The scheduler reads from the shared buffer and delivers the relevant changes to each stream table that needs them.

In plain terms: three stream tables on the same source now use the same buffer instead of three copies. Memory usage drops significantly in deployments with many stream tables on the same source.

LIST and HASH Partitioned Sources

v0.6.0 added support for RANGE partitioned source tables (the most common type). v0.13.0 extends this to LIST partitioning (where each partition covers specific discrete values, like country codes) and HASH partitioning (where rows are distributed across partitions by hash of a key column).

TPC-H 22/22 DIFFERENTIAL

All 22 TPC-H benchmark queries now pass in DIFFERENTIAL mode — the incremental engine handles all of them correctly without falling back to full recomputation. This is a significant milestone: the industry-standard analytical benchmark is fully supported with incremental refresh.

DVM Engine Improvements

Several improvements to the internal differential view maintenance engine:

  • The EXCEPT ALL operator (used internally for delta computation) was optimised for the common case of small deltas
  • A new MERGE execution strategy was profiled and adopted for cases where the number of changed rows is large relative to the stream table size

Scope

v0.13.0 delivers meaningful performance improvements through column-aware filtering and shared buffers, and completes the TPC-H DIFFERENTIAL milestone. The partitioning extensions complete the coverage for all three PostgreSQL partition types.