Contents

• AGENTS.md — Development Guidelines for pg_trickle
  • Project Overview
  • Workflow — Always Do This
  • Coding Conventions
    • Error Handling
    • SPI
    • Unsafe Code
    • Memory & Shared State
    • Background Workers
    • Logging
    • SQL Functions
  • Module Layout
  • Testing
    • CI Coverage by Trigger
    • Running All Tests Locally
    • Triggering Full CI on a PR Branch
  • CDC Architecture
  • Code Review Checklist

AGENTS.md — Development Guidelines for pg_trickle

Project Overview

PostgreSQL 18 extension written in Rust using pgrx 0.17.x that implements streaming tables with incremental view maintenance (differential dataflow). Targets PostgreSQL 18.x.

Key docs: docs/ARCHITECTURE.md · docs/SQL_REFERENCE.md · docs/CONFIGURATION.md · INSTALL.md

---

Workflow — Always Do This

After any code change:

just fmt          # Format code
just lint         # clippy + fmt-check (must pass with zero warnings)

After changes to SQL-facing code, run the relevant test tier:

just test-unit         # Pure Rust unit tests (no DB)
just test-integration  # Testcontainers-based integration tests
just test-e2e          # Full extension E2E tests (builds Docker image)
just test-all          # All of the above + pgrx tests

E2E tests require a Docker image. Run just build-e2e-image if the image is stale, or use just test-e2e which rebuilds automatically.

When you’re done and have edited files always remember to output git commands for staging and committing the changes. The git commit message should summarize the changes made. Feel free to put discrete changes into separate git commit commands. Never commit directly to git without asking for permission.

---

Coding Conventions

Error Handling

• Define errors in src/error.rs as PgTrickleError enum variants.
• Never unwrap() or panic!() in code reachable from SQL.
• Propagate via Result<T, PgTrickleError>; convert at the API boundary with pgrx::error!() or ereport!().

SPI

• All catalog access via Spi::connect().
• Keep SPI blocks short — no long operations while holding a connection.
• Always cast name-typed columns to text when fetching into Rust String (e.g. n.nspname::text). The PostgreSQL name type (Oid 19) is not compatible with pgrx .get::<String>() (Oid 25).
• Catalog lookups must handle “not found” gracefully — CTEs, subquery aliases, and function-call ranges do not exist in pg_class. Return Option rather than .first().get() which panics on empty results.
• Separate pure logic from SPI calls so the decision logic can be unit-tested without a PostgreSQL backend (see classify_relkind, strip_view_definition_suffix for examples).

Unsafe Code

• Minimize unsafe blocks. Wrap pg_sys::* in safe abstractions.
• Every unsafe block must have a // SAFETY: comment.

Memory & Shared State

• Be explicit about PostgreSQL memory contexts.
• Use PgLwLock / PgAtomic for shared state; initialize via pg_shmem_init!().

Background Workers

• Register via BackgroundWorkerBuilder.
• Check pg_trickle.enabled GUC before doing work.
• Handle SIGTERM gracefully.

Logging

• Use pgrx::log!(), info!(), warning!(), error!().
• Never println!() or eprintln!().

SQL Functions

• Annotate with #[pg_extern(schema = "pgtrickle")].
• Catalog tables live in schema pgtrickle, change buffers in pgtrickle_changes.

---

Module Layout

src/
├── lib.rs          # Extension entry point, GUCs, shmem init
├── api.rs          # SQL-callable functions (create/alter/drop/refresh)
├── catalog.rs      # pgtrickle.pgt_stream_tables CRUD
├── cdc.rs          # Change-data-capture (trigger-based)
├── config.rs       # GUC definitions
├── dag.rs          # Dependency graph, topological sort, cycle detection
├── error.rs        # PgTrickleError enum
├── hash.rs         # Content hashing for change detection
├── hooks.rs        # DDL event trigger hooks
├── monitor.rs      # Monitoring / metrics
├── refresh.rs      # Full + differential refresh orchestration
├── scheduler.rs    # Background worker scheduling
├── shmem.rs        # Shared memory structures
├── version.rs      # Extension version
├── wal_decoder.rs  # WAL-based CDC (logical replication polling, transitions)
└── dvm/            # Differential view maintenance engine
    ├── mod.rs
    ├── diff.rs     # Delta application
    ├── parser.rs   # Query analysis
    ├── row_id.rs   # Row identity tracking
    └── operators/  # Per-SQL-operator differentiation rules

See plans/PLAN.md for the full design plan.

---

Testing

Six test tiers, each with its own infrastructure:

Tier	Location	Runner	Needs DB?
Unit	src/** (#[cfg(test)])	just test-unit	No
Integration	tests/*_tests.rs (not e2e_*)	just test-integration	Yes (Testcontainers)
Light E2E	tests/e2e_*_tests.rs (most)	just test-light-e2e	Yes (stock postgres:18.3)
Full E2E	tests/e2e_*_tests.rs (10 files)	just test-e2e	Yes (custom Docker image)
TPC-H	tests/e2e_tpch_tests.rs (#[ignore])	see below	Yes (custom Docker image)
dbt	dbt-pgtrickle/integration_tests/	just test-dbt	Yes (Docker + dbt)

Light E2E uses cargo pgrx package output bind-mounted into a stock postgres container (no custom Docker image build). 42 test files (~570 tests) are light-eligible; 10 files (~90 tests) require the full E2E image.

• Shared helpers live in tests/common/mod.rs.
• E2E Docker images are built from tests/Dockerfile.e2e.
• Use #[tokio::test] for all integration/E2E tests.
• Name tests: test_<component>_<scenario>_<expected>.
• Test both success and failure paths.
• dbt tests use just test-dbt-fast to skip Docker image rebuild.

CI Coverage by Trigger

Job	PR	Push to main	Daily schedule	Manual dispatch
Unit (Linux)	✅	✅	✅	✅
Unit (macOS + Windows)	❌	❌	✅	✅
Integration	✅	✅	✅	✅
Light E2E	✅	✅	✅	✅
Full E2E + TPC-H	❌	✅	✅	✅
Upgrade completeness	✅	✅	✅	✅
Upgrade E2E	❌	✅	✅	✅
Benchmarks	❌	❌	✅	✅
dbt integration	❌	❌	✅	✅
CNPG smoke test	❌	❌	✅	✅

Note: Full E2E and TPC-H tests are skipped on PRs (the Docker build is ~20 min). Light E2E tests run on every PR using cargo pgrx package + stock postgres container. To run the full CI matrix on a PR branch, use manual dispatch (see below).

Running All Tests Locally

just test-all          # unit + integration + e2e (builds Docker image)

# TPC-H tests (not included in test-all, require e2e image):
just build-e2e-image
cargo test --test e2e_tpch_tests -- --ignored --test-threads=1 --nocapture

# Control TPC-H cycle count (default varies per test):
TPCH_CYCLES=5 cargo test --test e2e_tpch_tests -- --ignored --test-threads=1 --nocapture

Triggering Full CI on a PR Branch

To run the complete CI matrix (including E2E, TPC-H, benchmarks, dbt, and CNPG) on a feature branch:

gh workflow run ci.yml --ref <branch-name>

This is recommended before merging any PR that touches the DVM engine, refresh logic, or CDC pipeline.

---

CDC Architecture

The extension uses row-level AFTER triggers (not logical replication) to capture changes into buffer tables (pgtrickle_changes.changes_<oid>). This was chosen for single-transaction atomicity — see ADR-001 and ADR-002 in plans/adrs/PLAN_ADRS.md for the full rationale.

---

Code Review Checklist

• [ ] No unwrap() / panic!() in non-test code
• [ ] All unsafe blocks have // SAFETY: comments
• [ ] SPI connections are short-lived
• [ ] New SQL functions use #[pg_extern(schema = "pgtrickle")]
• [ ] Tests use Testcontainers — never a local PG instance
• [ ] Error messages include context (table name, query fragment)
• [ ] GUC variables are documented with sensible defaults
• [ ] Background workers handle SIGTERM and check pg_trickle.enabled