pgContext Product Roadmap

pgContext Product Roadmap

This document describes pgContext’s product direction and release-engineering plans following the installable GitHub V1 launch.

It outlines broad product direction, not an implementation checklist. Items marked complete are implemented for 0.3.0; remaining items describe later work.

A feature listed here remains “planned” until its public capability row and release notes announce its arrival. The document began as a post-V1 plan and now records completed 0.3 work and remaining direction. A roadmap item only becomes release-blocking once it is selected into a future dependency-ordered build or release plan.

The product goal is a PostgreSQL-native AI search engine into which an application can load ordinary rows, text, dense/sparse/multi-vectors, and metadata, query them immediately through an exact path, and gain faster retrieval as derived indexes become ready. Source rows and their registered vector representation remain authoritative. A provider-native integer or bit embedding may itself be that source representation; separately generated index codes remain derived. Index choice, compression, and background optimization must never create a second user-visible truth.

Published research and public mathematical specifications may inform the algorithms, but pgContext implementations are written independently in pgContext-owned Rust. The project will not copy, vendor, or depend on another database engine’s index, quantization, or query implementation. Other engines remain useful behavioral and benchmark references.

PostgreSQL owns transactions, MVCC, ACL/RLS, SQL planning, source storage, WAL/recovery, replication, and index lifecycle. Rust owns validated vector representations, distance and quantization kernels, ANN algorithms, bounded query execution, and versioned derived formats. Model inference stays outside the backend. This boundary lets pgContext use each layer for what it does well without building a second database inside the extension.

Frequently asked since 0.1.0

For transparency, here is where the capabilities most often requested after 0.1.0 stand in 0.3.0 and where the remaining work is headed:

Faster index builds — pgvector currently builds HNSW indexes faster. Closing that gap is planned through parallel-build efficiency and construction-throughput work under Delivery Phases; a concurrent builder already scales to roughly 3.3–3.5× at eight workers.
IVFFlat — implemented as the experimental native pgcontext_ivfflat access method with deterministic external construction, source-authoritative reranking, DML/VACUUM/REINDEX/CIC/partition lifecycle, PG17/18 dump/restore, crash replay, physical replication, and SQ8/PQ posting codecs. Remaining work is drop-in pgvector conversion/naming, PostgreSQL progress-view integration, and matched production-scale frontier evidence. See Full IVFFlat Support.
PostgreSQL lexical search — full tsvector and tsquery support is a first-class retrieval track: stored/generated vectors, caller-supplied queries, language configurations, weighted fields, phrase/prefix/web-search forms, GIN/GiST candidates, ranking, highlighting, and hybrid fusion. See PostgreSQL-Native Lexical Retrieval.
Quantized vectors — scalar, product, and binary encoded candidates exist experimentally for mapped HNSW generations, but complete index-AM, configuration, lifecycle, and certification work remains. The roadmap now includes first-class asymmetric scalar/binary/product serving, RaBitQ, and independent implementations of TurboQuant-MSE and TurboQuant-PROD. QJL and PolarQuant are evaluated in their proper roles rather than presented as interchangeable codecs. Every approximate path retains authoritative full-precision reranking. See Full Quantized Serving and TurboQuant.
Provider-native int8 and uint8 embeddings — planned as first-class source representations, not as derived quantization. Exact distance kernels and representation-specific opclasses will score the stored integer values; a float copy is optional rather than invented as an authority. This does not weaken the rule that codes derived from a float source remain rebuildable acceleration artifacts. See Model-Native Integer and Binary Source Vectors.
Ten-million-vector proof — 10M is now an explicit certification tier, not just an unspecified “larger corpus.” It covers HNSW, IVFFlat, quantization, filtering, lexical/hybrid retrieval, concurrent writes, build cost, memory, WAL/recovery, and matched comparisons. See Reproducible Public Benchmarks.
halfvec, sparsevec, and bitvec maturity — their SQL types remain experimental, while the complete metric-bound HNSW opclass names are now stable as recorded in Non-Dense ANN Opclasses. Named sparse ANN is now implemented experimentally under Named Sparse ANN. Existing pgvector sparsevec columns can be indexed and converted when their dimensions fit pgContext’s current vector policy and HNSW record envelope. Full pgvector sparse coordinate-range compatibility is planned under Large-Dimension Sparse Vectors.
x86-64 performance — the AVX2+FMA kernels are implemented and correctness-verified, but no x86 speed claim is made until measured on real x86 hardware. Every published benchmark is Apple Silicon (NEON). See the x86-64 SIMD kernels item under Delivery Phases.
PostgreSQL majors — PostgreSQL 17 and 18 are the supported release targets; PostgreSQL 17 remains the primary deep-lifecycle and benchmark target. PostgreSQL 15 and 16 are not current roadmap targets. Adding another major requires an explicit support-policy change and its own compile, schema, install, upgrade, recovery, packaging, and platform evidence. A non-supporting PostgreSQL 19 beta/RC readiness lane may run once the pinned pgrx toolchain supports the selected prerelease; that lane produces no package, compatibility, or support claim.
Drop-in pgvector name compatibility — the certified main-extension binding builds pgContext indexes on existing vector, halfvec, and bounded sparsevec columns without data movement; the optional name facade can be enabled when the hnsw and ivfflat names are free. See pgvector Migration and Compatibility.
Changing embedding models without re-embedding everything — planned as mixed-profile late fusion rather than vector-space translation. Old and new model profiles keep separate named vectors and indexes, each query is embedded with every selected profile, and candidates are merged by stable chunk occurrence ID with weighted reciprocal rank fusion. A full backfill remains optional. See Multi-Model Retrieval, Semantic Reranking, and Automatic Chunking.
Automatic chunking and model-based reranking — available as Experimental, bounded workflows. PostgreSQL owns source versions, chunk lineage, permissions, jobs, and atomic publication; provider-neutral external workers own the certified text/Markdown/HTML parsing and reranking execution. OCR, network fetch, provider embedding calls, and model-driven chunking remain planned. See Multi-Model Retrieval, Semantic Reranking, and Automatic Chunking.
Replacing a dedicated RAG database — that is the product goal for the retrieval and evidence layer: PostgreSQL remains the source of truth while pgContext supplies dense, sparse, lexical, multi-vector, graph-assisted, and hybrid retrieval. Embedding, reranking, and generation models remain provider-neutral external workers; the extension will not put model calls or credentials in a PostgreSQL backend. See Evidence Assembly, Provenance, and RAG Evaluation.

Delivery Phases (post-V1 overview)

The detailed sections below are grouped into broad delivery phases, in order:

pgvector interoperability (implemented in 0.2). Install pgContext alongside an existing pgvector database and build pgContext indexes directly on existing vector, halfvec, and bounded sparsevec columns — no data movement — with a side-by-side comparison function and a migration report/adopt toolkit. Queries over pgvector-typed columns always return full results; an advisory notice (optional, on by default) recommends migration.
Write-path scalability. pgContext uses a segmented index design: constant-time WAL-logged inserts into a small delta segment, merged at query time with the main graph, crash-safe at every WAL boundary, and bounded compaction from the index’s own pages. Full deltas rotate into immutable graph segments; directory saturation compacts only the smallest adjacent pair after maintenance_work_mem admission.

Source-table INSERT, UPDATE, and DELETE already maintain HNSW incrementally; ordinary writes do not require a full index rebuild. A separate planned improvement is set-based bulk mutation execution. Array and batch APIs avoid client network loops today, but some point mapping, source-validation, and payload paths still issue work per key or per key/field pair inside the extension. Set-oriented PostgreSQL statements will remove that internal overhead without changing point IDs, transaction semantics, source authority, or index maintenance. See Set-Based Bulk Mutations.

The planned next step is background-worker compaction: moving the rebuild off the write path so writes are accepted at full speed while a separate process maintains the index, improving both sustained throughput and tail latency. It is sequenced after the multi-segment work below, because segmentation changes the unit of compaction from a whole graph to a single bounded segment, which is the right unit for a background worker to operate on. The complete lifecycle and promotion gates are in Segmented Serving and Background Compaction.
Large-corpus quality, build throughput, and memory. The goal is competitive quality and latency at million-vector scale while keeping the small-corpus latency pgContext already delivers. Planned work:
1. Scale-aware search-effort defaults. The search effort a graph needs to hit a recall target grows with corpus size, so a fixed default degrades recall as collections grow. This adds defaults that scale with corpus size, including a higher ef_construction for large builds.
2. x86-64 SIMD kernels. The dense distance kernels dispatch hand-written AVX2+FMA on x86-64 behind runtime feature detection, falling back to a scalar path on CPUs without those features. Performance measurement on x86 hardware and AVX-512 kernels are planned; no x86 speed claim is made until measured on real x86.
3. Parallel-build efficiency. Profile-guided improvement of the parallel HNSW builder’s scaling.
4. Segmented serving for per-query parallelism. Split a large index into bounded segments, each with its own graph, and search them in parallel per query, merging the results — the mechanism that lets a segmented engine deliver lower latency at high recall. It also makes builds embarrassingly parallel and bounds compaction to a single segment. Small collections stay a single segment, so their behaviour is unchanged.
5. Memory and quality features. A target-recall setting that auto-tunes search effort per index, quantized in-graph traversal with exact reranking, statistics-driven filtered search, and memory-budgeted external index builds for very large tables. Quantized traversal with exact rerank is implemented; the remaining tuning and external-build work is planned.
Index portfolio and compressed serving. Establish one typed, versioned quantization domain shared by every index; implement full IVFFlat as a complementary low-memory, fast-build ANN index; finish native quantized HNSW/IVF candidate serving; and evaluate RaBitQ, TurboQuant-MSE, TurboQuant-PROD/QJL, and PolarQuant against scalar, binary, and product quantization on the same recall/latency/memory frontier.
PostgreSQL-native lexical and universal query execution. Promote tsvector/tsquery, GIN/GiST-backed lexical candidates, configurable ranking, fuzzy matching, and richer nested fusion/reranking while preserving PostgreSQL MVCC, ACL/RLS, and exact source rechecks.
Broader pgvector compatibility. The 0.2 bridge and resumable ownership conversion cover the certified PG17 profile. When pgvector is not installed, pgvector-spelled SQL (types, operators, opclasses, USING hnsw, USING ivfflat, familiar settings) runs unmodified, validated by running the pgvector regression suite in CI, plus an in-place adoption tool that converts columns without rewriting tables.
Advanced retrieval. Named sparse ANN with exact recheck, typed composite fusion, quantized candidates, internally maintained late interaction, mapped HNSW, and automatic observability have experimental bounded serving paths. Planned research adds MUVERA-style multi-vector candidate generation, Matryoshka coarse-to-fine search, learned sparse retrieval, multimodal document retrieval, mixed-profile embedding migration, weighted rank fusion, and externally executed semantic reranking without moving model inference into PostgreSQL. A corpus-level lane adds global query statistics and coverage profiles, extractive corpus maps, and stratified evidence sampling as deterministic engine-owned surfaces. An explicitly opt-in multi-source lazy beam begins as an HNSW-only experiment and later admits the optional topology provider described under Graph-Augmented Retrieval.
Document ingestion, evidence quality, production, and 10M certification. Add bounded automatic document chunking, source-linked evidence assembly, citation/provenance output, answerability signals, source-linked summary artifacts — deterministic-partition rollups and eager/lazy hierarchical corpus summaries — and held-out RAG evaluation. In parallel, run model-checked and fuzz-tested concurrency, crash-recovery and replication test matrices, deeper independent PostgreSQL 17 and 18 certification, bounded erasure and operator-invocable structural verification, PostgreSQL-native wait/statistics/EXPLAIN integration, progress reporting, 10M-vector performance and quality gates, and removal of experimental labels surface by surface as certification rows go green.
Ecosystem. Reproducible public benchmarks—including a corpus-size scaling benchmark, 10M-vector tier, and third-party-harness/x86 coverage (see Reproducible Public Benchmarks) — framework integrations (see Ecosystem and Framework Integrations), broader packaging, and scale-out deployment playbooks using standard PostgreSQL replication and partitioning.

Dependency Order

PG17 V1 freeze
├── write-path and large-corpus performance
│   ├── set-based point, validation, and payload mutations
│   ├── scale-aware search/build defaults
│   ├── measured x86-64 SIMD + parallel-build efficiency
│   └── bounded multi-segment index and serving
│       ├── per-query parallel segment search + global top-k merge
│       └── per-segment compaction
│           ├── supervised background-worker compaction
│           └── bounded deletion propagation + physical artifact erasure
├── non-dense ANN opclasses
│   └── named sparse ANN
│       └── large-dimension sparse vectors
├── shared retrieval architecture, source authority, and typed contracts
│   ├── model-native int8/uint8 source types + exact kernels/opclasses
│   ├── HNSW storage, WAL, exact-recheck, and build contracts
│   │   ├── mapped HNSW
│   │   ├── operator-invocable structural verification
│   │   └── full IVFFlat lifecycle
│   │       ├── pgvector IVFFlat compatibility
│   │       └── SSD-resident and streaming ANN evaluation
│   └── pgContext-owned typed quantization domain
│       ├── first-class scalar/SQ8, binary, and product quantization
│       ├── RaBitQ + TurboQuant-MSE prototypes
│       │   └── QJL estimator + TurboQuant-PROD prototype
│       ├── standalone PolarQuant evaluation
│       └── HNSW + IVFFlat candidate-serving integration
├── non-dense ANN + HNSW + IVFFlat compatibility
│   └── full pgvector migration and name compatibility
├── internally maintained late interaction + model/version metadata
│   ├── MUVERA-style fixed-dimensional candidate generation
│   ├── multimodal document retrieval
│   └── Matryoshka coarse-to-fine retrieval
├── non-dense ANN + quantized serving + named sparse ANN + late interaction
│   └── composite query execution
│       ├── PostgreSQL-native lexical retrieval
│       └── query execution and operations governance
│           └── typed result-grouping, facet, and aggregation stages
│               over arbitrary child queries
└── composite query execution + mapped serving
    └── expanded automatic observability
        ├── PostgreSQL 17 custom wait events
        └── PostgreSQL 18 cumulative statistics + EXPLAIN options

stable source identifiers + composite execution + mapped serving
├── internal HNSW-only lazy cursor
│   └── internal bounded vector-only beam state arena + exact score ordering
└── pgGraph topology-boundary audit + provenance manifest
    └── pgContext-owned `context-topology` kernel
        └── versioned projection + optional residency
            └── query-owned topology-expansion port
                └── mixed vector/topology beam + graph-augmented retrieval

exact retrieval + composite execution
└── global query statistics and coverage profiles

stable source identifiers + clustering primitives + lexical term statistics
└── extractive corpus maps
    └── stratified global evidence sampling

stable source identifiers + model/profile metadata + composite execution
├── mixed-profile weighted RRF
│   └── external semantic-reranker envelope + final source revalidation
└── registered document sources + provider-neutral worker contract
    └── bounded automatic chunking + versioned embedding generations
        ├── optional contextual chunk enrichment
        └── atomic profile/chunk publication and rollback

worker contract + deletion propagation + aggregate-visibility contract
└── source-linked summary artifacts
    └── deterministic-partition rollup summaries

source-linked summary artifacts + extractive corpus maps
└── hierarchical corpus summaries (eager and lazy generation)

composite execution + lexical retrieval + source/model provenance
├── multi-model/chunking workflow
│   └── evidence assembly, citation records, and RAG evaluation
└── optional graph-augmented evidence expansion

collection registration + exact retrieval + advisor
└── load data and query immediately
    ├── exact-first readiness and explicit lifecycle states
    └── background HNSW/IVFFlat/lexical builds
        └── validated generation publication and automatic recommendations

10M harness + exact ground truth starts after PG17 V1 freeze
└── write-path track + segmented serving + HNSW + IVFFlat
    + quantization + lexical/query + multi-vector/multimodal paths
    └── 10M performance, quality, write-path, and recovery certification

stable query contracts + load/query workflow + certified packages
└── reference framework integrations + compatibility matrix

Shared Retrieval Architecture

Status: planned cross-cutting consolidation before the index and codec portfolio expands. This is an incremental refactor, not a rewrite.

Depends on: the stable SQL type/operator behavior, exact distance oracles, existing query IR, and versioned storage/generation contracts.

Goal: each retrieval concept has one canonical definition and validation path even though PostgreSQL adapters, Rust algorithms, storage formats, and public SQL have different responsibilities. The implementation audit will determine where duplication exists; the roadmap does not assume every current boundary needs to move.

The current workspace already has useful boundaries: context-core owns the canonical DistanceMetric, context-query owns candidate/readiness ports, and PostgreSQL integration is isolated in context-pg. The clearest consolidation pressure is quantization: algorithm/training types live in context-index, HNSW-specific persisted codebooks and approximate scoring live in context-storage, SQL/catalog policy is adapted in context-pg, and benchmark code performs another conversion. Query and fusion also carry separate score-direction types. That evidence calls for a shared domain and adapters, not a workspace rewrite.

Scope:

centralize typed definitions for vector representation, metric and score direction, index kind, codec specification/version, candidate source, source identity, model/configuration revision, and generation/readiness state;
parse SQL, JSON, GUC, catalog, and on-disk values at adapter boundaries, then pass validated domain values internally instead of propagating raw strings, loosely related booleans, or per-module copies of the same enum;
keep exact distance, metric compatibility, score normalization, deterministic ordering, candidate deduplication, and source-recheck rules in shared pure functions with property tests used by every index and query path;
separate domain algorithms from PostgreSQL callbacks, WAL/page persistence, mapped I/O, background workers, and telemetry through narrow ports; use static enum dispatch or specialized kernels in performance-critical loops where dynamic dispatch would be measurable;
distinguish a persisted format descriptor from a mutable runtime policy so a setting change cannot reinterpret old bytes or silently mix generations;
add compile-time exhaustive matching and cross-adapter contract tests so a new metric, codec, vector type, or candidate source cannot be registered in one layer and forgotten in another;
preserve SQL and storage compatibility through explicit adapters, backward-read fixtures, rebuild/cutover plans, and deprecation windows;
baseline build time, binary size, allocations, query latency, and index throughput before each consolidation step so architectural cleanup cannot hide a performance regression.

This track is complete when adding a new codec or candidate source requires one domain registration plus explicit supported adapters, and unsupported combinations fail in a single validation layer. Internal Rust type names and crate boundaries remain implementation details until an implementation plan is selected.

Set-Based Bulk Mutations

Status: planned write-path optimization. pgContext accepts array and batch inputs today, but some point-mapping, source-validation, and payload-mutation paths still execute per key or per key/field pair through SPI.

Depends on: stable source-key and point-ID semantics, collection resource limits, authoritative source rows, ACL/RLS enforcement, and PostgreSQL index maintenance.

Goal: process a bounded batch with a bounded number of PostgreSQL statements. Set-based execution removes extension-level row loops; it does not imply that PostgreSQL heap and index access methods physically process an entire batch as one tuple.

Scope:

convert validated batch input into typed rowsets with ordinality using arrays, unnest, or recordsets;
replace per-key point mapping calls with set-oriented INSERT ... ON CONFLICT ... RETURNING and UPDATE ... FROM; point deletion remains logical and preserves stable point IDs rather than becoming a physical delete;
validate active point mappings and source rows with joins and anti-joins, returning deterministic missing/invalid-key errors without an existence query for every key;
set, delete, or clear payloads with one bounded statement per operation or compatible field group, never one statement per key/field pair;
support heterogeneous recordsets where each row may carry different payload or vector values, while ordinary source-table COPY and multi-row SQL remain the primary ingestion path;
bound parameters, statement size, memory, and lock duration by chunking large calls without weakening transaction-level atomicity;
preserve stable point IDs, input ordering, duplicate-key behavior, inserted/reactivated/deleted classification, per-row results, rollback, source authority, ACL/RLS, triggers, generated columns, and SQLSTATEs;
quote dynamic identifiers safely while keeping values typed and parameterized;
certify COPY, multi-row inserts, point upserts/deletes, payload set/delete/clear operations, and heterogeneous updates with live HNSW and ordinary PostgreSQL indexes.

Promotion requires real PostgreSQL tests for duplicate and missing keys, validation failure before mutation, rollback, concurrent overlapping batches, deadlock/cancellation behavior, ACL/RLS, triggers, and exact per-row result parity. Benchmarks cover 1, 100, 1,000, and 10,000 rows and report statement count, throughput, p50/p95/p99 latency, CPU, memory, WAL per row, lock contention, and live-index maintenance cost against the pre-refactor and direct-SQL baselines.

This work and background compaction solve different write-path costs. Set-based execution removes repeated SPI statements inside one user mutation. Incremental HNSW maintenance already makes each changed source row searchable; segmented background compaction later prevents accumulated index maintenance from creating periodic write-path stalls.

Segmented Serving and Background Compaction

Status: partially implemented for the PostgreSQL 17 profile. The WAL-logged delta segment, bounded threshold, explicit pgcontext.compact(index), and synchronous threshold compaction exist. Bounded multi-segment serving, per-segment compaction, and supervised background-worker execution are planned. Incremental index writes are therefore current functionality; moving maintenance out of the foreground write and bounding it per segment are the remaining roadmap work.

Depends on: page-native HNSW and its WAL/recovery contract, authoritative source rechecks, immutable generation publication, bounded worker/resource policy, and automatic observability.

Goal: sustained writes append at predictable cost while queries search a bounded set of immutable graph segments plus the active delta. Maintenance rewrites one bounded segment at a time outside the source write statement. Small/read-mostly indexes remain a single segment and do not pay parallelism or coordination overhead.

Architecture and dependency boundaries:

keep segment selection, compaction eligibility, debt calculation, and state transitions in a deterministic policy layer that can be property-tested without PostgreSQL; page/WAL work and background-worker scheduling remain PostgreSQL adapters behind explicit ports;
represent each index as a versioned manifest over immutable searchable segments plus one bounded active delta, with checked segment identity, source/configuration revision, tuple range, metric, checksum, publication state, and retirement pins;
append inserts and VACUUM tombstones only to the active delta under a bounded WAL record; rotate a full delta into an immutable segment without rebuilding the entire index or changing source-row visibility;
search relevant segments with a bounded per-query parallelism budget, merge candidates through one deterministic global top-k, then resolve and score survivors from authoritative rows under the statement snapshot, filters, ACL, and RLS;
prevent worker multiplication from turning query concurrency into CPU or memory oversubscription: segment fan-out, distance work, candidate memory, and participating workers share one query budget and degrade to serial search when the budget or collection size does not justify parallelism;
compact one or a small bounded set of segments into a new immutable generation, validate checksums/counts/recall, WAL-log or otherwise durably record publication, atomically switch the manifest, retain old reader-pinned segments, and reclaim them only after all pins and recovery obligations end;
store durable, idempotent maintenance jobs keyed by database/index, configuration generation, and input segment set. A database-scoped supervised worker claims bounded leases, heartbeats progress, retries only safe stages, and recognizes already-published output after restart;
track every worker from registration through clean shutdown. PostgreSQL postmaster death, extension update/drop, database drop, cancellation, SIGTERM, statement timeout, worker crash, and server restart must leave either the old valid manifest or one fully validated new manifest—never an unowned partial publication;
define an explicit overload policy for compaction debt, temporary disk, WAL, memory, and worker exhaustion. Operators choose bounded inline compaction, write backpressure/error, or scheduled maintenance; the engine must not hide an unbounded pause inside an ordinary write;
support inserts, updates, deletes, HOT/non-HOT changes, VACUUM, REINDEX, partition attach/detach, concurrent index operations, backup/restore, WAL replay, standby promotion, and old-format rejection/rebuild across segment rotation and compaction;
report active/immutable segment counts, rows and bytes per segment, delta occupancy, tombstones, compaction debt, queue/lease state, last progress, worker/retry/failure reason, query fan-out, merge candidates, pinned/retired generations, temporary bytes, WAL bytes, and estimated time to threshold.

Test-first implementation order:

model and property-test the manifest/segment/job state machines, including arbitrary rotation, compaction, retry, reader-pin, and crash sequences;
implement serial multi-segment search and deterministic global merge against an exact oracle before adding parallel execution;
implement one bounded foreground per-segment compaction using the same job contract the worker will later consume;
add the supervised worker adapter, durable leases, cancellation/shutdown, retry, failpoints at every publication boundary, and restart recovery;
add bounded parallel query/build execution, then tune locality, segment sizing, and scheduling from the 1M/10M harness rather than fixed assumptions.

Promotion requires:

at least 500 sustained source updates/second on the owned G4 workload and a 24-hour mixed read/write run without unbounded relation growth, compaction debt, lost work, incomplete results, or a periodic write-latency cliff;
recall/latency frontiers no worse than the single-segment baseline at matched work, plus a demonstrated p95/p99 or throughput win where parallel segmented serving is selected;
bounded RSS/shared-memory/temp-disk/WAL at 1, 16, 32, and 64 clients, including proof that nested query and worker parallelism cannot oversubscribe the host;
deterministic crash/restart, failpoint, corruption, backup/restore, WAL replay, replica catch-up, and standby-promotion matrices over every job and publication state;
explicit tests with the worker disabled, saturated, cancelled, crashed, and restarted, proving that exact/query correctness remains available and the configured overload policy—not an implicit fallback—controls writes.

Deletion Propagation and Bounded Erasure

Status: planned cross-cutting lifecycle and enterprise-readiness track. Existing source rechecks, tombstones, generation retirement, reader pins, artifact diagnostics, and cleanup establish query correctness and safe reclamation, but they do not yet provide a complete physical-erasure contract.

Depends on: stable source/chunk occurrence identity, segmented compaction, immutable generation publication and retirement, token/chunk projection ownership, automatic observability, and backup/replication documentation.

Goal: make deletion observable and bounded across every pgContext-owned representation without claiming control over storage that PostgreSQL or the operator owns. The contract has three separately reported clocks:

Logical invisibility: a snapshot established after the source deletion commits may not return the row through exact, ANN, lexical, token, chunk, fusion, reranking, or evidence paths. A transaction retaining an older snapshot keeps PostgreSQL’s documented MVCC view and is a physical reclamation blocker.
Live pgContext storage erasure: a configured deadline covers compaction or rebuild without the deleted item, retirement of old generations, unlink of pgContext-owned mapped/segment files, and removal of owned token, chunk, cache, job, and observability references once declared blockers clear.
Recoverability horizon: WAL archives, base/incremental backups, replicas, snapshots, and operator-owned source/projection tables follow explicit PostgreSQL and operator retention policies. pgContext reports this boundary but cannot truthfully certify deletion from external backup media.

Every advertised duration names whether its clock starts at the purge request or after the final declared blocker clears. A request-relative bound is offered only by a strict managed-retention mode whose snapshot, reader, replica, and backup preconditions can actually be enforced.

This is an operational privacy control informed by GDPR Article 17, not a blanket legal-compliance claim. PostgreSQL documents that DELETE does not immediately remove an old row version and that ordinary VACUUM normally makes its space reusable rather than returning it to the operating system: routine vacuuming. Its backup and PITR contracts also make archive retention an operator concern.

Scope:

define an idempotent explicit purge operation, with the final SQL spelling frozen during API design, that accepts a registered source occurrence, document, collection, or validated predicate plus a retention class and request identifier;
persist a crash-safe purge state machine with requested/deadline/completed times, representation inventory, retry progress, and typed blockers such as an old snapshot, reader pin, replication slot, failed worker, unavailable archive policy, or operator-owned table;
traverse registered lineage rather than searching by vector similarity: source occurrence, document/chunk generations, embeddings and named vectors, HNSW/IVF segments, mapped generations, late-interaction token relations, lexical projections, caches, evaluation records, jobs, and privacy-bounded observability records must each report deleted, not-applicable, externally-owned, or blocked;
force or schedule bounded compaction/rebuild that excludes purged identities, validate and atomically publish the replacement, wait for legitimate reader pins, then retire and unlink obsolete pgContext-owned files without resurrecting data after crash, replay, retry, or standby promotion;
distinguish deadline policies that may wait for readers from a stricter mode that rejects new work, cancels eligible pgContext jobs, or takes a documented maintenance lock. If a blocker outlives the deadline, status becomes blocked/breached; the system never records a false completion;
keep audit evidence content-free: store request state, counts, timestamps, policy revision, and opaque/pseudonymous correlation where required, but not the erased text, vector, source key, query, tenant identifier, or reversible content hash;
document the user-owned-table boundary. pgContext can delete registered projection rows and advise on VACUUM, table rewrite, partition drop, or encryption-key retirement, but cannot promise secure byte overwrite for an ordinary PostgreSQL heap or storage device;
expose live-storage completion and backup recoverability as different fields, including configured WAL/archive/base-backup/replica retention, and provide a checklist for backup expiration or cryptographic erasure outside the extension;
make collection/drop, profile retirement, source supersession, and ordinary deletion reuse the same lineage and reclamation machinery so the explicit purge path is not a separate, untested deletion engine.

Promotion requires failpoint and restart tests at every purge state; old snapshot, live/stale reader-pin, replication-slot, replica-lag, worker-failure, and deadline-breach fixtures; verification that no query branch or evidence surface returns the item under a post-delete snapshot; byte-level absence checks across every pgContext-owned retired generation and orphan path after completion; and documented restore tests showing exactly when older backups can and cannot reintroduce the erased content.

Experimental Promotion Queue

Status: ordered certification work. This section does not reclassify a capability; the parity matrix and release notes change only after the listed exit gates pass on the same release candidate.

Promotion is intentionally split by contract. A stable exact SQL type or query surface does not need to wait for a stable ANN page format, and a stable opclass name does not imply that every index lifecycle using it is stable.

Exact halfvec SQL surface. Nearest promotion candidate. Freeze text/binary I/O, typmods, rounding/casts, operators, exact metrics, aggregates, ordering, NULL/non-finite errors, and dump/restore; run the selected pgvector compatibility fixtures on PostgreSQL 17 and 18. Keep the HNSW storage row experimental independently.
Exact bitvec SQL surface and explicit opclass names. Promote after Hamming/Jaccard property tests, cast/typmod boundaries, zero-length and all-zero/all-one fixtures, planner/operator ordering, dump/restore, and cross-major tests. Bit HNSW pages remain separately lifecycle-gated.
Bounded sparsevec SQL plus named sparse exact/fusion. Promote the documented bounded profile after sparse-native arithmetic/overflow tests, source-authority and RLS gates, DML/restart/dump coverage, and stable limit errors. Full pgvector coordinate-range parity remains owned by Large-Dimension Sparse Vectors, and sparse ANN remains a separate row until its page/serving gates pass.
Model-native integer source types. Promote signed and unsigned 8-bit exact SQL types independently after I/O, typmod, cast, metric, wide-accumulator, SIMD-oracle, provider-fixture, planner, DML, dump/restore, and cross-major gates. Promote each integer HNSW/IVF representation/metric opclass separately; an exact type does not freeze an ANN storage format.
Dense HNSW and filtered ANN. Promote together only after the remaining serving-memory envelope is dispositioned, sustained mutation reaches the existing G4 target of at least 500 updates/second without unbounded index growth or a tail-latency cliff, the second-environment reproducibility gate is green, and the on-page compatibility/rebuild policy is frozen. Existing 1M matched-recall, filtered-recall, and build-time results are necessary but not sufficient.
Non-dense HNSW. After dense HNSW certification, promote each representation/metric tuple independently when it passes the same build/DML/VACUUM/WAL/recovery/replica/corruption, exact-oracle, bounded-work, memory, and second-environment gates.
Baseline quantized serving and per-vector configuration. Promote scalar, SQ8, binary, and product codecs independently once configuration is consumed end to end, codec/index generations cut over atomically, full-precision reranking is deterministic, and the 1M/10M recall/latency/memory/lifecycle matrix passes per codec. These codecs do not wait for TurboQuant, QJL, PolarQuant, or RaBitQ research.
Research quantization codecs. RaBitQ, TurboQuant-MSE, TurboQuant-PROD/QJL, and standalone PolarQuant remain planned until a real prototype exists, then enter the same per-codec promotion gate without inheriting maturity from the baseline codecs or one another.
Multi-vector and late interaction. Keep experimental until owned token maintenance, backfill/repair, memory and comparison budgets, write amplification, cancellation, 10M-document latency, and recovery/replication gates pass without requiring a user-maintained shadow table.

The first three entries are the best candidates for earlier, narrow promotion because their exact SQL contracts can be certified without freezing an ANN storage format. Dense HNSW/filtered ANN is the highest-impact promotion, but its open operational gates make it riskier than the type-only surfaces.

Non-Dense ANN Opclasses

Status: complete.

Depends on: PG17 V1 freeze, dense pgvector HNSW, and the shared metric semantics.

Scope:

promote half-vector L2, inner-product, cosine, and L1 HNSW opclasses;
promote sparse-vector metrics supported by the exact kernel;
promote bit-vector Hamming and Jaccard where the graph metric satisfies the required ordering and pruning contract;
remove non-dense storage-boundary aborts while keeping representation conversion explicit;
cover create, scan, insert, update, delete, VACUUM, REINDEX, restart, dimensions, casts, NULL/non-finite rules, SQLSTATEs, and exact oracles.

Validated by end-to-end serving tests for every representation/metric pair with exact-oracle and bounded-work assertions.

Model-Native Integer and Binary Source Vectors

Status: implemented experimentally as a first-class source-type track, separate from derived quantization. The Rust/SQL representation set now includes dense f32, half-precision, sparse, bit, signed 8-bit, and unsigned 8-bit vectors. Promotion remains per representation/metric tuple and does not promote the HNSW storage lifecycle.

Depends on: shared metric semantics, typed model/profile metadata, the vector type and operator-family compatibility policy, architecture-dispatched exact kernels, and the HNSW/IVFFlat lifecycle.

Providers including Cohere and Voyage AI can return signed 8-bit, unsigned 8-bit, and packed binary embeddings directly. When an application stores one of those model outputs without a float vector, the integer or bit value is the authoritative source representation. It must not be mislabeled as a rebuildable code derived from unavailable floats.

Scope:

define first-class signed and unsigned 8-bit dense SQL/Rust source types (working names int8vec and uint8vec) with typmods, canonical text and binary I/O, array support, checked dimensions, casts, aggregates where meaningful, dump/restore, and explicit compatibility rules;
treat the existing bitvec as a model-native binary source only when its logical dimensions, packing order, padding, and metric contract match the registered profile. Provider bytes are decoded through a checked boundary; incompatible signedness, bit order, or padding is never silently reinterpreted;
add scalar reference kernels and architecture-dispatched SIMD for squared L2, inner product, cosine, and any later advertised metric, using sufficiently wide accumulation and explicit overflow behavior. “Exact” means exact scoring of the stored integer/bit source under the declared metric, not reconstruction of a float vector that was never retained;
add representation-specific operators, operator families, and exact and ANN opclasses rather than routing integer values through hidden float conversion; planner costing, score direction, zero-norm behavior, and deterministic ties remain aligned with the shared retrieval contracts;
extend immutable embedding profiles with output representation, signedness, dimensions, normalization, metric, provider/model/revision, query/document templates, packing/bit order, and any provider-declared scale or zero point. Absence of dequantization metadata is explicit and does not block exact integer-space search;
require query vectors to use the same registered profile and representation that produced the stored source. Checked explicit conversions may be offered, but no implicit cross-model or signed/unsigned cast may change retrieval semantics;
if both float and integer embeddings are stored, require one declared authority and expose whether final exact rescoring used float or integer source values. If only integer values exist, candidate traversal and exact reranking use them; optional semantic reranking can still judge authorized source text;
allow HNSW, IVFFlat, mapped, segmented, and SSD-resident indexes to consume the source type directly while keeping their graph/list layout and any additional compression derived, versioned, checksummed, and rebuildable;
separate source-type lifecycle labels and promotion gates from derived scalar/PQ/binary codec labels so maturity in one track cannot imply maturity in the other.

Promotion is per representation/metric tuple. It requires exhaustive I/O and boundary fixtures; scalar/SIMD oracle equality on arm64 and x86-64; wide accumulator and zero-norm tests; PostgreSQL planner/operator/opclass, DML, VACUUM, WAL/recovery, replication, REINDEX, upgrade, and dump/restore matrices; provider-produced fixture interoperability; and matched quality, latency, memory, and storage results against float/halfvec baselines. The public source-authority documentation must be updated before the first integer source type is marked stable.

Quantized HNSW

Status: stable for scalar/SQ8, product, and binary HNSW serving on PostgreSQL 17 and 18. The retained one-million-row certification is documented in Quantized HNSW one-million-row certification.

Depends on: PG17 V1 freeze, dense HNSW, and resumable generation infrastructure.

Scope:

property-test scalar, product, and binary encoding, dimensions, codebooks, error bounds, and corrupted codes;
add the quantized_codes fuzz target and bounded corpus smoke;
consume quantization policy during real build/generation;
traverse encoded candidates with bounded work and rerank from authoritative source vectors;
bind configuration revisions from registration through build and scan;
cover concurrent configuration/source changes, restart, invalidation, replacement, corruption, deterministic recall, and serving diagnostics.

Validated by an end-to-end serving test with exact-oracle and bounded-work assertions before promotion.

Full Quantized Serving and TurboQuant

Status: baseline scalar/SQ8, product, and binary HNSW serving is stable. Native IVF-SQ8 and IVF-PQ serving is experimental and lifecycle-certified with exact source rerank. RaBitQ, TurboQuant, QJL, PolarQuant, and residual-PQ research remain planned and are promoted independently.

Depends on: versioned vector-codec and configuration contracts, mapped HNSW serving, authoritative full-precision source vectors, exact reranking, and the shared index build/generation lifecycle. Quantized IVF additionally depends on the full IVFFlat lifecycle.

pgContext will implement the selected methods independently in pgContext-owned Rust from their papers and public mathematical descriptions. It will not copy, vendor, translate, or depend on another database engine’s implementation or storage format. The primary research inputs are TurboQuant, QJL, PolarQuant, and RaBitQ and its multi-bit extension, together with the Google Research TurboQuant overview. External engines remain comparison targets only.

The algorithms are not treated as interchangeable labels. Scalar, product, and binary quantization are codec families. RaBitQ is a competitive binary/multi-bit candidate. TurboQuant-MSE and TurboQuant-PROD are evaluated separately because they optimize different scoring paths. QJL is primarily an inner-product estimation method and may be used directly or as a TurboQuant-PROD component. PolarQuant is evaluated both as the quantization component described by the TurboQuant work and, if its standalone path proves useful, as an independent experimental codec. Public names will follow the implemented mathematics, not collapse all of these methods into a single turboquant switch.

First-class support means a documented typed SQL/catalog configuration, automatic build and online replacement, HNSW/IVFFlat integration, planner and advisor support, explain/observability, upgrade behavior, and a published support matrix. Applications do not manage hidden code columns or codebooks. Quantized bytes remain derived acceleration artifacts; the registered full-precision PostgreSQL vector remains the correctness and rebuild source. This rule applies to codecs derived from a float source; it does not reclassify provider-native integer or bit values covered by Model-Native Integer and Binary Source Vectors.

Scope:

centralize quantization in one pure-Rust domain shared by HNSW, IVFFlat, exact candidate scans, mapped generations, and benchmark tooling; PostgreSQL adapters parse SQL/catalog values into that domain, storage adapters persist its versioned artifacts, and index implementations consume it without redefining codec choices or metric rules;
use one canonical typed configuration for codec family, format version, bit depth, metric, transform, training state, oversampling, and exact-rescore policy. Raw strings and duplicated per-index enums may exist only at external parsing boundaries and must map into the same validated representation;
define shared operations for validation, training, encoding, preparing a query scorer, batched approximate scoring, reconstruction where applicable, byte accounting, corruption detection, and upgrade/rebuild behavior. Prefer enum/static dispatch in the hot scoring loop and narrow trait boundaries at storage, build, and testing edges;
store fixed-width codes in contiguous, aligned buffers with checked stride/offset arithmetic rather than one allocation per vector; use the same code views for page-native, mapped, and benchmark paths, with scalar reference kernels as the oracle for every SIMD implementation;
make the generic codec format index-independent, explicitly versioned, checksummed, and rebuildable from source vectors; incompatible older acceleration formats fail closed and require rebuilding;
record the exact paper/version, independent derivation notes, generated test vectors, dependency/license inventory, and required project intellectual property review before exposing a research method as a public codec;
finish scalar/SQ8, product (including separately evaluated OPQ/residual variants), and binary 1-, 1.5-, and 2-bit serving as real candidate sources rather than helper functions or metadata;
prototype RaBitQ and TurboQuant-MSE before TurboQuant-PROD so each has an independent exact-oracle, build-cost, memory, and recall baseline;
prototype the current published TurboQuant family at 4-, 2-, 1.5-, and 1-bit targets with deterministic seeded transforms, independently test the QJL estimator and PolarQuant component, and retain separate feature flags and benchmark rows until their failure modes are understood;
implement full-precision query preparation and architecture-dispatched kernels for cosine, inner product, and L2 only where the algorithm supports a defensible estimator; L1 must publish its reconstruction cost and may not inherit a fast-path claim from another metric;
keep full-precision source vectors as the correctness and rebuild authority; quantized codes may narrow candidates but never become the only copy or the final score used for returned ordering;
expose per-named-vector codec, bit depth, memory placement, oversampling, candidate budget, and exact-rescore settings, and consume the same revision atomically from registration through build and scan;
support online codec changes through build-new, validate, publish, reader-pin, and retire-old generation semantics, without blocking source writes or silently mixing codebooks/configuration revisions;
implement architecture-dispatched scalar and SIMD kernels on arm64 and x86-64, with byte-for-byte reference tests and explicit fallback paths;
evaluate quantized traversal in HNSW, quantized residual/distance tables in IVFFlat, and hybrid arrangements such as IVF-PQ or IVF-TurboQuant; only combinations that improve a measured Pareto frontier become public APIs;
auto-tune oversampling and search effort against a caller-selected recall, memory, or latency objective, while allowing reproducible manual settings;
report codec, encoded bytes/vector, approximate comparisons, candidates, oversampling, source reranks, reconstruction count, cache/page behavior, fallback, and configuration revision through explain and observability.

Promotion is per codec/index/metric/architecture tuple. It requires property and fuzz tests for the encoding, end-to-end DML/VACUUM/REINDEX/restart/WAL/replica coverage, corruption and upgrade tests, deterministic exact reranking, and published recall/latency/index-size/build-cost curves at 1M and 10M. A codec does not become the default merely because it compresses more: it must beat an existing option on a declared, reproducible operating frontier. QJL, PolarQuant, RaBitQ, and the two TurboQuant modes graduate independently; success for one name or bit depth does not certify the others.

Full IVFFlat Support

Status: implemented and experimental. The native clean v4 format, advertised opclasses, SQ8/PQ codecs, bounded scan controls, diagnostics, native PostgreSQL parallel assignment, supervised compaction, and PostgreSQL lifecycle are functionally exercised on PG17 and PG18. Release-scale 1M/10M comparative certification, automatic pgvector conversion, and full pg_stat_progress_create_index phase reporting remain open and are not implied by this status.

Depends on: the stable metric/operator contract, authoritative exact scoring, the PostgreSQL index-AM/WAL lifecycle, parallel/external build infrastructure, filter selectivity statistics, and versioned index storage.

HNSW remains the high-recall, low-latency graph index. IVFFlat adds a complementary choice with faster builds, lower memory overhead, predictable list/probe control, and a natural home for product/residual quantization. “Full” means owning the complete lifecycle and pgvector-compatible behavior, not only producing centroids and returning a nearest-list demo.

Scope:

add a page-native pgcontext_ivfflat access method with L2, inner-product, cosine, and the representation/metric pairs supported by pgvector IVFFlat; additional L1, sparse, or quantized pairs require algorithm-specific quality evidence and explicit opclasses rather than implied support;
support trained list centroids, deterministic sampling, empty/small-table behavior, configurable or advised list counts, list occupancy/skew diagnostics, and retraining/rebuild guidance as distributions drift;
implement serial, parallel, and memory-budgeted external builds; use PostgreSQL progress reporting for sampling, k-means, tuple assignment, and page publication phases;
implement inserts, HOT/non-HOT updates, deletes, concurrent readers/writers, VACUUM, CREATE INDEX CONCURRENTLY, REINDEX, partitioned tables, dump/restore, WAL replay, physical replication, standby promotion, crash at every publication boundary, and explicit version rejection/rebuild;
provide lists, probes, max_probes, strict/relaxed iterative scans, cancellation, work/memory budgets, and planner costing compatible with familiar pgvector workflows where semantics can be preserved;
combine probes with PostgreSQL predicates and registered filter masks, widening into additional lists when post-filter candidates are insufficient; every survivor is resolved under MVCC/ACL/RLS and scored exactly from the source row;
expose list visits, candidate tuples, filtered survivors, exact reranks, centroid/list skew, fallback reason, and budget termination through EXPLAIN, progress views, and automatic observability;
make migration recognize existing pgvector IVFFlat definitions, preserve their type/metric/options where supported, validate exact and ANN results, and provide resumable rebuild/cutover/rollback rather than defaulting every case to HNSW;
evaluate IVF-Flat, IVF-SQ, IVF-PQ/residual, and IVF-TurboQuant using the same codec boundary and 10M harness; expose only the variants that materially add to the HNSW/IVFFlat speed/recall/build/memory frontier.

The behavioral reference is pgvector’s IVFFlat contract, including its supported representations, probe controls, iterative scans, parallel builds, and progress phases. Promotion requires exact-oracle and bounded-work tests for every advertised opclass plus the full write, maintenance, recovery, replication, and migration matrix. Performance promotion additionally requires matched 1M and 10M curves against HNSW and pgvector IVFFlat.

SSD-Resident and Streaming ANN

Status: planned research track after segmented and mapped serving. No new public access method is committed until it beats the simpler HNSW/IVFFlat choices at a declared operating point.

Depends on: segmented generations, mapped serving, background compaction, versioned codec storage, memory-budgeted builds, and the 10M harness.

The goal is to keep a 10M-vector collection useful when the complete graph, codes, and source vectors do not fit in RAM. Published DiskANN, FreshDiskANN, and SPANN design ideas are inputs, including Microsoft’s Project Akupara; any resulting pgContext implementation and storage format remain independently written and PostgreSQL-native.

Scope:

benchmark a tiered layout in which routing metadata and the hottest quantized codes stay memory-resident while colder graph/list pages and source vectors remain on SSD;
compare PostgreSQL buffer-managed pages, immutable mapped generations, and bounded direct/batched reads before selecting an I/O design. On PostgreSQL 18, explicitly benchmark a buffer-managed path that can benefit from the server’s asynchronous I/O subsystem and pg_aios observability; do not assume a custom access method or mapped file automatically uses it. No path may bypass PostgreSQL visibility, permissions, recovery, cancellation, or source rechecks;
use an immutable base generation plus small WAL-logged mutable segments so inserts are queryable immediately, then merge or compact in the background without stopping source writes;
coalesce reads, prefetch only within explicit I/O and memory budgets, and expose page reads, bytes, faults, queue depth, cache hit rate, amplification, and tail latency rather than hiding storage cost behind average QPS;
support streaming/external builds whose peak memory is configured independently of corpus size and whose intermediate artifacts are resumable, checksummed, and safe to discard;
compare the result with mapped HNSW, full IVFFlat, and quantized HNSW/IVF at identical recall, build window, update rate, RAM, SSD, and source-rerank budgets.

Promotion requires the ordinary DML/VACUUM/REINDEX/WAL/replica/crash matrix, bounded degradation under cold-cache and sustained-write workloads, and matched 10M evidence showing a material frontier improvement. A billion-scale claim requires its own later public dataset and certification tier; it is not implied by an SSD-oriented design.

pgvector Migration and Compatibility

pgContext is positioned as dedicated-engine-grade retrieval inside PostgreSQL, led by the registered-collection API. pgvector interoperability comes in two stages: coexist mode (install alongside pgvector and index existing vector columns directly) is the first deliverable, while full drop-in name compatibility is sequenced later as described below.

Status: the bounded PostgreSQL 17 coexistence and migration profile is implemented and certified; native pgContext IVFFlat is now available, while existing pgvector IVFFlat remains an explicit detect-and-plan input until the P7 conversion contract maps and validates it.

Depends on: PG17 V1 freeze, non-dense ANN opclasses, and quantized HNSW. Full drop-in coverage also depends on the native IVFFlat lifecycle.

Scope:

define and test whether pgvector and pgContext can coexist in one database; use explicit schemas and PostgreSQL type OIDs rather than assuming types with the same SQL spelling are interchangeable;
accept real pgvector database fixtures and provide a preflight inventory of vector columns, dimensions, operators, functions, HNSW/IVFFlat indexes, expression indexes, GUC use, and dependent views/functions;
provide lossless, resumable copy-based or in-place conversions for supported dense, half, sparse, and bit representations, with source-row counts, checksums, exact-distance fixtures, and rollback before destructive changes;
cover the pgvector helper and operator surface selected for compatibility, including normalization, subvectors, concatenation, vector arithmetic, and expression indexes, without duplicating metric semantics outside the shared core;
support subvector and binary-quantization expression-index migration with exact reranking against authoritative source vectors;
expose a compatibility facade for pgvector iterative-scan settings where the semantics can be preserved, and fail explicitly for settings or ordering modes pgContext cannot honor rather than silently accepting them;
add parallel HNSW construction and PostgreSQL progress reporting where benchmarks demonstrate that serial construction is an operational migration bottleneck;
detect pgvector IVFFlat and generate an explicit retain, exact-search, or rebuild-as-HNSW plan; add a resumable rebuild-as-pgContext-IVFFlat path that preserves supported types, metrics, list/probe settings, dependent queries, validation, rollback, and cutover;
test application queries and prepared statements against both extensions and publish a precise compatible, translated, and unsupported SQL inventory;
prove rollback to the untouched pgvector source objects and data without depending on pgContext index pages or catalogs.

Validated by an end-to-end serving test with exact-oracle and bounded-work assertions before promotion.

Large-Dimension Sparse Vectors

Status: planned after the 0.2 bounded sparse compatibility profile; not a 0.2 release blocker.

Depends on: non-dense ANN opclasses, named sparse ANN, versioned storage, and bounded HNSW serving.

Goal: support pgvector’s sparsevec coordinate profile—up to 1,000,000,000 logical dimensions and 16,000 nonzero entries—without allocating memory or performing work proportional to the logical dimension count. This enables lossless direct indexing and ownership conversion for the full certified pgvector sparsevec range rather than only values within pgContext’s current 16,000-dimension policy.

Scope:

split the shared dense-vector dimension ceiling into representation-specific policies, including a large sparse coordinate limit and a separately bounded nonzero-entry limit;
replace every sparse-to-dense HNSW build, insert, query, mapped-serving, and rerank boundary with sparse-native storage and distance/traversal, so a billion-dimensional vector with a handful of entries never creates a billion-element allocation;
define a versioned sparse graph/payload format with checked coordinate and offset arithmetic, corruption detection, upgrade behavior, and no silent reinterpretation of existing 0.1/0.2 pgContext sparse values;
carry large sparse typmods and dimensions through catalogs, registration, query IR, filters, telemetry, dump/restore, bridge preflight, and resumable ownership conversion without narrowing or truncation;
preserve exact scoring as the oracle and enforce explicit budgets for nonzero entries, candidate visits, decoded bytes, build memory, and mapped generations;
add boundary fixtures at dimensions 16,000, 16,001, and 1,000,000,000, malformed/overflow/corruption cases, and end-to-end pgvector direct-index and ownership-conversion tests covering DML, rollback, VACUUM, REINDEX, restart, and dump/restore.

Promotion requires exact-oracle parity for every sparse metric, bounded work proportional to nonzero entries and visited candidates rather than logical dimensions, and a live pgvector compatibility gate at the maximum coordinate range.

Named Sparse ANN

Status: implemented experimentally.

Depends on: non-dense ANN opclasses and metadata-filtered ANN.

Implemented scope:

add a real sparse ANN candidate source through the query-owned port;
retain exact sparse scoring as the correctness oracle and final recheck;
expose counters proving the default path does not score the full collection;
cover update, delete, VACUUM, REINDEX, restart, filter masks, and configuration changes.

Validated by an end-to-end serving test with exact-oracle and bounded-work assertions before promotion.

Internally Maintained Late Interaction

Status: implemented experimentally for the PostgreSQL 17 profile.

Depends on: execution ports, resumable generations, and persisted HNSW serving.

Scope:

make the registered source vector array authoritative;
replace the user-maintained companion table with a pgContext-owned token relation maintained in the same source DML transaction;
cover savepoint/rollback, insert, update, delete, bulk repair, schema change, stale/not-ready state, and crash/rebuild behavior;
property-test MaxSim, deduplication, token ordering, and exact rerank;
enforce token, comparison, hydration, memory, result, and cancellation budgets without per-query prerequisite scans;
make candidate generation replaceable so pooled-vector, token-ANN, and MUVERA-style fixed-dimensional encodings can share the same authoritative MaxSim rerank and lifecycle contract;
preserve ACL/RLS, drift, NULL/invalid-token, deleted-point, and source rechecks.

Validated by end-to-end serving tests before promotion.

MUVERA Multi-Vector Acceleration

Status: planned research and implementation after the late-interaction source and exact-MaxSim contracts are stable.

Depends on: internally maintained late interaction, named vector/model metadata, a single-vector maximum-inner-product candidate source, versioned generation storage, and the composite executor.

MUVERA maps a set of vectors to a fixed-dimensional encoding so an ordinary single-vector search can find candidates for an expensive multi-vector score. pgContext will independently implement and validate that method from its published description, then exactly rerank candidates with the authoritative source token vectors and MaxSim.

Scope:

create deterministic, versioned fixed-dimensional document and query encodings with explicit model, dimension, seed, repetition, and normalization metadata;
use existing HNSW/IVFFlat and quantization ports for the fixed-dimensional candidate stage instead of creating a second ANN stack;
support variable token counts, masks, named multi-vector fields, filters, partitions, and model-version aliases without changing exact MaxSim semantics;
compare MUVERA candidates with pooled-vector prefetch and token-level ANN at the same candidate, memory, build, and exact-rerank budgets;
expose encoding work, ANN candidates, source token hydration, MaxSim comparisons, recall, fallback, and stale-generation state.

Promotion requires held-out multi-vector quality data, exact-MaxSim oracle parity, DML/rebuild/restart coverage, and a reproducible advantage over at least one simpler candidate strategy. The fixed-dimensional score is never the final returned score.

Adaptive-Dimension and Coarse-to-Fine Retrieval

Status: implemented as an experimental correctness-preserving path; scan-based promotion is a performance no-go.

Depends on: registered model/version metadata, typed vector dimensions, exact source reranking, versioned codecs, and the query advisor.

Matryoshka Representation Learning trains embeddings whose useful information is nested in prefixes. pgContext uses this property only when the registered model contract explicitly guarantees it; arbitrary embeddings are never silently truncated.

Scope:

register the full dimension, supported prefix dimensions, model/version, normalization, and compatibility hash alongside each named vector;
search a smaller prefix for candidates only after a bounded visible-corpus preflight proves an exhaustive one- or two-stage schedule fits, then score every admitted authoritative vector exactly; otherwise choose full-vector exact search before prefix work;
let the advisor select a dimension, candidate budget, and rerank budget from a recall/latency/memory objective while preserving reproducible manual controls;
evaluate prefix indexing, expression/subvector indexing, and prefix quantization independently before combining their gains;
support online model and prefix-policy changes through build, validate, publish, alias cutover, and rollback rather than mixing incompatible embedding generations;
expose stage dimensions, candidates, full-vector reranks, expansion count, and why a query completed or fell back. Byte-read and advisor-selected objectives remain research work.

The implemented scan-based path preserves exact ordered results on PostgreSQL 17 and 18 and proves that missing, incompatible, or over-budget policies select the full-vector path. It is not promoted as a latency optimization: exhaustive prefix coverage performs more scan work than full-vector exact search, and larger corpora hit the 10,000-candidate ceiling and fall back before prefix work. Future prefix indexing or compression must establish held-out 1M/10M quality and latency curves before it can replace this no-go decision.

Multimodal Document Retrieval

Status: planned research track after late-interaction and model-provenance contracts.

Depends on: internally maintained late interaction, MUVERA evaluation, lexical/hybrid fusion, registered source identifiers, and the external-worker contract.

The target is page- and region-aware retrieval for documents whose meaning depends on layout, charts, tables, or images. ColPali-style visual late-interaction is an initial research direction based on the ColPali paper, not a commitment to run a vision model inside PostgreSQL.

Scope:

keep original documents, pages, extracted text, and metadata under ordinary PostgreSQL ownership while storing model-produced page/token embeddings as source-linked vector arrays;
fuse visual multi-vector, dense text, learned sparse, native tsvector, and structured-filter candidates through the same typed query plan;
retrieve stable page, region, and object references with offsets or bounding boxes so result hydration and citations can point to the evidence actually scored;
run OCR, parsing, embedding, and optional reranking in provider-neutral external workers; record model/version/provenance and make stale or partial outputs visible;
evaluate MUVERA-style prefetch and exact visual MaxSim under explicit token, image, candidate, hydration, and memory budgets;
benchmark on legally redistributable document-retrieval data with separate text-only, visual-only, and fused results rather than presenting one aggregate score.

Promotion requires source-linked exact-oracle fixtures, held-out retrieval quality, DML/backfill/restart/model-cutover coverage, RLS-safe hydration, and a measured gain over strong text and dense baselines.

Composite Query Execution

Status: shipped for PostgreSQL 17 and 18. The consolidated typed executor owns the bounded composite contract; external rerank and topology providers remain later-phase integrations.

Depends on: metadata-filtered ANN, quantized HNSW, named sparse ANN, and late interaction.

Scope:

execute the typed query IR rather than stopping at JSON constructors;
compose dense, filtered, sparse, full-text, quantized, and late-interaction adapters without infrastructure dependencies in context-query;
centralize candidate kind, metric/order direction, score normalization, source identity, and stage status in one typed contract shared by query parsing, execution, adapters, explain output, and observability;
accept bounded multi-query, query-expansion, and decomposition branches produced by an application or external model, with explicit model/prompt provenance and no model call from the PostgreSQL backend;
define a stable candidate envelope for optional external cross-encoder or late-interaction rerankers, then revalidate source version, permissions, and hydration before presenting the final evidence set;
property-test weighted fusion, reciprocal-rank fusion, deduplication, ties, stage ordering, rerank, and exact oracles;
deterministically cover empty/unavailable stages, malicious plans, cancellation, budget exhaustion, and semantic errors;
add the query_plan fuzz target and bounded smoke.

Validated by end-to-end serving tests before promotion.

Multi-Model Retrieval, Semantic Reranking, and Automatic Chunking

Status: mixed-profile retrieval shipped as a Stable SQL surface after the frozen equal-weight held-out contract passed at one million rows on PostgreSQL 17 and 18. Immutable profile lifecycle and versioned coverage, bounded weighted-RRF execution, explicit degraded policy, equivalent filter/ACL/RLS rechecks, and profile-backed migration records are implemented. Provider-neutral semantic reranking is also shipped through a detached bounded prepare/finalize API and a digest-verified no-network Rust worker. Worker output is untrusted ordering input and PostgreSQL rechecks current source, filter, ACL, and RLS state before returning any row. The private fixture adapter certifies that contract without claiming broad transformer compatibility or bundling weights. Tokenizer-aware automatic chunking now ships for plain text, Markdown, and HTML with immutable profiles, transactional outbox or explicit enqueue, fenced leases, a bounded external worker, canonical PostgreSQL revalidation, fake-embedding work, source-authoritative current reads, bounded rollback, delete invalidation, and atomic current aliases. Its complete PG17/PG18 lifecycle suites pass, but both one-million-row source-cardinality lanes miss the frozen publication-throughput floor (PG17: 10.837 chunks per second; PG18: 33.325). Automatic chunking therefore remains Experimental and its Stable promotion is a measured no-go. See Multi-model retrieval, Semantic reranking, and Automatic document chunking.

Experimental exact-first readiness now registers complete dense-vector source scans before any acceleration exists, freezes explicit advisor plans, and supervises genuine top-level concurrent HNSW/IVFFlat builds without changing source authority or exact final ordering. Stable promotion remains gated on the unchanged PostgreSQL 17 and 18 ten-million-row lifecycle reports. Both reports now preserve bit-exact results and 100% top-10 recall while exceeding the build throughput floor, but miss building/indexed p95 latency and temp ceilings (PG17: 7.850 s / 490 ms / 12.81 GB; PG18: 6.894 s / 468 ms / 13.74 GB). Exact-first therefore remains Experimental and Stable promotion is a measured no-go. See Exact-first readiness.

The first graph-composition foundation is now implemented as an internal lazy HNSW cursor. Existing page, mapped, quantized, segmented, and delta-overlay entry points drain the same bounded traversal, so this changes no SQL surface, planner default, index format, or PostgreSQL authority check. Its frozen differential gate requires exact eager parity across metrics and batch sizes, identical work accounting, and bounded latency, retained memory, and process RSS. See Internal lazy HNSW cursor.

The next graph-composition foundation is also implemented internally: a provider-neutral vector-only beam owns bounded parent state, dominance and cycle pruning, separated score components, and reconstructed occurrence paths. Its frozen graph-off lane preserves the lazy cursor’s ordered occurrence, point, score, and work output within fixed latency, retained-memory, and RSS ceilings. Topology expansion remains rejected, and the kernel adds no SQL or planner surface. See Internal virtual beam engine.

Depends on: stable source and chunk occurrence identities, immutable model/profile metadata bound to named vectors, composite query execution, authoritative source rechecks, bounded work governance, and the provider-neutral external-worker contract.

Goal: let an application adopt a new embedding model without translating old vectors or immediately re-embedding its entire corpus. Historical chunks may remain searchable through Model A, new or selectively refreshed chunks may use Model B, and a query may search both spaces before one shared semantic reranker judges the best current source text. The same workflow should turn large documents into versioned, source-linked chunks automatically without putting file parsing, model calls, credentials, or provider retries in a PostgreSQL backend.

Scope:

evolve model-version metadata into immutable embedding profiles that record provider/model/revision, dimensions, metric, normalization, query/document templates or prefixes, tokenizer revision, configuration hash, lifecycle state, and the bound named vector; credentials remain external;
allow different dimensions and metrics to coexist as separate nullable named vector columns or profile partitions while preserving one logical chunk occurrence ID across profiles;
make embeddings eligible only when their recorded source/chunk content version matches the current published chunk, so an old profile never returns stale text after a source edit;
embed an incoming query independently with every selected profile, query only that profile’s vector/index, and apply equivalent authorization and metadata filters plus authoritative source rechecks to every branch;
add parameterized and weighted reciprocal rank fusion with per-profile candidate limits, configurable k, stable-ID deduplication, deterministic ties, contribution diagnostics, global work/result budgets, and explicit all-profile versus degraded/partial completion policies;
keep candidate fusion distinct from semantic reranking: raw cosine or distance scores from different profiles are never averaged, and RRF is never presented as a cross-encoder judgment;
define a bounded provider-neutral candidate envelope for external cross-encoder, sequence-classification, or listwise rerankers containing only authorized current text/references, stable candidate IDs, source-version tokens, fused rank, branch provenance, and allow-listed metadata;
provide an optional, independently implemented Rust reference reranker worker behind that envelope, isolated from PostgreSQL and built around immutable model manifests, native tokenization, digest-verified artifacts, bounded token batches, declared score semantics, and deterministic stable-ID output; select a Rust-native or FFI-backed inference runtime only after a published compatibility, parity, latency, memory, and packaging spike;
treat every model and tokenizer artifact as separately licensed from pgContext source, record its exact revision, digest, provenance, and redistribution policy, and bundle no weights until a release review passes;
treat external reranker output as untrusted ordering input, restrict it to candidates in the issued envelope, and reapply source version, content hash, deletion state, filters, ACL, and RLS before returning final evidence;
support caller-selected require_reranker versus visibly degraded fused-order fallback; permission, source-version, and profile-mismatch failures never use permissive fallback;
register ordinary user-owned document tables, immutable chunking profiles, and user-owned chunk projection tables while keeping the original document row authoritative and every chunk/embedding/index rebuildable;
add a durable, idempotent ingestion outbox with bounded leases, retries, cancellation, supersession, typed states, and atomic publication of one validated chunk/embedding generation while the prior ready generation keeps serving;
run parsing, HTML cleanup, PDF/layout extraction, OCR, model-specific tokenization, embeddings, and semantic reranking in scoped external workers; start with deterministic plain-text, Markdown, and HTML adapters before separately certifying binary/layout parsers;
chunk structurally by page/region, heading, section, paragraph, list/table/code block, and sentence before tokenizer-safe windows; record byte, character, token, page/region, structure-path, parent, and neighbor provenance;
support optional contextual chunk enrichment in the external worker: generate a bounded document- or section-specific prefix for embedding and lexical inputs, store it separately from the original chunk, and version its model/prompt/revision/content hash. The enriched input is a rebuildable retrieval aid; citations, display, and final source validation always use the original authoritative span;
keep stable chunk occurrence identity separate from content_hash: exact hashes may enable cache reuse or requested duplicate collapse, but repeated text at two source locations remains two citable occurrences;
enforce configured input-byte, extracted-text/token, page/region, nesting, chunk-count, overlap, parser/tokenizer/model-time, staging, statement-size, batch, retry, and lease budgets; oversized or unsupported input fails explicitly and is never silently truncated;
publish profile and chunk-generation aliases atomically so operators can shadow, promote, drain, roll back, and optionally retire old profiles without an all-row rewrite or query-visibility gap;
expose privacy-bounded profile coverage, stale embeddings, chunk/job state, branch completion, fusion contributions, rerank drops/fallback, generation readiness, worker attempts, limiting budgets, latency, and storage cost without logging query text, chunk text, source keys, tenant IDs, or credentials by default.

Promotion is staged. Mixed-profile retrieval first requires known-answer and property tests for weighted RRF; different-dimension named-vector, filter, ACL/RLS, source-edit, missing-profile, and ANN exact-recheck gates; and held-out A-only, B-only, fused, and degraded quality results. The frozen equal-weight, eight-query workload passes at one million rows on PostgreSQL 17 and 18: fused recall is no worse than the stronger single-profile baseline under the same declared global budget, so mixed-profile retrieval is Stable. Semantic reranking then requires bounded-envelope, arbitrary-ID injection, concurrent source/permission change, final-revalidation, timeout/fallback, and held-out lift/latency/cost evidence. Automatic chunking requires deterministic span/citation fixtures, malformed and oversized input handling, worker crash/lease/retry/supersession, atomic publish/rollback/delete, backup/recovery, large-document bounded-memory, and mixed-profile end-to-end tests. The combined workflow graduates only after 1M/10M quality, latency, storage, mutation, restart, and security evidence shows that historical Model A and new Model B coverage can coexist without a mandatory full-corpus re-embedding.

PostgreSQL-Native Lexical Retrieval

Status: shipped. See Lexical retrieval.

Depends on: composite query execution, the shared fusion layer, registered field/type metadata, and bounded candidate-source execution.

Registered lexical sources bind ordered weighted text or JSON-path fields, or a stored/generated tsvector column, to a resolved text-search configuration, ranker, normalization, rank weights, and optional per-row tsquery. The typed plain, structured, phrase, web_search, prefix, distance, boolean, weight_restricted, and registered_tsquery forms compile to native constructors with bound values, attached GIN/GiST indexes serve bounded candidate probes that are authoritatively rechecked, and optional pg_trgm sources add typo tolerance. The legacy arbitrary-column simple full-text branch has been replaced.

Delivered scope:

register raw text, stored or generated tsvector, and caller-provided tsquery fields without copying them; validate type OIDs, ownership, configuration, expression stability, and schema drift;
select a text-search configuration per field instead of hardcoding simple, including language dictionaries, stemming, stopwords, thesauri, and application-defined configurations already installed in PostgreSQL;
support to_tsquery, plainto_tsquery, phraseto_tsquery, and websearch_to_tsquery, plus native Boolean, phrase/distance, prefix, and weight restrictions; malformed structured input must fail as a typed query error while the web-search form remains safe for raw user text;
support multi-column document construction with setweight, tsvector concatenation, JSON/JSONB text extraction, configurable A/B/C/D field weights, and source-aware null handling;
use existing GIN and GiST indexes for candidate generation, expose whether the chosen plan was indexed, lossy/rechecked, or exact-scanned, and preserve PostgreSQL’s row recheck semantics rather than maintaining a second inverted index;
support ts_rank and ts_rank_cd, normalization choices, positional proximity, and typed rank output for fusion; add bounded ts_headline generation as optional result hydration and document that callers must still HTML-sanitize untrusted output;
still open: corpus-statistics-aware BM25/BM25F-style ranking as an optional PostgreSQL-native ranker with explicit statistics/version semantics, deferred because it did not beat the native ts_rank/ts_rank_cd suite without adding an unproven statistics/storage contract;
still open: accept learned sparse outputs such as SPLADE through the first-class sparsevec path and fuse them with tsvector; model inference remains in the external worker and learned sparse scoring is benchmarked separately from PostgreSQL lexical ranking;
trigram fuzzy matching: add a pg_trgm similarity candidate source (word_similarity/%/similarity, backed by a GIN or GiST trigram index) so typo-tolerant and partial-token lexical retrieval can be fused alongside the dense, sparse, and full-text branches;
expose lexical and fuzzy branches through the typed query IR, composable with dense, sparse, quantized, recommendation, and late-interaction retrieval by RRF, weighted RRF, DBSF, and bounded formula reranking;
preserve PostgreSQL collation, MVCC, ACL/RLS, partition pruning, cancellation, statement_timeout, and source-row semantics on every path; never accept a lexical candidate merely because an index reported it;
expose parsed query/configuration, index identity/type, lexemes/candidates, heap rechecks, ranking work, source reranks, fallback, and budget termination through explain and observability.

PostgreSQL documents tsvector and tsquery as its native document/query types, provides the four query constructors and relevance-ranking functions, and recommends GIN as the preferred text-search index while also supporting GiST. The compatibility target is that native contract, not merely a separate structured text-filter API: types, querying and ranking, and GIN/GiST indexes.

Promotion requires result/rank fixtures against direct PostgreSQL SQL for every configuration/query form, proof that GIN/GiST branches avoid full-corpus work when selected, mutation and generated-column coverage, partition and RLS tests, and 1M/10M lexical-plus-hybrid quality and latency gates.

Query and Operations Priorities

Status: planned gap-closing after the typed composite executor. Existing stable or experimental pgContext features are not re-listed as new work.

Depends on: composite query execution, PostgreSQL-native lexical retrieval, automatic observability, and collection resource limits.

Dedicated retrieval engines have established a broad query and operations baseline: nested prefetch and multi-stage queries, RRF and DBSF fusion, formula scoring and decay, random sampling and ordering, discovery, result grouping, metadata and geographic filters, strict resource controls, tenant-aware locality, on-disk storage, and distributed shards. This track closes useful gaps while retaining PostgreSQL-native joins, security, durability, partitioning, replication, and backup.

Priority gaps:

Universal query execution. Generalize prefetch/rerank into bounded nested stages; add distribution-based score fusion (DBSF), parameterized and weighted RRF, score-threshold stages, and typed formula expressions over score plus registered numeric, timestamp, categorical, and PostGIS-derived features. Include linear, Gaussian, and exponential time/geo decay with deterministic null and unit semantics.
Additional query sources. Add dedicated random sampling and registered-field order-by branches where they make client APIs simpler; keep general cross-collection lookup as a safe PostgreSQL join rather than a restricted proprietary lookup mechanism.
Composable result grouping, facets, and aggregations. Generalize the existing stable exact grouped_search API into a typed group_by stage that can consume an arbitrary dense, sparse, lexical, fused, formula-scored, or semantically reranked child query. Preserve stable per-group limits and deterministic group order by the best child result; add an explicit maximum group count; define null/missing, multi-valued field, pagination, and tie semantics; and require stage ordering so later reranking cannot silently violate the requested one-result-per-document or per-field cap. Group fields remain registered, authorized PostgreSQL columns or JSON paths, and all child provenance survives grouping. Alongside grouping, add facet value counts over registered filter columns and JSONB paths and typed aggregation stages (count, min/max, sum, and mean over registered numeric and timestamp fields) computed over a query’s matched or candidate set, with an explicit maximum distinct-value budget, deterministic value ordering, declared null/missing/multi-valued semantics, and results computed only from rows visible under the caller’s snapshot, ACL, and RLS. A candidate-bounded facet or aggregate is labeled as such and never presented as a corpus total; corpus-scoped measurement belongs to Global Query Statistics and Coverage Profiles.
Richer filters. Add registered full-text, value-count/cardinality, UUID, array/nested-element, and optional PostGIS radius/bounding-box/polygon conditions with precise missing/null/array semantics and ordinary B-tree/GIN/GiST/BRIN/PostGIS index use.
Strict query governance. Finish enforcement—not only storage—of per-collection result, candidate, vector-comparison, filter-complexity, nesting-depth, timeout, memory, exact-search, oversampling, batch, and concurrent-query budgets. Offer a fail-fast mode for production workloads that require an indexed predicate or bounded plan.
Tenant-aware locality. Teach the advisor and planner to recommend or use PostgreSQL list/hash/range partitions, partial indexes, and tenant-leading indexes so tenant queries read local pages; test shared-small-tenant and dedicated-large-tenant layouts without weakening RLS.
Online optimization and storage tiers. Finish background per-segment build/compaction, immutable generation cutover, hot quantized codes in memory, cold full vectors/index pages on disk, and explicit degraded/exact-only behavior while an index is not ready. Queries must remain complete unless a caller explicitly selects an indexed-only policy.
PostgreSQL-native scale-out. Publish and test partition fan-out/global top-k merge, read-replica routing, physical/logical replication, failover, and Citus/FDW-style deployment patterns. This is operational parity, not a plan to add a competing consensus or snapshot subsystem.

Public dedicated-engine contracts and production workloads inform this gap analysis. Promotion remains per capability; there is no single parity checkbox.

Load Data and Query Immediately

Status: planned product workflow built from PostgreSQL-native ingestion and the exact-first collection lifecycle.

Depends on: collection registration, schema introspection, exact retrieval, background builds, the index advisor, and explicit resource budgets.

Goal: an application can load data with ordinary INSERT, COPY, or its existing migration/ETL tool, register the searchable columns once, and issue correct queries immediately. Derived vector and lexical indexes then improve the same query without a second ingest API or a correctness-changing cutover.

Scope:

provide one idempotent registration/advisor workflow that discovers eligible vector, halfvec, sparsevec, bitvec, vector-array, text, tsvector, scalar, JSONB, timestamp, UUID, and optional PostGIS columns;
validate primary/source keys, dimensions, metrics, text configuration, nullable/invalid values, model and normalization metadata, permissions, and existing useful PostgreSQL indexes before recommending changes;
make exact vector, lexical, filtered, and hybrid queries available as soon as registration commits; expose ExactOnly, Building, Indexed, Stale, and Degraded states without returning incomplete results by default;
select HNSW versus IVFFlat and full-precision versus quantized serving from corpus size, update rate, filter shape, memory, build window, and requested recall/latency objectives; show the evidence and generated DDL, and require an explicit policy before applying recommendations automatically;
build and publish indexes in the background with bounded CPU, memory, I/O, WAL, and temporary-disk use; preserve writes, cancellation, restart, and resumability throughout;
use set-based point mapping, validation, and payload mutation during bounded backfill and bulk workflows rather than moving client-side row loops into extension-side SPI loops;
support multiple embedding generations/named vectors and atomic aliases so a new model can backfill and validate before traffic switches;
define a provider-neutral external-worker contract for optional embedding generation. Model calls, credentials, retries, and rate limits stay outside the PostgreSQL backend query path; rows record model/version/provenance so stale embeddings can be found and rebuilt;
report ingestion/backfill progress, invalid-row samples, index readiness, recall checks, drift/skew, recommended maintenance, and storage cost through SQL views suitable for automation.

Promotion requires restartable 10M-row COPY/backfill tests, concurrent query and source-DML tests throughout every state transition, bounded-resource failure injection, and proof that index publication changes performance rather than visibility, permissions, exact-score semantics, or ordering among the returned candidates.

Mapped HNSW Serving

Status: implemented experimentally and lifecycle-gated for the PostgreSQL 17 profile.

Depends on: resumable generation publication, metadata-filtered ANN, and the shared graph-read port.

Scope:

own a real OS mapping with an immutable generation lifetime;
prohibit normal serving from reading the whole file into a vector or copying it through SQL bytea;
validate checksum, version, dimensions, offsets, alignment, architecture, truncation, and corruption;
implement graph traversal over a mapped wrapper with bounded-copy decoding;
reuse candidate-mask and authoritative source-recheck contracts;
cover generation replacement, reader pins, retirement, cleanup, crash recovery, corruption, and source changes;
run targeted Miri for validated pure views and sanitizer-backed subprocess tests for real mappings.

Validated by an end-to-end serving test with exact-oracle and bounded-work assertions before promotion.

Operator-Invocable Index Verification

Status: planned operations track. pgContext already validates supplied artifact bytes and exposes mapped-segment diagnostics and cleanup, but it has no single DBA-invocable verifier for a live index and all of its owned generations.

Depends on: frozen page/manifest/segment/codebook formats, safe relation and artifact resolution, mapped-generation lifetime rules, corruption taxonomy, and bounded maintenance privileges.

Goal: provide an amcheck-style, read-only-by-default verification surface (working name pgcontext.verify(index, mode)) that distinguishes structural integrity from retrieval-quality/recall checks. PostgreSQL’s amcheck is the operational model: verify the invariants that scans rely on, report corruption precisely, and do not pretend that verification itself repairs it.

Scope:

resolve the target by relation OID/regclass under a safe search path and verify that its access method, owner, database, source registration, and artifact roots match before reading any extension-owned path;
offer bounded quick, standard, and full modes. Quick checks metadata, headers, sizes, checksums, and generation references; standard walks pages and graph/segment invariants; full may cross-check every live source/index membership and therefore has separately documented locks, memory, I/O, and runtime;
verify PostgreSQL page headers and every pgContext page envelope/checksum carried by the format; metapage/version/configuration identity; tuple/node IDs and bounds; level, neighbor, reciprocity and reachability where required by the format, entry-point, and tombstone invariants; and source TID/key mappings where the selected mode permits a heap cross-check;
verify manifest publication state, segment identity/ranges, active/delta/base overlap, row/count summaries, codec/codebook revision and checksum, mapped file size/hash/alignment, reader pins, retirement state, and missing, duplicate, or orphaned generation files;
return stable typed findings with severity, code, relation/generation/segment, block or logical location, invariant, and operator guidance. Default output excludes source values, vectors, keys, payloads, and paths not safe for the caller;
require owner/MAINTAIN-like privileges for ordinary checks and a stricter role for checks that can expose filesystem or cross-tenant structure; honor cancellation, statement_timeout, interrupts, and resource budgets;
run safely against a declared snapshot and document what concurrent DML, VACUUM, compaction, publication, and retirement can change. A mode that cannot obtain a consistent view returns inconclusive or a typed blocker, never a false clean result;
keep repair separate. Findings may recommend REINDEX, rebuild, generation retirement, or cleanup, but verification performs no mutation unless the operator invokes an independently authorized repair operation.

Promotion requires known-good fixtures for every certified format; targeted corruption injection for page, graph, manifest, segment, codebook, mapped-file, and orphan cases; concurrent DML/VACUUM/compaction and cancellation tests; privilege/privacy checks; bounded quick/full resource measurements at 1M and 10M; crash/restart and replica verification; and proof that structural verification and recall self-checks report their different guarantees clearly.

Expanded Automatic Observability

Status: implemented and lifecycle-gated for the PostgreSQL 17 profile.

Depends on: executable query outcomes and every serving strategy that it reports.

Scope:

automatically persist actual strategy, visits, candidates, filters, rechecks, quantization, fallback, latency, cancellation, and budget outcome;
on PostgreSQL 17 and later, register low-cardinality extension wait events for real query-candidate I/O, generation publication, build, compaction, purge, and worker-queue waits so blocked backends are visible through pg_stat_activity/pg_wait_events; CPU work and ordinary execution states are not mislabeled as waits;
on PostgreSQL 18, feed the server’s extension cumulative-statistics API from the same typed counters used by the current bounded DSM pipeline, with defined reset/drop/persistence and no double counting. PostgreSQL 17 keeps a documented compatibility view over the existing collector;
on PostgreSQL 18, compare and adopt the server’s custom EXPLAIN option support for pgContext stage, budget, candidate, recheck, quantization, grouping, reranking, and degradation details while preserving stable text/JSON output and a version-gated PostgreSQL 17 fallback;
bound cardinality and exclude vectors, payload values, secrets, and tenant identifiers;
cover success, typed error, cancellation, concurrent updates, rebuild-required, not-ready, and corruption.

The query backend uses a bounded nonblocking named-DSM queue; a database-scoped worker commits observations independently so aborted statements can be reported without adding a synchronous catalog write to query latency. Queue health is restricted to pg_monitor, and delivery limitations are documented as best-effort, may-duplicate, and fail-open. The PostgreSQL 17 gate covers the complete outcome matrix above, privacy, strategy/work accuracy, disabled-vs-enabled latency, queue health, and worker reclamation. PostgreSQL native adapters are compiled and certified per major around one shared pure telemetry contract; their promotion additionally requires wait-state accuracy, statistics reset/restart/drop behavior, EXPLAIN text/JSON stability, privilege/privacy checks, and measured overhead with each surface independently enabled and disabled.

Global Query Statistics and Coverage Profiles

Status: planned engine-owned analytic surface. It involves no model inference and produces no derived text artifacts.

Depends on: exact retrieval oracles, composite query execution, registered filter columns and JSONB paths, bounded work governance, and automatic observability.

Goal: answer the deterministic slice of corpus-level questions — how much of a collection relates to a query, where that relevance sits, and how it distributes across registered facets and time — exactly and reproducibly. Coverage profiles also feed the answerability signals in evidence assembly and act as the measurement harness for every later corpus-level tier: corpus maps, sampling, and summary layers are graded against these statistics rather than against impressions.

Scope:

typed statistics requests over any retrieval branch: similarity-score histograms with declared binning, count-above-threshold, quantiles, and matched-row counts, computed on the exact path by default under bounded work, memory, and cancellation budgets;
optional sampled estimation for large collections with a declared sample design, seed, and confidence labeling; an estimate is never presented as an exact count;
joint distributions of relevance mass with registered filter columns, JSONB paths, and timestamp buckets, sharing facet semantics with Query and Operations Priorities; a query-scoped facet count over a bounded candidate set and a corpus-scoped coverage profile are distinct outputs, and every result labels which one it reports;
respect row visibility: statistics are computed only over rows visible under the caller’s snapshot, ACL, and RLS, so a count or histogram can never disclose the existence of rows the caller cannot retrieve;
deterministic output shapes with stable ordering and versioned semantics, exposed through the composite executor and explain/observability, without logging query text or row content by default;
publish the coverage-profile suite used to grade later corpus-level tiers: held-out corpus-level questions with exact statistics as ground truth.

Promotion requires property tests for histogram and quantile math against the exact oracle, ACL/RLS fixtures proving visibility-bounded counts, bounded-work and cancellation coverage, deterministic reruns, and 1M/10M latency and memory envelopes for both exact and sampled modes.

Extractive Corpus Maps

Status: planned deterministic derived-structure track. Build and serving involve no model inference; a map is rebuildable structure over authoritative rows, never a second truth.

Depends on: stable source and chunk occurrence identities, the shared clustering and codebook-training primitives from the typed quantization domain and IVFFlat work, term statistics from PostgreSQL-native lexical retrieval, versioned generation publication, and the background job machinery.

Goal: a versioned, inspectable map of a collection — bounded strata with deterministic term labels and representative members — built from embeddings, term statistics, and registered metadata alone. The map gives the corpus-level surfaces (sampling, rollups, hierarchical summaries) their structure and gives operators an inspectable answer to “what is in this collection.”

Scope:

cluster registered dense vectors with seeded, budget-bounded algorithms from the shared training domain (the k-means family first; hierarchical variants with declared linkage and depth limits), producing identical strata for identical inputs, configuration, and seed;
reuse structure the engine already builds where evidence supports it: IVFFlat list centroids may seed or serve as strata, and HNSW layer sampling may provide uniform-sample baselines; both are evaluated options, not assumed equivalences;
label each stratum deterministically with contrastive term weighting over the lexical projections (class-based TF-IDF as formulated by BERTopic) plus registered metadata distributions; labels record the term-statistics revision that produced them;
select representative members per stratum by declared criteria — medoids or coverage/diversity objectives such as facility-location selection (Lin & Bilmes) — with deterministic ties;
persist maps as versioned generations with atomic publication, staleness tracking against source mutation, bounded incremental refresh, and a full rebuild path;
bind each map generation to a declared visibility scope (whole collection, tenant/partition, or registered predicate): stratum labels and sizes are aggregates over member rows, so a map is served only within its scope, while representatives are ordinary rows that are additionally rechecked per caller;
expose map inspection — strata, sizes, labels, representatives, staleness, and generation — as SQL surfaces usable by operators and by downstream corpus-level stages.

Promotion requires determinism fixtures (identical inputs reproduce identical maps), labeled-dataset quality checks against random-partition baselines, mutation/staleness/recovery matrices, visibility-scope enforcement fixtures, and bounded build cost at 1M and 10M vectors.

Stratified Global Evidence Sampling

Status: planned first corpus-level evidence surface. Engine-owned and model-free; the final SQL spelling is frozen during API design.

Depends on: extractive corpus maps or declared deterministic partitions, the evidence assembly budget and diversity machinery, composite query execution, and authoritative source rechecks.

Goal: one bounded call returns a corpus-level evidence pack — a diversified, token-budgeted selection of authoritative rows or chunks that covers a collection’s strata, optionally conditioned on a query and filters — with provenance describing the sampling design. An external model synthesizes corpus-level answers from rows the caller is entitled to see; the extension generates no text.

Scope:

declared sampling designs: proportional, equal-per-stratum, and relevance-weighted allocation; per-stratum selection by representative rank or query-conditioned top-k; deterministic output for the same map generation, design, and seed;
resolve every returned row from the authoritative source under the statement snapshot, filters, ACL, and RLS. A pack contains only individually visible rows, and stratum metadata inside a pack reports visible membership only;
apply evidence-assembly machinery within and across strata: deduplication, near-duplicate collapse, MMR-style diversity, per-source caps, and token budgets, with deterministic ordering;
record per-row provenance — basis (sampled or extractive), map generation, stratum identity and label, allocation rule, and selection criterion — so external synthesis can cite rows and evaluation can attribute misses;
integrate as a composite-query stage so global-over-a-filtered-subset — the most common production shape, such as one tenant’s documents over one time range — is the same call with a filter;
define degraded modes: a collection without a published map falls back to a declared seeded-uniform design or a typed refusal per caller policy, and a stale map is served with an explicit staleness label, never silently;
grade packs against an exhaustive partition-synthesis oracle on held-out corpus-level questions — every stratum’s full content mapped through the evaluation model and hierarchically reduced — which defines the evidence-recall ceiling a bounded pack is measured against.

Promotion requires determinism and coverage property tests, ACL/RLS fixtures proving packs and stratum metadata never disclose invisible rows, bounded work at 1M/10M, and held-out evidence-coverage results against both the exhaustive oracle and naive top-k retrieval.

Summary Artifacts and Deterministic-Partition Rollups

Status: planned first model-derived corpus-level tier. Summarization runs in provider-neutral external workers; the PostgreSQL backend never performs model inference.

Depends on: registered document sources and the provider-neutral worker contract, versioned generation publication, deletion-propagation lineage, evidence assembly, and the aggregate-visibility contract defined here.

Goal: one shared contract for any stored text derived from many source rows — the summary artifact — plus its simplest producer: rollup summaries over deterministic partitions such as time windows, tenants, or registered facets. Deterministic partitioning avoids clustering risk, keeps staleness append-mostly, and lets visibility scope align with the partition key, which is why this tier is sequenced before hierarchical summaries.

Summary-artifact contract:

every artifact records its producing profile (provider/model/prompt/revision and parameter hash), an explicit member set of stable source/chunk occurrence IDs, a content hash, a generation, a staleness state, and a declared visibility scope; artifacts are derived and rebuildable, never authoritative;
aggregate visibility: an artifact is served only to callers whose row visibility covers its declared scope. Scope-aligned builds — per tenant or per RLS partition — are the primary strategy; the engine never synthesizes per-caller partial summaries, and a failed scope check degrades to a model-free basis or a typed refusal per caller policy;
artifacts join deletion-propagation lineage: purging a member marks dependent artifacts for rebuild-without-member under the same deadline/state machinery, and audit evidence stays content-free;
artifacts may be embedded and indexed as registered derived chunks with full lineage, so summary retrieval is ordinary retrieval and citations chain answer → artifact → members → source rows;
the worker envelope carries only authorized member text references, the profile, and budgets; the engine validates returned text against the issued member set, versions it, and publishes atomically.

Rollup scope:

registered rollup definitions: a partition key (temporal bucket, tenant column, or registered facet), a summary profile, a resolution chain (for example day → week → month as chained artifacts with cross-level lineage), and a refresh policy;
incremental maintenance: only open or dirtied buckets re-summarize; closed buckets stay immutable unless member mutation or purge invalidates them;
serving: rollup artifacts join evidence packs with basis rollup, per-bucket provenance and staleness labels, and a declared fallback to the sampling basis when a bucket summary is missing or stale beyond policy;
observability: bucket coverage and staleness, job and lease states, and worker-reported token cost per bucket, without logging summary or source text.

Promotion requires visibility-scope and RLS fixtures, purge-lineage rebuild-without-member tests, atomic publication and crash/restart coverage, worker crash/lease/retry matrices, append and mutation staleness fixtures, and held-out temporal and corpus-level question results demonstrating lift over the model-free sampling basis at reported cost.

Hierarchical Corpus Summaries

Status: planned full-strength abstractive tier for corpus-level questions. Research inputs are RAPTOR for recursive cluster summarization and Microsoft’s GraphRAG and LazyGraphRAG for eager versus deferred summary generation; per the roadmap’s research policy, implementations remain independently written pgContext Rust and worker code.

Depends on: the summary-artifact contract and rollup machinery, extractive corpus maps for cluster structure, bounded automatic chunking and versioned embedding generations, and evidence assembly. The graph-community variant of this capability lives in Graph-Augmented Retrieval and additionally requires the pgContext-owned topology projection.

Goal: multi-level abstractive summaries over corpus-map strata — recursively up to a declared depth — so thematic corpus-level questions retrieve from summary levels while every summary stays source-linked and rebuildable. Two generation policies share every other contract: eager builds materialize the hierarchy at index time; lazy builds materialize a stratum’s summary the first time demand touches it and cache the artifact.

Scope:

produce level-N+1 summary artifacts over corpus-map strata and recurse under declared depth, fan-in, and token budgets; each level is embedded and indexed as derived chunks so collapsed multi-level retrieval and level-targeted branches are ordinary composite stages;
eager policy: resumable, budgeted background jobs build the full hierarchy; source mutations dirty only affected ancestor paths, and re-summarization is bounded to dirty strata rather than the corpus;
lazy policy: the map itself is built eagerly because it is cheap and model-free, but a request touching an unsummarized stratum immediately returns the declared degraded basis (sampled, extractive, or rollup) and enqueues a bounded summarization job; later requests serve the cached artifact. A backend query never blocks on model inference; callers choose a require-summary policy (typed not-ready outcome) or degrade-now per call;
cache and cost governance: per-collection budgets on artifact count, bytes, and worker tokens; a retirement/eviction policy for cold strata; profile hashes so regenerated summaries are comparable across time;
visibility: hierarchies are built per declared scope under the summary-artifact contract; mixed-visibility strata are handled by scope-partitioned builds, never per-caller synthesis;
map-side query surface: a bounded, ranked, token-budgeted selection of summary artifacts at a caller-chosen level — ranked by embedding similarity, coverage statistics, or both — with full member lineage; the reduce step, final synthesis across selected summaries, stays outside the backend with the caller or its workers;
grade each level against the exhaustive partition-synthesis oracle, the sampling basis, and the rollup basis on held-out corpus-level suites, reporting evidence recall, citation precision through summary lineage, staleness, and cost per query and per build.

Promotion requires every summary-artifact gate plus collapsed-retrieval correctness (a summary hit always resolves to visible members with an intact citation chain), eager/lazy equivalence fixtures for identical profiles, invalidation matrices under mutation and purge, deterministic degraded modes, and 1M/10M cost and quality curves. A summary level that cannot demonstrate lift over the model-free sampling basis on its declared question class does not promote.

Evidence Assembly, Provenance, and RAG Evaluation

Status: planned product and quality track after the core retrieval branches are stable.

Depends on: composite query execution, PostgreSQL-native lexical retrieval, stable source/model identifiers, the mixed-profile and automatic-chunking workflow, bounded hydration, and automatic observability. Graph expansion and multimodal evidence are optional inputs, not prerequisites for the first evidence contract.

Goal: make PostgreSQL sufficient as the retrieval and evidence authority for a RAG application. pgContext should return a compact, inspectable evidence set that an external model can use and cite. It does not claim that retrieval alone eliminates hallucinations, and it does not move generator inference, credentials, or provider retry logic into a PostgreSQL backend.

Scope:

define a typed evidence result containing stable source table/key identity, content version or hash, field/chunk/page/region location, model and configuration version, candidate branch, approximate and exact scores, filter/rerank decisions, and index generation where applicable;
assemble context with bounded deduplication, near-duplicate collapse, diversity/MMR, parent and neighbor expansion, per-source caps, token budgets, and deterministic ordering while retaining the provenance of every included span;
support parent/child documents and derived hierarchical summaries as rebuildable, source-linked artifacts. Evaluate late chunking and contextual retrieval prefixes generated by the authorized external worker, plus RAPTOR-style retrieval trees without allowing an enrichment or summary to replace or lose its source rows. The storage contract and generation policies for those summaries are specified in Summary Artifacts and Deterministic-Partition Rollups and Hierarchical Corpus Summaries; corpus-level evidence packs declare their basis — sampled, extractive, rollup, or summary — in provenance;
allow bounded graph expansion, multi-vector evidence, temporal/freshness rules, and structured joins as explicit query stages with their own budgets and provenance rather than hidden post-processing;
return answerability and evidence-coverage signals, including an explicit insufficient-evidence outcome, while leaving the application’s generator free to abstain or request a wider query;
expose a provider-neutral evaluation record that joins a versioned query, judged relevance/evidence, retrieved sources, assembled context, citations, and optional external answer scores without logging source text or tenant identifiers by default;
maintain held-out suites for single-hop, multi-hop, temporal, filtered, conflicting, duplicate, unanswerable, multi-turn, and permission-sensitive questions. The multi-turn lane will draw on the mtRAG benchmark where its data and license fit the public harness.

Promotion requires deterministic source/citation lineage, ACL/RLS-safe hydration, bounded context work, mutation and model-version tests, and published retrieval recall/nDCG/MRR plus citation precision/recall, evidence coverage, abstention, freshness, and latency. Generator-specific answer scores remain a separately versioned optional layer so a model change cannot silently rewrite retrieval claims.

Reproducible Public Benchmarks

Status: planned; the 10M harness and dataset work starts before all target indexes are complete, while certification waits for the relevant serving lifecycles.

Depends on: PG17 V1 freeze and the existing GloVe-100-angular comparison harness. Full 10M certification additionally depends on segmented HNSW, IVFFlat, the quantized codec matrix, background maintenance, and the lexical, composite, multi-vector, and evidence paths being tested.

The published pgContext-versus-pgvector comparison is measured at a single corpus size (GloVe-100-angular, roughly 1.18M vectors) on Apple Silicon. This track turns that useful result into a multi-scale engineering and release program.

Dataset and ground-truth matrix

retain GloVe-100-angular as the regression bridge, then add deterministic 100k, 1M, and 10M corpus tiers; use an archived 10M subset from a recognized large-scale ANN corpus such as Big ANN Benchmarks plus at least one legally redistributable modern embedding corpus;
cover representative dimensions (low-dimensional ANN plus 384, 768, and 1,536 where datasets/models allow), cosine, inner product, and L2, with L1, bit, sparse, and late-interaction lanes reported separately rather than averaged into a misleading total;
include clustered, skewed, duplicated, near-duplicate, normalized, non-normalized, and distribution-drift fixtures so an algorithm is not tuned only to one friendly embedding distribution;
add 10M-row metadata with controlled selectivities from 0.001% through 100%, correlated and anti-correlated predicates, arrays/JSONB, timestamps, and shared/dedicated tenant layouts;
add a 10M-document lexical/hybrid lane with stored tsvector, GIN candidates, dense and sparse branches, typo/phrase/prefix queries, and a versioned judged-query set for recall, nDCG, MRR, and fusion evaluation;
add held-out multi-vector, Matryoshka, and multimodal lanes with exact-MaxSim/full-dimension oracles, model/version metadata, page/region judgments, and separate candidate-versus-final quality measurements;
add a source-linked RAG evidence lane with answerable, unanswerable, multi-hop, temporal, contradictory, duplicate, and permission-sensitive queries; retrieval and citation quality are primary, while generator scores are optional and separately versioned;
compute exact top-100 ground truth with a blockwise, checksum-recorded full-precision implementation independent of the ANN path; validate a sample through PostgreSQL exact SQL and a second reference implementation before accepting the corpus.

Dataset manifests record source/license, checksums, preprocessing, normalization, model/version, train/base/query splits, dimensions, filters, relevance judgments, and ground-truth generator version. Generated datasets must be reproducible from a seed; downloaded datasets must be content-addressed and may not silently change.

Required workload lanes

Search frontiers: exact, HNSW, IVFFlat, and every promoted quantized variant at recall@10 targets including 0.90, 0.95, 0.99, and the highest attainable operating point; report the whole speed/recall frontier rather than a single preferred setting.
Filtering and tenancy: indexed and unindexed PostgreSQL predicates, adaptive exact/ANN crossover, ACORN-like graph expansion, IVFFlat probe widening, partition pruning, RLS, and global versus tenant-scoped queries across the controlled selectivity range.
Hybrid quality: dense + tsvector, dense + sparse, three-way fusion, DBSF/RRF/weighted RRF, formula/decay rerank, quantized-prefetch + full-precision rerank, learned sparse + native lexical, pooled-vector and MUVERA prefetch + exact late interaction, Matryoshka prefix + full-dimension rerank, and multimodal fusion.
Concurrency: cold and warm cache; single-client latency; 16-, 32-, and 64-client throughput; read/write mixes; connection pooling; and cancellation storms under fixed CPU, memory, and I/O limits.
Build and ingestion: binary COPY, multi-row and heterogeneous batch mutations at 1, 100, 1,000, and 10,000 rows, concurrent source writes, serial/parallel/external builds, k-means training, codec generation, index publication, background compaction, restart/resume, and time until exact versus indexed queries are available. Batch runs also report SPI statement count, WAL per row, memory, and lock contention.
Sustained mutation: insert/update/delete churn at and above the existing 500-updates/second G4 exit target, distribution drift, tenant promotion, overlapping batch writers, payload changes, compaction debt, VACUUM, REINDEX, and 24-hour steady-state behavior without unbounded index growth or p99 collapse.
Durability and operations: WAL volume, checkpoint pressure, replica lag, crash recovery, standby promotion, backup/restore, partition attach/detach, disk-full/low-memory behavior, and old-generation reclamation while the 10M source remains authoritative.
Evidence and RAG quality: source/citation lineage, context deduplication/diversity, parent/neighbor/graph expansion, token-budget behavior, citation precision/recall, evidence coverage, freshness, abstention, and multi-turn stability on held-out judgments.

Measurements and acceptance

Every run records recall@k, nDCG/MRR where applicable, citation and evidence metrics for RAG lanes, p50/p95/p99/max latency, QPS, CPU time/instructions/cycles/cache misses when available, RSS and shared_buffers, mapped/resident bytes, index and code bytes/vector, temporary disk, page faults and PostgreSQL buffer reads/hits, build/training wall time, WAL bytes, replica lag, source-rerank count, and result completeness. It also records hardware, NUMA/topology, storage, filesystem, PostgreSQL settings, extension versions, container/native boundary, cache state, concurrency, dataset checksum, commit, and date.

Acceptance thresholds are frozen in the benchmark manifest before an optimization is tuned. At minimum:

every result respects MVCC, ACL/RLS, filters, deletion state, exact source scores, and deterministic ordering of returned candidates; no latency or recall win can waive correctness;
public ANN claims state the recall target and confidence interval, and public hybrid claims use a held-out relevance set rather than vector-neighbor recall as a proxy for search quality;
the chosen 10M configuration fits its declared RAM and disk budget, completes build/recovery within the declared operational window, and remains within its p95/p99 and write-churn envelope for the full steady-state run;
matched comparisons give pgContext, pgvector, and each declared external engine the same hardware, source/query vectors, concurrency, build wall-time or memory budget, and target recall. Differences such as in-process SQL versus a local service API, exact source rechecks, segment parallelism, durability, and full-vector retention stay visible rather than being normalized away;
a result is publishable only after one repeat on x86-64 and one on arm64, with no architecture-specific correctness difference and all raw artifacts retained.

Performance work driven by the 10M profile

The following are benchmark hypotheses, not promises. Each lands only if it moves a measured frontier without weakening lifecycle or correctness:

cache-aware page/node/list layout, neighbor/identifier compression, vector-code interleaving, software prefetch, batched distance evaluation, and fewer heap/source fetches before exact rerank;
SSD-resident routing/code layouts, batched page reads, and streaming builds, benchmarked against mapped HNSW and IVFFlat under the same RAM and I/O budgets;
AVX-512/VNNI and future arm64 SVE2/SME kernels behind runtime dispatch, while retaining scalar, AVX2/FMA, and NEON reference paths;
per-query parallel segment/list search with a bounded global top-k merge, adaptive parallelism, NUMA-aware worker placement, and protection against concurrent-query oversubscription;
better ANALYZE statistics and planner costing for filter cardinality, tenant/partition locality, centroid/list skew, HNSW selectivity crossover, and source-recheck I/O;
streaming/external-memory HNSW and IVFFlat builds, parallel k-means, batched WAL/page publication, background compaction, and build-time quantization;
optional GPU acceleration for offline exact-ground-truth generation, k-means, codec training, and index construction. CPU serving remains the baseline; GPU serving becomes a product dependency only if end-to-end deployment and fallback evidence justifies it. Published GPU indexing systems remain comparison points.

Execution tiers and publication

pull requests run small deterministic property/regression fixtures and a reduced performance smoke that catches gross regressions without noisy pass/fail claims;
nightly lanes run the 1M matrix and compare against a pinned baseline with noise-aware thresholds;
scheduled or manually approved dedicated runners execute the full 10M matrix, with serial workload lanes where resource contention would invalidate a comparison;
release candidates repeat the relevant 10M certification lanes on both architectures and preserve one immutable report bundle.

The third-party lane uses a community-recognized harness such as ANN-Benchmarks and/or Big ANN Benchmarks in addition to the PostgreSQL-aware suite. Each result is archived with raw samples, summaries, plans, logs, dataset and ground-truth manifests, deployment/build budgets, and a one-command reproducer. Charts are generated from those artifacts; no published number is manually transcribed.

Post-V1 Release Engineering

Status: planned after the installable PostgreSQL 17 GitHub V1.

This track strengthens production claims and adds distribution choices. It is not evidence that the corresponding platform, package, or operational envelope is supported by V1.

Extended PostgreSQL 17 production certification

run targeted Miri on pure unsafe storage/page views and Linux ASan/TSan subprocess suites across pgrx/pg_sys, real mappings, callback containment, and concurrent readers;
repeat crash-before-checkpoint, WAL replay and standby promotion, MVCC/HOT/TID reuse, VACUUM/REINDEX, generation replacement, backup/restore, and low-memory recovery matrices on one unchanged candidate;
establish deterministic recall, latency, throughput, build-time, memory, page-fault, update-cost, cancellation, and filtered-search envelopes at useful collection sizes;
run long fuzz campaigns and extended failpoint sweeps, minimize failures, and retain deterministic regressions;
promote experimental capability labels only when the relevant evidence is complete. A GitHub V1 installation claim alone is not production certification.

PostgreSQL-major and operating-system certification

The current support matrix is PostgreSQL 17 and 18 only. PostgreSQL 15 and 16 are not implicit future targets; adding either requires a separately approved support-policy and pgrx/toolchain plan plus the complete evidence below. PostgreSQL 19 is a forward-readiness target only: the PostgreSQL development roadmap currently plans its final release for September 2026, and pgrx has begun publishing beta-era PG19 support. Readiness evidence is explicitly not a pgContext support, compatibility, package, or release claim.

verify real upgrade, rollback, dump/restore, pg_upgrade, and format rejection or migration paths from prior extension releases;
deepen PostgreSQL 17 and 18 certification independently with exactly one pgrx feature and a real pg_config for each claimed major;
once the exact pinned pgrx release supports the selected PostgreSQL 19 beta/RC, run a non-blocking, allowed-to-fail readiness lane for compilation, generated-SQL/schema diff, extension install, exact/vector smoke, HNSW create/query/DML/VACUUM/REINDEX, crash restart, and artifact-format rejection. Pin the server prerelease precisely and advance it deliberately;
after PostgreSQL 19 reaches RC, rehearse pg_upgrade from 18 and inventory changed extension, access-method, statistics, EXPLAIN, wait-event, AIO, and packaging APIs. Do not publish PG19 binaries/images or add it to the support matrix until GA plus a separately approved policy change and the same certification evidence required of 17 and 18;
run full extension install and smoke gates on real Linux and macOS hosts;
publish a support claim only for major/platform pairs backed by preserved evidence from the same candidate.

Additional packages, images, and signatures

produce reproducible native install packages only for certified major/platform pairs, then test install, uninstall, upgrade, and rollback from those packages rather than from a checkout;
preserve per-major healthcheck, playground, provenance, and vulnerability evidence for the PostgreSQL 17 and 18 images on both supported architectures;
add maintainer-controlled artifact and container signing with documented identity, timestamping, verification, rotation, and revocation procedures;
attach signed checksums, SBOMs, provenance, and license inventories to future releases without committing credentials;
maintain and release-test the shipped Evokoa Homebrew tap and PGXN distribution for every supported extension/PostgreSQL pair, including install, upgrade, uninstall, rollback, checksum, and source-build evidence;
add APT/Yum repositories, native .deb/.rpm packages, or other registries only after their support, signing, upgrade, and rollback contracts are certified.

Release cadence and maintenance

define supported-version windows, security patch policy, deprecation policy, and a tested release rollback procedure;
automate scheduled dependency and security-advisory review;
keep expensive certification manual or infrequent, and retain bounded checks on ordinary pull requests;
preserve complete benchmark and certification reports for each released version.

Ecosystem and Framework Integrations

Status: planned after the relevant SQL/query surface is stable. Framework support does not block core retrieval or 10M certification.

Depends on: stable collection/query contracts, the load-and-query workflow, typed SQLSTATE/status behavior, versioned examples, and at least one certified package usable in integration CI.

Scope:

keep SQL and ordinary PostgreSQL drivers as the primary, complete integration contract; framework adapters remain thin translations and may not invent retrieval semantics unavailable through SQL;
publish provider-neutral reference adapters and runnable examples for the leading Python and JavaScript AI retrieval ecosystems selected at implementation time, followed by other languages when maintained demand and contributor ownership justify them;
cover registration, bulk backfill, named dense/sparse vectors, metadata filters, lexical/hybrid queries, recommendation/discovery, prefetch/rerank, streaming/keyset pagination, model-version aliases, recall checks, readiness, cancellation, and typed errors without requiring a second vector service;
define one canonical request/result mapping and generate or mechanically test adapter-specific translations so score direction, filter missing/null/array semantics, limits, pagination, and source keys cannot drift by framework;
provide minimal embedding-worker examples that use the provider-neutral external-worker contract; credentials, model calls, retries, and rate limits remain outside PostgreSQL and outside the core adapter;
run a compatibility matrix against supported framework/client versions and PostgreSQL 17/18, with install-from-released-package tests rather than only workspace-path dependencies;
document ownership, semantic-version policy, deprecation window, release cadence, and the behavior when an upstream framework changes or an adapter is no longer maintained.

Promotion is per adapter. It requires end-to-end tests against a real packaged extension, exact/ANN/hybrid result fixtures shared with the SQL oracle, filter and score-direction compatibility, cancellation/error tests, and a maintained owner. An example repository or generated client without those gates remains community/experimental rather than an official integration.

Graph-Augmented Retrieval

Status: planned graph foundation with an experimental, explicitly opt-in beam search track. It is dependency-ordered after stable source identifiers, composite execution, mapped serving, and bounded stage budgets. Specific public APIs and dates remain uncommitted.

Depends on: shared row/source identities, composite query execution, bounded stage budgets, mapped-generation lifecycle, exact source reranking, and the source/model provenance contract.

pgContext is built by Evokoa, which also maintains pgGraph, a graph extension for PostgreSQL. The planned integration is deliberately not a runtime Rust-ABI link and does not require the pgGraph extension to be installed. pgGraph remains an independent product and the upstream design/source reference.

The goal is true graph-aware hybrid retrieval: topology participates in deciding which candidates are explored, not merely in a SQL post-filter or final reranking pass. PostgreSQL rows and relationships remain authoritative; topology projections, ANN graphs, summaries, and search state are derived and rebuildable.

Integration decision: a selective pgContext-owned fork first

There are two viable source-sharing options, but they are not equal roadmap branches:

Option	Benefit	Cost	Roadmap decision
Refactor pgGraph so a small topology kernel becomes a separately versioned pure-Rust crate	One implementation and easier two-way fixes	Requires an up-front pgGraph refactor, a second release contract, coordinated compatibility testing, and tighter release coupling	Deferred convergence path
Port selected pgGraph modules into pgContext and adapt them behind a pgContext-owned boundary	Lets pgContext control its release cadence, artifact format, memory policy, and retrieval-specific evolution	Some fixes must be reviewed and carried between repositories manually	Chosen for the first implementation

The initial implementation is therefore a selective first-party fork, not a wholesale copy of pgGraph. pgContext owns the imported code after the port, including its tests, format versions, lifecycle, and compatibility policy. A shared crate becomes preferable only when all of the following are true:

the reusable boundary has stabilized around topology primitives rather than either extension’s SQL or PostgreSQL runtime;
both repositories are repeatedly carrying the same fixes or algorithms;
the crate can depend on no pgrx, SPI, PostgreSQL catalog, backend-global, query-executor, or extension-specific types;
the crate can have independent semantic versions, exact dependency pins, and a compatibility suite without forcing synchronized pgContext and pgGraph releases.

If those conditions are later met, the shared crate replaces the internals of the pgContext topology adapter without changing the query-owned retrieval contract. The fallback is not to move a large catch-all portion of pgGraph into a library; only the stable topology kernel qualifies. Direct Rust-ABI calls into a separately loaded pgGraph extension are not a third option: private Rust layouts, compiler and feature choices, panic behavior, allocator ownership, PostgreSQL load order, and symbol visibility do not form a stable extension boundary.

Every initial port records the pgGraph repository commit, source paths, applicable license and notices, local changes, and validation fixtures. Updates are manual, reviewed ports in either direction; there is no automatic subtree merge and no promise that the two repositories move in lockstep. pgContext’s tests and performance gates are authoritative for code shipped by pgContext.

pgContext crate and dependency boundary

The planned workspace split follows the boundaries already used by pgContext:

a new context-topology crate owns pure topology identities, adjacency views, generation validation, bounded traversal primitives, and topology format versions; it depends only on context-core and ordinary pure-Rust dependencies;
context-query owns beam state, work budgets, diagnostics, and a small topology-expansion port. context-topology does not depend on context-query;
context-build constructs topology projections from authoritative edge rows, while context-storage provides generic versioned segment and mapping machinery. Neither layer decides retrieval ranking;
context-hybrid owns graph/vector/lexical score composition and calibration;
context-pg is the composition adapter. It joins the pure crates to PostgreSQL snapshots, SPI, catalogs, ACL/RLS checks, interrupts, collection lifecycle, and source-row exact rechecks.

The existing CandidateSource page contract remains the leaf-candidate boundary. Incremental graph expansion needs a separate, query-owned port so a beam can request a bounded neighbor batch for specific frontier states without exposing topology storage objects to the executor. The concrete adapter lives at the PostgreSQL edge and composes context-topology with context-storage. This direction avoids circular dependencies and keeps pgrx out of the topology kernel.

The topology artifact format is versioned separately from crate versions, pgContext extension versions, and any pgGraph format. An unsupported or corrupt generation is never interpreted optimistically: it reports rebuild_required, and the authoritative rows are used to rebuild it.

Exact pgGraph scope

The first source audit may select and adapt only these capabilities:

stable node and relationship identity primitives that can map to pgContext’s existing SourceKey and PointId authority;
directed, typed, and optionally weighted adjacency with forward and reverse traversal;
compact immutable adjacency images, active/tombstone state, and delta-overlay algorithms;
bounded neighbor and frontier expansion, cycle/hub controls, generation watermarks, and deterministic traversal fixtures.

The acceptance boundary is capability-based, not a target percentage of the pgGraph repository. The audit records the exact modules, lines, dependencies, and tests selected. If the useful slice starts pulling in SQL parsing, catalog state, or the graph database runtime, the port is too broad and must be reduced or reimplemented behind the pure boundary.

The port adapts or reimplements these pgContext-specific responsibilities rather than copying them from pgGraph:

PostgreSQL snapshot, transaction, interrupt, ACL/RLS, partition, and authoritative source-row recheck behavior;
mapped-generation lifecycle, eviction, memory accounting, artifact naming, rebuild, crash recovery, and publication;
beam state, work admission, multi-provider scoring, dominance, exact rerank, diagnostics, and evidence-path assembly.

pgContext does not initially import pgGraph’s SQL/GQL surface, catalog and schema-management layer, general graph analytics, hydration APIs, job system, background-worker framework, client/package surface, or complete graph database runtime. The source audit rejects modules that cannot be separated from those surfaces without pulling their dependency graph into pgContext.

A future optional adapter may ingest relationships authored through pgGraph by using a documented, versioned SQL/schema contract. It must still materialize a pgContext-owned projection; it never borrows pgGraph process memory or private symbols, and the retrieval engine continues to work when pgGraph is absent. The adapter publishes an explicit supported-version matrix and rejects unsupported pgGraph versions before reading relationship data.

Optional residency

The graph layer must be genuinely optional because even a compact topology can add substantial resident memory. Each query supports off, auto, and required topology modes, while each projection reports an explicit absent, cold, active, paused, or rebuild_required lifecycle state. In off mode pgContext maps no topology generation, allocates no topology frontier, and runs no topology-maintenance worker for that collection; rebuildable on-disk artifacts may remain. auto may lazily map a compatible generation within a declared residency budget and fall back visibly. required fails with a typed status rather than silently degrading. Mapped generations are pinned only for the queries using them and become evictable afterward. Results and EXPLAIN-like diagnostics state whether topology was used, skipped, or unavailable.

Runtime off is the required control because a compile-time Cargo feature does not help a database that serves graph and non-graph collections in the same process. A later build feature may omit graph code for smaller packages, but it does not replace the runtime contract. The graph-disabled RSS gate measures steady-state memory as well as allocations during query execution.

Experimental virtualized beam search

“Virtualized beam search” is the roadmap name for a multi-source lazy beam search; it is not presented as an established algorithm name. The engine maintains one bounded logical beam across HNSW, topology, lexical, and relational-anchor candidate providers while materializing graph pages, vectors, and complete evidence paths only when expansion requires them.

Beam states remain compact: source-row identity, optional HNSW point and topology-node identities, parent-state identity, transition kind, path-pattern state, hop count, and score components. Parents live in a bounded arena so states do not copy complete paths. Each round selects the best expandable states, groups work by provider for batched page/neighbor access, scores newly admitted states, applies authorization, cycle, hub, duplicate, and dominance pruning, and retains the next bounded beam. Only selected finalists receive authoritative row visibility checks, exact full-precision vector reranking, and evidence-path reconstruction.

Correctness and resource limits are part of the algorithm, not tuning notes. The request contract bounds beam width, expansion batch width, admitted states, visited keys, vector expansions, topology edges/pages or bytes, hop depth, exact reranks, evidence paths, parent-arena bytes, and elapsed time. Dominance keys include row identity, path-pattern state, and authorization context rather than collapsing every visit to the same node. Ranking preserves its vector, lexical, relationship, path-progress, hop, hub, and duplicate components so the final result is explainable and can be recalibrated without hiding provenance. With topology off, the same executor creates no topology state and behaves as an HNSW/vector-only search path.

Staged delivery

Boundary audit and import spike. Inventory candidate pgGraph modules and transitive dependencies, create the source/license/provenance manifest, and port the smallest useful fixture into context-topology. Reject PostgreSQL coupling at the crate boundary and write differential tests against the pinned pgGraph behavior. The default result is the selective fork; promote a shared-crate extraction only if the convergence conditions above are already demonstrably true.
HNSW-only experiment — implemented internally. The existing bounded best-first HNSW traversal now runs behind an internal lazy cursor with typed terminal diagnostics. Differential and release-mode gates cover ordering, work, cancellation, memory, RSS, and latency parity. No public default or SQL contract changed.
Virtual state and memory governance — implemented internally. The vector-only kernel owns a compact parent arena, provider batching, deterministic dominance/cycle pruning, hard admission and visited budgets, explicit incomplete outcomes, separated score telemetry, and bounded path reconstruction. Its graph-off gate preserves P15 ordered output and work within frozen latency, retained-memory, and RSS ceilings. Topology remains rejected and no SQL or planner surface is exposed.
Topology projection and residency. Add the versioned projection, build/publication path, query-owned expansion port, and runtime lifecycle. Prove that off is nonresident, auto fallback is observable, and required is deterministic before enabling graph-aware experiments outside development builds.
Mixed and deep graph retrieval. Add graph/vector transitions, typed path constraints, query-adaptive transition scoring, bidirectional or graph-seeded search where justified, and minimal evidence-subgraph reconstruction. Topology must be able to alter candidate discovery, not only rerank an already fixed vector set.
Promotion decision. Consider a stable API or default only after the mixed search clears the validation gates below. An experiment may be removed or redesigned without compatibility guarantees before promotion.

Scope:

share stable row/point identifiers and typed relationship references without duplicating source records;
support retrieval that seeds a bounded graph traversal, graph predicates that constrain retrieval, and staged interleaving of the two within one transaction;
add relationship-aware candidate expansion and reranking with explicit depth, fan-out, path, candidate, memory, and timeout budgets;
preserve PostgreSQL snapshots, ACL/RLS, partitioning, cancellation, and exact source rechecks at every traversal/retrieval boundary;
return path and edge provenance with each expanded source so evidence assembly can explain why a row was included;
build community summaries as the graph-structured counterpart of hierarchical corpus summaries, reusing the source-linked summary-artifact contract as rebuildable derived artifacts, informed by Microsoft GraphRAG, while keeping the underlying rows and relationships authoritative;
benchmark multi-hop relevance, citation/evidence coverage, latency, and graph work against vector/lexical retrieval alone on held-out questions.

Validation compares every bounded fixture with an exhaustive vector/topology oracle and compares the graph-disabled path with the current HNSW baseline. It also runs differential fixtures for every pgGraph-derived primitive against the pinned source revision; this is test-time provenance, not a runtime dependency. It records ANN recall, relevant-path and evidence-chain completeness, latency, page faults, mapped/resident bytes, admitted/pruned states, exact rechecks, and fallback reasons. Hub, cycle, paging-thrash, stale-generation, concurrent mutation, crash/recovery, cancellation, partition, MVCC, and ACL/RLS fixtures are mandatory. No graph-aware result bypasses the authoritative source-row visibility and exact-score boundary.

Each capability receives its own maturity row and release notes only after it clears those real-execution, bounded-work, permissions, mutation, recovery, graph-off memory, and source-provenance gates. The direction is committed; specific APIs and dates are not.

Roadmap Change Policy

A roadmap feature moves into an implementation checklist only when it is selected for a concrete release or milestone. At that point:

copy only the selected feature into a new dependency-ordered build plan;
define owners, prerequisites, child checkpoints, and executable gates;
update its public maturity classification;
do not weaken the real-execution, bounded-work, source-authority, or lifecycle requirements stated here;
retain the independent-implementation policy and record the primary research, derivation, dependency, and format provenance used by the work.

PGXN

PostgreSQL Extension Network

Contents

pgContext Product Roadmap

Frequently asked since 0.1.0

Delivery Phases (post-V1 overview)

Dependency Order

Shared Retrieval Architecture

Set-Based Bulk Mutations

Segmented Serving and Background Compaction

Deletion Propagation and Bounded Erasure

Experimental Promotion Queue

Non-Dense ANN Opclasses

Model-Native Integer and Binary Source Vectors

Quantized HNSW

Full Quantized Serving and TurboQuant

Full IVFFlat Support

SSD-Resident and Streaming ANN

pgvector Migration and Compatibility

Large-Dimension Sparse Vectors

Named Sparse ANN

Internally Maintained Late Interaction

MUVERA Multi-Vector Acceleration

Adaptive-Dimension and Coarse-to-Fine Retrieval

Multimodal Document Retrieval

Composite Query Execution

Multi-Model Retrieval, Semantic Reranking, and Automatic Chunking

PostgreSQL-Native Lexical Retrieval

Query and Operations Priorities

Load Data and Query Immediately

Mapped HNSW Serving

Operator-Invocable Index Verification

Expanded Automatic Observability

Global Query Statistics and Coverage Profiles

Extractive Corpus Maps

Stratified Global Evidence Sampling

Summary Artifacts and Deterministic-Partition Rollups

Hierarchical Corpus Summaries

Evidence Assembly, Provenance, and RAG Evaluation

Reproducible Public Benchmarks

Dataset and ground-truth matrix

Required workload lanes

Measurements and acceptance

Performance work driven by the 10M profile

Execution tiers and publication

Post-V1 Release Engineering

Extended PostgreSQL 17 production certification

PostgreSQL-major and operating-system certification

Additional packages, images, and signatures

Release cadence and maintenance

Ecosystem and Framework Integrations

Graph-Augmented Retrieval

Integration decision: a selective pgContext-owned fork first

pgContext crate and dependency boundary

Exact pgGraph scope

Optional residency

Experimental virtualized beam search

Staged delivery

Roadmap Change Policy