pg_fts roadmap

The single plan file. Open work first, in priority order, each with its status and the evidence behind it. Closed items are one line each; the full record of how they were closed (and the wrong turns on the way) is in bench/ROADMAP_HISTORY_2026-09-17.md and the files bench/INDEX.md lists.

Rules for this file: an item is either open, blocked, or closed. When it closes, collapse it to one line here and leave the detail in the CHANGELOG. Do not create a second plan file.

Open – repository and presentation (from REVIEW_2026-09-17.md)

These do not touch the index and are the cheapest, highest-leverage work in the project.

# item status
R1 README comparison paragraph contradicted the project’s own table (claimed a rare-term lead; called the doclen sidecar future work). Rewritten as a wins / loses / caveat structure; every figure grep-verified against BENCHMARK_SUMMARY.md and RESULTS_C1X. The paragraph now states it is derived from the summary and that the summary wins on disagreement. Rule 27 in AGENTS.md and RELEASING.md step 1 keep it in sync. done 2026-09-17
R2 Delete the 31 dead base SQL scripts in the root. Only pg_fts--<current>.sql is installed; the rest were snapshots left by each rename. Verified none referenced by Makefile/meson/flake before git rm. RELEASING.md step 1 now says git mv, not copy. done 2026-09-17
R3 bench/INDEX.md naming current-truth vs dated-record files. done 2026-09-17
R4 One plan file. HANDOFF/DEFERRED/CAPABILITIES folded: DEFERRED (all resolved) and the old ROADMAP moved to bench/ as history; CAPABILITIES moved to doc/ (it is user-facing Q&A, not a plan). done 2026-09-17
R5 C comments no longer cite bench/ files (9 sites -> 0). They cite the CHANGELOG release that closed the issue; the journal can be reorganised, the CHANGELOG cannot. done 2026-09-17
R6 t/010 in all CI matrices – and what verifying it uncovered. GitHub: added, and the run log confirms t/010_vacuum_delete_heavy.pl .. ok. Forgejo: added too, but the Codeberg CI has never executed a single run – every ci.yml run back to v0.1.0 is cancelled with started_at = None (queued, never picked up, superseded by the next push). Cause is account-side: Codeberg Actions needs a runner enabled per repository, which is not visible or fixable from the tree. The README’s first badge had therefore shown “waiting”/“cancelled” to every visitor since the repo existed; it now points at the GitHub CI, which is the pipeline that actually runs. Open sub-item R8 below. done (GitHub verified); Forgejo cannot run
R8 Codeberg Actions had never run – and PGXN was frozen at 0.2.0 because of it. Corrected diagnosis: Actions was enabled (has_actions: true); the cause is that Codeberg provides no shared runners, so all 115 queued runs (ci / release / docs) from v0.1.0 to v1.8.2 sat unstarted and were cancelled by the next push. The consequence was not cosmetic: PGXN publishing and the postgresql.org announcement lived only in the Codeberg release workflow (the GitHub one explicitly deferred to it), so PGXN served 0.2.0 for 45 releases – pre-dating every field fix – and the README’s second badge pointed at it. Also the Codeberg Pages docs link was a 404. Resolved by deletion, not by standing up a runner: a 24/7 self-hosted runner to duplicate a matrix GitHub already runs green is real cost for zero signal. .forgejo/workflows/ removed; PGXN publish + announce moved into the GitHub release workflow (409 = already published, idempotent); README badges now point at GitHub CI and the real latest tag; dead Pages link removed; RELEASING.md describes the one pipeline that exists. Both follow-ups closed the same day: PGXN_USER/PGXN_PASSWORD were already set as GitHub secrets (since 2026-07-08, unused until now); a workflow_dispatch re-publish of the existing v1.8.2 tag ran the new definition – PGXN upload HTTP 303, accepted – and api.pgxn.org now reports version: 1.8.2, with 0.2.0 retained in history. The GitHub release was re-targeted, not duplicated (1 asset). PGORG_* is unset so the announcement step skipped, as designed; set it if pgsql-announce posts are wanted. done – PGXN current
R7 Agent tooling out of the working tree’s face. .agent/, .claude/, .kiro/, .mcp.json, .agent-steering-domains.md are gitignored but visible. AGENTS.md is now tracked and is the one entry point; the rest stay local. Also found and fixed while doing this: result-1, a nix build-output symlink, was tracked (now git rm --cached, result* ignored). done 2026-09-17

Code quality (from REVIEW_2026-09-17.md) – all three resolved 2026-09-17

# item outcome
C1 Allocator state passed explicitly. The four file-scope globals (bm25_lowfree, _n, _i, bm25_alloc_extend_only) became one BM25AllocCtx struct reachable only through bm25_alloc_scope_enter(index, mode) / bm25_alloc_scope_exit(prev), which nest by returning the previous context – the hand-rolled save/restore at two sites became the mechanism. bm25_new_buffer() now elog(ERROR)s if it finds compaction state with no scope active: exactly the dangling-pointer failure from the 1.7.1 work that only t/007 caught. Deliberately an elog, not an Assert: the release gate is not a cassert build, and a check that only fires in a build nobody ships is documentation. Threading a struct through all 16 bm25_new_buffer callers was considered and rejected – compaction is single-writer, so the context is backend-scoped in effect either way, and the scoped-lifetime design makes the failure mode a hard error at the same cost as a comment. done
C2 bm25_collect_matches split: 412 -> 226 lines. The 176-line per-segment loop body became bm25_collect_segment() (153 lines, returns SEG_RESTART for the positional-phrase fallback the loop used to express as s = -1; continue), and the 71-line pending-list walk became bm25_collect_pending(). State shared with the extracted evaluator travels in a BM25CollectCtx. Behaviour-preserving; full gate green. While doing it, the 14 page + pd_lower reads in _scan.c and the one in pg_fts_trgm_index.c’s blob reader (a 9th instance of the 1.7.0 defect class, feeding a memcpy length) were routed through bm25_page_data_end(). done
C3 The 12,000-line translation unit: KEEP, and document why. Measured the cost of splitting: 15 am.c statics would go extern (13 for _scan, 5 for _trgm), 1 the other way, ~10 shared struct types would move into pg_fts_am.h (the on-disk-format header), and the hot-path static inline helpers (bm25_tid_to_docid, bm25_docid_to_tid, bm25_page_data_end, bm25_doclen_cursor_lookup – inside the 45%/37% profile) would stop inlining without LTO, which PGXS does not use. In return: three .o files and no behaviour change. Decision recorded at the #include site and in both included files' headers, with the two prerequisites (an internal header; a before/after latency measurement) if separate compilation is ever needed. decided: keep

Open – index behaviour

# item status evidence
I1 Bulk-ingest write amplification – FIXED for row-per-transaction ingest (1.8.3). Root cause was not the XID horizon (that binds only inside one multi-row statement) but the merge’s EXTEND_ONLY allocation never consulting the free list. BM25_ALLOC_SNAPSHOT (free list gathered once at entry, never re-read) keeps the recycle-race guard by construction and reuses earlier frees. Field shape, 30k then 100k docs of row-per-txn churn, autovacuum on, nothing manual: 1,823 -> 1,823 MB and 1,823 -> 1,875 MB with fts_vacuum finding nothing to reclaim; v1.8.2 on the same harness grew 3.2 GB per 5k rows and then deadlocked (a pre-existing concurrent-merge deadlock, also fixed, plus a live-page handout deadlock the fix exposed – both in the CHANGELOG). Residual: one very large INSERT ... SELECT of oversized rows still needs an fts_vacuum after; modelled the fewer-larger-merges alternative at ~2x, not worth its complexity. done; residual narrowed bench/RESULTS_I1_2026-09-18.md
I2 Common-term ranked latency – done in 1.11.0 (Approach A). Best-first block order with an exact effective-length block bound: year top-10 8.48 -> 0.81 ms (pg_search 2.63), 1,911 -> 22,369 tps at 16 clients. Conjunctive best-first for AND and phrase: united & states 23.9 -> 2.7 ms, "united states" 25.0 -> 3.6 ms. No format change. done 1.11.0 bench/RESULTS_A_2026-10-07.md
A1 Build time vs pg_search (257 s vs 71 s as expression indexes, 2.19M articles, 1.11.0 rig). Re-measured 2026-10-28 (c6id.8xlarge, PG 17.10, stored column, 8 workers, mwm 8 GB, bench/RESULTS_1144_2026-10-28.md): 103.1 s = parallel heap scan into 7 worker segments (~40 s) + the single-backend collapse merge (39.8 s) + the in-build pack, a second full merge of the one segment (23.7 s). Next step still the redesign: workers emit sorted runs to a shared tuplesort and the leader writes ONE segment (no collapse, no pack) – removes ~64 s of the 103 s if the scan stays ~40 s. A design change for its own minor release; unmeasured beyond this split. open (measured) bench/RESULTS_1144_2026-10-28.md
A2 Many-block conjunctions and wide ORs (world & war, 4-term OR; 1.11.0 rig: 5.87 vs pg_search 5.75, 13.97 vs 13.40). Offline (bench/RESULTS_1144_2026-10-28.md, 2.19M Wikipedia): best-first visits 643 of 2,086 war blocks at k=10. The proposed pairwise bound (max over co-occurring documents only) visits 642 – refuted. The bound is loose per TERM: storing the top 4 points of each 128-block’s (tf, quantized length) Pareto frontier (median frontier 6 points, p90 8) visits 55, exact; top 2: 136; top 1: 302; the exact block maximum: 10. That is a block-header format change (dual-read, per-segment flag, rewrite on merge: RELEASING.md) for a minor release. 4-term OR not simulated yet. open (measured) bench/RESULTS_1144_2026-10-28.md
D Two-level page bitmaps + SIMD (TIN-style). Standby: Approach A closed the common-term gap it was meant to close without it (I2). Format side is tractable via the 1.5.0 optional-per-segment-pointer + dual-read precedent (no REINDEX). Real cost: no SIMD infrastructure exists (no intrinsics, no runtime dispatch, no -mavx2 plumbing) and a scalar fallback must be kept for non-AVX and ARM – two implementations forever. Largest change the project has attempted, against a competitor that cannot be benchmarked. Needs explicit sign-off. Do not vectorize the vendored sparsemap. blocked on sign-off bench/NOTE_TIN_FEASIBILITY_2026-09-14.md, NOTE_SIMD_VENUE_2026-09-14.md
I5 Lazy phrase gate – done in 1.9.1. A ranked phrase/NEAR on a positions=on index checks adjacency per WAND candidate from the cursor’s own block instead of materializing every match first. "united states" k10: 138 (1.9.0) -> 34.8 ms, vs pg_textsearch 43.0 (same day, separate hosts). Identical results to the collect path on 28 phrase/k cases at 2.2M rows; phrase_gate test with planted mutants. done 1.9.1 bench/RESULTS_191_2026-10-06.md
I6 Rare-term throughput fell with backend count – fixed in 1.10.0. Each backend held a private ~4.8 MB copy of the C1 slot doclen array, and the copies competed for L3. 1.10.0 keeps one copy per segment per server in DSM (pg_fts.shared_doclen), refcounted per scan, keyed by segment identity so backends with different snapshots stay exact. Same-host A/B on Graviton3: rare-term tps at 64 clients 16,654 vs 13,283 (+25%), now flat 16 -> 64; per-transaction cost of bm25_doclen_cursor_lookup 21.2 -> 6.9 units. done 1.10.0 bench/RESULTS_110_2026-10-07.md, bench/PLAN_I6_SHARED_DOCLEN_IMPL_2026-10-07.md
I7 Per-query shared cache-line traffic – done in 1.12.0. Ranked lookups re-walked the dictionary-index chain (up to 85 pages, pinned and share-locked, read by every concurrent query) three times per term; each backend now binary-searches an in-memory copy of each segment’s dictionary index, kept in the existing relcache chunk. Rare top-10 278 -> 80 shared buffers; single-client rare 0.54 -> 0.27 ms; throughput rare +67-97%, common +74-92% at 16-64 clients on six hosts, results identical. Host-state variance itself is not addressable in pg_fts (bench/RESULTS_TPS_VARIANCE_2026-10-09.md); pg_fts.shared_doclen stays a documented trade. Remaining per-query costs: the doclen lookup and scoring (profile in bench/data_i7_2026-10-09/h1/). done 1.12.0 bench/RESULTS_112_2026-10-09.md
I8 Relcache cache leaked on an ERROR during its rebuild – FIXED 1.15.0. Attached at palloc, swapped before free, complete gate; t/015 (hooks build) 1.14.3 +11.5 kB per failed rebuild, fix flat. done CHANGELOG 1.15.0
I3 Managed-service validation – done for Neon (2026-10-29), the open-source compute/storage-separated backend. pg_fts 1.15.0 on Neon fa504217c (PG 17.5), a 16 MB compute with no local file cache, so index pages are rebuilt from WAL by the pageserver: build, deletes + VACUUM, inline flushes, fts_merge, fts_vacuum, a compute restart, and a branch at a mid-workload LSN (writes + merge on the branch) – index == seq scan at every stage (6 query shapes x 2 index forms, > 0), ranked top-20 == fts_bm25 oracle, 0 walredo errors. Vanilla: wal_consistency_checking=‘generic’ primary + standby, the full installcheck + churn, 20,536 verification images compared, 0 inconsistent pages (PG 17.10, 18.6). Not done: Aurora (closed; no custom C extensions). bench/RESULTS_150_2026-10-29.md s3. done (Neon); Aurora n/a bench/data_150_2026-10-29/i3/
I4 Independent review of WAL/crash/recovery paths. 1.15.0: two independent read-only sub-agent reviews against the RELEASING checklist (bench/data_150_2026-10-29/review/); the first found a UTF-8 split in I9 (an ERROR on CJK text) fixed before release. A HUMAN reviewer remains open – the ROADMAP item asks for one and no agent substitutes. open (human); agent reviews done per release bench/data_150_2026-10-29/review/
I9 Configured analyzer lost positions after 16,383 – FIXED 1.15.0. Chunked parsetext() with offset ordinals; the PR #12 example matches. Stored columns / expression indexes need a recompute or REINDEX to gain positions. done CHANGELOG 1.15.0
I9a 1.15.0’s chunked re-parse split HTML markup at chunk boundaries – FIXED 1.15.1. Cuts are configured-parser token boundaries verified against a whole-text parser-only pass; byte-identical to an unchunked wide parser on 28 of 28 long-document probes (1.15.0: 4 markup docs differed). Thesaurus phrases across a cut remain a limit. done 1.15.1 CHANGELOG 1.15.1
I13 VACUUM bulkdelete memory was O(postings) – FIXED 1.14.0. bm25_segment_docids decoded every posting of a segment into one array and bulk-added it: ~24 bytes per posting, independent of maintenance_work_mem or dead-tuple count. A production index (977M postings/segment) OOM-killed an autovacuum at 13.9 GB. Now a v4 segment reads the doclen sidecar (O(documents), zero posting decode) and an inline-doclen segment flushes in maintenance_work_mem/24 batches; both loops are cancellable + cost-delayed. 400k docs/120M postings: one-row INDEX_CLEANUP ON 1,429 -> 151 MB. vacuum_mem test. Present since 1.1.5. The 1.14.0 claim “5%-deleted VACUUM 2,820 -> 157 MB” held only with the sidecar on or a small maintenance_work_mem; with the reporter’s settings (sidecar off, mwm 4 GB) 1.14.0 measured 2,814 MB – retracted, see I13a. done 1.14.0 (completed 1.14.1, I13a) pg.ddx.io report 2026-10-15
I14 Manual VACUUM errored under parallel index vacuuming on PG18 – FIXED 1.14.0. bm25_truncate_lock took AccessExclusiveLock in the parallel-vacuum leader’s cleanup; under wal_level >= replica that assigns an xid (hot-standby lock logging), an ERROR in parallel mode, rolling back the whole VACUUM so relfrozenxid stalled. Now declines truncation when IsInParallelMode() (tail stays in FSM, reclaimed by a later non-parallel VACUUM/fts_vacuum). Reproduced + fixed on PG 18.6; autovacuum and VACUUM (PARALLEL 0) never affected. Introduced 1.11.0 (ada32aa). vacuum_parallel test (fails on the mutant on PG18). done 1.14.0 pg.ddx.io report 2026-10-15
I15 @@@ estimated every row whenever anything was pending – FIXED 1.14.1. 1.14.0’s fts_matchsel used the ordering scan’s upper bound (bm25_query_maxhits), which returns N when npending > 0; with continuous inserts that is always, so plans flipped to Seq Scans (production: a 96-row count estimated at 1,651,568 and timed out). Now segment df / segment documents (bm25_query_segdf); falls back to tsmatchsel for a modifier term, a term in no segment while documents are pending, or an index whose segments hold fewer documents than its pending list. 100k rows + 3k pending: rare est 103,000 -> 103, Seq Scan 20.54 ms -> Bitmap 0.143 ms (medians of 7). planner_pending test (fails on the 1.14.0 estimator). done 1.14.1 pg.ddx.io report 2026-10-15
I16 VACUUM grew the index when it could not truncate – FIXED 1.14.1. The cleanup’s compaction ran its extend-only vacate after a declined conditional truncation (any session holding a lock on the index, since 1.11.0; parallel mode, 1.14.0), so each such VACUUM grew the file by about its live size: reader held across 3 VACUUMs 1,507 -> 4,085 blocks (1.13.0, 1.14.0); 3 parallel VACUUMs 7,848 -> 10,198 (1.14.0). The vacate now needs AccessExclusiveLock (fts_vacuum, build); cleanup packs only. Fixed: 1,506 held, 7,845 held; a later lockless VACUUM reclaims. bm25_vacuum_growth isolation test + vacuum_parallel size checks (both fail on 1.14.0). done 1.14.1 found re-validating I14, 2026-10-16
I13a I13’s batch was unbounded at a large maintenance_work_mem – FIXED 1.14.1. mwm/24 at 4 GB = 178M entries > a 60M-posting segment, so doclen_sidecar=off bulkdelete was still one sort: 1,430 MB, cancel 3.5-7.3 s at 200k docs (= 1.13.0). Capped at 4M entries and flushed before a term that would overfill the batch, so a batch holds at most max(4M, df) (cap 256k/1M/4M/8M measured: 16.7/13.3/13.9/14.1 s, 76/92/163/259 MB at 400k docs): 155/169 MB at 200k/400k, cancel <= 0.43 s (shipped tree). done 1.14.1 re-validation 2026-10-16
I17 count(*) pushdown read indexes no query may use – FIXED 1.14.1. It (and the 1.14.0 @@@ estimator) took the index from RelationGetIndexList: a partial fts index made count(*) count only the predicate’s rows (100 vs 2,000), an indcheckxmin index too new for the snapshot and a cancelled-CIC invalid index gave wrong counts (1/49 vs 0/50; 0 vs 1), and during CIC’s first wait count(*) (1.13.0, 1.14.0) and in 1.14.0 every @@@ plan failed (empty file). From the code: partial index since 0.1.0; the others since 1.3.0 for a stored-column index (expression index only with its expressions already loaded); measured on 1.10.0/1.13.0/1.14.0. Both now use the planner’s rel->indexlist and skip partial indexes. bm25_cic_planner spec + planner_indexes test (fail on 1.14.0). Open: a predicate-implied partial index could serve the pushdown (not attempted). done 1.14.1 review of the I15 fix, 2026-10-17
I18 Index returned rows whose own @@@ is false, one slot after a true match – index data from <= 1.10.0, REINDEX (1.14.2). pg.ddx.io, 1.14.1 / PG 18.6, an index written by 1.10.0 then upgraded in place: an entry of a row that died while pending (rollback, failed batch/COPY, duplicate key, delete) survived VACUUM in <= 1.10.0, and the row that reused the slot inherited its terms, returned as exact (no recheck). Fixed in 1.11.0 but documented as “no REINDEX”; the 1.11.0 lost-insert bug likewise left missing rows. Reproduced: 199/399 false on 1.10.0, kept after upgrade + VACUUM + merge, 0 after REINDEX; 1.14.1 clean in every arm. 1.14.2 = docs + SQL check + pending_reuse test (fails on the mutant). Open: a SQL-callable verifier (a minor release); not tried on 1.14.1: CIC/REINDEX CONCURRENTLY during aborting inserts, parallel build, standby. vacuum_parallel’s serial_reclaims size check failed once in 26 nix PG17 runs and once more on 1.14.3: another backend’s snapshot (any database) blocks page recycling; the test now waits for other snapshots, bounded (1.14.3, bench/RESULTS_1143_2026-10-28.md section 3). done 1.14.2 (docs); users REINDEX pg.ddx.io report 2026-10-24
I19 Four user reports against 1.14.2 (robinlahtinen, Codeberg #6-#10) – FIXED 1.14.3. #9 fts_bm25/fts_bm25_opts read dfs[i] past a short array (stale value / NaN; now df = 1). #7 MSVC link (__builtin_clz, since 1.5.0). #10 fts_index_df left pending documents out (now segments + pending, as the ordering scan); sibling: ranked prefix/fuzzy/regex/weighted queries (bm25_topk_exact, since 1.13.0) had the same gap. #8 fts_matchsel now reads a predOK partial index (the count pushdown still never does); <=> docs say the BM25 order comes only from the ordering scan. #6 absent & (a | b) skips the segment walk (10.8 -> 0.066 ms, 400k rows). Tests bm25_dfs, index_df_pending, planner_partial, ranked_empty_and (all fail on 1.14.2). done 1.14.3 Codeberg #6-#10, bench/RESULTS_1143_2026-10-28.md
I19a Ranked rare & (a | b) walked the OR group – FIXED 1.15.0. Driven from the cheapest positive operand of the root AND when it is <= 1/8 of the postings: 3.6 -> 0.2 ms (400k rows, PG 17/18). Open: the seeks still read the OR group’s block headers (buffers 501 -> 417). done CHANGELOG 1.15.0
I19b Collect path built a whole OR group before an empty AND operand – FIXED 1.15.0 for a plain required term absent from a segment: zzzq & (apple | ban:*) 3.2 -> 0.07 ms. 1.16.0: also an operand that is an OR of plain terms all absent ((zzzq | zzzx) & kur:*, a word with two lemmas; Codeberg #6 measured 17.7 vs 0.35 ms on 1.15.1): 3.79 -> 0.077 ms on the 400k corpus, 618 -> 0.076 ms on a 640k-row index (bench/RESULTS_I23_2026-11-01.md). Open: a present but rare required term (rare & (apple | ban:*), 3.2 ms) still builds the group. done (plain-term and plain-readings cases) CHANGELOG 1.15.0, 1.16.0
I19c Ranked prefix / phrase-with-prefix from its dictionary expansions – candidate, from an unsubmitted fork branch (cepymm/pg_fts perf/ranked-phrase-prefix 79bb338, 2026-10-15, 41 commits behind 1.15.0): expand a prefix to <= 64 dictionary terms and rank with the plain-term top-k (phrase: last term any expansion, adjacency checked). Reviewed 2026-10-29, not ported: it changes the SCORE of a prefix query (today the collect path scores the literal term: measured on 1.15.0, a document holding only the expansion zqxvaa has fts_bm25(d, 'zqxv*') = 0 while one holding the literal zqxv scores 0.88, and the ranked scan puts every literal holder first; the code calls expansion scoring ‘a separate ranking change’), so <=> values and order change and fts_bm25, the documented seq-scan oracle, would disagree with the index unless it changes too. Its test checks ids only with LIMIT covering every match, so it cannot see a score change. Needs: a ranking-semantics decision (expansion scoring everywhere: index, fts_bm25, fts_index_df), score-level tests, a minor release. Would also close the I19b remainder (rare & (apple | ban:*)). open (design)
I20 Flush publish window returned a pending row twice – FIXED 1.15.0. One metapage update publishes the segment and clears the pending list; t/014 (hooks build) 1.14.3 100/50 rows, fix 50/50. done CHANGELOG 1.15.0
I21 Concurrent UPDATEs under ranked readers are slow – measured 2026-10-29 (M4 W3, 59.7M posts, expression index to_ftsdoc('english', content) positions on): 19,038 UPDATEs in 600 s (25/s of a 1,000/s target) vs pg_search 599,489; OR readers 1.7 tps (p99 47 s) vs 26.7; index 26 -> 44 GB. Not diagnosed: per-UPDATE re-analysis through the expression index, the pending list and inline flushes (pg_fts.pending_flush_threshold 20,000), and reader/flush interaction are the candidates – instrument before theorising (rule 12). open bench/RESULTS_M4_2026-10-29.md
I22 Parallel build memory is per participant, not total – measured 2026-10-29 (M4): with maintenance_work_mem = 24GB, shared_buffers = 24GB and 3 participants the build was OOM-killed at 32 and 64 GB and peaked at 132-141 GB (documented bound sb + (workers+1) x 2 x mwm = 168 GB). Core PG’s parallel builds divide mwm among participants; pg_fts gives each up to 2 x mwm (the 1.0.6 ceiling). Candidate: divide the flush budget by the participant count (fewer, smaller segments to merge – measure build time). open bench/RESULTS_M4_2026-10-29.md
I23 Pure OR about 5-8% slower than 1.14.3 – NOT REPRODUCED (reported 2026-10-30, Codeberg #6: build | (10 terms) on a 640k-row corpus, 519-547 -> 557-582 ms, PG 18.6 x86_64, .so swapped under one catalog). Measured 2026-11-01 on c6id.metal (pinned, turbo off), the .so swapped the same way, 12 ABBA samples per build: a 10-term OR differs from 1.14.3 by +0.04% / +0.25% (400k reporter corpus, PG 17 / 18) and -0.45% / +0.28% (640k x 60-token index, 98 / 41 ms, PG 18), A/A (same tree built twice) up to 0.21%; 400k buffers 1,779 on both. No new per-posting work on that path (no gate, so the I19a dispatch is skipped). Open only for the reporter’s environment: their figures include their access filtering. not reproduced bench/RESULTS_I23_2026-11-01.md
I24 Ranked top-k within a caller-supplied allowed set – candidate, from an unsubmitted fork branch (jimdawdy-hub/pg_fts prototype/native-subset-20261016 7519fb6, authored 2026-10-16, pushed 2026-11-02, on their 1.13.0 branch a0b5355; their ROADMAP calls it fixture-only, not installed, with the allowed-ID and latency gates unproven). fts_search_subset(index, query, tid[], k [, tie_key bigint[]]) maps each allowed TID to its HOT root under a page lock, keeps the snapshot-visible ones, seeks every term cursor to each allowed docid in order and scores with the index’s own BM25 (same scores and order as fts_search restricted to the set, which its test checks). Today the same question is WHERE d @@@ q AND id = ANY($allowed) ORDER BY d <=> q LIMIT k: the ordering scan ranks in BM25 order and the executor filters; when a batch runs out the scan grows k x4 and recomputes, skipping the rows already returned, so a selective allowed set may cost several passes over most of the match set – unmeasured. Both downstream users we know filter by access (Codeberg #6’s reporter: ‘our access filtering’). Reviewed 2026-11-02 (read only, not built): sound visibility/TID validation, SELECT privilege checked, REVOKEd from PUBLIC; limits a port must lift: plain positive AND/OR only, ERROR when the index has pending rows (common under ingest; bm25_topk_add_pending exists), non-partial index, no RLS, not SERIALIZABLE. Needs: a measurement of the planner form against a driven form on a selective set first (rule 12), then an API decision (a function, or the planner pushing = ANY into the ordering scan) and a minor release. open (design)
I25 Ranked top-k 60-300x slower after deletes scattered over a large heap – FIXED 1.15.2 (reported 2026-11-02 from a production index: 1.88M rows, doclen_sidecar=off, 23,313 rows moved, then fts_vacuum and VACUUM; REINDEX restored it). Tombstone lookups head-walked a sparsemap of one chunk per scattered tombstone: the sm cursor ignored a cached chunk starting after the index (every lookup in a gap), and the best-first paths reset it per block. Reproduced on c6id.metal (1.9M rows): top-200 3,640 -> 64 ms (fresh 54), top-10 62 -> 3.5 ms (0.35), the common term’s top-200 ids identical. 1.16.0: fts_vacuum rewrites a single packed segment that carries tombstones (bba830f; 1.15.2 kept them until a REINDEX, the report’s steps E/F): back to fresh speed in 69-89 s against 136-184 s for REINDEX; and a per-backend memo of blob validation + checkpoints (eca5e75): top-10 3.49 -> 2.02 ms. Open: a backend’s first query still validates, and every scan reads the blob and tests each posting (top-200 1.15-1.7x fresh until the next fts_vacuum or merge); fts_index_stats shows no tombstone counts (the report’s third suggestion; a new SQL function, so a minor release). done bench/RESULTS_CHURN_2026-11-02.md
I26 Block-max WAND skipped the pivot (wrong top-k) – FIXED 1.15.2, since 1.3.0 (1.3.0-1.10.0 jumped to pivot + 1; 1.11.0 kept that as the “always make progress” clamp). From jimdawdy-hub/pg_fts e4bba08 (Jim Dawdy, never submitted): his bmw_block_end fails on 1.3.0, 1.10.0, 1.11.0, 1.13.0 and 1.15.1 (top-1 id 1, exact 5000) and passes with the clamp removed (PG 17 + 18). Cost: +0.3-2.4% on 2-3-term ORs, A/A up to 1.7% (NOISY). The 1.13.0 ‘no wrong top-k’ line is retracted. done bench/RESULTS_CHURN_2026-11-02.md
I27 jimdawdy-hub 7d347ff scan-path ideas – MEASURED, NOT PORTED (2026-11-03): the four that still apply, each alone on main (c6id.metal, 640k rows, 12 ABBA samples, A/A pair): p1 skip the sort of a presorted TidSet: count(*) through the collect path +5.0% (ranges do not overlap; a prefix leaf’s TidSet is per-term runs, so the check scans then sorts anyway); p2 one dictionary lookup per (term, segment): -2.0% to +1.6%, ranges overlap (NOISY); p3 reuse the cursor’s block buffer: inside noise; p4 one heap slot per visibility loop: within 0.25%. The fifth (append-then-sort pending matches) has been upstream since 1.13.0. Not measured: many segments (p2), pending-heavy, cold cache. declined bench/RESULTS_P7_2026-11-03.md
I28 CBMC proofs – added 1.16.0 (2026-11-03, test/cbmc). Bounded model checking of the shipped code (verbatim copies checked by check_verbatim.py, each property with a planted historical bug that must be caught): ranked traversals == brute-force top-k (BMW, MaxScore), block bounds >= scores (IEEE; plus an 11.1G-case native sweep), on-page guards, query-shape shortcuts, FOR codec / doclen byte, sparsemap cursor, shared-doclen protocol. Found and fixed (CHANGELOG 1.16.0): wand_load_block decoded FOR columns without sizing them against the block (a corrupt width byte -> ranked read past the block; reproduced, t/003), the same class in the doclen sidecar readers and bm25_decode_term, five dictionary walks without bm25_dict_entry_fits, pointer-past-page formation in the guards, a 1-ulp-low BMW bound (operation order). Open: the full proofs need a 256 GB host and ~2 h (CI runs only the drift check); the IEEE harness_bound with symbolic tf/len did not finish in 90 min on minisat (run with CBMC_FP=1); not covered: fts_search_bestfirst_n / _and_bestfirst / _driven, phrase/NEAR. added test/cbmc/README.md
I29 sparsemap: a cursor’s left-neighbour hint is only partly validated (found by test/cbmc/smcursor_harness.c C3x, 2026-11-03). The 5.8.4 gap resume (__sm_get_chunk_offset) trusts prev_offset if the bytes there “end exactly where the cached chunk begins”; a hint that is not a chunk start can pass that and start the walk mid-chunk (CBMC counterexample, replayed natively: wrong sm_contains). The API leaves such hints after an add/remove without resetting the cursor (8% of random trials) – a use the cursor contract calls undefined, though the function’s comment says it self-validates against it; 12M lookups after them gave no wrong answer. Not reachable from pg_fts (tombstone maps are read-only during scans; bulkdelete resets its cursor after the last insert). A sound fix needs the hint’s chunk start in the cursor (sm_cursor_t grows: ABI change, so a sparsemap minor release); cheap plausibility checks (aligned start, ends before the cached start) reject the found counterexample but are not a proof (55,555 non-chunk hints still pass them in 200k random maps). Written up for the sparsemap team 2026-11-04 with a deterministic repro and four fix options: bench/data_cbmc_2026-11-03/sparsemap_issue/SPARSEMAP_ISSUE_cursor_prev_offset.md. open (upstream) harness_stale_hint
I17a count(*) from a partial fts index – FIXED 1.15.0 when the other quals are provably equivalent to the predicate (both directions, predicate_implied_by); anything else keeps the normal plan. done CHANGELOG 1.15.0
I10 ASan signal over the SQL paths – done. ASan-built PostgreSQL + test/asan_smoke.sql (1.14.0), fails on a false check (1.14.1), and (1.15.0) on the cassert allocator’s chunk-overflow / double-pfree reports, which ASan cannot see; checked with a planted overflow mutant. done CHANGELOG 1.15.0
I11 Ranked LIMIT >= 250 on an expression index took the bitmap + Sort plan – FIXED 1.14.0. fts_matchsel estimates @@@ from the index dictionary df (a term’s df, AND/phrase the min, OR the sum) instead of tsmatchsel’s fixed default, so the planner prices a common term apart from a rare one; it falls back to tsmatchsel for a non-constant query / no bm25 index / join clause. year LIMIT 5000 at 50k rows: 21.6 s -> 44 ms, staying on the ordering scan. SQL change (new estimator, both @@@ operators rewired); pg_fts--1.13.0--1.14.0.sql. planner_cost test. done 1.14.0 metal smoke test 2026-10-15
I12 Ranked reads under sustained inserts – FIXED 1.14.0. Every ranked query read the whole unflushed pending list (since 1.13.0, so pending documents rank), so latency grew with inserts since the last merge (0.22 ms none pending, 205 ms at 1M). pg_fts.pending_flush_threshold (default 20000, 0 disables) makes the inserting backend fold the pending list into a segment at the threshold, taking the maintenance lock only if free (inserters never wait); and a single term across many segments now runs one global best-first pass instead of MaxScore, so the extra flushed segments do not slow single-term reads (7 segments: rare top-10 3.80 -> 0.26 ms, identical results). Mixed load 8+8 at 2.19M rows: ranked reads 97 -> 3,223 tps at 10% fewer inserts. pending_flush test. Metal write-band A/B still to re-run for the release. done 1.14.0 (metal A/B pending) metal benchmark 2026-10-15

Open – measurement debt

# item
M1 C2 cross-engine ingest – measured 2026-10-29 in one common form (TIN rig W3: OR top-10 readers + a 1,000 UPDATE/s target, 600 s, 59.7M posts): pg_search 0.25.2 599,489 UPDATEs (readers 26.7 tps), pg_textsearch 1.4.0 1,033 (46.6 tps), pg_fts 1.15.1 19,038 (1.7 tps), GIN OOM. One run each. bench/RESULTS_M4_2026-10-29.md. Bulk INSERT rate across engines still open.
M2 C3 NDCG vs rivals – measured on 2 BEIR sets (2026-10-28). SciFact NDCG@10: pg_fts 0.6848 (English) / 0.6650 (simple), pg_search 0.26.1 0.6339, ts_rank 0.5684; NFCorpus 0.3240 / 0.3016, 0.3000, 0.3001 (bench/RESULTS_1144_2026-10-28.md s5). Open: larger BEIR sets (e.g. TREC-COVID, FiQA), pg_textsearch, VectorChord-bm25.
M3 Longer ingest run done 2026-09-18: 20 rounds / 100k row-per-txn docs. No decay – throughput oscillates 33-56 rows/s with round durations at exactly 1x / 1.33x / 1.67x a 90 s floor, i.e. autovacuum cycles landing in the round or not. The absolute rate is the harness (psql building 1,660-term strings), not publishable. Appended to RESULTS_I1_2026-09-18.md.
M4 The published competitor set on TIN’s rig – done 2026-10-29 (bench/RESULTS_M4_2026-10-29.md): i7i.8xlarge, PG 18.6, 8 CPU / 32 GB cgroups, the article’s settings and versions (pg_search 0.25.2, pg_textsearch 1.4.0, GIN) + pg_fts 1.15.1, 59.7M Stack Overflow posts (their 150M export is unpublished), the article’s trace recipe. pg_fts: W2 AND+phrase 125 tps vs pg_search 53; W1 mixed 26 vs 45 (long ORs); W3 updates far behind; build RAM 132-141 GB. One run per engine (rule 4: reproduce before quoting).

Declined (with the measurement that declined them)

  • Impact-ordered postings – breaks the docid ordering that count(*)/AND/phrase/prefix need.
  • Early termination – breaks exact top-k.
  • Heap-side positions=off – saves ~16% of a STORAGE=extended column; no-go.
  • Parallel ranked scan – built, measured, reverted (bench/NOTE_PARALLEL_RANKED.md).
  • Parallel merge – 1.45x slower and 19% larger at scale; mpmw=8 silently serial.
  • df-threshold bulk load – cost ~linear in df, no fixed floor.
  • Verbatim posting copy on merge (TIN item C) – the merge re-encodes through the build hash table; no splice point.
  • Vectorizing sparsemap – never a query hotspot; the one time it was the bottleneck (P0, 99.75%) the fix was algorithmic; its compressed layout is SIMD-hostile; it is vendored byte-identical to upstream on purpose.

Closed (one line each; detail in CHANGELOG)

  • 1.11.0 (Approach A): best-first single-term and conjunctive top-k with exact block bounds; MaxScore wrong top-k for 4+ terms (every release) and ranked term:LABEL fixed; merge 5x faster with identical output and no fts_vacuum after a plain build; maintenance buffer ring; prefetch where chains jump and for the ranked heap fetches; VACUUM pending-delete fix and four concurrency fixes (lost inserts during flush, truncation without AccessExclusiveLock, freed-page link overwritten by the recycle XID, double free on scan retry), all in every earlier release, regression test t/012. Champion lists (Approach C) not needed. bench/RESULTS_A_2026-10-07.md.

  • 1.12.0: ranked dictionary lookups from a per-backend in-memory copy of each segment’s dictionary index (I7); pg_fts.doclen_cache_mb changes now take effect in the session. bench/RESULTS_112_2026-10-09.md.

  • 1.9.0 ranked-retrieval release (LIMIT pushdown, resident doclen, dense high-df scoring, score reuse) and two exactness fixes present since 1.x (negative idf after deletes; WAND last-block skip). Leads pg_textsearch 1.4.0 on every latency and throughput band except phrase (3.2x behind).

  • 1.8.6 sparsemap 5.7.0 -> 5.8.0: bulk sm_add_many_grow merges instead of inserting per bit (1.4-4x faster standalone; unmeasured in pg_fts at scale), plus a coalesce over-read fix. Byte-identical wire output.
  • 1.8.5 sparsemap 5.6.0 -> 5.7.0: small-set tombstone encoding, verified live through pageinspect and by a new sanitizer fuzz target over the vendored library; the 1.8.4 open guard holds in both encodings.
  • 1.8.4 sparsemap 5.5.1 -> 5.6.0 (upstream security hardening); every sm_open now goes through bm25_sm_open_checked, so a corrupt tombstone/trigram bitmap raises ERRCODE_DATA_CORRUPTED instead of silently opening as EMPTY – which for a tombstone map would have resurrected every deleted row. t/003 proves it (red unguarded, green guarded).
  • 1.8.3 two deadlocks that shipped in every prior version (insert-path full-directory merge ran without the maintenance mutex; a live page could be handed out as merge output), and the bulk-ingest bloat root cause (merge was extend-only, never reused pages) – zero net growth over 100k row-per-txn docs.
  • 1.8.2 allocator state is a scoped struct with misuse an elog(ERROR); bm25_collect_matches 412 -> 226 lines; the single translation unit kept as a costed, documented decision; a 9th pd_lower read site guarded.
  • 1.8.1 count-path: df fast-count gate tests (10, non-vacuous); block-run VM checking measured ~1%, kept as cleanup.
  • 1.8.0 intra-word - . / are terms, not operators (pkg-config no longer parses as pkg & !config).
  • 1.7.2 insert-time merge gated on segment pressure: bulk-ingest growth -31%.
  • 1.7.1 pd_lower guard generalised to all 8 page-read sites; “one WAL record per page” known issue retracted (0.005 ms/page measured).
  • 1.7.0 P0: unvalidated pd_lower in the merge dict walk made an index permanently unvacuumable at field shape; fixed. Huge-alloc gaps in doclen/tombstone arrays fixed. Cleanup no longer grows the index (18/18/18 MB vs 35/52/69).
  • 1.6.1 P0: VACUUM never completed on a delete-heavy index (4h39m -> 393 s; dense tombstone bitmap sized by sm_maximum). sparsemap 5.5.1.
  • 1.6.0 phrase over positionless docs returns false, not a silent conjunction (matches OP_PHRASE).
  • 1.5.9 / 1.5.10 non-UTF-8 case folding; common-term 1.56x via ascending-resume + word-load bm25_for_get.
  • 1.5.0 doclen sidecar (format v3 -> v4) with dual-read, no REINDEX – the precedent for all future format changes.
  • COUNT pushdown (CustomScan), fts_search under-fetch, reserved keywords as literals, sparsemap error-path leaks, recovery guard on fts_merge/fts_vacuum, privilege lockdown, recently-dead exclusion from corpus stats, parallel-build memory ceiling – all shipped; see history file.