pg_fts roadmap

The single plan file. Open work first, in priority order, each with its status and the evidence behind it. Closed items are one line each; the full record of how they were closed (and the wrong turns on the way) is in bench/ROADMAP_HISTORY_2026-09-17.md and the files bench/INDEX.md lists.

Rules for this file: an item is either open, blocked, or closed. When it closes, collapse it to one line here and leave the detail in the CHANGELOG. Do not create a second plan file.

Open – repository and presentation (from REVIEW_2026-09-17.md)

These do not touch the index and are the cheapest, highest-leverage work in the project.

# item status
R1 README comparison paragraph contradicted the project’s own table (claimed a rare-term lead; called the doclen sidecar future work). Rewritten as a wins / loses / caveat structure; every figure grep-verified against BENCHMARK_SUMMARY.md and RESULTS_C1X. The paragraph now states it is derived from the summary and that the summary wins on disagreement. Rule 27 in AGENTS.md and RELEASING.md step 1 keep it in sync. done 2026-09-17
R2 Delete the 31 dead base SQL scripts in the root. Only pg_fts--<current>.sql is installed; the rest were snapshots left by each rename. Verified none referenced by Makefile/meson/flake before git rm. RELEASING.md step 1 now says git mv, not copy. done 2026-09-17
R3 bench/INDEX.md naming current-truth vs dated-record files. done 2026-09-17
R4 One plan file. HANDOFF/DEFERRED/CAPABILITIES folded: DEFERRED (all resolved) and the old ROADMAP moved to bench/ as history; CAPABILITIES moved to doc/ (it is user-facing Q&A, not a plan). done 2026-09-17
R5 C comments no longer cite bench/ files (9 sites -> 0). They cite the CHANGELOG release that closed the issue; the journal can be reorganised, the CHANGELOG cannot. done 2026-09-17
R6 t/010 in all CI matrices – and what verifying it uncovered. GitHub: added, and the run log confirms t/010_vacuum_delete_heavy.pl .. ok. Forgejo: added too, but the Codeberg CI has never executed a single run – every ci.yml run back to v0.1.0 is cancelled with started_at = None (queued, never picked up, superseded by the next push). Cause is account-side: Codeberg Actions needs a runner enabled per repository, which is not visible or fixable from the tree. The README’s first badge had therefore shown “waiting”/“cancelled” to every visitor since the repo existed; it now points at the GitHub CI, which is the pipeline that actually runs. Open sub-item R8 below. done (GitHub verified); Forgejo cannot run
R8 Codeberg Actions had never run – and PGXN was frozen at 0.2.0 because of it. Corrected diagnosis: Actions was enabled (has_actions: true); the cause is that Codeberg provides no shared runners, so all 115 queued runs (ci / release / docs) from v0.1.0 to v1.8.2 sat unstarted and were cancelled by the next push. The consequence was not cosmetic: PGXN publishing and the postgresql.org announcement lived only in the Codeberg release workflow (the GitHub one explicitly deferred to it), so PGXN served 0.2.0 for 45 releases – pre-dating every field fix – and the README’s second badge pointed at it. Also the Codeberg Pages docs link was a 404. Resolved by deletion, not by standing up a runner: a 24/7 self-hosted runner to duplicate a matrix GitHub already runs green is real cost for zero signal. .forgejo/workflows/ removed; PGXN publish + announce moved into the GitHub release workflow (409 = already published, idempotent); README badges now point at GitHub CI and the real latest tag; dead Pages link removed; RELEASING.md describes the one pipeline that exists. Both follow-ups closed the same day: PGXN_USER/PGXN_PASSWORD were already set as GitHub secrets (since 2026-07-08, unused until now); a workflow_dispatch re-publish of the existing v1.8.2 tag ran the new definition – PGXN upload HTTP 303, accepted – and api.pgxn.org now reports version: 1.8.2, with 0.2.0 retained in history. The GitHub release was re-targeted, not duplicated (1 asset). PGORG_* is unset so the announcement step skipped, as designed; set it if pgsql-announce posts are wanted. done – PGXN current
R7 Agent tooling out of the working tree’s face. .agent/, .claude/, .kiro/, .mcp.json, .agent-steering-domains.md are gitignored but visible. AGENTS.md is now tracked and is the one entry point; the rest stay local. Also found and fixed while doing this: result-1, a nix build-output symlink, was tracked (now git rm --cached, result* ignored). done 2026-09-17

Code quality (from REVIEW_2026-09-17.md) – all three resolved 2026-09-17

# item outcome
C1 Allocator state passed explicitly. The four file-scope globals (bm25_lowfree, _n, _i, bm25_alloc_extend_only) became one BM25AllocCtx struct reachable only through bm25_alloc_scope_enter(index, mode) / bm25_alloc_scope_exit(prev), which nest by returning the previous context – the hand-rolled save/restore at two sites became the mechanism. bm25_new_buffer() now elog(ERROR)s if it finds compaction state with no scope active: exactly the dangling-pointer failure from the 1.7.1 work that only t/007 caught. Deliberately an elog, not an Assert: the release gate is not a cassert build, and a check that only fires in a build nobody ships is documentation. Threading a struct through all 16 bm25_new_buffer callers was considered and rejected – compaction is single-writer, so the context is backend-scoped in effect either way, and the scoped-lifetime design makes the failure mode a hard error at the same cost as a comment. done
C2 bm25_collect_matches split: 412 -> 226 lines. The 176-line per-segment loop body became bm25_collect_segment() (153 lines, returns SEG_RESTART for the positional-phrase fallback the loop used to express as s = -1; continue), and the 71-line pending-list walk became bm25_collect_pending(). State shared with the extracted evaluator travels in a BM25CollectCtx. Behaviour-preserving; full gate green. While doing it, the 14 page + pd_lower reads in _scan.c and the one in pg_fts_trgm_index.c’s blob reader (a 9th instance of the 1.7.0 defect class, feeding a memcpy length) were routed through bm25_page_data_end(). done
C3 The 12,000-line translation unit: KEEP, and document why. Measured the cost of splitting: 15 am.c statics would go extern (13 for _scan, 5 for _trgm), 1 the other way, ~10 shared struct types would move into pg_fts_am.h (the on-disk-format header), and the hot-path static inline helpers (bm25_tid_to_docid, bm25_docid_to_tid, bm25_page_data_end, bm25_doclen_cursor_lookup – inside the 45%/37% profile) would stop inlining without LTO, which PGXS does not use. In return: three .o files and no behaviour change. Decision recorded at the #include site and in both included files' headers, with the two prerequisites (an internal header; a before/after latency measurement) if separate compilation is ever needed. decided: keep

Open – index behaviour

# item status evidence
I1 Bulk-ingest write amplification – FIXED for row-per-transaction ingest (1.8.3). Root cause was not the XID horizon (that binds only inside one multi-row statement) but the merge’s EXTEND_ONLY allocation never consulting the free list. BM25_ALLOC_SNAPSHOT (free list gathered once at entry, never re-read) keeps the recycle-race guard by construction and reuses earlier frees. Field shape, 30k then 100k docs of row-per-txn churn, autovacuum on, nothing manual: 1,823 -> 1,823 MB and 1,823 -> 1,875 MB with fts_vacuum finding nothing to reclaim; v1.8.2 on the same harness grew 3.2 GB per 5k rows and then deadlocked (a pre-existing concurrent-merge deadlock, also fixed, plus a live-page handout deadlock the fix exposed – both in the CHANGELOG). Residual: one very large INSERT ... SELECT of oversized rows still needs an fts_vacuum after; modelled the fewer-larger-merges alternative at ~2x, not worth its complexity. done; residual narrowed bench/RESULTS_I1_2026-09-18.md
I2 Common-term ranked latency – done in 1.11.0 (Approach A). Best-first block order with an exact effective-length block bound: year top-10 8.48 -> 0.81 ms (pg_search 2.63), 1,911 -> 22,369 tps at 16 clients. Conjunctive best-first for AND and phrase: united & states 23.9 -> 2.7 ms, "united states" 25.0 -> 3.6 ms. No format change. done 1.11.0 bench/RESULTS_A_2026-10-07.md
A1 Build time vs pg_search (257 s vs 71 s as expression indexes, 2.19M articles, 1.11.0 rig). Re-measured 2026-10-28 (c6id.8xlarge, PG 17.10, stored column, 8 workers, mwm 8 GB, bench/RESULTS_1144_2026-10-28.md): 103.1 s = parallel heap scan into 7 worker segments (~40 s) + the single-backend collapse merge (39.8 s) + the in-build pack, a second full merge of the one segment (23.7 s). Next step still the redesign: workers emit sorted runs to a shared tuplesort and the leader writes ONE segment (no collapse, no pack) – removes ~64 s of the 103 s if the scan stays ~40 s. A design change for its own minor release; unmeasured beyond this split. open (measured) bench/RESULTS_1144_2026-10-28.md
A2 Many-block conjunctions and wide ORs (world & war, 4-term OR; 1.11.0 rig: 5.87 vs pg_search 5.75, 13.97 vs 13.40). Offline (bench/RESULTS_1144_2026-10-28.md, 2.19M Wikipedia): best-first visits 643 of 2,086 war blocks at k=10. The proposed pairwise bound (max over co-occurring documents only) visits 642 – refuted. The bound is loose per TERM: storing the top 4 points of each 128-block’s (tf, quantized length) Pareto frontier (median frontier 6 points, p90 8) visits 55, exact; top 2: 136; top 1: 302; the exact block maximum: 10. That is a block-header format change (dual-read, per-segment flag, rewrite on merge: RELEASING.md) for a minor release. 4-term OR not simulated yet. open (measured) bench/RESULTS_1144_2026-10-28.md
D Two-level page bitmaps + SIMD (TIN-style). Standby: Approach A closed the common-term gap it was meant to close without it (I2). Format side is tractable via the 1.5.0 optional-per-segment-pointer + dual-read precedent (no REINDEX). Real cost: no SIMD infrastructure exists (no intrinsics, no runtime dispatch, no -mavx2 plumbing) and a scalar fallback must be kept for non-AVX and ARM – two implementations forever. Largest change the project has attempted, against a competitor that cannot be benchmarked. Needs explicit sign-off. Do not vectorize the vendored sparsemap. blocked on sign-off bench/NOTE_TIN_FEASIBILITY_2026-09-14.md, NOTE_SIMD_VENUE_2026-09-14.md
I5 Lazy phrase gate – done in 1.9.1. A ranked phrase/NEAR on a positions=on index checks adjacency per WAND candidate from the cursor’s own block instead of materializing every match first. "united states" k10: 138 (1.9.0) -> 34.8 ms, vs pg_textsearch 43.0 (same day, separate hosts). Identical results to the collect path on 28 phrase/k cases at 2.2M rows; phrase_gate test with planted mutants. done 1.9.1 bench/RESULTS_191_2026-10-06.md
I6 Rare-term throughput fell with backend count – fixed in 1.10.0. Each backend held a private ~4.8 MB copy of the C1 slot doclen array, and the copies competed for L3. 1.10.0 keeps one copy per segment per server in DSM (pg_fts.shared_doclen), refcounted per scan, keyed by segment identity so backends with different snapshots stay exact. Same-host A/B on Graviton3: rare-term tps at 64 clients 16,654 vs 13,283 (+25%), now flat 16 -> 64; per-transaction cost of bm25_doclen_cursor_lookup 21.2 -> 6.9 units. done 1.10.0 bench/RESULTS_110_2026-10-07.md, bench/PLAN_I6_SHARED_DOCLEN_IMPL_2026-10-07.md
I7 Per-query shared cache-line traffic – done in 1.12.0. Ranked lookups re-walked the dictionary-index chain (up to 85 pages, pinned and share-locked, read by every concurrent query) three times per term; each backend now binary-searches an in-memory copy of each segment’s dictionary index, kept in the existing relcache chunk. Rare top-10 278 -> 80 shared buffers; single-client rare 0.54 -> 0.27 ms; throughput rare +67-97%, common +74-92% at 16-64 clients on six hosts, results identical. Host-state variance itself is not addressable in pg_fts (bench/RESULTS_TPS_VARIANCE_2026-10-09.md); pg_fts.shared_doclen stays a documented trade. Remaining per-query costs: the doclen lookup and scoring (profile in bench/data_i7_2026-10-09/h1/). done 1.12.0 bench/RESULTS_112_2026-10-09.md
I8 Relcache cache leaked on an ERROR during its rebuild – FIXED 1.15.0. Attached at palloc, swapped before free, complete gate; t/015 (hooks build) 1.14.3 +11.5 kB per failed rebuild, fix flat. done CHANGELOG 1.15.0
I3 Managed-service validation – done for Neon (2026-10-29), the open-source compute/storage-separated backend. pg_fts 1.15.0 on Neon fa504217c (PG 17.5), a 16 MB compute with no local file cache, so index pages are rebuilt from WAL by the pageserver: build, deletes + VACUUM, inline flushes, fts_merge, fts_vacuum, a compute restart, and a branch at a mid-workload LSN (writes + merge on the branch) – index == seq scan at every stage (6 query shapes x 2 index forms, > 0), ranked top-20 == fts_bm25 oracle, 0 walredo errors. Vanilla: wal_consistency_checking=‘generic’ primary + standby, the full installcheck + churn, 20,536 verification images compared, 0 inconsistent pages (PG 17.10, 18.6). Not done: Aurora (closed; no custom C extensions). bench/RESULTS_150_2026-10-29.md s3. done (Neon); Aurora n/a bench/data_150_2026-10-29/i3/
I4 Independent review of WAL/crash/recovery paths. 1.15.0: two independent read-only sub-agent reviews against the RELEASING checklist (bench/data_150_2026-10-29/review/); the first found a UTF-8 split in I9 (an ERROR on CJK text) fixed before release. A HUMAN reviewer remains open – the ROADMAP item asks for one and no agent substitutes. open (human); agent reviews done per release bench/data_150_2026-10-29/review/
I9 Configured analyzer lost positions after 16,383 – FIXED 1.15.0. Chunked parsetext() with offset ordinals; the PR #12 example matches. Stored columns / expression indexes need a recompute or REINDEX to gain positions. done CHANGELOG 1.15.0
I9a 1.15.0’s chunked re-parse split HTML markup at chunk boundaries – FIXED 1.15.1. Cuts are configured-parser token boundaries verified against a whole-text parser-only pass; byte-identical to an unchunked wide parser on 28 of 28 long-document probes (1.15.0: 4 markup docs differed). Thesaurus phrases across a cut remain a limit. done 1.15.1 CHANGELOG 1.15.1
I13 VACUUM bulkdelete memory was O(postings) – FIXED 1.14.0. bm25_segment_docids decoded every posting of a segment into one array and bulk-added it: ~24 bytes per posting, independent of maintenance_work_mem or dead-tuple count. A production index (977M postings/segment) OOM-killed an autovacuum at 13.9 GB. Now a v4 segment reads the doclen sidecar (O(documents), zero posting decode) and an inline-doclen segment flushes in maintenance_work_mem/24 batches; both loops are cancellable + cost-delayed. 400k docs/120M postings: one-row INDEX_CLEANUP ON 1,429 -> 151 MB. vacuum_mem test. Present since 1.1.5. The 1.14.0 claim “5%-deleted VACUUM 2,820 -> 157 MB” held only with the sidecar on or a small maintenance_work_mem; with the reporter’s settings (sidecar off, mwm 4 GB) 1.14.0 measured 2,814 MB – retracted, see I13a. done 1.14.0 (completed 1.14.1, I13a) pg.ddx.io report 2026-10-15
I14 Manual VACUUM errored under parallel index vacuuming on PG18 – FIXED 1.14.0. bm25_truncate_lock took AccessExclusiveLock in the parallel-vacuum leader’s cleanup; under wal_level >= replica that assigns an xid (hot-standby lock logging), an ERROR in parallel mode, rolling back the whole VACUUM so relfrozenxid stalled. Now declines truncation when IsInParallelMode() (tail stays in FSM, reclaimed by a later non-parallel VACUUM/fts_vacuum). Reproduced + fixed on PG 18.6; autovacuum and VACUUM (PARALLEL 0) never affected. Introduced 1.11.0 (ada32aa). vacuum_parallel test (fails on the mutant on PG18). done 1.14.0 pg.ddx.io report 2026-10-15
I15 @@@ estimated every row whenever anything was pending – FIXED 1.14.1. 1.14.0’s fts_matchsel used the ordering scan’s upper bound (bm25_query_maxhits), which returns N when npending > 0; with continuous inserts that is always, so plans flipped to Seq Scans (production: a 96-row count estimated at 1,651,568 and timed out). Now segment df / segment documents (bm25_query_segdf); falls back to tsmatchsel for a modifier term, a term in no segment while documents are pending, or an index whose segments hold fewer documents than its pending list. 100k rows + 3k pending: rare est 103,000 -> 103, Seq Scan 20.54 ms -> Bitmap 0.143 ms (medians of 7). planner_pending test (fails on the 1.14.0 estimator). done 1.14.1 pg.ddx.io report 2026-10-15
I16 VACUUM grew the index when it could not truncate – FIXED 1.14.1. The cleanup’s compaction ran its extend-only vacate after a declined conditional truncation (any session holding a lock on the index, since 1.11.0; parallel mode, 1.14.0), so each such VACUUM grew the file by about its live size: reader held across 3 VACUUMs 1,507 -> 4,085 blocks (1.13.0, 1.14.0); 3 parallel VACUUMs 7,848 -> 10,198 (1.14.0). The vacate now needs AccessExclusiveLock (fts_vacuum, build); cleanup packs only. Fixed: 1,506 held, 7,845 held; a later lockless VACUUM reclaims. bm25_vacuum_growth isolation test + vacuum_parallel size checks (both fail on 1.14.0). done 1.14.1 found re-validating I14, 2026-10-16
I13a I13’s batch was unbounded at a large maintenance_work_mem – FIXED 1.14.1. mwm/24 at 4 GB = 178M entries > a 60M-posting segment, so doclen_sidecar=off bulkdelete was still one sort: 1,430 MB, cancel 3.5-7.3 s at 200k docs (= 1.13.0). Capped at 4M entries and flushed before a term that would overfill the batch, so a batch holds at most max(4M, df) (cap 256k/1M/4M/8M measured: 16.7/13.3/13.9/14.1 s, 76/92/163/259 MB at 400k docs): 155/169 MB at 200k/400k, cancel <= 0.43 s (shipped tree). done 1.14.1 re-validation 2026-10-16
I17 count(*) pushdown read indexes no query may use – FIXED 1.14.1. It (and the 1.14.0 @@@ estimator) took the index from RelationGetIndexList: a partial fts index made count(*) count only the predicate’s rows (100 vs 2,000), an indcheckxmin index too new for the snapshot and a cancelled-CIC invalid index gave wrong counts (1/49 vs 0/50; 0 vs 1), and during CIC’s first wait count(*) (1.13.0, 1.14.0) and in 1.14.0 every @@@ plan failed (empty file). From the code: partial index since 0.1.0; the others since 1.3.0 for a stored-column index (expression index only with its expressions already loaded); measured on 1.10.0/1.13.0/1.14.0. Both now use the planner’s rel->indexlist and skip partial indexes. bm25_cic_planner spec + planner_indexes test (fail on 1.14.0). Open: a predicate-implied partial index could serve the pushdown (not attempted). done 1.14.1 review of the I15 fix, 2026-10-17
I18 Index returned rows whose own @@@ is false, one slot after a true match – index data from <= 1.10.0, REINDEX (1.14.2). pg.ddx.io, 1.14.1 / PG 18.6, an index written by 1.10.0 then upgraded in place: an entry of a row that died while pending (rollback, failed batch/COPY, duplicate key, delete) survived VACUUM in <= 1.10.0, and the row that reused the slot inherited its terms, returned as exact (no recheck). Fixed in 1.11.0 but documented as “no REINDEX”; the 1.11.0 lost-insert bug likewise left missing rows. Reproduced: 199/399 false on 1.10.0, kept after upgrade + VACUUM + merge, 0 after REINDEX; 1.14.1 clean in every arm. 1.14.2 = docs + SQL check + pending_reuse test (fails on the mutant). Open: a SQL-callable verifier (a minor release); not tried on 1.14.1: CIC/REINDEX CONCURRENTLY during aborting inserts, parallel build, standby. vacuum_parallel’s serial_reclaims size check failed once in 26 nix PG17 runs and once more on 1.14.3: another backend’s snapshot (any database) blocks page recycling; the test now waits for other snapshots, bounded (1.14.3, bench/RESULTS_1143_2026-10-28.md section 3). done 1.14.2 (docs); users REINDEX pg.ddx.io report 2026-10-24
I19 Four user reports against 1.14.2 (robinlahtinen, Codeberg #6-#10) – FIXED 1.14.3. #9 fts_bm25/fts_bm25_opts read dfs[i] past a short array (stale value / NaN; now df = 1). #7 MSVC link (__builtin_clz, since 1.5.0). #10 fts_index_df left pending documents out (now segments + pending, as the ordering scan); sibling: ranked prefix/fuzzy/regex/weighted queries (bm25_topk_exact, since 1.13.0) had the same gap. #8 fts_matchsel now reads a predOK partial index (the count pushdown still never does); <=> docs say the BM25 order comes only from the ordering scan. #6 absent & (a | b) skips the segment walk (10.8 -> 0.066 ms, 400k rows). Tests bm25_dfs, index_df_pending, planner_partial, ranked_empty_and (all fail on 1.14.2). done 1.14.3 Codeberg #6-#10, bench/RESULTS_1143_2026-10-28.md
I19a Ranked rare & (a | b) walked the OR group – FIXED 1.15.0. Driven from the cheapest positive operand of the root AND when it is <= 1/8 of the postings: 3.6 -> 0.2 ms (400k rows, PG 17/18). Open: the seeks still read the OR group’s block headers (buffers 501 -> 417). done CHANGELOG 1.15.0
I19b Collect path built a whole OR group before an empty AND operand – FIXED 1.15.0 for a plain required term absent from a segment: zzzq & (apple | ban:*) 3.2 -> 0.07 ms. Open: a present but rare required term (rare & (apple | ban:*), 3.2 ms) still builds the group. done (plain-term case) CHANGELOG 1.15.0
I19c Ranked prefix / phrase-with-prefix from its dictionary expansions – candidate, from an unsubmitted fork branch (cepymm/pg_fts perf/ranked-phrase-prefix 79bb338, 2026-10-15, 41 commits behind 1.15.0): expand a prefix to <= 64 dictionary terms and rank with the plain-term top-k (phrase: last term any expansion, adjacency checked). Reviewed 2026-10-29, not ported: it changes the SCORE of a prefix query (today the collect path scores the literal term: measured on 1.15.0, a document holding only the expansion zqxvaa has fts_bm25(d, 'zqxv*') = 0 while one holding the literal zqxv scores 0.88, and the ranked scan puts every literal holder first; the code calls expansion scoring ‘a separate ranking change’), so <=> values and order change and fts_bm25, the documented seq-scan oracle, would disagree with the index unless it changes too. Its test checks ids only with LIMIT covering every match, so it cannot see a score change. Needs: a ranking-semantics decision (expansion scoring everywhere: index, fts_bm25, fts_index_df), score-level tests, a minor release. Would also close the I19b remainder (rare & (apple | ban:*)). open (design)
I20 Flush publish window returned a pending row twice – FIXED 1.15.0. One metapage update publishes the segment and clears the pending list; t/014 (hooks build) 1.14.3 100/50 rows, fix 50/50. done CHANGELOG 1.15.0
I17a count(*) from a partial fts index – FIXED 1.15.0 when the other quals are provably equivalent to the predicate (both directions, predicate_implied_by); anything else keeps the normal plan. done CHANGELOG 1.15.0
I10 ASan signal over the SQL paths – done. ASan-built PostgreSQL + test/asan_smoke.sql (1.14.0), fails on a false check (1.14.1), and (1.15.0) on the cassert allocator’s chunk-overflow / double-pfree reports, which ASan cannot see; checked with a planted overflow mutant. done CHANGELOG 1.15.0
I11 Ranked LIMIT >= 250 on an expression index took the bitmap + Sort plan – FIXED 1.14.0. fts_matchsel estimates @@@ from the index dictionary df (a term’s df, AND/phrase the min, OR the sum) instead of tsmatchsel’s fixed default, so the planner prices a common term apart from a rare one; it falls back to tsmatchsel for a non-constant query / no bm25 index / join clause. year LIMIT 5000 at 50k rows: 21.6 s -> 44 ms, staying on the ordering scan. SQL change (new estimator, both @@@ operators rewired); pg_fts--1.13.0--1.14.0.sql. planner_cost test. done 1.14.0 metal smoke test 2026-10-15
I12 Ranked reads under sustained inserts – FIXED 1.14.0. Every ranked query read the whole unflushed pending list (since 1.13.0, so pending documents rank), so latency grew with inserts since the last merge (0.22 ms none pending, 205 ms at 1M). pg_fts.pending_flush_threshold (default 20000, 0 disables) makes the inserting backend fold the pending list into a segment at the threshold, taking the maintenance lock only if free (inserters never wait); and a single term across many segments now runs one global best-first pass instead of MaxScore, so the extra flushed segments do not slow single-term reads (7 segments: rare top-10 3.80 -> 0.26 ms, identical results). Mixed load 8+8 at 2.19M rows: ranked reads 97 -> 3,223 tps at 10% fewer inserts. pending_flush test. Metal write-band A/B still to re-run for the release. done 1.14.0 (metal A/B pending) metal benchmark 2026-10-15

Open – measurement debt

# item
M1 C2 cross-engine ingest. Rivals' ingest paths differ fundamentally; needs per-engine forms chosen as carefully as C1X’s.
M2 C3 NDCG vs rivals – measured on 2 BEIR sets (2026-10-28). SciFact NDCG@10: pg_fts 0.6848 (English) / 0.6650 (simple), pg_search 0.26.1 0.6339, ts_rank 0.5684; NFCorpus 0.3240 / 0.3016, 0.3000, 0.3001 (bench/RESULTS_1144_2026-10-28.md s5). Open: larger BEIR sets (e.g. TREC-COVID, FiQA), pg_textsearch, VectorChord-bm25.
M3 Longer ingest run done 2026-09-18: 20 rounds / 100k row-per-txn docs. No decay – throughput oscillates 33-56 rows/s with round durations at exactly 1x / 1.33x / 1.67x a 90 s floor, i.e. autovacuum cycles landing in the round or not. The absolute rate is the harness (psql building 1,660-term strings), not publishable. Appended to RESULTS_I1_2026-09-18.md.
M4 The published competitor set on TIN’s exact rig (i7i.8xlarge, 8 vCPU / 32 GB container, Stack Exchange corpus) plus pg_fts – places us on their axis without asserting anything about TIN. Several EC2 hours.

Declined (with the measurement that declined them)

  • Impact-ordered postings – breaks the docid ordering that count(*)/AND/phrase/prefix need.
  • Early termination – breaks exact top-k.
  • Heap-side positions=off – saves ~16% of a STORAGE=extended column; no-go.
  • Parallel ranked scan – built, measured, reverted (bench/NOTE_PARALLEL_RANKED.md).
  • Parallel merge – 1.45x slower and 19% larger at scale; mpmw=8 silently serial.
  • df-threshold bulk load – cost ~linear in df, no fixed floor.
  • Verbatim posting copy on merge (TIN item C) – the merge re-encodes through the build hash table; no splice point.
  • Vectorizing sparsemap – never a query hotspot; the one time it was the bottleneck (P0, 99.75%) the fix was algorithmic; its compressed layout is SIMD-hostile; it is vendored byte-identical to upstream on purpose.

Closed (one line each; detail in CHANGELOG)

  • 1.11.0 (Approach A): best-first single-term and conjunctive top-k with exact block bounds; MaxScore wrong top-k for 4+ terms (every release) and ranked term:LABEL fixed; merge 5x faster with identical output and no fts_vacuum after a plain build; maintenance buffer ring; prefetch where chains jump and for the ranked heap fetches; VACUUM pending-delete fix and four concurrency fixes (lost inserts during flush, truncation without AccessExclusiveLock, freed-page link overwritten by the recycle XID, double free on scan retry), all in every earlier release, regression test t/012. Champion lists (Approach C) not needed. bench/RESULTS_A_2026-10-07.md.

  • 1.12.0: ranked dictionary lookups from a per-backend in-memory copy of each segment’s dictionary index (I7); pg_fts.doclen_cache_mb changes now take effect in the session. bench/RESULTS_112_2026-10-09.md.

  • 1.9.0 ranked-retrieval release (LIMIT pushdown, resident doclen, dense high-df scoring, score reuse) and two exactness fixes present since 1.x (negative idf after deletes; WAND last-block skip). Leads pg_textsearch 1.4.0 on every latency and throughput band except phrase (3.2x behind).

  • 1.8.6 sparsemap 5.7.0 -> 5.8.0: bulk sm_add_many_grow merges instead of inserting per bit (1.4-4x faster standalone; unmeasured in pg_fts at scale), plus a coalesce over-read fix. Byte-identical wire output.
  • 1.8.5 sparsemap 5.6.0 -> 5.7.0: small-set tombstone encoding, verified live through pageinspect and by a new sanitizer fuzz target over the vendored library; the 1.8.4 open guard holds in both encodings.
  • 1.8.4 sparsemap 5.5.1 -> 5.6.0 (upstream security hardening); every sm_open now goes through bm25_sm_open_checked, so a corrupt tombstone/trigram bitmap raises ERRCODE_DATA_CORRUPTED instead of silently opening as EMPTY – which for a tombstone map would have resurrected every deleted row. t/003 proves it (red unguarded, green guarded).
  • 1.8.3 two deadlocks that shipped in every prior version (insert-path full-directory merge ran without the maintenance mutex; a live page could be handed out as merge output), and the bulk-ingest bloat root cause (merge was extend-only, never reused pages) – zero net growth over 100k row-per-txn docs.
  • 1.8.2 allocator state is a scoped struct with misuse an elog(ERROR); bm25_collect_matches 412 -> 226 lines; the single translation unit kept as a costed, documented decision; a 9th pd_lower read site guarded.
  • 1.8.1 count-path: df fast-count gate tests (10, non-vacuous); block-run VM checking measured ~1%, kept as cleanup.
  • 1.8.0 intra-word - . / are terms, not operators (pkg-config no longer parses as pkg & !config).
  • 1.7.2 insert-time merge gated on segment pressure: bulk-ingest growth -31%.
  • 1.7.1 pd_lower guard generalised to all 8 page-read sites; “one WAL record per page” known issue retracted (0.005 ms/page measured).
  • 1.7.0 P0: unvalidated pd_lower in the merge dict walk made an index permanently unvacuumable at field shape; fixed. Huge-alloc gaps in doclen/tombstone arrays fixed. Cleanup no longer grows the index (18/18/18 MB vs 35/52/69).
  • 1.6.1 P0: VACUUM never completed on a delete-heavy index (4h39m -> 393 s; dense tombstone bitmap sized by sm_maximum). sparsemap 5.5.1.
  • 1.6.0 phrase over positionless docs returns false, not a silent conjunction (matches OP_PHRASE).
  • 1.5.9 / 1.5.10 non-UTF-8 case folding; common-term 1.56x via ascending-resume + word-load bm25_for_get.
  • 1.5.0 doclen sidecar (format v3 -> v4) with dual-read, no REINDEX – the precedent for all future format changes.
  • COUNT pushdown (CustomScan), fts_search under-fetch, reserved keywords as literals, sparsemap error-path leaks, recovery guard on fts_merge/fts_vacuum, privilege lockdown, recently-dead exclusion from corpus stats, parallel-build memory ceiling – all shipped; see history file.