Changelog

Changelog

All notable changes to pg_fts are documented here.

1.8.2 - 2026-09-17

Code-quality release from the fresh-eyes review (REVIEW_2026-09-17.md), plus one real fix found while doing it. No on-disk format change; no REINDEX required.

Fixed

A ninth unvalidated pd_lower read, in the trigram blob reader (pg_fts_trgm_index.c). It computed a memcpy length as pd_lower - contents_offset; on a corrupt or recycled page whose pd_lower is below the contents offset that wraps to a huge Size before the Min() clamps it, and the copy runs past the page. Same defect class as the 1.7.0 P0 that made an index permanently unvacuumable. Now routed through bm25_page_data_end(), as are the 14 remaining reads in pg_fts_am_scan.c. 1.7.1 claimed to have fixed “all eight” sites; this makes it nine, and the count is now every read of pd_lower in the tree.

Changed (behaviour-preserving; full gate green after each)

Allocator state is a scoped struct, and misuse is a hard error. The four file-scope globals (bm25_lowfree, _n, _i, bm25_alloc_extend_only) became one BM25AllocCtx reachable only through bm25_alloc_scope_enter() / bm25_alloc_scope_exit(), which nest by returning the previous context. In the 1.7.1 work, code read those globals without owning them and handed out garbage block numbers (“could not open file … target block 829694001: previous segment is only 527 blocks”); only t/007_segment_cap.pl noticed. bm25_new_buffer() now elog(ERROR)s on that condition. Deliberately an elog, not an Assert – the release gate is not a cassert build, and a check that fires only in a build nobody ships is documentation, not enforcement.
bm25_collect_matches split, 412 -> 226 lines. The per-segment loop body is now bm25_collect_segment() (returns SEG_RESTART for the positional-phrase fallback the loop used to express as s = -1; continue) and the pending-list walk is bm25_collect_pending(). Shared state travels in a BM25CollectCtx.
The single translation unit is now a documented decision, not an accident. pg_fts_am.c #includes pg_fts_am_scan.c, pg_fts_trgm_index.c and pg_fts_lev.c. Splitting was costed: 15 statics would go extern, ~10 shared types would move into the on-disk-format header, and the hot-path static inline helpers inside the 45%/37% common-term profile would stop inlining (PGXS does not use LTO). In return, three .o files and no behaviour change. Kept; the reasoning is at the #include site and in both included files' headers, with the two prerequisites if separate compilation is ever needed.

1.8.1 - 2026-09-17

Counting-path work from the TIN feasibility review. No on-disk format change; no REINDEX required.

Added

Ten gate-refusal tests for the single-term count(*) fast path. That fast path (answering from the dictionary’s df with no posting decode and no heap access) has existed since the COUNT pushdown work, but shipped with a single positive test and nothing proving its gates actually refuse — the shape where a missed gate silently returns a plausible wrong count. Each case is now compared against a ground truth computed without the index: prefix, conjunction, disjunction, negation, unmerged pending documents, tombstones, and a not-all-visible heap all fall back and agree exactly, and the fast path is confirmed to resume after VACUUM.

Changed

fts_count() and the COUNT(*) pushdown now consult the visibility map once per run of matches on the same heap page instead of once per matching tuple, and create one tuple slot per call instead of one per probed tuple. Safe because matches arrive sorted and de-duplicated and a docid is block × MaxHeapTuplesPerPage + offset, so matches on a page are contiguous; the visibility map is page-granular.

Measured effect: none significant (~1–2%, inside run-to-run noise). An apparent 8.5% improvement came from a single high outlier in the baseline; repeated same-arm runs overlap (base 397.3–408.2 ms, fix 397.4–402.7 ms). The loop is dominated by table_index_fetch_tuple heap probes, not by visibility-map lookups. Counts are identical in both arms. Shipped as a code-quality change, not a performance feature.

Documentation

README and the SGML manual now describe when the df fast count applies and, more importantly, when it refuses.
bench/RESULTS_ABC_2026-09-17.md records the measurements, including the two corrections above; bench/NOTE_TIN_FEASIBILITY_2026-09-14.md is annotated with the outcome. A third proposed item (copying posting bytes verbatim during a merge) was withdrawn after reading the merge path: it decodes through the build hash table and re-encodes at flush, so there is no byte-stream splice point.

1.8.0 - 2026-09-14

Query parsing fix with a behaviour change. No on-disk format change; no REINDEX required — stored data was never affected.

Fixed

-, . and / inside a word are terms, not operators. Reported from the field:

query	before	after
`pkg-config`	`('pkg' & !'config')`	`'pkg-config'`
`install-info`	`('install' & !'info')`	`'install-info'`
`foo/bar`	`'foo'` (rest swallowed)	`'foo/bar'`
`python3.14`	`('python3' & '14')`	`'python3.14'`

The hyphen case was the damaging one: the ! clause actively excluded the documents being searched for, so searching pkg-config returned everything except pkg-config, and install-info matched 1 row instead of 10. / was worse in a different way — it opened a regex and swallowed the remainder of the query. As the reporter put it, this is a worse failure mode than operator injection: injection raises a visible error, this silently returns a different, wrong answer.

The separator set is not a guess — it is what the document analyzer already joins, verified against to_ftsdoc('simple', 'a-b c/d e.f g_h i+j'), which yields 'a-b' 'a' 'b' 'c/d' 'e.f' 'g' 'h' 'i' 'j': -, . and / stay inside a token while _ and + split. PostgreSQL’s own parser agrees, classifying them asciihword, file and file. So the query lexer was the only side that disagreed, and the tokens needed to match these queries were already stored.

Behaviour change

A - between two word characters no longer negates. a -b still excludes b (prefix position), and !b is unchanged, but an application relying on a-b meaning “a AND NOT b” must now write a !b or a - b.
c++ and gtk+ still lex to 'c' and 'gtk'. A trailing separator is dropped, which is what to_tsvector and our own document analyzer do, so this is parity rather than a bug — noted because the original report listed it alongside the others.

Regression cases covering all four inputs, plus prefix negation, leading -, and /regex/, are pinned in sql/pg_fts.sql.

1.7.2 - 2026-09-14

Measured the cause of the bloat known issue and removed ~31% of it. No on-disk format change; no REINDEX required.

Fixed

Bulk ingest grows the index ~31% less. At high terms-per-document every document exceeds one pending page, so each one mints a one-document segment — and the insert-time merge then folded it in immediately, rewriting a whole level-0 run for every single document. Measured: 23–30 index pages extended per document (linear) against roughly 2 pages of actual postings, a ~12–15× write amplification with page reuse at ~0.3%.

The merge is now gated on there being BM25_MERGE_FANOUT small runs waiting. Below that threshold the leveled compactor would find no level over capacity and do nothing anyway, so this skips work without changing behaviour. Over six 5,000-document batches the index peaked at 21,874 MB instead of 31,537 MB, and the size after one fts_vacuum is byte-identical (124 MB).

The segment-directory bound this protects was re-verified under the worst case for segment minting — one row per transaction, 4,000 transactions — reaching a maximum of 15 segments against the hard cap of 128. t/007_segment_cap.pl now asserts <= 64 rather than <= 128, since a bound at the cap only fails once the index is already in the state that motivated the eager merge (a field deployment went 8 → 128 segments in ~1 h and could then neither merge nor VACUUM).

Known issues

The bloat is reduced, not eliminated. Growth is still ~3.7 GB per 5,000 documents at field shape, and fts_vacuum after bulk ingest is still recommended (it is fast — tens of seconds for millions of pages — and recovers the space completely).

The mechanism is now measured rather than guessed: freed pages are found and then rejected by bm25_page_recyclable(), because they were freed by the inserting transaction itself and GlobalVisCheckRemovableXid() cannot yet clear them (norecyc=3,169 of 5,924 allocations). That gate is correct and must stand — bypassing it previously corrupted a concurrent reader. So in-transaction reuse is impossible by construction, and fully fixing this means moving the merge out of the inserting transaction: a design change, not a point-release edit.

Measurements: bench/RESULTS_KNOWN_ISSUES_2026-09-14.md.

1.7.1 - 2026-09-14

Follow-up to 1.7.0’s page-corruption fix, plus a retraction of one of 1.7.0’s known issues. No on-disk format change; no REINDEX required.

Fixed

The pd_lower bounds guard is now applied at every page-read site. 1.7.0 fixed the dict walk in merge_source_load_page, where an unvalidated pd_lower made the merge request an impossible allocation and left the index permanently unvacuumable. Auditing the siblings found eight sites forming page + pd_lower from unvalidated on-page data — including both walks in bm25_free_segment and the doclen and posting readers. All now route through one helper, bm25_page_data_end(), which validates in the integer domain (forming the pointer at all is undefined behaviour for an absurd value) and returns an empty range for anything out of bounds, so a caller degrades to “this page has nothing to read” rather than walking off the page. 1.7.0 fixed one instance of this defect; this fixes the class.

Retracted

1.7.0’s “bm25_free_page emits one WAL record per page” known issue was wrong. Measured directly: fts_vacuum freed 8,686,917 pages in 46 seconds — 0.005 ms/page, roughly 2,800× cheaper than the ~14 ms/page I published, and a second run confirmed it (7,912,288 pages in 33 s). There is no per-page WAL problem and no WAL batching is needed. My figure came from a 113-minute run on an index that had already hit the 1.7.0 allocation bug repeatedly; gdb showed the backend inside bm25_free_page and I turned “where it is” into “why it is slow”. A stack sample gives a location, not a bottleneck.

Known issues

The transient bloat spike is confirmed and larger than reported: ~210×, not 45×. Measured at field shape (1,660 terms/doc), inserting 5,000 documents at a time with no maintenance: the index grows ~5.6 GB per batch with nsegments pinned at 8, reaching 61,814 MB after ten batches, and a single fts_vacuum returns it to 236 MB. The space is freed-but-never-reused, not live.

The cause is not yet known. Four hypotheses were eliminated: one-doc segment accumulation (impossible — BM25_MAX_SEGMENTS is 128 and the insert path forces a merge), 128-segment cycling (nsegments sits at 8), freed pages failing the recyclability XID gate (instrumentation showed the free-list scan is never reached: probe=0 reject=0), and loop-wide bm25_alloc_extend_only (scoping it per merge produced byte-identical growth — reverted rather than shipped as a fix). The next step is a counter on each of bm25_new_buffer’s three outcomes rather than another hypothesis.

Practical guidance unchanged: run fts_vacuum after bulk ingest. It is fast (tens of seconds for millions of pages) and recovers the space completely.

Details and measurements: bench/RESULTS_KNOWN_ISSUES_2026-09-14.md.

1.7.0 - 2026-09-13

Two field-blocking fixes found by reproducing the reported ~2.87M-doc email-body index shape. No on-disk format change; no REINDEX required.

Fixed

An index could become permanently unvacuumable. The dict-page walk in merge_source_load_page took its end bound from the page’s pd_lower with no validation, and stepped by an untrusted termlen. On a recycled or malformed page the walk ran past the page and counted garbage entries, so the caller’s doubling asked for an impossible allocation:

  ERROR:  invalid memory alloc request size 3406063183

Because this runs under bm25_merge_segments_streaming, it failed every merge, every autovacuum cleanup, and fts_vacuum — the index could never be vacuumed or reclaimed again. This is the likely cause of the reported “VACUUM/merge do not reclaim bloat”. Isolated with gdb; both the counting and filling walks are now bounds-checked, and pd_lower is validated as an integer before any pointer is formed from it (forming page + pd_lower for a corrupt value is itself undefined behaviour — the new fuzz target for this loop caught that with UBSan). - Huge-allocation gaps in bm25_doclens_load’s resident docid array and bulkdelete’s carry/newdead tombstone arrays, which used plain palloc/repalloc and so failed the same way on a large or delete-heavy index. The FTS_ALLOC_MAYBE_HUGE macros already existed for the per-term posting arrays; these sites were missed. - Index cleanup no longer grows the index. bm25_vacuum_compact now skips a compaction pass when its free space is not yet reusable: bm25_page_recyclable() gates on GlobalVisCheckRemovableXid(), so pages freed by the same cleanup’s merge are all rejected, and the pass would relocate live data upward while reclaiming nothing. Measured before: 35 → 52 → 69 MB across three cleanups with no rows added. After: flat.

Added

Fuzz target for the dict-page walk (test/fuzz/fuzz_block.c), asserting the walk stays inside the page and can never report more entries than a page can physically hold, for arbitrary page bytes and arbitrary pd_lower.
t/010_vacuum_delete_heavy.pl now asserts no growth across repeated cleanups, and additionally that cleanup still reclaims after deletes.

Known issues

A transient bloat spike at nsegments=8, reproduced at field shape: an index went 299 MB → 66,796 MB → 1,016 MB across three churn rounds, settling at 1,480 MB. The index is not permanently bloated — it inflates ~45× while segments accumulate and collapses once merges catch up, so an index sampled during that window looks like unbounded bloat. Not yet fixed.
bm25_free_page emits one WAL record per page. fts_vacuum on a 3.8 GB index ran 113+ minutes without finishing (progressing, not hung): ~489k pages × a full GenericXLog delta each. Needs WAL batching. Not yet fixed.

Both are documented with reproductions in bench/RESULTS_FIELDSHAPE_2026-09-13.md.

Documentation

README and the SGML manual no longer recommend scheduling a periodic fts_vacuum. Measured at 1M docs with autovacuum on and no manual maintenance: flat at 511 MB over five cleanups, flat at 875 MB over six churn rounds, and 875 → 106 MB after deleting half the table, with results exact throughout.

1.6.1

P0 fix: VACUUM could consume CPU indefinitely and never complete on an index with many tombstones. C-only, no SQL objects, no on-disk format change (BM25_VERSION stays 4), no REINDEX — ALTER EXTENSION is the whole upgrade.

Also re-vendors sparsemap 5.4.0 → 5.5.1.

The bug

On a 2,188,038-document index, deleting 312,166 rows and running VACUUM produced a backend pinned at 100% CPU that never finished — 4h39m of CPU consumed, with 99.75% of perf samples in __sm_get_chunk_offset, reproduced twice. There was no error and no log line; the index simply never got cleaned.

Because a VACUUM that never completes also never reclaims space, an index that appeared never to shrink after deletes is a likely symptom of this. If you have seen either behaviour, this release is the fix.

Measured after the fix, same workload: 393 s and completes, with results identical to sequential-scan ground truth.

stage	before	after
`VACUUM` with 312k tombstones	4h39m CPU, never finished	393 s, rc=0
further round (376k more deleted)	—	270 s
further round (375k more deleted)	—	178 s
`fts_vacuum`	—	135 s

Correctness verified at every stage — index counts equal sequential-scan counts (year 504368/504368, 378037/378037, 251902/251902; slovakia 5605/5605, 3751/3751).

Root cause: one mistake in two places

Both sites probed the tombstone sparsemap in a pattern that defeats every acceleration structure it offers, so each membership test walked the chunk chain from the head. At ~1,068 chunks × millions of probes, that does not terminate in practical time. Fixing the first site only exposed the second.

bm25_merge_segments_streaming (the merge/compaction path, reached by bm25_vacuumcleanup). This loop walks terms in sorted order, and each term’s postings ascend from a low docid — so the docid sequence resets at every one of millions of term boundaries. Neither the 8-way MRU cache used here, nor a forward-resume cursor, nor batched sm_contains_many survives that: each pays an O(chunks) startup per term. Fix: the tombstone map is read-only for the whole merge, so decode it once per source into a dense bitmap (sm_next_member, a single forward pass) and test each posting in O(1).
bm25_bulkdelete (the per-index delete path). A cursor was used, but was declared inside the walk and therefore reset every iteration, restarting each lookup from the head. The enclosing walk is monotonically ascending — exactly the cursor’s contract — so the cursor was right in intent and defeated by its scope. Fix: hoist the declaration out of the loop.

This is the same class of pathology fixed for the ranked scan in 1.4.1 (24 s → 2.5 ms); these two paths never received the equivalent treatment.

sparsemap 5.4.0 → 5.5.1

Re-vendored as exactly upstream plus our one namespacing block; vendor/sm.h is byte-identical to upstream, and the public header changes only its version macros, so this is a drop-in.

Two fixes reach code pg_fts actually executes:

Big-endian chunk-descriptor flag reads (5.5.0). Ten sites aliased the 64-bit descriptor as uint8_t *, walking its 2-bit flags in reverse on big-endian and breaking every counting and navigation path. sm_contains was unaffected because it shifts the word directly — which is precisely why the bug hid behind a working membership test. pg_fts is exposed through sm_next_member, used to iterate tombstones. (Note: big-endian remains untested in our CI.)
__sm_append_data now returns bool and is warn_unused_result (5.5.1). In 5.5.0 the append inside __sm_map_set was unchecked, so a full buffer silently overflowed instead of reporting ENOSPC. pg_fts reaches that path via sm_add_many_grow → __sm_add_c → __sm_map_set, and our grow-and-retry loop depends on ENOSPC being signalled rather than the buffer being overrun.

The three headline 5.5.0 fixes (sm_difference RLE data loss, sm_offset invalid maps, sm_split ENOSPC) are in functions pg_fts does not call.

A note on how this was found, and on our test coverage

The hang was found while qualifying the sparsemap bump at scale — not by the local test suite. installcheck (PG 17/18), the full TAP set, alloc/ascii guards and the block fuzzer all pass on the unfixed code, and passed on two wrong fixes before the real cause was isolated with gdb. None of those checks deletes a large fraction of a large indexed table and then vacuums.

The gap was known: ROADMAP item 9’s outstanding task was exactly “quantify the merge path under a delete-heavy workload”, and that measurement had never been completed. Details, including the failed attempts and why each failed, are in bench/P0_VACUUM_HANG_2026-09-10.md.

1.6.0

Correctness release: an unverifiable phrase no longer silently answers as a conjunction. C-only – no SQL objects change, no on-disk index format change (BM25_VERSION stays 4), no REINDEX. MINOR rather than patch because query results change, for the better.

Also re-vendors sparsemap 5.4.0 -> 5.5.0, which fixes a big-endian corruption that reaches our tombstone iteration.

1. A phrase whose adjacency cannot be verified now returns false

phrase_step() fell back to presence-only AND whenever either operand lacked positions – “recall preserved, precision degraded” – so a phrase query could report a NON-ADJACENT document as a match, with nothing to distinguish it from a real phrase hit.

This is now false, which is what PostgreSQL does: absent TS_EXEC_PHRASE_NO_POS, OP_PHRASE “always returns false if lexeme position information is not available” (tsearch/ts_utils.h). Upstream returns false for strip(to_tsvector('simple','quick brown')) @@ 'quick <-> brown' even though the words are adjacent; pg_fts_match.c claims to mirror TS_execute and on this branch did the opposite.

Four routes reached the bad path, all confirmed on 1.5.10:

input	1.5.10	1.6.0
`$$'brown':1 'quick':1$$::ftsdoc` (canonical literal, no `@`)	t	f
`to_ftsdoc(strip(to_tsvector(...)))`	t	f
`ftsdoc \|\| ftsdoc` where one side lacks positions	t	f
`quick <-> (brown & fox)` on a fully positioned doc, via the tsquery cast	t	f

The fourth is the important one: the boolean arms null out positions, so the document’s own flag cannot distinguish “no positions anywhere” from “this operand lost them”. The fix therefore carries an explicit reason flag on the operand, set at exactly one producer.

Prefix-inside-phrase is unchanged. "quick bro*" stays deliberately permissive, because prefix positions are genuinely not tracked – that is shipped, documented lossiness rather than a bug, and it now has test coverage it lacked.

Why false and not an error: @@@ is a match operator, so raising an error would be order-dependent (a predicate that errors on a full scan can succeed under a LIMIT) and would turn a handful of unverifiable rows into a whole-table failure.

2. Field-zone restrictions on a positionless document match nothing

Zone labels are carried in each position’s high bits, so a document without positions carries no label information – it is unknown, not “D”. Previously term:D matched every positionless document while term:A matched none, and a concatenation that dropped one side’s labels silently answered term:D = true. A zone restriction that cannot be evaluated now does not match.

3. sparsemap 5.4.0 -> 5.5.0

Re-vendored as exactly upstream 5.5.0 plus our one namespacing block (SPARSEMAP_PREFIX=__pg_bm25_); vendor/sm.h is byte-identical to upstream. Header change is purely additive (one new function), so it is a drop-in.

The fix that reaches pg_fts is chunk-descriptor flag reads on big-endian hosts: ten sites aliased the 64-bit descriptor as uint8_t *, walking the 2-bit flags in reverse on big-endian and breaking every counting and navigation path. sm_contains was unaffected because it shifts the word directly – which is precisely why the bug hid behind a working membership test. pg_fts is exposed through sm_next_member, used to iterate tombstones, so on a big-endian host that iteration could silently go wrong. Upstream measured a map with bits 42 and 1024 set reporting cardinality 1, minimum 768, maximum 1792 on sparcv9.

The three headline 5.5.0 fixes (sm_difference RLE data loss, sm_offset structurally invalid maps, sm_split ENOSPC) are in functions pg_fts does not call – included, but not our exposure.

Upgrade notes

ALTER EXTENSION pg_fts UPDATE TO '1.6.0' is the whole upgrade. No REINDEX, no data migration.

If an application relied on the old lossy phrase behaviour as a cheap conjunction, write the conjunction explicitly ('quick & brown'). If you issue phrase or NEAR queries at scale, note separately that WITH (positions = on) is what makes them fast – 36x on a 2.19M-document corpus (see 1.5.10’s notes and bench/NOTE_PHRASE_PROFILE_2026-09-06.md).

Validation

All three phrase evaluation paths were checked for agreement on a 3,000-row corpus where only a third of rows have the phrase adjacent: sequential scan (heap matcher) 1000, index scan with positions=off 1000, index scan with positions=on 1000, regex ground truth 1000. installcheck (PG 17/18), full TAP set, alloc/ascii guards, block fuzzer all pass; upstream sparsemap’s own suite passes 10/10 including its property tests.

1.5.10

Performance release: common-term ranked top-k is 1.56x faster. C-only, no SQL objects changed, no on-disk index format change (BM25_VERSION stays 4), no REINDEX. Both wins came from profiling the scan with perf on real hardware, and both are ordinary read-path fixes – no format, ordering or exactness change.

query	1.5.9	1.5.10	change
common k10 (`year`, df 734,896)	56.30 ms	36.16 ms	1.56x
common k100	69.78 ms	46.01 ms	1.52x
rare k10 / mid k10 / 2-term OR	5.85 / 10.69 / 4.32	5.89 / 10.64 / 4.12	flat

1. Ascending-resume hint in the doclen lookup

bm25_doclen_cursor_lookup() ran an unconditional binary search over the resident 128-entry doclen block – ~7 branchy iterations per posting, on a term whose docids are consecutive. perf annotate put ~45% of a common-term query in that search (the BM25 math itself was 1.68%).

The WAND scan probes docids in strictly ascending order, so the next answer is almost always the next entry: try a short linear walk from a resume hint, fall back to the binary search on a miss, and reset the hint whenever a new block is decoded. 56.30 -> 42.67 ms.

2. `bm25_for_get()` was still decoding bit-by-bit

bm25_for_unpack() (batch) had long since been optimized to a word-load/shift/mask extraction – its comment even says it “replaces the per-bit inner loop that dominated posting decode” – but the random-access twin bm25_for_get() never got that treatment and still ran one bit-test per bit of width, per call. It is on the hot path twice: wand_contrib_cur() reads tf through it for every scored posting, and the v3 inline-doclen path reads |D| through it too.

Gave it the same extraction, including the shift == 0 undefined-behaviour guard the batch version documents for a corrupt on-disk width byte. 42.67 -> 36.16 ms. The bm25_for_get(buf, i) == bm25_for_unpack(buf)[i] equivalence is already asserted by both test/fuzz/fuzz_for.c and test/hegel/test_for.c.

Rejected by measurement (recorded so they are not retried)

Partial/lazy block decode for rare-term latency. Gains ~1%. The motivating “71x decode amplification” was an arithmetic artifact of ours: the doclen sidecar is keyed by all docids, so a rare term’s target sits at an arbitrary offset in its block – instrumented at 60-82 entries decoded per block, which is the unavoidable gap-decode prefix (delta-encoded docids have no random access), not waste. Forcing a small decode window made it worse (rare 5.83 -> 7.74 ms) via tens of thousands of geometric re-walks.
Impact-ordered postings / a precomputed block-max score / finer block granularity / early termination – all previously disproven or rejected; see bench/NOTE_WAND_PRUNING_2026-09-04.md.

After this release rare and mid are at their floor: what remains in bm25_doclen_cursor_load_page() is inherent gap-decoding. Moving them further would need a format change (periodic absolute docids within a block) worth at most ~2x of a portion of the query.

Validation

Ranked top-k parity PASS on all 10 cases (single-term, AND, OR at k=10 and k=100) against an exact fts_bm25 sort, on 2.19M Wikipedia articles at 1421 MB. installcheck (PG 17/18), full TAP set, alloc/ascii guards and the block fuzzer (== ALL CLEAN ==) all pass. Measured on EC2 r6id.4xlarge, medians of runs 4-8.

1.5.9

Correctness fix for non-UTF-8 databases, plus two ranked-latency optimisations. C-only, no SQL objects, no on-disk index format change (BM25_VERSION stays 4).

1. Non-ASCII case folding was broken on non-UTF-8 server encodings

Who is affected: databases whose server encoding is NOT UTF-8 (LATIN1, WIN1252, etc) and whose locale is not C. UTF-8 databases were never affected.

fold_token() case-folded ASCII but passed every byte >= 0x80 through unchanged, so upper- and lower-case accented letters were different terms and case-insensitive search silently failed for all non-ASCII text. Measured on LATIN1 + de_DE.iso88591: a document Apfel (A-umlaut, 0xC4) did not match a query apfel (a-umlaut, 0xE4), while PostgreSQL’s own to_tsvector() did. We were diverging from PostgreSQL text search on exactly the deployments most likely to be non-UTF-8.

The non-UTF-8 path now delegates to str_tolower() – the same locale/collation-aware primitive tsearch’s lowerstr() uses – so pg_fts produces byte-identical terms to to_tsvector() on those servers.

Behaviour under locale C is deliberately unchanged: the C library has no case mapping for high bytes there. (PostgreSQL appears to “match” under C only because its parser discards the accented character; pg_fts keeps it, which loses less information.) The contract is now explicit: non-ASCII folding on a non-UTF-8 server follows the database locale, exactly as PostgreSQL does.

Upgrade action for affected databases: terms already stored by an older version are unfolded and will not match the newly-folded query form. Re-derive them – UPDATE t SET d = to_ftsdoc('<cfg>', body) for a stored ftsdoc column, or REINDEX INDEX <name> for an expression index. Until then those rows keep the old (case-sensitive for non-ASCII) behaviour. Nothing to do on UTF-8.

Also hardened: the UTF-8 folding loop called utf8_to_unicode() without checking that the character’s bytes fit within the token. Not reachable today (the tokenizer never splits a well-formed UTF-8 character and pg_fts imposes no term length cap), but the bound is now explicit.

New regression coverage in t/004_encodings.pl asserts an upper-case accented document matches a lower-case query on LATIN1 with an ISO-8859-1 locale; it skips cleanly where no such locale exists. The pre-existing LATIN1 probes only tested exact-case round-trips, which is why this bug survived.

2. Rare-term ranked latency: 1.7x faster (block-granular doclen decode)

The page-directory cursor decoded a whole sidecar page (~31 blocks, ~4,000 doc entries) to answer one doclen lookup. A rare term scattered across the docid space touches nearly every sidecar page, so scoring 10,875 postings cost ~2.2M doc-decodes – a ~200x amplification, measured as ~7.9 ms of a 10.2 ms rare-term ranked query. Now only the covering 128-doc block is decoded: the page’s block headers are walked (no FOR-unpack) to locate it, then that one block is unpacked.

rare k10 10.2 -> 6.05 ms. Mid is ~flat (a denser term’s consecutive postings already shared a block), common 60.9 -> 56.5 ms.

3. Multi-term ranked latency: 1.44-1.63x faster (shared resident block)

Each (term, segment) has its own doclen cursor, but the scoring loop asks every cursor sitting at the pivot docid for its contribution – i.e. N cursors look up the SAME docid, and each decoded the same sidecar block independently. The resident decoded block is now hoisted into a per-segment slot shared by all of that scan’s cursors, so the 2nd..Nth lookup of a docid is a pure in-memory binary search.

query	before	after	change
1-term	6.02 / 10.83 ms	6.15 / 11.06 ms	flat (one cursor)
2-term OR	6.17 ms	4.27 ms	1.44x
3-term OR	10.95 ms	6.71 ms	1.63x

(The originally-planned “LRU of decoded blocks” was dropped: the WAND visits a cursor’s docids monotonically ascending, so a block is never revisited by the same cursor and an LRU would have nothing to hit. The duplication is across cursors, which is what this fixes.)

Rejected by measurement (recorded so it is not retried)

A df-threshold “bulk-load the whole sidecar for small-df terms” fast path was planned and then disproven: measured ranked cost is ~linear in df with no fixed floor (df 2,560 -> 2.35 ms, already the plain @@@ count floor), while a bulk load reads ~547 sidecar pages regardless of df. There is no crossover; the page directory already wins at every df. Details in bench/NOTE_RARE_MID_LATENCY_OPTIONS_2026-09-05.md.

Validation

Ranked top-k parity PASS on all five query shapes (single-term, AND, OR, at k=10 and k=100) against an exact fts_bm25 sort, on 2.19M Wikipedia articles. Encoding fix verified against real LATIN1/WIN1252/UTF-8 clusters and against to_tsvector on identical bytes. installcheck (PG 17/18), full TAP set (incl. t/004_encodings running for real with an ISO-8859-1 locale, and t/006_concurrent_extend), alloc/ascii guards and the block-fuzzer all pass.

1.5.8

Performance + robustness release. C-only, no SQL change, no on-disk index format change (BM25_VERSION stays 4), no REINDEX.

1. Doclen-sidecar ranked latency: 2.5x faster rare/mid terms. The doclen_sidecar=on default (the ~4.7x smaller index) had a fixed per-scan tax: 1.5.4-1.5.7 decoded the ENTIRE segment sidecar once per ranked scan into a scan-local array (~18 ms on a 2.19M-doc segment: ~534 page reads plus a FOR-unpack of every block), which dwarfed the actual scoring for anything but a very common term. A fresh 5-way benchmark on real Wikipedia caught it (rare ranked 25.6 ms vs 1.6 ms with inline doclen).

Replaced with a page-directory cursor: one tiny (first_docid, blk) entry per sidecar PAGE, built by walking only page HEADERS (no block decode) and cached in the index relcache (rd_amcache) as ONE contiguous chunk keyed by the metapage generation – so it is built at most once per backend, not per scan. A lookup binary-searches the directory to the covering page, decodes ONLY that page, and keeps it resident (the WAND scan visits docids ascending). Single-chunk is deliberate: it satisfies the rd_amcache “pfree()d wholesale on relcache invalidation” contract that 1.5.4’s multi-chunk 20 MB decoded array violated (the 1.5.5 crash). The whole-segment bulk decode is retained for the MERGE path, which legitimately reads every doc sequentially.

Measured on 2.19M Wikipedia articles (median, warm, doclen_sidecar=on): rare 25.6 -> ~10 ms, mid 29.3 -> ~11 ms. Ranked top-k parity PASSES on all bands (k=10 and k=100, single-term and AND), and a 90 s concurrent fts_merge + fts_vacuum + 6-reader soak kept the ranked top-10 stable with zero mismatches and zero errors.

Known remaining gap (documented, not a regression): a very common term (735k df, 34% of the corpus) is still ~70 ms, and that is NOT the doclen path – a plain @@@ count of the same term is 2.4 ms while the ranked top-10 reads 8,115 buffers, i.e. the block-max WAND is not pruning it. That is a separate lever.

2. fts_search() SRF could return fewer than k rows. The top-k engine over-fetches for MVCC (wantk = k*4); the amgettuple ordering scan retries and grows on its own, but the fts_search SRF called the engine once, so a heavy-delete workload where most of the top candidates are invisible could yield nvis < k. bm25_topk_visible now grows wantk and re-generates when the visibility loop ends short AND more candidates existed, with a bounded growth cap (and keeps the directory-generation retry inside each attempt).

3. Sparsemap error-path leaks. sm_create() maps are libc-malloc (no palloc allocator is installed), so an ereport(ERROR) between create and sm_free leaked past transaction abort. bm25_bulkdelete and bm25_segment_docids now free them via PG_TRY/PG_FINALLY. (The cleanup pointers are resynced before each throw because sm_add_many_grow updates *map even on a partial grow-then-fail – freeing the pre-call pointer would be a double free.) bm25_read_blob buffers are palloc’d and are left alone.

4. Reserved keywords are literal words inside a phrase or NEAR. The query lexer recognized and/or/not/near as operators unconditionally, so "the and clause" or NEAR(near y, 2) failed to parse. Keyword tokens now carry their folded text and the phrase/NEAR operand loops accept them as terms, matching to_tsquery (which lexes them as lexemes). The ambiguous BARE top-level form (and & x) is deliberately unchanged.

Also: a genuine bug in the vendored sparsemap (an __sm_insert_data offset/length convention mismatch – a latent buffer over-write currently masked by compensating capacity slack) was found during this work and reported upstream; it is not triggered by pg_fts today.

1.5.7

Concurrency crash-fix release: three pre-existing races surfaced by a read+insert+merge+vacuum soak, all present since at least 1.5.3. C-only, no SQL change, no on-disk format change (BM25_VERSION stays 4), no REINDEX.

Who is affected: any index under concurrent write + maintenance load (ingestion plus fts_merge/fts_vacuum/autovacuum). 1.5.3 crashed into recovery under such a soak; 1.5.7 stays up and error-free.

1. Merge crash (severe). Every operation that mutates the segment directory or frees + recycles pages – bm25_flush_pending, the tiered bm25_merge_segments, bm25_vacuum_compact, bulkdelete’s livedocs swap – ran with no mutual exclusion: an INSERT’s tiered merge holds only RowExclusiveLock, a user fts_merge/fts_vacuum holds ShareUpdateExclusiveLock/AccessExclusiveLock on the INDEX, and autovacuum cleanup holds ShareUpdateExclusiveLock on the TABLE – lock tags that do NOT conflict. Two of these running at once let one free + recycle a segment’s pages while the other’s streaming merge was still reading them -> SIGSEGV in merge_source_load_page. Fix: a per-index maintenance serialization lock (a heavyweight page lock on the metapage block, the same mechanism GIN uses to serialize pending-list cleanup) – blocking for explicit/ required maintenance, conditional (skip if busy) for the opportunistic insert-time tiered merge.

2. Index-relkind assert / API misuse. The freed-page recycle gate called GlobalVisCheckRemovableXid(index, xid) with the INDEX relation; that routine expects a table (or NULL) and tripped an assertion under –enable-cassert (and is latent API misuse otherwise). Fixed to pass NULL (the global visibility horizon), a sound and slightly conservative bound.

3. Scan read past a concurrent truncation. fts_vacuum (and autovacuum compaction) truncate the freed tail of the index file back to the OS. A scan follows the segment directory it snapshotted (dict/posting chain heads by block number), re-checking the metapage generation afterward and retrying if it moved – but a block number from the pre-truncation snapshot points past EOF, and ReadBuffer raised a hard “could not read blocks N: read only 0 of 8192” ERROR before the generation re-check could discard the stale result (a transient query error under heavy read+vacuum churn). Fix: the scan’s chain-following reads now treat an out-of-range block as end-of-chain (a truncated block can only mean a concurrent vacuum bumped the generation), so the existing generation guard restarts the scan from a fresh directory and the count/ranked result stays exact.

Validation (assert build, on the exact concurrent workload): t/006_concurrent_extend 45 consecutive runs with zero crashes (1.5.3 crashed ~13/15); a read+insert+merge+fts_merge+fts_vacuum+VACUUM soak across many runs with zero “could not read blocks” errors, zero crashes, and the concurrently-churned corpus’s fixed-term count staying exact throughout. The nix tap-* check now runs the full t/003-t/008 set (was only t/005) so this class of crash is caught by nix flake check locally.

1.5.6

Crash-fix release. C-only, no SQL change, no on-disk format change (BM25_VERSION stays 4), no REINDEX.

Who is affected: anyone on 1.5.4 or 1.5.5 with the default doclen_sidecar=on. Upgrade to 1.5.6. (1.5.3 and earlier lack the doclen cursor and are unaffected.)

The bug: the doclen-sidecar cursor added in 1.5.4 has an owned flag that tells bm25_doclen_cursor_free whether the cursor allocated its own decoded arrays (and must free them) or merely borrowed the scan cache’s (and must not). bm25_doclen_cursor_init set owned only on the rarely-taken self-decode path and left it UNINITIALIZED on the common borrow and v3-segment paths. The cursor lives in a palloc’d (not zeroed) WandCursor, so owned held stale heap bytes: when they were non-zero, cursor teardown pfree()d a pointer it did not own – a borrowed interior/shared pointer – corrupting the allocator (ERROR: could not find block containing chunk ... on a following multi-term fts_search) or segfaulting a concurrent backend. Intermittent: it fired only when the reused chunk’s bytes happened to be non-zero, which is why fresh backends often looked fine but the CI regression + concurrent-extend suites tripped it.

The fix: initialize owned = false at the top of bm25_doclen_cursor_init so every path is defined; only the genuine self-decode path sets it true. Hardening in the same release: the scan-time sidecar decode (bm25_doclens_load) now bounds its page walk to the relation’s block count and stops at any page that is no longer a BM25_DOCLEN page, so a chain broken by a concurrent merge/vacuum recycling its pages (the A1 race, already retried via the metapage-generation guard) cannot spin or read unrelated pages before the retry.

Validation: clean under AddressSanitizer over the exact multi-term fts_search sequence that regressed (200-iteration loop, plus a garbage-owned poison test); installcheck (PG 17/18), the full TAP set (t/003-008, including the concurrent-extend crasher), alloc/ascii guards, and the block-fuzzer all pass. The nix tap-* check now runs the full t/003-008 set (previously only t/005) so this class of crash is caught by nix flake check locally, not only in downstream CI.

1.5.5

Concurrency crash-fix release. C-only, no SQL change, no on-disk format change (BM25_VERSION stays 4), no REINDEX.

Who is affected: anyone running 1.5.4 with the default doclen_sidecar=on under concurrent write + query load. Upgrade to 1.5.5. (1.5.3 and earlier do not have the 1.5.4 doclen cache and are unaffected.)

The bug: 1.5.4 cached the decoded doclen sidecar in the index relcache entry (rd_amcache). rd_amcache must be a single palloc’d chunk because PostgreSQL pfree()s it wholesale on a relcache invalidation (e.g. one raised by a concurrent fts_merge/segment extend via RelationReloadIndexInfo); the 1.5.4 cache was multi-chunk (a header plus per-segment decoded arrays), so the invalidation freed only the header, corrupting the allocator and/or leaving a concurrently-scanning backend’s cursor pointing at freed memory – an intermittent backend crash under the exact ingest+merge+query overlap the concurrent-extend TAP test drives.

The fix: the decoded-sidecar cache is now scan-local – decoded once per scan into the scan’s own memory context and shared across that scan’s per-(term,segment) cursors, freed when the scan ends. This keeps 1.5.4’s read-locality win (a common term’s sidecar is decoded once per scan, then every doclen lookup is an in-RAM binary search – no per-posting sidecar page reads) while being invalidation-safe: nothing is stored in rd_amcache, so a concurrent merge cannot free memory a live cursor borrows. (Cross-query caching was dropped; if cold-scan decode ever dominates at scale, a persistent build-time sidecar directory is the follow-up.) Verified by running the concurrent-extend TAP test 20x with no crash (it reproduced ~1-in-10 on 1.5.4).

1.5.4

Performance + correctness release making the default doclen_sidecar=on (v4) layout both fast and exact on multi-term AND. C-only, no SQL change, no on-disk index format change (BM25_VERSION stays 4), no REINDEX.

1. Doclen-sidecar read locality (performance). Since 1.5.0 the per-doc length lives in a per-segment sidecar instead of inline in each posting. That saves ~35% index size but a ranked scan of a common term (whose postings are scattered across the whole docid space) then read roughly one sidecar buffer per scored posting – effectively the entire sidecar chain per query. Measured on a 2.19M-doc corpus: a common term touched 16,887 buffers with the sidecar vs 1,432 inline, and warm ranked latency was ~2x inline and far behind competitors. 1.5.4 decodes a segment’s whole sidecar once per backend into a sorted (docid, byte) array cached in the index relcache entry (rebuilt only when the metapage generation moves), then answers every doclen lookup with an in-RAM binary search – 0 buffer reads after the first build. The common term now touches ~2,000 buffers (level with inline) and mid-frequency ranked queries are ~5 ms. (An ultra-common term – e.g. one in a third of all docs – remains dominated by the posting scan itself, the same cost inline pays.)

2. Block-max WAND multi-term AND recall (correctness). A pre-existing soundness gap in the block-max block-skip: the fast path skipped a whole posting block whenever a single term’s cursor sat at or before the WAND pivot – but a cursor being past the pivot does not mean it is past the block, so a later document in the skipped block that contained BOTH query terms (and thus scored the SUM of their contributions, which the one-term block bound never covered) could be dropped from the top-k. On a 2.19M Wikipedia corpus the ranked top-10 for slovakia & hungary missed the true #1 and admitted lower-scoring docs. The fix only takes the whole-block-skip fast path when no other term’s cursor falls within the skipped block’s docid range; otherwise it advances past the pivot exactly. This was present on BOTH the sidecar and inline layouts (it is in the shared WAND traversal), and single-term ranked was always exact.

3. Sidecar block-max bound vs quantized doclen (correctness). The v4 sidecar stores a quantized length byte (SmallFloat, rounded DOWN), so scoring can see a slightly shorter |D| – and thus a slightly higher score – than the exact minimum |D| the block header records. The block-max WAND bound now dequantizes the block’s min |D| through the same codec, so the bound stays a true upper bound for the quantized scores and cannot wrongly prune a top-k document.

Validation: ranked top-k now matches the exact fts_bm25 top-k for AND queries on both layouts (new regression test and_sidecar_topk_exact / and_inline_topk_exact), verified at 2.19M docs on the exact slovakia & hungary case that first exhibited the gap; a concurrent soak (readers + writer + merge/vacuum loop) on a sidecar-on index stayed correct and bounded; installcheck (PG 17/18), TAP, alloc/ascii guards, and the block-fuzzer all pass.

1.5.3

Correctness + performance bug-fix release: the segment MERGE path produced a broken v4 doclen sidecar (empty), so a merged v4 segment’s ranked scores were wrong and its ranked scan was pathologically slow. C-only, no SQL change.

Who is affected: any index built with the default doclen_sidecar=on (v4) under 1.5.0-1.5.2 that has undergone a segment merge or fts_vacuum/fts_merge compaction – i.e. essentially every non-trivial actively-used v4 index. A freshly-built, never-merged single-segment v4 index was correct; the corruption was introduced by merge. Indexes built with doclen_sidecar=off (inline) were never affected.

The bug: the streaming merge wrote merged postings in the 2-column (no inline doclen) layout but never populated the merged segment’s doclen collector, so bm25_write_doclen_sidecar returned InvalidBlockNumber for the merged segment. A segment with 2-column postings but doclenstart = Invalid is then read as if doclen were inline – so scoring read a garbage “doclen” from past the tf column. Wrong doclen corrupts the BM25 length-normalization (wrong ranking) and defeats block-max WAND pruning (the ranked scan scores far more of the posting list than the top-k needs – the multi-second Index Searches: 0 scans reported from the field).

The fix: the merge now feeds each surviving posting’s (docid, doclen) into the merged segment’s doclen collector, so the merged segment gets a correct, populated sidecar and a valid doclenstart. A regression test builds several segments, forces a merge, and asserts the merged sidecar index returns the same ranked top-k as an inline-built twin.

Action for operators on 1.5.0-1.5.2 with a default (v4) index: REINDEX, or rebuild under doclen_sidecar=off, to correct any already-merged segment’s doclen. New merges under 1.5.3 are correct. (If you were already on doclen_sidecar=off per the 1.5.2 note, you are unaffected and need do nothing.)

1.5.2

Bug-fix release: the native-v4 ranked-scan slowdown on large, many-segment indexes (a field-reported 1.5.x adoption blocker), plus an escape hatch. C-only, no REINDEX (ALTER EXTENSION pg_fts UPDATE TO '1.5.2').

Doclen sidecar lookup is now random-access, not a forward walk. The v4 scoring path reads each scored doc’s quantized length from the per-segment sidecar. 1.5.1 read it through a forward cursor, which is O(1) amortized only when scored docids are dense; on an index with a long merge/delete history the surviving docids are sparse, so a term whose postings sit at high/scattered docids forced the cursor to decode the whole sidecar up to that docid – a ranked top-k that touched tens of thousands of buffers (even for a rare term). The cursor now builds a small per-segment page directory (first-docid per sidecar page, one buffer/page, no block decode) once and BINARY-SEARCHES it to jump straight to the covering page, so a lookup is O(log pages) and total sidecar reads are bounded by the pages actually covering scored docids.
New WITH (doclen_sidecar = on|off) reloption (escape hatch). Default on (the v4 quantized sidecar). off stores doclen inline in each posting (the pre-1.5 layout) – the same ranked-scan behavior as 1.4.x – for a workload that prefers it while keeping the 1.5.x crash fixes. Both layouts are read by the same self-describing decoder, so an index can mix sidecar and inline segments and the option can be changed without REINDEX (new segments follow the current setting).
Note. Block-max WAND still does not early-terminate a very-high-df term whose per-block score bounds cluster near the top-k threshold; that is pre-existing (1.4.x) and unchanged. This release fixes the v4-SPECIFIC cost (the sidecar walk) that made a many-segment v4 index slower than the same query on v3.

1.5.1

Bug-fix release: three field-reported regressions in the 1.5.0 v3->v4 doclen sidecar upgrade (one caused a production search outage). C-only, no SQL change, no REINDEX (ALTER EXTENSION pg_fts UPDATE TO '1.5.1'). A 1.5.0 index – whether an upgraded-in-place v3 or a native v4 – is read correctly by 1.5.1 with no REINDEX.

Fixed corrupt dual-read of a pre-existing v3 index (the outage). 1.5.0 added doclenstart INSIDE BM25SegMeta, which grew that struct (48->56 bytes). Because segment descriptors are stored inline in the metapage’s segs[] array, a v3 metapage laid them out at the old 48-byte stride; the 1.5.0 reader cast the page straight to the larger v4 struct and read every segs[1..] field (and generation) from the wrong offset -> a garbage livedocslen became palloc(4294967295) and a garbage dictstart became an out-of-range block seek. The metapage read is now VERSION-AWARE: a v3 metapage is deserialized at the v3 stride into the in-memory v4 struct (doclenstart = Invalid, i.e. inline doclen), and a v3 metapage is upcast to v4 in place on the first metapage mutation. A compile-time assert now pins the v3/v4 head-layout contract so a future field insertion cannot silently reintroduce this.
Fixed the v4 ranked-scan slowdown on many-segment indexes. The per-segment doclen sidecar was BULK-LOADED (whole segment) at scan start, so a ranked/@@@ scan was O(segment docs) per query regardless of matches – on an index with a long merge history (many segments) this read tens of thousands of buffers for a small top-k. Scoring now reads the sidecar through a FORWARD CURSOR that touches only the pages covering the docids actually scored (bounded by the WAND’s ascending docid walk), matching the posting scan’s own block-skip.
Note on high-df ranked latency. Block-max WAND does not early-terminate a high-frequency term whose per-block score bounds cluster near the top-k threshold; this is a pre-existing property of the docid-ordered index (present in 1.4.x, not introduced by v4 – measured v4 is faster than v3 on the same high-df term). It is unchanged here; a future impact-ordered format is the only lever and is not in this release.

1.5.0

Storage + performance release: per-document length moves out of the posting lists into a per-segment quantized sidecar (on-disk format v3 -> v4). C-only, no REINDEX (ALTER EXTENSION pg_fts UPDATE TO '1.5.0').

Doclen sidecar (the size + common-term-latency win). BM25 needs each document’s length for its length-normalization, but pg_fts stored it once per posting (once per doc x term) – the widest posting column. 1.5.0 stores it once per document as a single quantized byte (a Lucene/Tantivy-style fieldnorm: 5-bit exponent + 3-bit mantissa) on a per-segment sidecar page chain, and scoring reads that byte instead of decoding a per-posting column. Measured on 2M high-vocabulary docs:
- index 37.5% smaller (954 MB -> 596 MB);
- common-term ranked top-10 7.7x faster (17.5 ms -> 2.3 ms), top-100 5.6x faster (18.4 ms -> 3.3 ms), because each posting block is ~56% smaller and length-normalization is now a byte lookup;
- rare/mid ranked, boolean AND, phrase, prefix, and count(*) unchanged. avgdl stays EXACT (from the per-segment sumdoclen/ndocs); only the per-doc normalization denominator is quantized, which changes BM25 score ORDERING by at most a fraction of a percent at tie boundaries (the accepted fieldnorm tradeoff; verified <= 0.2% on the 2M rig).
No REINDEX; dual-read + lazy migration. A v4 build reads existing v3 segments (inline doclen) and new v4 segments (sidecar) in the same index; the block’s column count is self-describing from its byte length, so decode is correct across mixed-version segments. Segments migrate to v4 as merge/vacuum rewrites them – an existing index keeps working untouched and converges to the smaller format over time with no operator action.

1.4.1

Bug-fix release: two field-reported robustness fixes on high-vocabulary corpora under heavy churn. C-only, no index format change, no REINDEX (ALTER EXTENSION pg_fts UPDATE TO '1.4.1').

Ranked scan no longer degrades under tombstone bloat. A ranked (ORDER BY d <=> query) scan checks each candidate docid against the segment’s deleted-doc (tombstone) map. That check used an 8-way MRU chunk cache which degenerates to an O(chunks) head-walk per lookup once an ascending scan runs past its eight cached chunks – so a segment carrying millions of tombstones (e.g. after a full-table UPDATE + VACUUM) turned a common-term top-k from ~30 ms into tens of seconds (measured 24 s for a 735k-df term at 2.2M docs; ~99.9% of the time was in the sparsemap walk). The tombstone check now uses a forward-resume cursor (sm_cursor_t) that resumes the walk from the last located chunk, restoring O(postings + chunks). Validated at 2M docs with ~4M tombstones: 24 s -> 2.5 ms.
Fuzzy/regex/NOT candidate scan is memory-bounded. When no trigram acceleration is available (trigrams = off, the default, or a pattern too short to yield trigrams), fuzzy/regex/NOT queries fall back to a whole-segment candidate scan. That fallback accumulated every posting (Sum of df across the dictionary – the whole expanded inverted index) before de-duplicating, so on a high-vocabulary corpus it could attempt a multi-gigabyte allocation and fail with “invalid memory alloc request size”. It now folds duplicates as it collects, keeping peak memory O(ndocs) regardless of Sum(df). Results are unchanged (the fallback still returns the exact set).

1.4.0

Feature release: field-targeted (weight-zone) search. No index format change, no REINDEX (ALTER EXTENSION pg_fts UPDATE TO '1.4.0').

Field zones via tsvector-style weight labels A/B/C/D. Tag a sub-document with a weight and concatenate labelled parts so a query term can restrict itself to a field:
```
CREATE INDEX ... USING fts ((
  to_ftsdoc('english', subject, 'A') ||
  to_ftsdoc('english', body,    'C'))) WITH (positions = on);
... WHERE d @@@ to_ftsquery('english', 'vacuum:A & tgl:B')
```
New: to_ftsdoc(regconfig, text, "char"), setftsweight(ftsdoc, "char"), the ftsdoc || ftsdoc concatenation operator, and the query syntax term:A / term:AB (a term restricted to one or more zones), mirroring standard to_tsquery('english','term:A'). to_ftsdoc(tsvector) now also carries the tsvector’s own A/B/C/D weights.
Upgrade is a no-op for existing indexes. Weight labels live only in the ftsdoc VALUE (the top 2 bits of each token position); the on-disk index posting format is unchanged. A field-restricted query is answered via the heap recheck (like fuzzy/regex), so existing indexes keep working and queries without a :label behave exactly as before. To get field provenance, rebuild a table’s ftsdoc from labelled to_ftsdoc(...,weight) || ... documents (opt-in per table – NOT a global reindex). An unlabelled document reads as label D, so term:D matches it and term:A does not.
Field restriction requires the ftsdoc to carry positions (the labels ride on positions); build the index WITH (positions = on) for field-restricted ranked/count queries. BM25 scoring stays document-level – a zone filter changes which documents match, not how a matching document scores.
Limitation: weight labels apply to plain terms; term:A* / term:A~k (weight
- prefix/fuzzy/regex) are rejected as a syntax error.

1.3.2

Performance bug-fix release. No on-disk format change from 1.3.1; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.3.2').

Fixed a ~5x common-term ranked-latency regression introduced in 1.3.0. The 1.3.0 ranked-exactness hardening replaced the block-max WAND scan’s O(1) per-block skip with a per-posting re-pivot on every prune, which on a common term turned each block-max prune into ~128 re-pivots (measured on 2.19M Wikipedia, PostgreSQL 18: year top-10 rose 31.8ms -> 163ms; the slowdown scaled with document frequency – mid-frequency terms ~2.6x, rare terms unaffected). The exactness bug that hardening guarded against is latent (not reachable in practice – segments own contiguous, non-overlapping docid ranges), so the trade was a real regression for defense against an unreachable bug. The fix restores the O(1) block-skip in the common single-segment / few-term case while keeping the safe per-posting seek for the densely- interleaved multi-segment case, so ranked exactness is fully preserved. After the fix, common-term ranked latency is competitive with the fastest block-max-WAND BM25 extensions again (year top-10 ~28ms, top-100 ~29ms), and rare/mid terms lead. Anyone on 1.3.0 or 1.3.1 doing ranked (<=>) searches on common terms should upgrade.

1.3.1

Correctness bug-fix release. No on-disk format change from 1.3.0; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.3.1').

Stopwords are now dropped from the query, not just the document. to_ftsdoc(regconfig, text) removes a configuration’s stopwords (e.g. the, a, of), but to_ftsquery(regconfig, text) previously kept them verbatim, so a stopword query term was unsatisfiable and silently zeroed a boolean AND (the & postgres matched nothing; the vacuum problem under implicit-AND returned far fewer results than vacuum problem). to_ftsquery now runs each term through the same dictionary pipeline and elides stopword terms from the query tree – matching standard to_tsquery: to_ftsquery('english','the & postgres') reduces to postgres, and an all-stopword query becomes empty (matches nothing). Prefix/fuzzy/regex terms are matched literally and are never stopword-dropped. Reported against a 2.8M-document English email-body index in production.
Known limitation (tracked, not fixed here): and/or/not/near are reserved query operators and cannot be searched for as literal words; on natural-language corpora they are stopwords and dropped anyway.

1.3.0

Feature + hardening release. No on-disk format change from 1.2.2; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.3.0').

WITH (trigrams = on|off) reloption (default OFF). The per-segment trigram tier accelerates only regex and long fuzzy queries; the query side already falls back to a full dictionary scan when it is absent, so the default now omits it – a smaller index (~18% in a 2.19M-doc measurement) at no correctness cost. Build WITH (trigrams = on) for regex- or long-fuzzy-heavy workloads. Results are identical either way.
Faster count(*). A single plain term over a tombstone-free, pending- free, fully-all-visible index is now counted straight from the dictionary document frequency – no posting decode, no heap probe (measured: a common term at 2.19M docs, ~756 ms -> ~2 ms). Set-membership decodes skip the tf/ doclen columns they never use. The count(*) index pushdown now also fires for a plain-column fts index (the recommended stored-ftsdoc-column form), not only an expression index – previously a stored-column count(*) fell back to a bitmap heap scan.
Managed-service hardening. fts_merge() and fts_vacuum() now refuse to run during recovery (a hot standby is read-only) and require the caller to own the target index (they open it by OID and take heavy locks). Two functions that emit indexed content by index OID – fts_search() and fts_anomalous_docs() – are revoked from PUBLIC (the index owner and superusers keep access; an owner may grant explicitly). Corpus statistics (BM25 IDF + length normalization) now exclude recently-dead tuples surfaced to the build with tupleIsAlive = false – such tuples are still indexed (an old snapshot may need them) but no longer inflate the document count / total length.
Ranked-exactness hardening (internal). The block-max WAND pivot skip now advances every cursor at or before the pivot rather than skipping one cursor’s whole block, removing a provably-unsound (though not field-reachable) over- skip. No behavior change on pg_fts’s contiguous-docid-range segments.

1.2.2

Bug-fix release. No on-disk format change from 1.2.1; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.2.2'). Two fts_vacuum fixes.

Fixed fts_vacuum() growing the index instead of shrinking it. The 1.2.1 deletion-XID recycle gate (which protects a concurrent scan from reading a just-freed page) also blocked fts_vacuum()’s compaction from repacking into the low pages it had itself just freed, so the vacate+pack phase extended the relation and fts_vacuum() GREW the index on every call and never truncated a tail. Compaction now reuses freed pages again and fts_vacuum() compacts to a stable floor (measured: a churned index 143 MB -> 60 MB, idempotent). If you ran fts_vacuum() on 1.2.1 and it did not shrink, re-run it on 1.2.2.
Fixed a rare crash from fts_vacuum() (or autovacuum) concurrent with reads. fts_vacuum() ran under a lock that does not block scans, so a reader could still be copying a segment’s pages while compaction recycled and overwrote them, corrupting the read (a rare SIGSEGV under heavy simultaneous read + insert + merge + vacuum). Page recycling during compaction is now bypassed only under an exclusive lock, and fts_vacuum() takes AccessExclusiveLock on the index (like REINDEX) so its in-place shrink is safe; autovacuum’s cleanup keeps the gate and reclaims space across passes without blocking or corrupting concurrent scans. (Root-caused with AddressSanitizer; a concurrent read+vacuum regression test now gates CI.)

1.2.1

Bug-fix release. No on-disk format change from 1.2.0; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.2.1'). Fixes three production failures on a continuously-written index and hardens the scan-vs-merge path.

Fixed the 128-segment cap becoming an unrecoverable outage. On an index whose rows mostly exceed one page, every insert creates a segment; the live count could reach the internal 128-segment maximum and then reject all further writes – and neither VACUUM nor fts_merge() could recover (merging needs a free slot to flush pending into, a chicken-and-egg deadlock; only REINDEX escaped). Adding a segment now merges to free a slot and retries instead of erroring, so a write is never refused because compaction fell behind. In addition, the index now compacts continuously on the write path (leveled LSM merge after each segment-creating flush), keeping the segment count bounded (O(log N) tiers) automatically – no periodic VACUUM/fts_merge() needed to stay healthy under continuous ingestion.
Fixed ERROR: unexpected data beyond EOF when fts_merge()/VACUUM ran concurrently with ingestion. Relation extension was only locked during a parallel build, so two ordinary backends extending the index at once (an insert flush and a merge/vacuum) could race. The extension is now always locked around the single page add, as heap and the core index AMs do; concurrent writers still proceed in parallel. fts_merge() is now safe to run while writes continue.
Fixed a crash / wrong count from count(*) on an @@@ query under plan caching or concurrency. The count-pushdown plan stored the query as a bare internal pointer, which dangled once its planning memory was freed (e.g. a count(*) in a PL/pgSQL loop, or concurrent re-execution), corrupting the query and crashing the backend. The query is now deep-copied into the plan.
Hardened concurrent scan vs. merge/vacuum with a deletion-XID page-recycle gate (a freed page is not reused until no in-progress scan could still reference it) plus bounds checks on all page-derived lengths, so a scan that races page recycling degrades to a retry rather than a crash. Format- preserving (no REINDEX).

1.2.0

Minor release, re-numbered from the 1.1.6 and 1.1.7 patch releases. It contains exactly the 1.1.6 + 1.1.7 changes below.

Reindex recommended. 1.1.6 changed ranked-scan (ORDER BY <=>) results. Because the query behavior over an index changes, we group this as a minor release (not a patch) and recommend rebuilding every fts index after upgrading so ranked results are consistent for anyone who observed the old truncated output. (There is no on-disk format change in 1.1.6/1.1.7/1.2.0 – the read-path fix is correct against an existing index without a rebuild – but the re-numbering exists precisely so this class of behavior change is never shipped as a silent patch again.) If you already upgraded to 1.1.6 or 1.1.7, upgrade to 1.2.0 (ALTER EXTENSION pg_fts UPDATE TO '1.2.0').

Upgrade steps:

ALTER EXTENSION pg_fts UPDATE TO '1.2.0';
REINDEX INDEX CONCURRENTLY your_fts_index;   -- recommended; repeat per fts index

Fixed ORDER BY doc <=> query (ranked) index scans silently returning fewer rows than match. A ranked query retrieved through the KNN/ordered index scan capped at ~4096 rows regardless of how many documents actually matched (e.g. 6057 or 19347 matches both returned ~4096), independent of LIMIT – so ranked search dropped and mis-ordered results. The ordered scan had an internal top-k ceiling meant to bound worst-case latency, but a KNN index scan must return every matching row in score order (the query’s LIMIT is the only bound). The ceiling is removed: the scan now returns the complete match set in order. A small LIMIT (a page of results) is still served cheaply. The plain @@@ match path was always complete and is unaffected.
Build progress logging. A build over a large corpus of long documents (full email bodies, source code) is dominated by per-document text analysis (tokenize + stem). A serial build (max_parallel_maintenance_workers = 0) on a multi-gigabyte corpus can legitimately run for many minutes before the first segment is flushed – the in-memory buffer fills only after a whole budget’s worth of large documents, during which the segment count does not change and nothing is written yet, which is hard to tell apart from a hang. The build now emits a LOG-level line as documents are analyzed and at each segment flush (set log_min_messages = log to see them), so a long build is distinguishable from a stuck one. Verified end to end on a 1.97M-document / 54 GB-of-text high-vocabulary corpus: a serial english-configuration build completes, collapses to a single segment, and is valid and queryable; the wall-clock cost is the inherent analysis cost, which parallel workers reduce proportionally.
Documentation: expanded the large-build guidance with a build-time/throughput section (analysis is the dominant cost and is embarrassingly parallel; how to read the new progress logs).

1.1.7

Bug-fix / observability release. Superseded by 1.2.0 (which relabels 1.1.6+1.1.7 as a minor release and documents the REINDEX requirement). No on-disk format change from 1.1.6.

Build progress logging. A build over a large corpus of long documents (full email bodies, source code) is dominated by per-document text analysis (tokenize + stem). A serial build (max_parallel_maintenance_workers = 0) on a multi-gigabyte corpus can legitimately run for many minutes before the first segment is flushed – the in-memory buffer fills only after a whole budget’s worth of large documents, during which the segment count does not change and nothing is written yet, which is hard to tell apart from a hang. The build now emits a LOG-level line as documents are analyzed and at each segment flush (set log_min_messages = log to see them), so a long build is distinguishable from a stuck one. Verified end to end on a 1.97M-document / 54 GB-of-text high-vocabulary corpus: a serial english-configuration build completes, collapses to a single segment, and is valid and queryable; the wall-clock cost is the inherent analysis cost, which parallel workers reduce proportionally.
Documentation: expanded the large-build guidance with a build-time/throughput section (analysis is the dominant cost and is embarrassingly parallel; how to read the new progress logs).

1.1.6

Bug-fix release. No on-disk format change from 1.1.5; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.1.6').

Fixed ORDER BY doc <=> query (ranked) index scans silently returning fewer rows than match. A ranked query retrieved through the KNN/ordered index scan capped at ~4096 rows regardless of how many documents actually matched (e.g. 6057 or 19347 matches both returned ~4096), independent of LIMIT – so ranked search dropped and mis-ordered results. The ordered scan had an internal top-k ceiling meant to bound worst-case latency, but a KNN index scan must return every matching row in score order (the query’s LIMIT is the only bound). The ceiling is removed: the scan now returns the complete match set in order. A small LIMIT (a page of results) is still served cheaply; only an explicit large/unbounded ranked scan does the deeper work. The plain @@@ match path was always complete and is unaffected.

1.1.5

Bug-fix and usability release. No on-disk format change from 1.1.4; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.1.5').

Fixed a very slow CREATE INDEX CONCURRENTLY finalization on a large, high-vocabulary index. After the merge converged, the CONCURRENTLY validation phase could run for a very long time at 100% CPU without completing. It built a per-segment live-document set one dictionary term at a time, which was quadratic on a segment with millions of low-frequency terms. It now builds that set in one linear pass. On a ~1.9M-document body corpus the validation phase drops from over an hour (not completing) to a few minutes.
New GUC pg_fts.build_mem_ceiling_mb (default 0 = previous behavior): the per-participant flush-budget ceiling, in MB. Raising it lets a large build flush fewer, larger segments so the segment count stays well under the internal limit – useful when you have spare RAM and want less post-scan merging. Peak build memory is about shared_buffers + (max_parallel_maintenance_workers + 1) * pg_fts.build_mem_ceiling_mb.

1.1.4

Bug-fix release. No on-disk format change from 1.1.3; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.1.4').

Fixed segment merges that could fail to converge, or crash, on a large, high-vocabulary index. Two issues in the merge path, both exposed only at scale (validated on a ~1.9M-doc / 40GB+ index):
- A merge discarded its finished output and retried if the segment directory changed at all while it ran (for example a concurrent flush appending a segment). On a big index each merge takes minutes, so it could read a great deal and never commit – the reported “reads hundreds of GB, segment count never drops” non-convergence. A merge now re-locates its inputs by content and commits alongside concurrent flushes.
- A committed merge freed its input pages, which the next merge could recycle for its output while still reading the previous chain – leading to a rare crash (SIGBUS) on very large merges. Merges now write to freshly extended pages during the merge loop and reclaim the freed space afterward. These are memory/scale-path fixes only; the on-disk format is unchanged.

1.1.3

Bug-fix release. No on-disk format change from 1.1.2; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.1.3').

Fixed non-converging merges on a large, high-vocabulary index. On a multi-tens-of-GB index with many similarly-sized segments (e.g. a ~1.9M-doc / ~45GB email-body corpus), fts_merge and the build’s compaction could run for hours without finishing: the segment merge was size-tiered with unbounded fan-in, so it tried to merge essentially all segments into one in a single pass over the whole index. The merge is now leveled (LSM/HanoiDB-style) with bounded fan-in: a segment’s level is derived from its size and no single merge combines more than a bounded number of segments, so compaction proceeds in small, discrete, observable steps (each merge logs merging N of M segments at DEBUG1) with bounded write amplification and always converges. Validated on a 1.9M-doc / 26GB corpus: build to a bounded tiered set in ~17 minutes; fts_merge collapse to a single segment in ~8 minutes; previously it did not complete. The segment level is computed from size, not stored, so the on-disk format is unchanged and no REINDEX is needed.

1.1.2

Bug-fix release. No on-disk format change from 1.1.1; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.1.2').

Fixed a hang in the final phase of a large CREATE INDEX CONCURRENTLY. On a large, high-vocabulary corpus, a build could get all the way through the merge (the 1.1.1 fix) and then wedge in finalization – the build leader parked waiting on parallel workers while another backend blocked on the relation-extension lock, with the index never becoming valid. The build finalization started a second parallel worker set to merge the segments; on a large index under CONCURRENTLY on a busy host, the participants could contend on the relation-extension lock and stall indefinitely. Finalization is now serial (the 1.1.1 O(N) merge made the parallel pass unnecessary for convergence), so no relation-extension contention arises and the build completes. fts_merge() still merges in parallel when run on its own.

1.1.1

Bug-fix release. No on-disk format change from 1.1.0; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.1.1').

Fixed quadratic (O(N²)) time in VACUUM / bulk-delete on a large index. 1.1.0 fixed the same pattern in the build’s trigram writer; three more instances remained on the delete/vacuum path (building a segment’s live-docid set and the tombstone sets one member at a time re-walked the sparsemap from the start on every insert). Harmless on a small index, but on a large index a VACUUM or a large DELETE could spend a very long time in tombstone construction — the same “time explodes at scale” behavior the build had. All corpus-scale sparsemap builds now use the bulk O(N) path. Tombstone results are unchanged (verified against a delete + VACUUM cycle).
Docs: added a build time model and serial/low-parallelism recommendation (the final single-segment collapse is a single-backend pass over the whole index, so budget it in a maintenance window or leave the index tiered), and noted that fts_index_stats() (like fts_index_nsegments()) can be polled on an in-progress index.

1.1.0

Build-convergence and operability release. No on-disk format change from 1.0.8; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.1.0').

Fixed a non-converging index build on large, high-vocabulary corpora. Building an index over many long, high-vocabulary documents (full email bodies, source code, patches – many distinct low-frequency terms per doc) could enter a merge phase that ran for hours with no forward progress and never completed, even though memory stayed bounded. The cause was a quadratic (O(N²)) construction of each trigram’s term-set during the merge; it is now built with a bulk O(N) path. A build that previously did not finish in 8.5 hours completes in minutes.
Large builds now always converge, in bounded steps. Instead of always collapsing to a single segment at the end of a build (a single-backend pass over the whole index), a build whose total size exceeds the new pg_fts.build_collapse_max_mb GUC (default 4096 MB) stops at a bounded, size-tiered set of segments. The index is valid and fully queryable; ranked scans traverse a bounded handful of segments (a small fixed cost). Run fts_merge(index) to collapse to a single optimal segment in a maintenance window. Set the GUC to 0 to always collapse (the historical behavior), or raise it to collapse larger indexes during the build.
Build monitoring. Per-merge progress is logged at DEBUG1 (segments in, terms/docs written, elapsed), and a LOG line reports when a build stops at a tiered set. fts_index_nsegments() now works on an in-progress (indisvalid = f) index, so a build can be polled to watch its segment count fall as merges complete.
Docs: a new “Building indexes on large or high-vocabulary corpora” section covers the build-memory formula (including shared_buffers), the collapse GUC, monitoring, and partitioning.

1.0.8

Bug-fix release. No on-disk format change from 1.0.7; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.0.8').

Fixed out-of-memory / worker SIGKILL during a large index build’s merge phase. The 1.0.6 flush-budget ceiling bounds the scan phase, but the merge phase held two vocabulary-proportional working sets that no setting bounded: it loaded every input segment’s entire dictionary into memory (and the final reduction opens all segments at once), and it accumulated the whole merged vocabulary before writing the dictionary. On a large, high-vocabulary corpus (tens of millions of distinct terms) each could reach multiple gigabytes regardless of maintenance_work_mem, so a big build could exhaust host memory in the merge phase even at a configuration whose scan phase fit comfortably. The merge now reads each input dictionary a page at a time and spills the output dictionary metadata to a temporary file, so its memory no longer scales with the corpus vocabulary. Measured peak build memory roughly halved on a high-vocabulary corpus, with a much flatter growth curve; build time did not regress. (Memory-only change: on-disk format is byte-identical, no REINDEX.)

1.0.7

Bug-fix and usability release. No on-disk format change from 1.0.6; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.0.7').

Fixed a concurrent scan-vs-merge stale read (the A1 hazard). A scan snapshotted the segment directory, released the metapage lock, then walked segment pages holding only per-page share locks; a concurrent fts_merge / fts_vacuum / autovacuum could free those pages and a concurrent insert recycle them, so the scan followed a now-stale segment descriptor and returned a wrong (too-low) match count or ranking. The metapage now carries a generation counter bumped on every segment-directory change (segment add/merge/free and the bulkdelete livedocs-pointer swap); a scan records it at its snapshot and re-checks it before trusting the result, restarting from a fresh snapshot if it moved. The counter lives after the segment array, so existing indexes are unaffected – no format change, no REINDEX. Reproduced and verified deterministically (test/a1_recycle/).
to_ftsdoc(tsvector): build an ftsdoc directly from an existing tsvector (lexemes and positions mapped straight across, no re-analysis). Lets you index a stored tsvector column, or migrate a to_tsvector workload, without re-parsing text. Positions are preserved only when every lexeme has them (a stripped tsvector indexes positionless).
Long-running scans, CREATE INDEX CONCURRENTLY validation, and fts_count are now promptly cancellable. Added interrupt checks at lock-free points in the scan and trigram-scan page walks, so a query stuck on a large or pathological index responds to statement timeout / Ctrl-C instead of spinning to completion.

1.0.6

Bug-fix release. No on-disk format change from 1.0.5; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.0.6').

Fixed runaway memory during a long index build over a large corpus (the phase after the 1.0.5 fix). The per-participant build memory budget grew geometrically with no ceiling, so a large build’s in-memory working set could climb to tens of gigabytes before flushing; combined with one budget per build participant (the leader plus each parallel worker), a big parallel build could exhaust host memory and thrash into swap hours in. The budget is now capped at 2 x maintenance_work_mem, so peak build memory is bounded to (max_parallel_maintenance_workers + 1) x 2 x maintenance_work_mem regardless of corpus size. (Peak build memory scales with the worker count – size maintenance_work_mem with that multiplier in mind.)
A freshly built or REINDEXed index now reclaims the free tail its final merge leaves on disk, so it ships closer to its compacted size instead of carrying the merge’s freed-input pages until the next VACUUM. (Full compaction of interior free space still comes from fts_vacuum / VACUUM.)

1.0.5

Bug-fix release. No on-disk format change from 1.0.4; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.0.5').

Fixed sustained, unbounded-looking memory growth during a long index build over a high-vocabulary text column (e.g. email/message bodies with quoted chains, patches, and code – millions of distinct terms). The build bounds its memory by flushing a segment when the in-memory working set exceeds maintenance_work_mem, but the size check did not count the term hash table, which lives in a child memory context – so on a vocabulary-dominant corpus the check undercounted the real working set and the flush fired far too late, letting a multi-hour build grow well past maintenance_work_mem (observed ~19 GB resident+swap on an 83 GB / 1.8M-row build before it was killed). The check now counts child contexts, so a build settles at roughly maintenance_work_mem regardless of vocabulary size. Affects both serial and parallel builds; no change to results or on-disk format.

1.0.4

Scale-hardening release. No on-disk format change from 1.0.3; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.0.4').

CREATE INDEX on a large table with a modest maintenance_work_mem no longer fails with “reached the maximum of 128 segments.” The build flushed one segment per maintenance_work_mem of accumulation with no intermediate merge, so a large enough index overflowed the segment directory before the end-of-build merge. Each build participant now grows its own flush budget geometrically, so the flush count grows only logarithmically with corpus size (and stays far under the cap) – parallel-safe, with no on-disk or behavior change to the finished index.
Corrected integer-width overflows that only surface at extreme scale (wrong results, not crashes): document-frequency sums used in IDF/BM25 scoring and in fts_index_df() are now 64-bit (a term in more than ~4 billion documents no longer wraps); the anomaly-scan “skip common terms” filter no longer misfires on a term whose df exceeds ~2.1 billion.
fts_index_stats’s nterms output is now bigint (was int), so an index with more than ~2.1 billion total distinct terms reports a correct count. This widens the function’s output column; the upgrade replaces the function definition (a DROP/CREATE, applied automatically by ALTER EXTENSION ... UPDATE).

1.0.3

Bug-fix release. No on-disk format change from 1.0.2; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.0.3').

Fixed an invalid memory alloc request size crash in a parallel index-build worker (reported against a ~13-hour parallel CREATE INDEX CONCURRENTLY over a large table with a very large vocabulary). This was a different site from the 1.0.2 fix: the trigram inverted-index builder and the streaming segment merge sized several allocations from the merged-group vocabulary, which – unlike the per-segment build path – is not bounded by maintenance_work_mem, so a hot trigram’s term list (and the merge output arrays) could exceed the 1 GB allocation limit on a large enough corpus. All corpus/vocabulary-scale allocations in the build, merge, and analyze paths now use a huge-safe allocation, closing this crash class off across the board.
A single document that would assemble into an ftsdoc larger than 1 GB now reports a clear “document is too large” error instead of an opaque allocation failure. An ftsdoc is a variable-length value limited to 1 GB.

1.0.2

Bug-fix release. No on-disk format change from 1.0.1; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.0.2').

Fixed a read-path crash (invalid memory alloc request size) when decoding a posting block with a very large per-block position count. The positions-decode path in bm25_decode_term sized its scratch buffers with an unguarded allocation; when a block’s summed term frequency pushed the buffer past the 1 GB MaxAllocSize limit, the allocation threw and aborted whatever triggered the decode – a CREATE INDEX CONCURRENTLY validation scan in the reported case, but the same path is reached by ordinary scans (count, ranked, phrase), merges, and vacuum. A legitimately large position count now uses a huge-safe allocation; a corrupt or inflated on-disk term-frequency (a class the existing block-header and column-length corruption checks did not catch) is now detected and rejected with a WARNING (a bounded miss, REINDEX to rebuild), rather than reading past the block. This was the read-side counterpart of the build-time huge-allocation fix; both sides are now guarded.
The corruption/fuzz harness gained a dedicated planted-bug (“teeth”) build for this class, so a regression that removed the guard would fail CI.

1.0.1

Bug-fix release. No on-disk format change from 1.0.0; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.0.1').

Fixed an out-of-memory crash when building or merging a large index. The segment-merge phase (used by the final compaction of an index build, by fts_merge, and by parallel builds) decoded every posting of every term from all merged segments into memory at once before writing the result, so compacting a large index could hold the entire index’s postings in RAM and OOM the server. Merging is now a bounded, streaming k-way merge that holds only one term’s postings at a time; peak merge memory is independent of index size. Measured on a 3M-document build: peak merge memory dropped from 2240 MB to ~1 MB, with the same result and no build-time regression. No on-disk format change and results are unchanged (index-vs-sequential-scan parity, with positions, tombstone drops, and phrase queries all preserved).

1.0.0

First stable release. No on-disk format change from 0.3.x; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '1.0.0'). The on-disk format (BM25_VERSION 3, FTS_DOC_VERSION 3) and the SQL surface are now considered stable; future 1.x releases keep backward compatibility.

fts_vacuum now converges and reclaims space in a single call, never grows the index, and is interruptible. Compaction previously could oscillate (transiently grow the index before shrinking) and, in a first correctness pass, could stabilize without reclaiming dead space; and no pg_fts operation checked for interrupts, so a long build/merge/vacuum could not be cancelled. fts_vacuum now compacts to the size floor in one call, is stable across repeated calls, and never returns larger than it started. It honors pg_cancel_backend and statement_timeout (nine interrupt-check points along the merge/vacuum path); a cancelled or out-of-disk run leaves the index valid and correct, just not fully compacted. Because compaction rewrites live data before freeing the old copy (for crash safety), it transiently needs free disk space of roughly the live index size, like VACUUM FULL / CLUSTER / pg_repack.
The transparent count(*) ... WHERE col @@@ q fast path is now chosen at scale. The FtsCount custom-scan cost model was priced against the whole heap and lost to a bitmap index scan on large tables; it is now priced as the index-only visibility-map count it actually performs, so the planner uses the faster path automatically.
Supported versions. PostgreSQL 17 and 18 are fully supported and gated in CI (regression + isolation + TAP). PostgreSQL 19 / master-devel builds and is exercised best-effort (it is unreleased).
Testing. The TAP suite (crash recovery, replication, torn-page recovery, server-encoding install/parity) is now a gating part of CI on both forges, alongside regression + isolation, AddressSanitizer / UndefinedBehaviorSanitizer builds, a fuzz harness for the posting-decode path, property-based tests, and a line-coverage floor.

0.3.6

fts_snippet default ellipsis is now ASCII ... (was the UTF-8 …). The non-ASCII default made CREATE EXTENSION pg_fts FAIL on a non-UTF-8 server database (LATIN1, EUC_JP, …) with “invalid byte sequence for encoding” – pg_fts was uninstallable there. The install SQL is now pure ASCII (guarded by make check-ascii in CI on both forges), so pg_fts installs on every server encoding. Callers who want the … glyph pass it explicitly: fts_snippet(doc, q, ellipsis => Eu2026).
Character-encoding / multi-script correctness is now permanently tested. A UTF-8 regression block asserts pg_fts @@@ == native to_tsvector @@ across 14 scripts (Latin, Windows-1252 punctuation, CJK Han, Japanese, Hangul, NFC vs NFD combining marks, emoji + 4-byte astral, CJK Ext-B, Arabic/Hebrew RTL, Turkish dotless-i, German sharp-s), plus a corner-case block (fold-length changes, multi-mark combining, ZWJ, BOM, fullwidth, Cyrillic) exercising the built-in analyzer; and t/004_encodings.pl gates LATIN1 + EUC_JP server encodings (install + native parity on high/multibyte bytes).

0.3.5

Hardening + testing release. No on-disk format change; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '0.3.5').

Further hardened the segment posting-decode path against corrupt/torn pages (three issues found by the new fuzz harness, extending the 0.3.4 fix): the per-block count clamp was one-sided (a uint32 count >= 2³¹ cast to a negative int and slipped past > BM25_BLOCK_SIZE) and is now tested on the unsigned value; the three FOR columns' width-driven byte consumption is now bounded against the block’s declared bytelen before decoding, so a corrupt width byte cannot read past the page; and a shift-by-64 undefined behavior in bm25_for_unpack on a corrupt width (> 64) is fixed (valid widths unaffected). A corrupt block remains a bounded miss with a WARNING, never a crash.
New testing regime (see doc/testing.md), run in CI on both forges: an AddressSanitizer+UBSan build of the regression + isolation suite; a gating fuzz/corruption harness (test/fuzz/) over the FOR codec, the stored-document validator, and the block decoder, with planted-bug “teeth”; property-based tests (test/hegel/) for the codec + validator invariants; a torn-page crash-safety TAP test (t/003_corruption.pl); and a 90% line-coverage gate on the pg_fts sources. fts_doc_is_valid’s logic was factored into a pure header (pg_fts_docvalid.h) shared with the fuzzer (behavior-identical).

0.3.4

Crash-safety bug-fix release. No on-disk format change; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '0.3.4').

Fixed an intermittent crash in the segment posting-decode path (fts_doc_matches <- bm25_collect_matches <- bm25_gettuple), a follow-on to the 0.3.3 pending-list fix that its validator did not cover. bm25_decode_term read a posting block header’s count from disk and unpacked that many values into fixed 128-element (BM25_BLOCK_SIZE) stack arrays (gaps/tfs/dls) with no bound check – a torn or corrupt block header with count > BM25_BLOCK_SIZE overflowed the stack (an AddressSanitizer heap/stack-buffer- overflow, reproduced against the FOR codec). The WAND block loader already clamped its count; this decoder (used by the boolean/@@@/count-pushdown/ ranked/anomaly/trigram scan paths – every bm25_decode_term caller) did not. It now clamps the per-block count to BM25_BLOCK_SIZE and stops decoding a block whose declared column byte lengths (bytelen/posbytelen) run past the page, so a corrupt block is a bounded miss (with a WARNING hinting REINDEX) instead of a crash. Fixing it inside bm25_decode_term protects all callers at once. Valid indexes are unaffected; a regression test decodes a large multi-block posting list (df >> 128, with positions).

0.3.3

Crash-safety bug-fix release. No on-disk format change; no REINDEX required (ALTER EXTENSION pg_fts UPDATE TO '0.3.3').

Fixed two backend crashes on the pending-list path (reported on 0.3.2 / PostgreSQL 18 under heavy concurrent write load): a _FORTIFY_SOURCE buffer overflow (SIGABRT) in add_posting during the autovacuum pending-list flush, and a SIGSEGV in fts_doc_matches while scanning pending documents. Both stemmed from the pending-list readers casting raw index-page bytes to an ftsdoc and trusting its term metadata (nterms, per-term len/tf/ posoff) without validation, so a malformed or torn page turned a bad length into a wild memcpy or a bad offset into an out-of-bounds read. The flush (bm25_vacuumcleanup -> bm25_flush_pending) and scan (bm25_gettuple -> bm25_collect_matches) paths now validate each pending document’s structure against its own byte length (fts_doc_is_valid) before trusting any offset, and skip a malformed document with a WARNING (hinting REINDEX) instead of crashing the backend. add_posting’s fixed-size term key also clamps its length defensively as a last line of defense. Valid documents are unaffected. A crash-regression test covers the long-token pending -> scan -> flush cycle.

0.3.2

Additive release. No on-disk format change; no REINDEX required for the extension upgrade (ALTER EXTENSION pg_fts UPDATE TO '0.3.2'). Indexes over non-ASCII text should be REINDEXed to pick up the new Unicode lowercasing; ASCII-only indexes are unaffected.

pg_stat_user_indexes now reflects bm25 index usage (PR #5, dinesh-salve). Every query path that reads the index registers an index scan (pgstat_count_index_scan), so idx_scan/last_idx_scan are no longer stuck at 0: the bitmap scan, the plain + ranked (ORDER BY <=>) index scans, the count(*) pushdown / fts_count(), and native fts_search() top-k. idx_tup_read is reported on the bypass paths too (the AM scan paths already get it from the generic index layer). A bare ORDER BY <=> with no @@@ filter is a seq-scan+sort and correctly stays at 0.
Unicode lowercasing in the built-in analyzer (PR #4, dinesh-salve). The built-in to_ftsdoc(text)/to_ftsquery(text) analyzer folded only ASCII A-Z, so accented text never matched case-insensitively ('CAFÉ' missed 'café'). A shared fold_token() (used by the document analyzer, the query lexer, and the aux tokenizer, so both sides fold identically) now lowercases non-ASCII tokens per Unicode code point via unicode_lowercase_simple() in UTF-8 databases; non-UTF-8 databases keep byte-wise ASCII folding. Simple lowercasing, not full case folding ('ß' stays 'ß'), matching pg_search’s default, and to_ftsdoc() stays IMMUTABLE. No on-disk format change; ASCII-only indexes are unaffected. Indexes over non-ASCII text built before this change should be REINDEXed (their stored terms are unfolded).
Build-time huge-allocation fix for very high-df terms. At large diverse corpora (found while benchmarking at 20M docs – see bench/RESULTS_20M.md) a single ultra-common token’s build-time posting arrays can exceed MaxAllocSize (1 GB), which plain palloc rejects, aborting the index build. add_posting, bm25_decode_term, and bm25_write_postings now use the ...Huge allocation variants past 1 GB. No format change.

0.3.1

Additive feature release. No on-disk format change (read-only over the existing index) and no REINDEX required. ALTER EXTENSION pg_fts UPDATE TO '0.3.1'.

Lexical anomaly detection: fts_anomalous_docs(index, k, max_df). A set-returning function that surfaces the top-k most lexically-anomalous documents in an fts index – those containing globally rare terms. A document’s anomaly score is the maximum idf over its terms (driven by its single rarest term), using the same rarity value BM25 uses: idf = log(1 + (N - df + 0.5)/(df + 0.5)) on the global df (a term’s df is summed across all segments before scoring, so a document split across two segments is not made to look artificially rare). Returns (ctid tid, score float8, rarest_term text, min_df int) ordered by score DESC, limit k.
- Cheap because it walks only the low-df tail. The rarest terms have the shortest posting lists, so the function walks the term dictionary and skips any term whose global df exceeds max_df before decoding a single posting – the common, high-df bulk of the dictionary is never decoded. On a 1M-document corpus with a handful of injected unique tokens, the query returns those docs in well under a millisecond (measured 0.6 ms), not a full-corpus scan. max_df defaults to max(N/1000, 1) when NULL, keeping the walk on the low-df tail.
- The returned ctids are index-resident heap pointers (like fts_search); this is an analytic/heuristic result, so no per-doc heap visibility check is done – join ctid back to the table and filter for visibility if needed. Per-segment tombstones are honored, so deleted documents are not reported as anomalies.
- Lexical only: it catches rare/novel wording and tokens, not semantic novelty (see bench/NOTE_ANOMALY_DETECTION.md). Bench harness in bench/anomaly.sql.
Fixed ftsdoc text I/O round-trip (Codeberg #3). ftsdoc_out emitted the canonical 'term':tf form but ftsdoc_in re-tokenized that string as raw text, so ftsdoc_in(ftsdoc_out(x)) != x – text COPY/pg_dump --inserts of stored ftsdoc columns corrupted the data. ftsdoc_in now parses the canonical grammar 'term':tf[@p1,p2,...] (falling back to raw-text analysis for the ergonomic 'the quick brown fox'::ftsdoc cast), and ftsdoc_out, ftsdoc_send/ftsdoc_recv now carry per-token positions so both text and binary I/O are faithful, position-preserving round-trips. Input is validated at the trust boundary (ascending/distinct terms, tf>=1, ascending positions, tf positions per term; corrupt binary bounded before palloc). The ftsdoc binary wire version bumped 2 -> 3; ftsdoc_recv still accepts v2 so a pg_dump -Fc taken under an older pg_fts restores cleanly (v2 docs are position-free). No on-disk index format change.

0.3.0

Feature release with an on-disk index format change (BM25 v2 -> v3). Existing bm25 indexes must be REINDEXed; the format guard rejects a v2 index with a REINDEX hint. No ftsdoc/ftsquery type change.

Token positions in the postings, gated by a new positions reloption. CREATE INDEX ... USING fts (...) WITH (positions = on) stores per-token positions in the posting blocks (a 4th, lazily-decoded frame-of-reference column after docid-gaps/tf/doclen). Phrase and NEAR queries are then answered directly from the posting lists – intersect on docid, verify adjacency from the stored positions via the same phrase_step logic the heap recheck uses – with zero heap access and no recheck. This removes the phrase/NEAR count cliff (a common two-word phrase count over an expression index dropped from seconds to the AND-count range in local tests) for both the expression index (to_ftsdoc(col)) and the stored-ftsdoc-column shapes.
- Default is positions = off: positions roughly double the posting bytes on high-term-frequency corpora, so the size-sensitive majority who never phrase-search pay nothing. Phrase/NEAR is always correct either way; it is only fast (index-only, no recheck) with positions = on. With positions = off it falls back to the correct-but-slower heap recheck.
- Positions are decoded lazily: plain BM25 ranked / boolean AND / count queries never read or decode the positions column (a posbytelen-guided pointer skip, mirroring the existing tf/doclen skip), so a non-phrase query pays ~zero for positions existing (measured: no regression vs v2 on a common-term ranked/count query).
- fts_vacuum / merge carry positions through the compaction rewrite and keep reclaiming space; a pathological per-(term,doc) term frequency whose positions would overflow a page drops that block’s positions and phrase falls back to recheck for those docids (correctness preserved).

0.2.4

Bug-fix release. The fix is in the shared library; no SQL objects change and no REINDEX is required. ALTER EXTENSION pg_fts UPDATE TO '0.2.4'.

Phrase / NEAR queries silently returned wrong results on a stored ftsdoc column. The positions[] region of a document was addressed as MAXALIGN(absolute-pointer), but the analyzers lay it out at base + MAXALIGN(offset). For a heap-resident (detoasted) document whose base is not itself MAXALIGN’d, MAXALIGN(base+off) != base+MAXALIGN(off), so the position array was mis-addressed and phrase/NEAR degraded to a plain AND (matching any document containing the terms, ignoring adjacency). Fixed to the offset-based address. Expression indexes on to_ftsdoc(col) were unaffected by this bug (freshly-analyzed, always-aligned documents); short documents in the existing tests happened to remain aligned, which is why it was missed.
Note: phrase count over an expression index on a common two-word phrase is still slow (it rechecks the whole AND-set against the heap, re-analyzing each document). A positional-index format change in a later release removes that heap recheck; this release only fixes the stored-column correctness bug.

0.2.3

Performance release. The change is in the shared library; no SQL objects change, results are unchanged, and no REINDEX is required. ALTER EXTENSION pg_fts UPDATE TO '0.2.3' after installing the new library.

Ranked ORDER BY d <=> q LIMIT k over a boolean AND/NOT query is much faster. The 0.2.1 boolean-structure correctness fix pre-collected the entire exact @@@ match set before the ranked scan filtered against it, which was slow on common terms (e.g. year & hungary top-10 took ~37 ms at 2M docs because the whole year posting list was materialized). The ranked scan now evaluates the query’s boolean structure lazily during the WAND traversal (from which terms are present at each candidate), with no collect pass: year & hungary top-10 drops to ~1 ms at 2M (measured), and a near-universe NOT like year & !hungary from ~415 ms to ~42 ms. Results are byte-identical (the ground-truth ranked-parity test passes unchanged). Pure-OR / single-term queries were already on the fast path and are unchanged; phrase/NEAR/fuzzy/ regex keep the exact-recheck path.

0.2.2

Bug-fix release. The fix is in the shared library; no SQL objects change. ALTER EXTENSION pg_fts UPDATE TO '0.2.2' after installing the new library.

Phrase ("a b c") and NEAR queries now enforce term adjacency on all query paths. They previously degraded to AND (matching any document containing the terms, regardless of order/adjacency) on the primary to_ftsdoc(regconfig, text) path — e.g. to_ftsdoc('english', body) — because that analyzer did not store token positions; and the index candidate path did not request the heap recheck that would have enforced adjacency, so non-adjacent documents leaked through @@@, the bitmap scan, and the ranked <=> scan alike. Now: the config analyzer stores positions; @@@/bitmap enforce adjacency via the executor recheck; and the ranked <=> scan and fts_count() recheck the exact match set against the heap document (bm25_recheck_exact). Phrase / NEAR / boolean ranking is exact on all paths.
No REINDEX required (bm25 on-disk format unchanged). Phrase correctness on a stored ftsdoc column populated by the old analyzer requires re-analyzing those rows; expression indexes on to_ftsdoc(...) are correct immediately.
Known limitation (documented): ranked <=> over fuzzy/prefix/regex returns a correct subset of the @@@ matches (never a wrong document, but may be incomplete, since the ranked scan builds cursors from the literal term). Use @@@ for exhaustive fuzzy/prefix/regex retrieval.

0.2.1

Bug-fix release. The fix is entirely in the shared library; no SQL objects change and existing fts indexes need no REINDEX (on-disk format unchanged from 0.2.0). ALTER EXTENSION pg_fts UPDATE TO '0.2.1' after installing the new library.

Ranked <=> scan now respects boolean AND/NOT/PHRASE structure. The ORDER BY d <=> q LIMIT k ordering scan previously ranked the term disjunction (it flattened the query to its terms) and never intersected with the boolean match set that @@@ uses, so AND/NOT/PHRASE queries could return documents that fail @@@ — e.g. a & !b ranked documents that contain b. The ranked scan now gates results by the exact @@@ match set, so every row it returns satisfies @@@. @@@ matching and pure-OR / single-term ranking were already correct and are unchanged.

0.2.0

Breaking: the index access method was renamed bm25 → fts (CREATE INDEX ... USING fts (to_ftsdoc('english', body))). This lets pg_fts coexist in the same database as Timescale pg_textsearch (whose AM is named bm25), so a pg_textsearch workload can be migrated one index at a time rather than in a single hard cutover. Existing USING bm25 indexes must be recreated as USING fts. The BM25 scoring functions (fts_bm25, fts_bm25f, fts_bm25_opts) are unchanged — BM25 is the ranking algorithm, fts is the access method.

On-disk format version check: opening an index whose stored format version does not match the loaded shared library now raises a clear error (... has pg_fts on-disk format version N, but this build expects M) with a REINDEX hint, instead of silently misreading the index.
New doc/MIGRATING_FROM_PG_TEXTSEARCH.md: query/DDL rewrite table, the multi-column → concatenated-to_ftsdoc pattern, and index build sizing (CREATE INDEX bounds build memory to maintenance_work_mem).

0.1.0 — initial public release

First public release. The extension was developed as an internal, qualified feature series (each stage clean under --enable-cassert, regression-green) that reached internal version 1.20 before being squashed to a single 0.1.0 install script for release. Versioning starts at 0.1.0 to signal that the on-disk format and ranked-query performance will iterate before 1.0.

Included in 0.1.0:

ftsdoc / ftsquery types, the @@@ match operator, and the <=> relevance-ordering operator (ORDER BY d <=> q LIMIT k plans as an index scan, no Sort).
The bm25 inverted-index access method: WAL-logged via GenericXLog (crash-safe, physical-replication safe), MVCC-correct (per-segment tombstones), segmented (Lucene/Tantivy-style) on-disk format with a size-tiered background merge, block-max WAND / MaxScore top-k with lazy per-column decode.
Okapi BM25 scoring with the lucene / robertson / atire / bm25+ / bm25l variants; BM25F multi-field weighting; index-maintained corpus statistics (N, avgdl, per-term df) so ranking needs no heap recheck.
A rich query language over one operator: boolean, phrase "a b c", NEAR, prefix term*, fuzzy term~k (Levenshtein DFA), and regex /re/, with a trigram pre-filter for fuzzy/regex.
fts_highlight() / fts_snippet(); tsquery_to_ftsquery() migration helper and cast.
Incremental maintenance (INSERT appends to a pending list, no REINDEX); fts_merge() and fts_vacuum() (compact + truncate) for on-demand maintenance.
fts_count() and a transparent count(*) ... WHERE @@@ CustomScan pushdown for MVCC-correct bulk counts from the index — a capability the specialist BM25 extensions do not expose.
Parallel index build/merge; standalone PGXS build plus Nix flake and a Windows/MSVC meson recipe; supported on PostgreSQL 17, 18, and 19/devel.

Known performance position (see bench/RESULTS_VS_VCHORD_PGTEXTSEARCH.md): pg_fts is far faster than the built-in tsvector/GIN + ts_rank stack on ranked retrieval (up to ~40×), but trails the specialist BM25 extensions (VectorChord-bm25, Timescale pg_textsearch) on raw ranked latency and index size. Closing that gap is a posting-codec rewrite tracked in ROADMAP.md; 0.1.0 ships on its distinguishing strengths — query-language breadth, index-native COUNT, and MVCC/crash correctness — and will iterate on ranked performance.

PGXN

PostgreSQL Extension Network

Contents

Changelog

1.8.2 - 2026-09-17

Fixed

Changed (behaviour-preserving; full gate green after each)

1.8.1 - 2026-09-17

Added

Changed

Documentation

1.8.0 - 2026-09-14

Fixed

Behaviour change

1.7.2 - 2026-09-14

Fixed

Known issues

1.7.1 - 2026-09-14

Fixed

Retracted

Known issues

1.7.0 - 2026-09-13

Fixed

Added

Known issues

Documentation

1.6.1

The bug

Root cause: one mistake in two places

sparsemap 5.4.0 → 5.5.1

A note on how this was found, and on our test coverage

1.6.0

1. A phrase whose adjacency cannot be verified now returns false

2. Field-zone restrictions on a positionless document match nothing

3. sparsemap 5.4.0 -> 5.5.0

Upgrade notes

Validation

1.5.10

1. Ascending-resume hint in the doclen lookup

2. bm25_for_get() was still decoding bit-by-bit

Rejected by measurement (recorded so they are not retried)

Validation

1.5.9

1. Non-ASCII case folding was broken on non-UTF-8 server encodings

2. Rare-term ranked latency: 1.7x faster (block-granular doclen decode)

3. Multi-term ranked latency: 1.44-1.63x faster (shared resident block)

Rejected by measurement (recorded so it is not retried)

Validation

1.5.8

1.5.7

1.5.6

1.5.5

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.1

1.4.0

1.3.2

1.3.1

1.3.0

1.2.2

1.2.1

1.2.0

1.1.7

1.1.6

1.1.5

1.1.4

1.1.3

1.1.2

1.1.1

1.1.0

1.0.8

1.0.7

1.0.6

1.0.5

1.0.4

1.0.3

1.0.2

2. `bm25_for_get()` was still decoding bit-by-bit