pg_sift 2.0.2: a decoder hardening against damaged index data and a fix for VACUUM running next to queries. Install the new files: the extension SQL does not change (no ALTER EXTENSION pg_sift UPDATE step), the on-disk format does not change (no REINDEX), the results do not change.

Highlights

  • Security: a damaged index can no longer make a query read or write outside memory it owns. The query path follows the lengths, counts and offsets that a term blob stores without the exhaustive checks of sift.verify, and used them unchecked. Bytes that are not what the writer wrote (a bad disk or controller, a torn page, a file edited with the server stopped, a physical copy from a damaged or hostile source) could make a backend read past the blob, write past a stack array, or return one group without end. Fuzzing the decoder found 35 distinct crash inputs on 2.0.1 in five minutes; 2.0.2 is clean on all of them, on the six reproductions of the audit and on about 4 million further executions.
    • Affected: any sift index whose pages are damaged. There is no remote vector: no SQL role can write index pages; it takes damage on disk or write access to the data directory.
    • What to do: install 2.0.2. No REINDEX is needed for healthy indexes. A damaged index now fails the query with XX001 (sift: corrupt posting data: ..., naming what is wrong) instead; rebuild it with REINDEX, and use sift.verify to find out which are damaged. The checks are made once per blob header, per group and per block, not per posting. sift.verify itself also read one nibble past a tf stream whose bits counted more nibbles than it holds; fixed.
    • Cost (user instructions per operation against 2.0.1, bench/steps/cpu_gate.sh, interleaved, 7 rounds): queries 0.995x to 1.006x (the cells with a tight interval 1.000x to 1.003x), VACUUM after deleting 80% of the rows 1.007x (it opens every term blob of the segment it rewrites), CREATE INDEX 1.004x, COPY, inserts, sift.merge and sift.promote 1.000x to 1.003x.
  • Fixed: a lazy VACUUM running next to queries could reuse the index blocks of a segment while a query was still reading them. A VACUUM (manual or autovacuum) is left out of snapshots and of the xid horizon, its own xid included, so a query that began after the VACUUM had taken its xid saw that xid as passed, and the blocks the VACUUM released with it became reusable while the query still held the directory that listed them. The query then failed with sift index "t" is corrupted (expected page type 4 generation 107, found type 8 generation 0); the page checks every read makes turn the reuse into an error, and no wrong result was seen. Reproduced on 2.0.1 within seconds by 24 pgbench clients (inserts, deletes and searches of stable rows with a content check) with VACUUM in a loop, and not without the VACUUM loop; with the fix the same run passes.
    • Affected: indexes that are searched while VACUUM or autovacuum runs on their table and rewrites segments (dead documents, merges by the VACUUM). Only the reuse of released blocks changes, never the contents of a segment: the error is transient (retry), nothing is stored wrong, no REINDEX.
    • Now: a VACUUM releases with the next xid, so the blocks it frees are reusable once a later transaction has started; its own truncation of the free end of the file (an exclusive lock keeps every query out) counts them as reusable at once, so the file still shrinks in the VACUUM that freed the space.
  • Documentation: sample queries and documents use neutral wording; no change for users.

Compatibility

  • PostgreSQL 17 and 18; PostgreSQL 19 is experimental (CI builds it from git on every push). No extension SQL change (it stays 1.2) and no on-disk format change: existing indexes keep working, with no REINDEX. After installing the new files nothing else is required.

Packages

x86-64 packages for PostgreSQL 18 are attached below (SHA256SUMS covers every asset); other versions build from source with make USE_PGXS=1.

  • Debian 13, Ubuntu 24.04 (PGDG postgresql-18): sudo apt install ./postgresql-18-pg-sift_2.0.2-1_amd64.deb
  • RHEL, Rocky, Alma 9 and 10 (PGDG postgresql18-server): sudo dnf install ./pg_sift_18-2.0.2-1.el9.x86_64.rpm
  • Arch Linux: AUR pg_sift.

Every change: CHANGELOG.md.

SQLite extension

pg_sift for SQLite 2.0.2 (sift, a loadable extension; a separate component from the PostgreSQL extension above). Rebuilt on the shared core of the PostgreSQL 2.0.2 release: no change to the storage format or the SQL interface.

  • Shared engine: the posting decoder is the one hardened for PostgreSQL. A table file with damaged bytes now fails the query with a data-corruption error instead of reading outside the term it decodes; the damaged table still has to be rebuilt from its source. The unit and feature tests pass under ASan and UBSan, and the stress test under TSan and ASan. No speed-up or slow-down is claimed: the figures above are PostgreSQL’s. The VACUUM fix does not apply: SQLite has no shared page store.
  • Needs SQLite 3.38 or newer; Linux x86-64, glibc 2.34 or newer. Built on Debian 12, loaded into the SQLite of Debian 12, Ubuntu 24.04 and AlmaLinux 10 before release.
  • Storage format 2: nothing to upgrade from 2.0.1.

Download sift-sqlite-2.0.2-linux-x86_64.tar.gz (below, covered by SHA256SUMS with the packages), unpack it and .load ./sift from the sqlite3 shell (sha256sum --ignore-missing -c SHA256SUMS). Documentation: sqlite/README.md. Every change: sqlite/CHANGELOG.md.

MariaDB extension

pg_sift for MariaDB 2.0.2 (sift, a UDF library and two SQL files; a separate component from the extensions above). Rebuilt on the shared core of the PostgreSQL 2.0.2 release: no change to the index format or the SQL files.

  • Shared engine: the posting decoder is the one hardened for PostgreSQL. Segment files with damaged bytes now fail the query with a data-corruption error instead of reading outside the term; CALL sift_rebuild rebuilds such an index from its table. The functional, live and stress tests pass on MariaDB 13.0.2. No speed-up or slow-down is claimed: the figures above are PostgreSQL’s. The VACUUM fix does not apply: MariaDB has no shared page store.
  • One library per MariaDB series, because it calls the server’s own symbols and refuses to load into another series: 10.11, 11.4, 11.8, 12.1 and 13.0; Linux x86-64. Each was built in the stock mariadb:<series> image against that server’s headers and loaded into a server of the same image before release. BUILD-INFO in the tarball lists the server version and the glibc each library needs; a server whose glibc is older than that of its library does not load it.
  • Index format 2: an index of 2.0.1 opens as it is, nothing to rebuild.

Download sift-mariadb-2.0.2-linux-x86_64.tar.gz (below, covered by SHA256SUMS with the packages), unpack it, copy lib/<series>/libsift_udf.so of your server’s series (SELECT VERSION()) into its plugin_dir, then SOURCE sql/functions.sql once per server and sql/sift.sql once per database (sha256sum --ignore-missing -c SHA256SUMS). Documentation: mariadb/README.md. Every change: mariadb/CHANGELOG.md.