pg_sift 3.5.2: fixes two crashes of a backend (signal 11), one in the load of a lookup table whose text column has a dictionary kept in blocks, one in an aggregate scan that reads several wide columns beside a dictionary key; the second also gave rows to the wrong groups. Update with the files of this release and restart the server; no ALTER EXTENSION step, no REINDEX.

Highlights

  • Fixed: a crash loading a lookup table with a large text dictionary. A lookup table loaded from the leaves of a sift_col index crashed the backend (signal 11) when it had a text column of 1,024 or more distinct strings in a segment, whose dictionary the segment keeps in blocks and the backend has not loaded: SELECT county, sum(x) FROM facts, addresses WHERE facts.k = addresses.k GROUP BY county killed the server on the first statement of a connection with 1,100 counties. The string of an entry is now read with the call every other reader of a dictionary uses; the rows are those of the normal plan. Test agg_lookup_lazy_dict.
  • Fixed: wrong groups or a crash with several wide columns. An aggregate scan that read several wide leaves (numeric beyond 64 bits, interval, money) for the selected rows of its units, with a text or other dictionary key beside them, gave rows to the wrong groups or ended the backend with signal 11: the eight slots of the executor’s per-dictionary maps were handed out round robin, so a dictionary made for the read of a wide leaf (one for every unit) took the slot of a key map still in use. The least recently used slot goes now. Test agg_dict_slots.

Who is affected and what to do

  • Every install of 3.5.0 and 3.5.1 that runs one of the two statements above: install 3.5.2 and restart the server. A backend that already loaded the old library keeps it until its connection ends, so after the install restart the server (a standby too), or reconnect every pool and worker. Earlier releases were not checked.
  • Until then: SET sift.enable_agg = off for the role restores PostgreSQL’s plan for the statements affected.
  • Nothing else changes: plans, answers and formats are those of 3.5.1.

Compatibility

  • PostgreSQL 17, 18 and 19; the packages below cover all three (other builds from source with make USE_PGXS=1). A drop-in replacement of 3.5.1: extension SQL 3.5 (ALTER EXTENSION pg_sift UPDATE has nothing to do and there is no pg_sift--3.5--3.5.2.sql), no change of the BM25, sift_col or sift_table formats, no setting changes its default. pg_get_loaded_modules() on PostgreSQL 18 says 3.5.2+<commit>; the extension version stays 3.5.

Packages

x86-64 and arm64 packages for PostgreSQL 17, 18 and 19 are attached below (SHA256SUMS covers every asset); other versions build from source with make USE_PGXS=1. The packages depend on liblz4 and libzstd. The 17 and 19 packages and all the arm64 ones were added after the release, built from the same tag; the 19 ones are built and tested against PGDG’s 19~rc1. Each package was installed in fresh containers of its architecture (Debian 13, Ubuntu 24.04, Rocky 9, Alma 9 and Alma 10; the arm64 ones on a hosted arm64 runner), where CREATE EXTENSION, an update from 1.0, a sift index and its queries ran. The arm64 .deb for PostgreSQL 19 is the tag’s source with one change of a vector type’s alignment in src/col/codec_vec.h (GCC 14 on arm64 ends with an internal compiler error on it; the compiled code of the other builds is byte for byte the same with and without it).

  • Debian 13, Ubuntu 24.04 (PGDG postgresql-N, N = 17, 18 or 19): sudo apt install ./postgresql-N-pg-sift_3.5.2-1_amd64.deb (_arm64.deb on arm64)
  • RHEL, Rocky, Alma 9 and 10 (PGDG postgresqlN-server): sudo dnf install ./pg_sift_N-3.5.2-1.el9.x86_64.rpm (.aarch64.rpm on arm64)
  • Arch Linux: AUR pg_sift.

Every change: CHANGELOG.md.

SQLite extension

pg_sift for SQLite 3.5.2 (sift, a loadable extension; a separate component from the PostgreSQL extension above). Rebuilt with the PostgreSQL 3.5.2 release: no change to the SQL interface or the shared format.

  • No change: the fixes of this release are in the PostgreSQL column index, the aggregate planner and sift_table, which are not in this extension. No speed-up or slow-down is claimed.
  • Needs SQLite 3.38 or newer; Linux x86-64, glibc 2.34 or newer. Built on Debian 12, loaded into the SQLite of Debian 12, Ubuntu 24.04 and AlmaLinux 10 before release.
  • Storage format 2: tables of 3.4.0 to 3.5.1 open as they are, nothing to rebuild.

Download sift-sqlite-3.5.2-linux-x86_64.tar.gz (below, covered by SHA256SUMS with the packages), unpack it and .load ./sift from the sqlite3 shell (sha256sum --ignore-missing -c SHA256SUMS). Documentation: sqlite/README.md. Every change: sqlite/CHANGELOG.md.

MariaDB extension

pg_sift for MariaDB 3.5.2 (sift, a UDF library and two SQL files; a separate component from the extensions above). Rebuilt with the PostgreSQL 3.5.2 release: no change to the SQL files.

  • No change: the fixes of this release are in the PostgreSQL column index, the aggregate planner and sift_table, which are not in this extension. No speed-up or slow-down is claimed.
  • One library per MariaDB series, because it calls the server’s own symbols and refuses to load into another series: 10.11, 11.4, 11.8, 12.1 and 13.0; Linux x86-64. Each was built in the stock mariadb:<series> image against that server’s headers and loaded into a server of the same image before release. BUILD-INFO in the tarball lists the server version and the glibc each library needs; a server of another distribution or of a series not listed is built from source (make -C mariadb).
  • Index format 2: an index of 3.4.0 to 3.5.1 opens as it is, nothing to rebuild.

Download sift-mariadb-3.5.2-linux-x86_64.tar.gz (below, covered by SHA256SUMS with the packages), unpack it, copy lib/<series>/libsift_udf.so of your server’s series (SELECT VERSION()) into its plugin_dir, then SOURCE sql/functions.sql once per server and sql/sift.sql once per database (sha256sum --ignore-missing -c SHA256SUMS). Documentation: mariadb/README.md. Every change: mariadb/CHANGELOG.md.