Contents

pg_lion: a roaring-bitmap inverted index AM for PostgreSQL — design v0

Status: prototype targeting PostgreSQL master (20devel). Built and tested against the assert-enabled install in .local/pg (.local/pg/bin/pg_config). This document is the single source of truth for on-disk format, locking protocol, and module boundaries. Change it before changing the code.

Extension name: pg_lion. Access method name: roaring. C symbol prefix: lion_ / RBI.

CREATE EXTENSION pg_lion;
CREATE INDEX ON fact USING lion (country);

1. Goals and non-goals for v0

Goals - Single-column equality index whose posting lists are roaring-style containers (array / bitset / run). - Bitmap scans (amgetbitmap) so the planner can use it in Bitmap Index Scan / BitmapAnd / BitmapOr. - Correct under concurrent INSERT and VACUUM. Crash-safe via generic WAL. - Bulk build via tuplesort (replaced by appends per key, §24 “Build”). Inserts, VACUUM (ambulkdelete) that removes TIDs and shrinks containers. - SQL-callable lion_index_stats() and lion_index_verify() for tests and debugging. - Layered so that phase 2 can add: a visibility-map-interlocked lion_index_count() and a CustomScan that answers count(*) ... GROUP BY key from containers without visiting the heap.

Non-goals for v0 (documented limitations; NULL keys and IN lists arrived in v1, §14 and §15; entry deletion and page recycling in §18) - Multi-column indexes, INCLUDE columns, ordered scans, amgettuple (arrived in §29), parallel build (arrived in §24, “Build”) and scan, fine-grained write concurrency (inserts serialize on the directory leaf that holds the key, §21), key sizes above LION_MAX_KEY_SIZE (§4; longer varlena keys are stored compressed, §4 “Long keys”, and tsvector, text array and jsonb keys past it are hashed, §17 and §42.1).

2. TID encoding

Every heap TID is mapped to a 64-bit code and split into a container key and a 15-bit low part.

LION_OFFSET_BITS      = 9 at BLCKSZ 8192 (MaxHeapTuplesPerPage = 291 < 512)
                       10 at 16K (585), 11 at 32K (1169)   -- computed at compile time, see lion_tid.h
                       (BLCKSZ below 8192 does not compile: see the end of this section)
code(tid)            = ((uint64) block << LION_OFFSET_BITS) | offset        -- 41 bits at 8K
LION_CONTAINER_BITS   = 15
ckey(code)           = code >> 15                                          -- uint32, block >> 6 at 8K
lo(code)             = code & 0x7FFF                                       -- 0 .. 32767
blocks per container = 1 << (15 - LION_OFFSET_BITS) = 64 at 8K

Offsets ≥ (1 << LION_OFFSET_BITS) are an ERROR at insert/build time (“table AM not supported”). The meta page records offset_bits and container_bits; opening an index with a mismatch is an ERROR.

Table access methods: the heap only (implemented 2026-09-23). The encoding has room for the heap’s line pointers and nothing else, and the count of §9 reads the heap’s visibility map under an interlock that is a statement about heap VACUUM. Another table AM may hand out TIDs of any shape (citus_columnar’s are stripe row numbers, with offsets far past 511) and keeps no visibility map, or one that means something else. So a lion index may only sit on a table whose table AM ROUTINE is the heap’s: lion_table_am_supported() (lion_am.c) tests rel->rd_tableam == GetHeapamTableAmRoutine(), the test the count’s heap recheck already made to choose its per-block path. The routine and not the AM’s name or Oid, because CREATE ACCESS METHOD heap2 TYPE TABLE HANDLER heap_tableam_handler is the heap under another name and works unchanged, while a handler that returns anything else - even a copy of the heap’s callbacks - may have changed anything.

  • ambuild is the gate: lionbuild() refuses first thing, rows or no rows, with FEATURE_NOT_SUPPORTED, access method "lion" does not support table access method "<am>", a detail naming the table and a hint to use the heap. Every way an index comes to sit on a table builds it: CREATE INDEX, REINDEX, ALTER TABLE … SET ACCESS METHOD (a rewrite that rebuilds the indexes, so a heap table with a lion index cannot move to another AM and the ALTER rolls back), a partition created, attached or indexed under a partitioned lion index (each leaf is built on its own, so one leaf of another AM fails the CREATE INDEX on the parent, the CREATE TABLE … PARTITION OF … USING, and the ATTACH), and on 17+ a partitioned table’s SET ACCESS METHOD, which only sets the default for partitions created later. aminsert needs no check: a table’s AM cannot change without that rewrite. Before this check an index build on a non-heap table failed somewhere inside: at lion_check_key_offset() for columnar’s TIDs, in heap_getnext() (“only heap AM is supported”) for a copied heap routine, or not at all for an AM with small offsets.
  • The count paths refuse as well, for an index that got past the gate (none can today; the catalogs can be edited, and a later release may loosen the build rule): the planner declines a leaf of another AM in lion_collect_targets(), so one such partition declines the whole partitioned query (§10, §16); the executor’s lion_open_relation() and the SQL counts' lion_count_open_indexes() raise the ambuild error.
  • The other option the §23 citus_columnar item named, a per-AM TID encoding with the bitmap path only, was not taken: no table AM other than the heap is known to hand out heap-shaped TIDs, and a wider offset field is a format change that belongs with level (2) of that item, which needs its own visibility source anyway.
  • Tests: test/sql/tableam.sql (the heap under another name: builds, pushdown and SQL counts on clean and dirty pages, SET ACCESS METHOD in both directions with the index rebuilt and verified, partitions of both AMs under one index) and test/modules/lion_hooktest’s tableam test under make hookcheck, whose lion_heapcopy AM returns a copy of the heap routine: every refusal above, and the three count paths declining or refusing after a table with lion indexes is moved to it by editing pg_class.relam. Not tested: a real non-heap AM (citus_columnar and TimescaleDB’s compressed chunks are not in the test installs; §23).

Why not GIN’s 11 offset bits: bitset containers would be 86% empty. Why 15 container bits and not 16: a 16-bit bitset is 8192 bytes and cannot be a page item; 15 bits gives a 4096-byte bitset, so every container fits on a page of PostgreSQL’s default BLCKSZ or larger - a BITSET item is 4104 bytes plus its line pointer, against 8136 bytes of item space at 8K - and VACUUM can always fall back to a bitset when a run split would otherwise grow a container past its slot. (The first version said “on any page”. It does not hold below 8K: a 4K page has 4040 bytes of item space, and a build or an insert failed on the first BITSET. A BLCKSZ below 8192 is now refused at compile time, by an #error in lion_tid.h - the first header every file includes - and, as the property that refusal stands for, a static assertion in lion.h that MAXALIGN(LION_CONTAINER_MAX_SIZE) + sizeof(ItemIdData) <= LION_PAGE_CAPACITY. 2026-09-25 review.)

3. Containers (module lion_container.[ch], no backend dependencies beyond c.h + pg_bitutils)

typedef struct LionContainer
{
    uint32  ckey;          /* container key (code >> 15) */
    uint16  cardinality;   /* number of members, 0..32768 */
    uint8   type;          /* LION_CT_ARRAY=1, LION_CT_BITSET=2, LION_CT_RUN=3,
                              LION_CT_NARROW=5 (§38); 4 is a sparse segment (§13) */
    uint8   flags;         /* a NARROW's width (§38); reserved, 0, otherwise */
    /* payload follows immediately (no padding; header is 8 bytes) */
} LionContainer;

Payloads (all little values are host-endian uint16/uint64, like every other PG on-disk structure): - ARRAY: uint16 lo[cardinality], strictly ascending. Max cardinality LION_ARRAY_MAX_CARD = 2048. - BITSET: uint64 words[512] (4096 bytes, fixed), bit i set ⇔ lo i is a member. - RUN: uint16 nruns; struct { uint16 start; uint16 len_minus_1; } runs[nruns], runs ascending and non-adjacent (merged). Max LION_RUN_MAX_NRUNS = 1023 (payload ≤ 4094 bytes). - NARROW (§38, format version 8): uint64 words[64 * k] (512 * k bytes), k words a heap block, k the width in flags, 1 .. 5 at 8K, for a container whose members are all at offsets below 64 * k: member lo is bit (lo >> 9) * 64 * k + (lo & 511). A BITSET is the same layout at the full width, 8 words a block. - LION_CONTAINER_MAX_SIZE = 8 + 4096 = 4104 bytes. Invariant: every container ≤ this size.

Representation policy - Insert into ARRAY beyond 2048 members ⇒ convert to BITSET. Insert into RUN that would exceed 1023 runs ⇒ convert to ARRAY if the members, the new one included, fit one (≤ 2048), else BITSET. - Remove from BITSET leaving ≤ 2048 members ⇒ convert to ARRAY. Remove from RUN that would exceed 1023 runs (split) ⇒ the same: ARRAY if the members fit one, else BITSET, which the removal shrinks to ARRAY if it can. Cardinality may reach 0; the caller deletes empty containers. (The RUN insert used to go straight to BITSET, 2026-09-25 review: 20 runs of 10 (a 90-byte RUN) plus 1003 scattered inserts are 1023 runs of 1203 members, and the next isolated insert made a 4104-byte BITSET where a 2416-byte ARRAY holds the same - an ARRAY that fits is never larger than a bitset - and the insert path does not optimize afterwards, so the bitset stayed, and a 4104-byte item cannot share an 8K page with another. The split already ended as an ARRAY, through a bitset; it now converts the same way the insert does.) - lion_container_optimize() picks the smallest of the four representations, NARROW of the narrowest width that holds the members (§38). It is called at bulk build time for every container, by VACUUM after modifying a container, and by an insert for every container it writes as a new item; an insert that adds the member in place (below) keeps the representation, so a container grows by its slack between two of them. A NARROW given a member past its width widens to the narrowest width that holds it, and one shrunk to 256 members a word of its width (LION_WIDTH_ARRAY_CARD(k)) or fewer is an ARRAY. - Mutators operate on a caller-supplied buffer of LION_CONTAINER_MAX_SIZE bytes. Page code copies a container out of the page into such a buffer, mutates, and writes it back (in place when it fits). The one exception is growth in place: the hot-key insert (lion_insert_container_inplace()) and its redo (LION_OP_CONTAINER_ADD, §25) call lion_container_add() on the page item itself, whose allotted length is only its size plus slack. What they rely on, and all they rely on, is that add() writes at most 2 bytes past the size of an ARRAY below 2048 members, 4 past a RUN below 1023 runs, and nothing past a BITSET or past a NARROW given a member within its width, and never changes the representation of any of them; the conversions above happen only at those limits, which the in-place path checks and refuses - both it and its redo through one table, lion_container_inplace_need() (§38). The sparse insert has the same shape: it writes exactly the lion_sparse_insert_size() bytes the in-place path and its redo check first, at most 11 past the segment (§13). Stated in lion_container.h and lion_sparse.h, and tested on exact-size heap allocations (test/unit).

Damaged containers (2026-09-25 review). A container read from disk is data, and a reader checks its header and its size (lion_inline_fetch()) - not its payload: lion_container_check() costs as much as using the container (0.8 us for a BITSET, a popcount of all 512 words, which is what an and-cardinality of two bitsets costs; 0.12 and 0.14 us with the AVX2 kernels below), and the count engine reads millions per query. The library used to trust the payload as far as its own buffers went, and a review found, with AddressSanitizer, three ways a container that passes the size check wrote outside one: an ARRAY member of 32768 or more indexed a bitset word past the 512th (65535 wrote 4 KB past a 4104-byte stack buffer), a run reaching past 32767 made lion_container_to_array() write past its 32768-entry output, and a BITSET whose header understated its members overflowed the 2048-entry array VACUUM’s shrink-to-ARRAY extracts into. Every function is now memory-safe for ANY payload behind a header of a valid type, at the cost of a mask or a comparison where the payload meets an index:

  • bitset positions are masked into range (lo & LION_LO_MASK); a run’s last value is clamped to 32767 and a run that starts past it is empty; a RUN is walked strictly ascending, skipping what an earlier (overlapping) run covered, so no container ever yields more than 32768 values;
  • member and run counts are clamped to 2048 and 1023 wherever they size a loop, so nothing reads more than LION_CONTAINER_MAX_SIZE bytes of a container whatever its header claims - which is what makes VACUUM’s work buffer, filled by the line pointer’s length rather than the header’s, safe - and a mutator rewrites such a claim to what it uses before it starts, so what it leaves behind is never larger than LION_CONTAINER_MAX_SIZE;
  • the header’s cardinality is a claim, not a bound: an extraction into a fixed array is bounded by the array, and a representation change the claim allows but the payload contradicts (to ARRAY, from a BITSET that holds more than 2048 members) is not made, which leaves the container as damaged as it was for verify() to report rather than silently shorter;
  • every lo value handed to a caller is below 32768, which the callers' per-block arrays rely on, and so is every member a result of the set algebra stores, because a result is handed on.

For a well-formed container all of this is a no-op and every result is byte for byte what it was; for a damaged one the results are unspecified but deterministic, which WAL replay needs. Assert() states only the caller’s side of a contract, never what a container’s bytes say, so an assert-enabled build does not stop in the library on a damaged page either. Cost, in instructions (callgrind, -O2): and-cardinality and membership tests of every type pair within 0 - 8% of what they were (a RUN against a bitset pays six per run; against an ARRAY or a RUN nothing, because those merge loops only compare run ends and may take them unclamped), iteration one per ARRAY member, and materialising a 1000-member ARRAY 1800 against the 260 of the memcpy() it was. The unit tests feed every function hand-made and random damaged containers in exact-size heap buffers (test/unit, “damaged”), which a -fsanitize=address build checks to the byte. What remains the READER’s job is the size check itself: a container used straight from a page must lie inside its item, as lion_inline_fetch() checks, because reading up to 4104 bytes from the start of a short item can leave the page.

The 2026-09-27 review found the readers outside the library skipping that job, and code outside it reading containers without the library’s masks. So: every reader of a posting-tree leaf - the count cursors and the copies they materialize, the bitmap scan, VACUUM, the insert path and the page searches - takes its items through lion_page_item_fetch(), which checks the line pointer (a normal one, inside the page’s item space at a MAXALIGNed offset, the test verify() and amcheck make), the item kind, the size the header gives against the line pointer’s length and 4104, and that a segment is not empty (§13), and raises INDEX_CORRUPTED naming the block otherwise; an unchecked item used to be read wherever its line pointer and header said, up to 256 KiB past the page image. And the count engine’s OR of k containers into a bitset image and its visibility-map block mask are the library’s own (lion_container_or_into_bitset(), lion_container_block_mask(), unit-tested with the rest): its copies of them took an ARRAY member or a RUN past 32767 as it was, which wrote up to 12 KiB past a 4 KiB image, and shifted by 64 or more, which is undefined, could drop a block with members from the mask - its rows then counted without a look at the map - and at BLCKSZ 16K and 32K set flags past the engine’s per-block array. A WAL redo of the two in-place calls (LION_OP_CONTAINER_ADD, LION_OP_SPARSE_INS, §25) checks its item the way the writer did before it logged the call.

The 2026-09-29 review found the WRITERS trusting line pointers: they copy, move, overwrite and delete items by them inside a WAL record, which in rmgr mode is a critical section (§25). A line pointer that claimed 4000 bytes for an 1816-byte ARRAY passed every reader, and the next push-down of its set copied the item by that length, failed half way through its record and PANICked - and again at the first insert after recovery; one that reached into the next item within its slack had the in-place insert grow the container into that item’s header, logged. So lion_page_item_fetch() also bounds what the line pointer claims past the item’s size by LION_ITEM_SLACK_BOUND, verify()’s rule, and a writer checks what its record is going to do before the record opens: every item of the page inside the item space and no two overlapping (lion_page_check_items(), before a push-down, a split, VACUUM’s filtering of a page and a directory split), the one item it overwrites or deletes in place alone in the bytes the page allots it (lion_page_check_alone(), before the in-place insert, a replacement and VACUUM’s delete - and for the CHAIN entry that every record of its posting set’s insert or VACUUM rewrites, before the first: lion_insert_chain(), lion_vacuum_entry_copy()), and the directory entry a record rewrites a whole one, of exactly its size when it is a CHAIN entry (lion_page_entry_fetch()). Each is an ERROR (INDEX_CORRUPTED), and verify() reports an overlap too. test/sql/corrupt_items.sql.

The 2026-10-18 review’s damaged pages added two checks to that list. The entry a record rewrites is exactly one of INLINE and CHAIN (lion_page_entry_fetch(), verify()’s rule): every writer picks its path by those flags while the header form follows CHAIN alone, so an entry flagged both, at a CHAIN entry’s length, passed every check, went down the INLINE path of an insert of its key and was rewritten, damage and all, in a record. And a page a record only ADDS an item to has its header checked first (lion_page_check_header(), before the add of a directory entry, of a posting leaf’s item and of a posting downlink; lionvec_check_page() for every lionvec page, §44.5): PageAddItemExtended() answers a header it finds damaged with a PANIC of its own, in either WAL mode, and the buffer manager’s check of a page it reads (PageIsVerifiedExtended()) lets in a pd_lower below the end of the header - one of the things it PANICs on - with which the page reads as one with no items and room to spare. test/sql/corrupt_records.sql and test/sql/vector_corrupt_records.sql, which damage the file and have the server read it back by moving the index to another tablespace (ALTER INDEX ... SET TABLESPACE copies the FILE, after writing out only what is dirty), and so run on PostgreSQL 16 as well.

Whole-bitset kernels (2026-09-30). Every pass over all 512 words of a BITSET that counts what it sees is one call of a fused kernel (lion_container.c, “whole-bitset kernels”), which reads the operands, writes the result if the operation has one and counts, in the same pass:

pass what callers
COUNT a BITSET’s cardinality lion_container_bitset_recount(), lion_container_check(), the count engine’s OR images (lion_bits_to_container(), through lion_container_image_cardinality()), lion_container_add_many()
RUNS the number of its runs lion_container_optimize() of a BITSET, and a RUN rebuilt by remove_if()
AND_COUNT the cardinality of a BITSET x BITSET AND, nothing stored lion_container_and_cardinality()
AND, OR, ANDNOT the result written and counted lion_container_and(), _or(), _andnot() and their _raw twins, when the right operand is a BITSET: two BITSETs read straight from their payloads, a RUN or an ARRAY on the left filled into the result first
OR_NEW a BITSET ORed into one in place, the new members counted lion_container_or_inplace() (§32’s dense unions)

Before, the AND of two BITSETs was a 4 KB copy of one, a pass ANDing in the other and a pg_popcount() of the result, and_cardinality() an AND into a 4 KB stack image and a pg_popcount() of it, and the run count and the in-place OR a pg_popcount64() a word - which on x86-64 before PostgreSQL 19 is an indirect call, as is every word of pg_popcount() itself before 17. There are three implementations, picked once per process at the first call: a static function pointer started at a resolver, which asked __builtin_cpu_supports() and overwrote the pointer with its choice, as PostgreSQL’s own pg_popcount64 does - and since 2026-10-26 every pass dispatches on the extension’s one choice instead, lion_simd_current(), made the same way and to the same kernel on every CPU (§50, “One choice of implementation”):

  • AVX2 (x86-64): Muła’s popcount - each byte’s count looked up a nibble at a time in a 16-entry table with VPSHUFB - summed in bytes over 16 vectors, at most 128 a byte, before one VPSADBW;
  • POPCNT (x86-64 without AVX2): a word at a time, four sums;
  • portable, on every other target and compiler and under make LION_NO_SIMD=1: the same word-at-a-time loop where __builtin_popcountll() is an instruction (CNT on aarch64, which clang vectorizes with NEON), and what each caller did before - a pg_popcount() of what it counts - where it is not (x86 built for its baseline ISA, where the builtin is libgcc’s table).

The x86-64 kernels are GCC’s and clang’s __attribute__((target("avx2"))) and ("popcnt"), so the extension builds with the default -march and runs on any x86-64; __builtin_cpu_supports("avx2") also checks that the OS saves the AVX registers. No AVX-512, on purpose: Intel’s client CPUs lack it, AMD’s Zen 4 runs it as two 256-bit halves, and outside L1 a pass waits on the cache rather than the popcount - with the operands cycling through 2 MB the AVX2 kernel is within a tenth of the POPCNT one - so a fourth implementation to test would buy little beyond a hot L1. (pg_popcount() itself uses VPOPCNTDQ where it finds it, from PostgreSQL 17 on, and the portable kernel on x86 counts through it.) lionvec’s plane-1 kernel is the case this leaves out - a query’s four bit-planes, 768 bytes at 1536 dimensions, stay in L1 while every centroid’s and row’s sign bits pass them - and takes VPOPCNTDQ where the CPU has it, on 256-bit vectors (§44.3, “Kernels”). Every implementation gives the same counts and writes the same words, so the same representations are chosen and plans do not change (the golden plans of the regression suite are identical).

Measured through the library (a harness beside it; Xeon at 2.8 GHz with AVX2 and no VPOPCNTDQ, gcc -O2, PostgreSQL 16’s libpgcommon; one pair of bitsets of 30% and 50% density in L1 / 256 pairs, 2 MB), in ns:

call before AVX2 POPCNT
and_cardinality() 775 / 878 144 / 255 202 / 277
and_raw() (or_raw(), andnot_raw() alike) 914 / 1182 133 / 390 211 / 407
and(), optimizing 1603 / 1814 357 / 623 601 / 809
or_inplace() 876 / 887 138 / 252 318 / 333
bitset_recount() 636 / 641 121 / 139 163 / 166
optimize() of a BITSET that stays one 644 / 651 184 / 190 347 / 351

PostgreSQL 17 and 18 run POPCNT inside pg_popcount(), which about halves the “before” of what counts through it (and_cardinality() 404 / 500 ns, bitset_recount() 290 against an -O1 build of 18’s libpgcommon). End to end, on the assert-enabled PostgreSQL 18 build (-O1), 10M rows of three random 0/1 columns with a lion index each, all-visible, 845 container keys, a count’s backend CPU time (600 runs in one session, medians of three rounds alternating the builds, on a shared and busy machine): WHERE a = 1 AND b = 1 3.85 -> 2.68 ms, AND c = 1 too 6.12 -> 4.12 ms, a = 1 OR b = 1 4.22 -> 3.27 ms - a fifth to a third less.

Left as they were, with a reason each: a partial range’s count (bits_range_cardinality(): a RUN against a BITSET a run at a time, range_cardinality(), remove_range(), a RUN ORed in place), which is a few words per call; the count near the cap in bits_extract_array(); and the bitmap scan’s window, which ANDs an image into another and then hands it to lion_bits_to_container(), whose count is the kernel’s now but a second pass (lion_scan.c, lion_source_window_next()). The cost model is unchanged: its fitted constants (§10, “The merge”: a bitset costs what an array of a thousand members does, 800 ns) were measured with the old passes and now overcharge a dense count somewhat, which a refit may take up with the rest of the model.

lion_container_simd_force() (a -DFRONTEND build only) makes the library use a given implementation - through lion_simd_force() since §50, so every kernel of the extension that dispatches on the choice follows it: the unit tests force each one the build and CPU have, check every pass against a reference that counts a bit at a time - 22 x 22 pairs of patterns: empty, full, densities from 0.05% to 99.95%, the first or last bit of every word, runs across words and across the AVX2 kernel’s 16-vector blocks - in exact-size heap buffers at every 8-byte alignment modulo 32, check what the public functions make of them (counts, words, and the representation the optimizing forms choose, against the policy above), and rerun the set algebra of every type pair under it. test/unit/container_test_nosimd is the same program built with -DLION_NO_SIMD (and -mpopcnt on an x86-64 host, which makes its portable kernel the fused loop other architectures run).

Full API: src/lion_container.h. Unit tests: test/unit/container_test.c (make unit), which must cover every type transition, boundary cardinalities (0, 1, 2047, 2048, 2049, 32767, 32768 members; 256k - 1, 256k and 256k + 1 members and offsets 64k - 1 and 64k for a NARROW of every width k), run merging/splitting, and set algebra against a brute-force 32768-bit reference.

The kernels above take a word count and run a pass compiled for a BITSET’s 512 words or one for a NARROW’s 64k (§38); every NARROW payload bit is a legal member, so its cardinality and its width are the claims a damaged NARROW can make wrongly: the cardinality is handled as a BITSET’s is, and a width the build does not have is refused.

4. Page layout (module lion.h, implemented in lion_pages.c, lion_meta.c, lion_state.c, lion_entry.c and lion_posting_put.c)

All pages are standard PG pages (PageInit) with a special area:

typedef struct LionPageOpaqueData
{
    BlockNumber rightlink;     /* next page in this chain, or InvalidBlockNumber */
    uint32      minckey;       /* container pages: smallest ckey on page (0 if empty) */
    uint32      maxckey;       /* container pages: largest ckey on page (0 if empty) */
    uint32      owner_hash;    /* container pages: owner stamp of the entry that owns the chain (§18, §21) */
    BlockNumber owner_head;    /* container pages: head block of that chain (§18) */
    uint16      flags;         /* LION_PAGE_META | LION_PAGE_BUCKET | LION_PAGE_CONTAINER,
                                * plus LION_PAGE_DELETED for a freed page (§18) */
    uint16      page_id;       /* LION_PAGE_ID = 0xFF87, for identification by inspection tools */
} LionPageOpaqueData;          /* 24 bytes */

Superseded in part by §24 (format version 6). An entry tuple carries the KEY COLUMN it belongs to in a uint16 attno of its header (§21, “Entry tuples”). An attno of 0 belongs to the minus-infinity downlink alone.

Block 0: meta page. Payload struct LionMetaPageData { magic 0x52424931, version 6, offset_bits, container_bits, nbuckets, inline_limit, unused padding to 64 bytes }. (Version 7, §32: an index with summary posting sets, which records them in summary_cols and summary_tids, two words that were reserved; an index without them is still version 6, byte for byte. Version 8, §38: an index with NARROW items, with or without summaries, which builds before it refuse. Version 9, §13: sparse segments of delta varints. Version 10, §21: compact entry tuples. Version 11, below: keys of up to 2047 bytes. Version 12, §21: key-less first downlinks. Version 13, §17: the NULLELEMS entry. Version 14: the EMPTYDIM and EMPTYINNER entries (§17), and NARROW items wherever a container goes, with no meta page flag for them (§38) - which every index this build writes is and the only one it reads; whether there are summaries is summary_cols.) Version 2 was the first that indexes NULL keys (§14); version 3 (§18) added owner_hash/owner_head, which grows the special area from 16 to 24 bytes and therefore moves every item on every page. An index of an older version is structurally readable by nothing in this code, so opening one is an ERROR that asks for a REINDEX rather than a wrong answer.

Superseded by §21 (format version 4). Everything in the next two paragraphs - the bucket directory, nbuckets, lion_bucket_of() - is gone: the entry directory is a B-tree keyed by the index key, entry tuples live on its LEAVES (which keep the flag name LION_PAGE_BUCKET), and the meta page holds root/height/dirpages where it held nbuckets. The entry tuple and everything below it are described below and in §21 (“Entry tuples”).

Blocks 1 .. nbuckets: bucket head pages. Bucket b for a key with 32-bit hash h is h % nbuckets (lion_bucket_of() in lion.h, the single place that mapping lives); its head page is block 1 + b. A bucket is a rightlink chain of bucket pages holding entry tuples. nbuckets is NOT a power of two: ambuild sizes it from the bytes the entries need (§5), and the buckets reloption means an exact count.

Entry tuple (an item on a directory leaf): a header - flags (LION_ENTRY_INLINE or LION_ENTRY_CHAIN, plus LION_ENTRY_NULLKEY for the NULL entry, §14), the key length, the key column, and the posting set’s container and TID counts, with the head and tail of its container pages for a CHAIN entry - then the key, then for an INLINE entry its containers back to back in ascending ckey, their total bytes the item’s size less the payload offset. §21 (“Entry tuples”) gives the byte layout, which has a short header for INLINE entries and a longer one for CHAIN entries.

Key data is stored with datumCopy semantics, except that a by-value type is stored at its own width (typlen bytes); fixed-length by-reference types as typlen bytes; varlena as a detoasted, 4-byte-header varlena, or a compressed one when that is too long (below); cstring as strlen+1 bytes. Keys larger than LION_MAX_KEY_SIZE = 2047 bytes, the 4-byte header included, are an ERROR, except where an opclass files a longer one under a hash: tsvector_ops a lexeme of more than 2043 bytes (§17, “Long lexemes”), array_ops a text, varchar or bytea element of more than 2043 bytes (§17, “Long elements”), jsonb_contains_ops any key past the limit (§42.1). (The limit was 2000 bytes until format version 11, which a tsvector lexeme of up to 2047 bytes passes; a version 10 index is refused, since its jsonb keys of 2001 to 2047 bytes are hashes that this build would look for whole.) Entry lookup compares the key’s hash, computed from the key and never stored, then calls the opclass equality operator’s function (strategy 1 of the opfamily, looked up once per relation and cached in rd_amcache) with the index collation.

Long keys

btree stores a long value compressed (index_form_tuple() compresses a varlena past TOAST_INDEX_TARGET) and checks its ~2704-byte limit against the compressed size, so it takes a value of any length that compresses well enough: repeat('abc', 20000) in 699 bytes, a char(2500) of 'code1' and its blanks in a few dozen, a 5000-digit numeric in 49. A lion key that a plain 4-byte header would put past LION_MAX_KEY_SIZE is stored the same way, COMPRESSED (lion_key_form(), lion_state.c): an inline compressed varlena, which every function of the type detoasts as it detoasts a heap value, so the key compares, hashes, prints and is emitted (GROUP BY, min/max, LionOrdered) as the value it is. Nothing else in the index looks at a key’s bytes as the value: an entry is found by the opclass’s comparison and equality, and the bytewise tail of the order (§21) and the owner stamps (§18) only need the stored bytes to be the entry’s own.

  • A key that fits plain is stored plain, as before, so an index of short keys is unchanged.
  • One that arrives compressed (inline in the heap) and fits as it is is stored as it came, pglz or lz4; any other is compressed with pglz, asked only to fit - not to save the quarter TOAST asks of a heap value - and to give up as soon as its output passes the limit, so an incompressible value of a megabyte costs a couple of kilobytes of work. Two equal values may therefore be stored in different bytes, which the directory already allows (citext’s ‘Alice’ and ‘alice’ are one entry).
  • A stored compressed key is checked like any other (§21, “Readers”): an inline compressed header whose length is the key’s, a method this build knows, and a raw length past the limit, since no shorter key is ever compressed.
  • A value still over the limit compressed is the ERROR btree raises, worded alike, naming the index and, from an insert or a build, the heap row (lion_key_prepare(), which the insert and the build call on every key before they write anything): index key size N exceeds lion maximum 2047 for index "...", Index row references tuple (b,o) in relation "...". btree’s limit is about 650 bytes larger; an incompressible value between the two is the one value btree takes that lion does not (below).
  • Not done: incompressible ordered keys past the limit. The pivot of an internal page is a key, and three of them and a high key must fit a page (§21, “Split”), which is what sets the limit. A hashed key - a prefix and a hash, rechecked - keeps neither the type’s order nor its value, so for an ordered scalar column it would have to live in an order of its own (a kind after VALUE), every range and the ordered walks (LionOrdered, GROUP BY, count(DISTINCT), min/max, the aggregates over keys, the semi and FK joins, the summaries) would have to merge it in from the heap or decline at run time once the index held one, and the planner would have to cost that. A multi-key column needs none of it: its keys are only ever looked up, which is why text array elements are hashed (§17, “Long elements”) and scalar keys are not.

INLINE entries hold their containers in the entry tuple while the payload is ≤ inline_limit bytes (reloption, default 4096 = LION_MAX_INLINE_LIMIT). When an insert would exceed that, the entry spills: allocate a container page, move the containers there, set head = tail = that page, flags = CHAIN (keeping every LION_ENTRY_RESERVED bit: a key-less entry that loses its flag becomes an ordinary entry with a zero-length key that no reader looks for, so the next row of its kind starts a second one - the 2026-09-20 review reproduced exactly that), and shrink the entry tuple. Entries never convert back from CHAIN to INLINE. The default is the maximum on purpose: a CHAIN posting set owns whole container pages, so a key whose set is a few hundred bytes costs a whole page once it spills (§13 measured a 20000-key index at 164 MB with a 1024-byte limit against 35 MB with 4096).

Superseded in part by §22 (format version 5). A CHAIN entry’s container pages are the LEAVES of a B-tree over container keys now: the entry’s head is that tree’s ROOT, and internal posting pages (the same page kind, told apart by level > 0) hold downlinks above them. Everything the next three paragraphs say about a leaf - the items, the ordering, min/max, splits, free space, growth slack - is unchanged, and so is the right-link list they form; what changes is only how a container key is turned into a block, and §22 says so where it does.

Container pages (per key, CHAIN entries): items are LionContainer structs, ascending ckey within a page; all ckeys on page P are smaller than all ckeys on P.rightlink. minckey/maxckey in the special area are maintained on every change. Page P owns free space like any heap/index page; use PageAddItemExtended (with explicit offset to keep order), PageIndexTupleOverwrite (handles size change), PageIndexTupleDeleteNoCompact/PageIndexMultiDelete + PageRepairFragmentation as needed.

Locating the page for a ckey (lion_chain_find_page): a DESCENT of the posting tree since §22. The insert path still tries the append case first - the set’s last leaf, taken with the exclusive lock the insert needs anyway - and falls back to the descent for anything else; an empty tail has minckey 0 and must not be trusted, so it falls through too.

Growth (lion_chain_put_container): overwrite in place if the page has room; otherwise split page P: move the items at/after the insert position to a freshly allocated page N linked after P; if the container still does not fit on P, place it alone on a second new page M linked between P and N (P → M → N → old right). This guarantees progress in one WAL record (P, N, M, entry page = 4 buffers, the GenericXLog maximum). Items only ever move right and only to pages that are linked immediately right of the page they came from - which is the property the §9/§11 interlock needs, and it holds whether the new page was extended onto the end of the relation or recycled out of the free space map (§18). Since §22 the split also puts a downlink for each new page into the parent, in a record of its own, with the page to its left flagged LION_PAGE_INCOMPLETE_SPLIT in between; and a split of the ROOT is a push-down that keeps the root at its block. A whole posting set is freed when its entry is deleted (§18); individual pages of a live one are never unlinked, so a leaf VACUUM has emptied stays in the list and still owns the range its parent gave it. INLINE payloads are packed without padding, so containers inside them are unaligned: read them with lion_inline_fetch() into an aligned buffer; never cast into the payload. Containers stored as page items are MAXALIGNed and may be used in place.

Free space accounting: a container page is “full” for a given container when PageGetFreeSpace(page) < MAXALIGN(size) + sizeof(ItemIdData). Splits guarantee progress because every container ≤ 4104 bytes and a fresh page holds at least one.

Growth slack inside an item (v1 write wave). An item on a container page may be allotted MORE bytes than its header needs, so that the next few members can be added inside it. An item therefore has two sizes, and both are needed: lion_item_size() is its LOGICAL size, derived from the header, and is what every reader uses to find where it ends; ItemIdGetLength() is its ALLOCATED length, which is what it may grow to in place. lion_item_alloc_size() chooses the second whenever an insert writes an item: MAXALIGN(size) plus size/8 clamped into [LION_ITEM_SLACK_MIN = 8, LION_ITEM_SLACK_MAX = 64] bytes, capped at LION_CONTAINER_MAX_SIZE, dropped entirely when the page has no room for it, and never for a BITSET (4104 bytes is already the maximum an item can be) or a NARROW (§38: a member it holds changes no size, and one it does not widens it, which no slack would hold). The slack is a fraction of the item rather than a fixed 64 bytes on purpose: a key whose TIDs are spread thinly owns dozens of ~50-byte items per page, and 64 bytes of slack each would nearly halve what a page holds; MAXALIGN padding, which the page spends either way, is part of the slack and therefore free. Slack bytes are zeroed when the item is written, so that the page image is deterministic and a GenericXLog delta of a later change stays small. An item that is written back over itself keeps the allocated length it already has whenever that is still enough (and not wastefully more), because then PageIndexTupleOverwrite() moves no other item on the page at all; the exception is an item that has far more room than it can use - a segment replaced by the container one of its container keys was promoted to - which gives the excess back. (2026-10-27: an INSERT’s rewrite keeps the length only while it leaves the item LION_ITEM_SLACK_MIN bytes to grow into, and takes fresh slack otherwise when the page has room. A RUN whose new run used up its slack was rewritten at the same length, with no room, by every later member of that run - an extended run needs no byte but the in-place add asks for a run’s four whatever it does - so a quarter of the inserts into a sliced column’s existence slice took the general path: §51, “Insert cost, revisited”.)

An insert whose item has room inside it adds the member there and nowhere else (lion_insert_container_inplace(), lion_insert_segment_inplace() in lion_insert.c): the item keeps its offset and its allotted length, no other item on the page moves, minckey/maxckey change only when a segment’s range really grew, and the WAL delta is the handful of bytes that changed. A BITSET always qualifies, a NARROW does for a member within its width (§38), an ARRAY needs 2 spare bytes and a cardinality below LION_ARRAY_MAX_CARD, a RUN needs 4 and a run count below LION_RUN_MAX_NRUNS (both would otherwise turn into a bitset), and a sparse segment needs the bytes lion_sparse_insert_size() gives for the pair (one varint at the tail, at most 11 bytes anywhere, nothing once its stream would pass 255 bytes) and a container key that stays below LION_SPARSE_THRESHOLD members (else the key is promoted, which is not an in-place change). Otherwise the general path runs - copy out, mutate, write back, split if needed - and leaves fresh slack behind, which is where the slack of a growing key comes from in the first place. Inserts add slack, and the build only when asked: VACUUM writes items at their exact size (lion_chain_put_items_locked() is the no-slack form of lion_chain_put_items_locked_ext()), and so does ambuild by default, because a bulk-built index is read-mostly and the space would be pure loss. The build_slack reloption (auto | on | off, default off) has the build allot every posting item lion_item_alloc_size(), the slack an insert would leave, zeroed past the container (lion_builder_spill()), and every INLINE entry lion_entry_alloc_size() (below; lion_builder_flush()). auto is on for an index built for generic WAL and off for one built for the resource manager; what it buys in each mode is measured below (“Slack at build time, measured”). Nothing reads the option after the build: a bulk-built item with slack is the item an insert would have written, and every reader, VACUUM and verify() already take it.

Slack is bounded by LION_ITEM_SLACK_LIMIT so that it can never be mistaken for corruption: verify() accepts an item whose allocated length is up to that much above its logical size, and rejects anything else (including any item above LION_CONTAINER_MAX_SIZE, which is what lets every reader copy an item out by ItemIdGetLength() into a fixed work buffer). lion_index_stats() reports the total as slack_bytes; container_bytes counts logical bytes only, and free_bytes comes from PageGetFreeSpace(), which does not see intra-item slack at all.

Growth slack inside an INLINE payload (implemented; §25 left it undone and said what shape it should take). An INLINE entry’s payload is packed, so adding a member to a container inside it moves every byte above that container - and, before this, the entry itself was rebuilt at its new length and written back with PageIndexTupleOverwrite(), which moved every OTHER entry on the directory leaf as well. That is 1.5-2.9 KB of WAL per insert in generic mode, where the record is a byte diff of the page, and a memmove of the page tail in both modes. An entry an insert rewrites is therefore allotted a few bytes MORE than its payload needs, by the same rule as an item (lion_entry_alloc_size(): paylen/8 clamped into [LION_ITEM_SLACK_MIN, LION_ITEM_SLACK_MAX]), and the next insert whose payload still fits inside them keeps the entry’s allocated length exactly as it is: no other entry on the leaf moves, and the record covers this entry alone.

The slack needs no header field, and deliberately does not use the two spare bytes of §24: it is ZEROED, and lion_inline_fetch() already stops at the first zero item header because no item kind is 0. That is the convention VACUUM’s shrink-in-place has used since §18, so the two kinds of slack meet in one entry with no further rule, and verify() bounds both at LION_ENTRY_SLACK_BOUND and requires every byte of them to really be zero.

Two caps keep it from changing anything else. Slack may never push the STORED payload past inline_limit, and it is never worth a leaf split: the entry grows only into the free space the page already has (PageGetExactFreeSpace()), else it is written at its exact size and lion_dir_place() splits as it always did. And the SPILL decision is made on the payload the key really uses, before the allocation is chosen, so slack can never make a posting set spill early - the test in lion_insert_inline() is on newlen, not on what the entry is allotted. As for items, inserts add it, ambuild only under build_slack (with the same inline_limit cap, and only to an entry that keeps its set inline), and VACUUM keeps the length it finds. lion_index_stats() reports it as inline_slack_bytes, apart from slack_bytes, because it is an entry’s and not an item’s.

Measured (release, 1M rows, 10,000 single-row inserts into one index, §25’s harness): the c20k column falls from 2918.7 to 173.0 bytes of WAL per insert in generic mode and from 422.4 to 180.5 in rmgr mode, and c1m from 1682.2 to 106.8 and from 124.1 to 82.0. The 8-index portfolio insert falls from 53.53 to 9.48 MiB (generic) and from 29.21 to 9.08 MiB (rmgr) in steady state, and the index is the same size afterwards to the byte - the slack lands in space the leaves already had.

Slack at build time, measured. PostgreSQL 18.6 release build, wal_consistency_checking off, the synthetic 5M-row table and update wave of §22 (“Posting leaves with room, measured”): six indexes on columns of 2 to 1M keys, one row in a hundred moved to the next key of the 20-key column, which inserts into every index inside a container or entry the build wrote. Bytes of index WAL per updated row without full-page images, and index size at build -> after the wave:

generic WAL       build_slack = off              build_slack = on
20 keys           1167.7 B  10.8 -> 18.8 MB      434.6 B   12.0 -> 12.0 MB
200 keys          2402.2 B  13.1 -> 25.2 MB      104.7 B   14.8 -> 14.8 MB
2,000 keys        2401.0 B  32.9 -> 50.0 MB      335.5 B   47.9 -> 59.4 MB
20,000 keys       1493.1 B  18.1 -> 19.3 MB      402.6 B   18.9 -> 18.9 MB
1M keys           2289.7 B  81.6 -> 81.6 MB       65.0 B   90.6 -> 90.6 MB
six indexes                 158.1 -> 196.4 MB              185.7 -> 197.1 MB

In generic mode the record is a diff of the page, and without slack the first member added inside a bulk-built entry or container moves everything stored after it on the page; with slack the change stays inside the item. The WAL of the wave falls 2.7-fold on the 20-key column and up to 35-fold on the 1M-key one. What it costs is space, at build: 11% on the 1M-key column, whose entries are small and each get the 8-byte floor, and 46% on the 2,000-key column, whose sets fill their leaves almost exactly, so that the slack pushes many of them onto another leaf - and which still splits under the wave. The 20-key column’s in-place insert into an ARRAY still diffs the members it shifts, hence 434.6 B. posting_fillfactor alone does not help generic mode: at 90 the leaves stop splitting (six indexes 160.9 -> 162.1 MB) but a rewritten item that moves the rest of its leaf diffs as much as before, 1102.6 / 2857.4 / 2810.2 B on the 20-, 200- and 2,000-key columns. Under the resource manager, where the record is the item’s own change either way (§25’s DELTA and SPLICE), slack at build changes little - the 20,000-key column’s entry records stay at about 166 B - and costs the same space, which is why auto leaves it off there. It stays off by default in both modes so that an index is built the same whatever its WAL mode, and an append-only or read-mostly index does not pay for room it never uses.

Page allocation: lion_alloc_page() takes a page - a recycled one from the free space map when one is safe to take, else a fresh block from ExtendBufferedRel - and lion_wal_init_buffer() initialises it and registers it in the caller’s record, so a crash cannot leave an initialised page that nothing links to; a page that nothing links to yet - the root of a set a spill makes, a new position tree - is initialised in a PAGE_INIT record of its own. The two are separate calls since §25: taking the page is the fallible half and has to happen before the record opens, because an rmgr-mode record is written inside a critical section. The recycling rule, what heaprel is for and why a chain’s head page never comes from the map are in §18. verify() reports DELETED pages as the ordinary free pages they are, unreferenced never-initialised or empty pages as WARNINGs (the next VACUUM’s sweep turns them into free pages) and anything else as an ERROR - on a primary only once the writers that were in flight are done and the page is still not where a search for it lands, since a live page nothing reached may be one an INSERT made behind the walk (§7, “verify() beside writers”).

5. Locking protocol (deliberately coarse in v0)

Two B-trees are involved, and this section is the protocol across them: the entry DIRECTORY, keyed by the index key, whose leaves hold the entry tuples (§21), and for every CHAIN entry a POSTING TREE keyed by container key, whose leaves are the container pages and whose root is the entry’s head (§22). What v0 decided and every format since has kept: one lock order, one WAL record per atomic step with an entry’s counters in the same record as the containers they count, and every writer of a key - aminsert and VACUUM - serialised on one page, the directory leaf that holds the key’s entry. That leaf is what the bucket head page of formats 1-3 was; the historical note at the end of this section says what went away with it.

A MULTICOLUMN index changes none of it (§24). One relation holds every key column’s entries in one directory, so one aminsert is n independent single-key inserts - one per column, each taking and releasing the leaf that holds ITS entry - and two columns of one index contend exactly as two single-column indexes would, except that they may land on the same leaf. The lock ordering below is unchanged because a directory leaf is a directory leaf whatever column’s entry is on it.

Lock ordering:

  • Directory pages before posting pages, never the reverse. Nothing waits for a directory lock while it holds a posting page; a READER also never asks for one while it holds a PIN on a posting page (§11’s deadlock rule for readers: VACUUM holds the leaf and waits for cleanup locks on that entry’s posting pages, and buffer locks have no deadlock detector). VACUUM’s pass 2 is the one path that holds a posting page and wants the leaf, and it only ever tries (ConditionalLockBuffer), letting go of the posting page before it waits (§11). Replay takes them the other way round (§25): a record lists the posting pages it changes before the entry’s leaf, and redo locks its blocks in that order and holds them to the end of the record. So on a hot standby no backend may hold a directory page while it WAITS for a posting page either - which no reader does, holding one page at a time; lion_index_verify() did until the 2026-09-29 review, and replay deadlocked against it for good (§7).
  • Within one tree, nbtree’s rules (§21 “Locking summary”, §22): a descent takes SHARE locks and releases the parent BEFORE locking the child, and takes the target leaf directly in the mode the caller needs, so no lock is ever upgraded; a searcher whose key a concurrent split has moved off the page it lands on moves right, which is where the split put it; on one level pages are only ever taken left to right; an ascent - a split inserting its downlink, a repair finishing an INCOMPLETE_SPLIT - holds the child while it locks the parent, which is exactly why a descent must not couple downwards.
  • A new page last. It is taken (from the free space map or by extending the relation, §18) before the record that initialises it opens, and it is always linked immediately right of the page whose split or spill allocated it - the property §9/§11 build on.
  • The meta page is read at relation open and cached; the directory root is cached in LionState and validated by the LION_PAGE_ROOT flag, so a lookup reads the meta page only when the root has really moved (§21). A directory split writes it (it counts dirpages) and a root split rewrites it, both taking it as the LAST buffer of their record; nothing else writes it.

INSERT (lion_insert.c: lioninsert() calls lion_insert_one() once per key column and, for a multi-key opclass, once per extracted key) 1. Hash the key and build the search key - kind VALUE, or a reserved entry: NULL, or one of the key-less entries of §17 (EMPTY, EMPTYDIM, EMPTYINNER, NULLELEMS) (§14, §17). Descend the directory (lion_dir_find()) and take the LEAF EXCLUSIVE; a write descent that lands on an INCOMPLETE_SPLIT page finishes that split first (§21, the repair rule). The leaf is held until the insert is complete: it is the page every writer of this key serialises on. 2. Look the key up on the leaf: binary search to its prefix run, then the opclass equality over the run (§21 “The order”). If it is absent, add an INLINE entry holding a one-pair sparse segment (§13) at its exact position (lion_dir_add_entry()); find-or-create is one serialised operation, because the leaf is held from the unsuccessful lookup to the insert - and for the rare prefix run that spans pages, the leaf it is entered from is held as a GUARD while the scan couples rightwards (§21 Operations). A leaf with no room splits: left, right, old right sibling and meta page in one record, the downlink into the parent in a second, the INCOMPLETE_SPLIT flag cleared in a third; a root split is one atomic record (§21). This is also the one path that makes the index grow a key, so the §17 cardinality guard is checked here: the entries on this leaf times the number of leaves, against max_entries. 3. INLINE: rebuild the payload with the member added. If it is at most lion_inline_max() - inline_limit, capped so that the entry never exceeds LION_MAX_ENTRY_SIZE (§21) - overwrite the entry: at its allotted length when the INLINE growth slack left by the last insert holds the new payload (only this entry’s bytes change, no other entry on the leaf moves), else at the new length plus fresh slack (§4). A grown entry that no longer fits its leaf splits the leaf (lion_dir_place()) instead of spilling, which the hash directory had to do. Otherwise SPILL: allocate one container page, move the containers there, make the entry CHAIN with head = tail = that page - the head of a posting set of one leaf and no level above it (§22), keeping every LION_ENTRY_RESERVED bit - and fall through to CHAIN. The leaf is still held. 4. CHAIN: find the posting LEAF that owns the ckey and lock it EXCLUSIVE. The tail is tried first (lion_insert_lock_chain_page()), with the exclusive lock the insert needs anyway, and kept when it is still the rightmost leaf, not incompletely split, and its minckey is at or below the ckey - which is where every insert into a growing posting set lands. Anything else releases it and descends the posting tree from head (lion_posting_search()); because the directory leaf held in step 1 serialises every writer of the key, no posting split can be in flight, the separators say exactly which leaf owns the ckey, and a write descent needs no move-right (§22). Then copy the item out, add the member - or let the pair join or start a sparse segment, promoting a ckey that reaches LION_SPARSE_THRESHOLD members to a container of its own (§13) - and write it back, splitting the leaf if it no longer fits: P → M → N in one record (P, M, N, entry leaf), a record per downlink, then the flags cleared; a split of the ROOT is a push-down that keeps the root at its block, so head never changes (§4, §22). The entry’s counters (ncontainers, ntids, tail) are written in the same record as the items. Fast path: when the item already on the page can take the member without changing the number of bytes the page has allotted it, the member is added directly in the page the WAL shim hands back (same locks, same single record carrying the entry tuple) instead of being copied out, mutated and written back with PageIndexTupleOverwrite. Nothing else on the page changes: same allotted length, same offset, same ckey, so min/max stay put (a sparse segment may extend its range, and then only min/max move). A BITSET always qualifies - it is 4104 bytes whatever its cardinality - and an ARRAY, a RUN or a segment qualifies when the item has growth slack (§4), which the general path leaves behind whenever an insert writes an item. Trying the tail first saves the SHARE acquisition a lookup-then-lock would take on the very same tail page just to read its minckey: two lock acquisitions per appending insert instead of three, all of them inside the leaf-lock window. It did NOT move the concurrent-insert ceiling of step 6 (643 tps either way, measured with the hash directory and generic WAL by swapping the two builds in one session), which is what step 6 explains; it is kept because it strictly removes work from inside that window, and an insert whose ckey is not on the tail pays the same number of acquisitions as before (an exclusive lock on the tail where a shared one would have done). The tail is good only for the in-place fast path (2026-09-25). An insert that needs the general path - which rewrites the item and may split the page - lets the tail go and DESCENDS, even when the tail owns its ckey, because the tail may be the right half of a split that never finished, and only a descent finishes one (§22 addendum, “an unfinished split and the writers that do not descend”). The descent lands on the tail again; what it costs is its own locks, one per level, on the general path only - never for a BITSET, and otherwise once each time an item’s growth slack (§4) runs out. 5. Every page modification goes through the WAL shim of §25 (lion_wal_begin(), lion_wal_register_buffer(), lion_wal_op(), lion_wal_finish()): one record per atomic step, registering at most 4 buffers - GenericXLog’s limit, which the resource manager’s records keep so that both modes share every call site; a split touches P, N, possibly M, and the entry leaf. In rmgr mode the record is written inside a critical section, so everything fallible - taking a new page above all - happens before it opens. 6. Release everything. Inserts whose keys live on different directory leaves do not block each other; inserts of one key serialise, and so do inserts of keys that share a leaf. Documented and accepted for v0, and measured for v1 with the hash directory and generic WAL (bench/write_micro.sh, 8 clients × 100 rows per transaction, fsync on, synchronous_commit on, 1 index):

   transactions/s, 100 rows each      1 client   4 clients   8 clients
   roaring, 2 key values                 387        629         643
   roaring, 1000 key values              371       1078        1890
   btree, 2 key values                   460       1128        1466
   no index, 2 key values                511       1170        2296

With a thousand key values the index side all but disappears (5.1× scaling, 82% of the no-index ceiling); with two it flatlines at 1.7×. The wait-event profile of the two-value case was 57% Buffer/BufferExclusive against btree’s 31%, so the buffer content locks of the two keys' pages were indeed where the time went - but shortening the hold is NOT what would fix it, and the numbers said why. A single-client insert cost ~7.7 µs of which the serialised part was the GenericXLog record: registering a buffer copies the whole 8 KB page and GenericXLogFinish() diffs it against the copy and applies the image, so a record covering the container page and the entry page moved ~48 KB of memory per single-TID insert, all of it inside both locks because the entry’s counters travel with the container change. Repeating the burst on an UNLOGGED table - no delta, no XLogInsert - lifted it only from 643 to 868 tps (btree 1466 → 2150, no index 2296 → 19538), so the cost was the page-sized copying rather than the logging. So the protocol that would shorten the leaf hold (release the leaf, walk and lock the posting page, then ConditionalLockBuffer() the leaf and re-read the entry, retrying from the top on failure, as ambulkdelete does in §11) was NOT implemented: it does not remove the record write from the window where both pages are held, and it adds an acquisition and a retry path. The structural fix was the custom WAL resource manager of §25, which logs the bytes that changed with no image and no diff: the same 8-client, 2-key burst went from 765 tps generic to 1275 rmgr (btree 1632, release build), past the unlogged ceiling above.

SCAN (lion_scan.c, amgetbitmap; §29 says what the plain index scan does differently) 1. Descend the directory with SHARE locks to the leaf that holds the search key’s entry - the probe resolved once per scan key type, which is also how a cross-type key is handled (§21 lion_probe_init()/lion_probe_find()) - holding one page at a time and moving right past a concurrent split. IS NULL descends to the column’s reserved NULL entry; IS NOT NULL that nothing else answers and the multi-key fallback walk the column’s whole run of leaves instead (beside quals that answer, IS NOT NULL is the NULL entry taken away, step 5), and a range the part of it between its bounds (§28), copying each leaf into backend-local memory and releasing it before its entries are emitted. 2. On finding the entry: INLINE → copy the payload out, release the leaf, emit. CHAIN → copy the entry out and release the leaf BEFORE any posting page is pinned (the reader half of §11’s deadlock rule), then descend the posting tree from head to its leftmost leaf and walk the leaves holding one SHARE lock at a time: copy the page - items and rightlink - under the lock, release, lock the rightlink. Splits only move items to a new page immediately to the right and require an EXCLUSIVE lock on the source, so a reader sees every container exactly once (it read the items and the rightlink atomically). The descent hands the first leaf back LOCKED, which is what keeps a root push-down from slipping in between. Between releasing the leaf and reading the posting pages the scan holds nothing of the set but its head, so VACUUM may free the set and an insert may take its pages back; the owner stamp on every page (§18) is what makes the walk stop there, and VACUUM frees a set only once it holds nothing any snapshot could see, so stopping loses nothing. 3. Emit each container into the TIDBitmap: for each of the ≤ 64 heap blocks the container covers, collect its offsets into an ItemPointerData array and call tbm_add_tuples once per heap block (never one call per TID). Return the number of TIDs. 4. A NULL scan key (col = NULL, a NULL array) returns 0 TIDs, and so does a NULL element of an array. SK_SEARCHNULL emits the NULL entry and SK_SEARCHNOTNULL every other entry of the column (§14); SK_SEARCHARRAY emits one entry per non-NULL element (§15). A multi-key opclass answers its own strategies through its extracted key tree (§17). 5. The planner may hand one key column more than one qual (tags && '{a}' AND tags && '{b}', b = ANY (x) AND b = ANY (y), an equality next to a range or a null test). The scan answers EVERY qual that is a set tree - an equality, a list, IS NULL, a multi-key query - and ANDs their trees, a column’s with each other exactly as with the other columns' (§24), so the bitmap holds the rows the quals select together (lion_scan_choose()). What is not a set tree is one WALK per column, taken only by a column without sets: its range (every range key of the column at once, or an op ANY (array) with a range strategy, §28), else IS NOT NULL, which is walked only when nothing else answers: beside anything that answers, its column’s NULL entry is taken away from their answer, or its column walked beside them where that reads less - exact either way (§14, “IS NOT NULL in the scans”). A range beside its column’s sets is left out - the recheck tests it per row of the sets, where the walk would read every entry it holds - and so is a qual the sets cannot express (a multi-key query in mode ALL, §17); the scan then passes recheck = true to tbm_add_tuples() for every TID, so the bitmap heap scan re-applies the original quals. Emitting a superset with recheck set is correct; silently dropping a qual would not be. IS NOT NULL beside a strict qual of its own column is implied by it and needs neither; IS NULL beside one, or beside IS NOT NULL, and any comparison with NULL, select no row at all, and two sets that contradict each other (k = 5 AND k = 6) AND to nothing. (Deviation from the version before 2026-09-28, which answered ONE qual per column - the most selective-looking, ranked equality, list, range, null test - and left the others to the recheck: two overlaps on one array column read the sets of one and handed the heap every row of it to throw away (a benchmark, 2026-09-28). test/sql/samecolumn_scan.sql.) (Deviation, 2026-10-19: IS NOT NULL beside anything that answers was dropped here too, and the heap handed every row of the others' answer to test for a NULL; §14.)

VACUUM (lion_vacuum.c, ambulkdelete) 1. Walk the directory leaves left to right from the leftmost one, each in the two passes §11 sets out. Pass 1 cleanup-locks the leaf with nothing else held, filters its INLINE entries and writes every changed one back in ONE record for the leaf, and notes the offset, head and KEY of every CHAIN entry; it then drops the content lock and keeps the pin. Pass 2 takes each CHAIN entry’s posting tree in turn and never waits for a lock while it holds another (§11’s waiting rule): it waits for a posting page’s cleanup lock with nothing else held and then only TRIES the leaf, and when the leaf is busy it lets go of the page, waits for the leaf with nothing else held and then only tries the page, starting over if that fails. The noted offset is only a hint - a sorted leaf shifts entries when an insert lands in the middle and moves them when it splits - so every use re-validates the entry by kind and stored key, and one that moved is found again by a descent made with NOTHING held (§21 “Offsets are not names any more”). 2. For each entry: INLINE → filter the payload through the callback and repack. Removal can GROW a container (every-other-member deletion turns a RUN into a 4104-byte BITSET), so a filtered INLINE payload may exceed lion_inline_max(); then the entry spills during VACUUM, exactly as an insert would. The growth is not bounded by inline_limit either, only by the number of items the payload had: a clustered key’s RUN container is a few hundred bytes for 64 heap pages of rows, and with a tenth of those rows deleted at random it is a 4104-byte BITSET, two of which never share a page. So VACUUM’s spill may need a LEAF PER CONTAINER - seven for a key of a 300,000-row table of three, eleven for 150,000 rows of one key with every other row gone - and lion_entry_spill() writes as many as it takes: a record per leaf, each leaf allocated just before the record of the one in front of it (which has to carry its rightlink), then the posting tree’s root and the entry together in a last record (§18, “The spill”). It used to allocate every leaf up front and refused more than four as unreachable, which it is for an INSERT, whose payload is within inline_limit and always fits one leaf; for VACUUM it meant that the table’s every VACUUM failed, autovacuum’s and the anti-wraparound one’s included, until the failsafe stopped vacuuming indexes at all (2026-09-25 review). CHAIN → descend from head to the leftmost leaf with a cleanup lock on every page of the way, the root first (lion_vacuum_descend(), §11, §22: the head of a set with no internal level is a leaf a reader may have copied containers from), then walk the leaves left to right, each under LockBufferForCleanup whether or not the page has anything to remove, because this is the interlock the count path relies on (§9: a heap-skipping reader keeps the page pinned while it consults the visibility map). Filter every container with lion_container_remove_if, run lion_container_optimize, write back in place or, if it grew and no longer fits, re-place it through the posting-tree placement code in a window of its own (the regrow of §11 and §18, which may split the leaf or push the root down); delete empty containers with one PageIndexMultiDelete per page; update min/max. A root that an insert pushed down under the walk - VACUUM finds an internal page where it expected a leaf - makes it descend again. Every page record also carries the updated entry (ncontainers/ntids), applied as deltas to the entry as it stands, so a crash cannot desynchronise the counters. A leaf the walk leaves empty is then taken out of the set while the key lives on (§22, “Empty leaves”); internal pages stay in the tree; an entry whose ntids reaches 0 is DELETED, and its whole posting tree freed bottom up, in a final step for its leaf (§18). A page VACUUM cleanup-locked without writing to it joins the standby barrier of §25. 3. Report stats: num_pages, num_index_tuples = the ROWS the index holds (§18, “Statistics”: Σ ntids over the entries of its first scalar key column, the heap’s row count when it has none), tuples_removed (postings), and the free pages (§18, “Page counts”): pages_newly_deleted, the pages this VACUUM freed - including what the leak sweep at the end recovered from blocks the walk never reached - ADDED UP over its ambulkdelete calls; pages_deleted, the free pages the index holds at the end of the call; and pages_free, those of them the next allocation could take already. 4. amvacuumcleanup: if stats is NULL (no bulkdelete was needed) return a fresh stats struct by counting pages, with the heap’s row count marked as an estimate; otherwise pass it through. An index of multi-key columns only reports the heap’s row count either way, exact when the heap’s is (§18, “Statistics”).

BUILD (lion_build.c, lion_spool.c) 1. table_index_build_scan’s callback APPENDS each row’s code to its key’s entry in a per-key-column accumulator (§24 “Build”): a hash table of the column’s keys, each with its codes as varbyte deltas, since the scan delivers TIDs in ascending order. A NULL goes to the column’s reserved NULL entry (§14), a row a multi-key opclass extracts nothing from to the key-less entry its value names (§17). The columns share maintenance_work_mem; when it is full, the largest are spilled to a logical tape as sorted runs. On PostgreSQL 17 and later the scan can be parallel: every participant fills an accumulator of its own and hands the leader one tape. 2. The spool gives the entries back in directory order - the distinct keys sorted, the runs and participants merged by key and each key’s codes by code - and ONE pass groups the codes of each key into containers. There is no sizing pass any more: the old pass 1 counted distinct keys only to size the bucket directory, and a B-tree directory is not sized, it is built to fit. 3. Keep one open builder per distinct key in progress - one, unless the hash function of an unordered opclass collides - using lion_container_append_sorted with a per-key “last ckey” and finishing each container with lion_container_optimize when the ckey changes; a ckey with fewer than LION_SPARSE_THRESHOLD members goes into a sparse segment instead (§13). When the key changes, flush: the builders are put in directory order first (§21, a hash collision), then a payload within lion_inline_max() becomes an INLINE entry, and anything larger a posting tree - its leaves written left to right, each filled until the next container does not fit (rightlink, min/max), with the ROOT’s block reserved the moment a second leaf is needed and the internal levels built bottom-up in the same pass (§22). The entry then goes to the directory’s leaf level, which is built the same way, nbtree’s _bt_buildadd shape: one open page per level, leaves filled to fillfactor, internal pages to 70% with at least two downlinks, each page written when the item that does not fit arrives - that item’s key is the page’s high key - and the level whose last page is also its first is the root (§21). An empty table gets a single leaf that is also the root. 4. Pages are written through the bulk-write API (storage/bulk_write.h), which is what the nbtree and GiST builds use: smgr_bulk_start_rel() once for MAIN_FORKNUM, smgr_bulk_get_buf() for each page image, smgr_bulk_write() when that page is final, smgr_bulk_finish() at the end (PostgreSQL 16 gets a reduced copy of 17’s API in lion_build.c). Pages go straight to the file without passing through shared buffers, are WAL-logged in batches of up to 32 as full-page images (or not logged at all for an unlogged relation or under wal_level = minimal, in which case the relation is registered for the next sync), and each page is written EXACTLY ONCE. Under wal_level = minimal the build’s COMMIT then makes the file durable as core does for any relation the transaction created: one of at least wal_skip_threshold is synced, a smaller one is WAL-logged page by page (log_newpage_range()), which reads every page into shared buffers. Nothing may touch these blocks through the buffer manager until smgr_bulk_finish() has returned. The build therefore never goes through the WAL shim, and wal_mode changes nothing about it (§25). Writing each page once is the point: the old route through the buffer manager logged one GenericXLog record per entry tuple, so a build of 632k keys paid 632k page diffs and 632k WAL records to fill 15k pages (measured, 1M rows, one column: 4.7 s and 76 MiB of WAL against 1.9 s and 39 MiB with the bulk writer; the eight-index portfolio of bench/COMPARISON.md went 13.0 s / 95 MiB to 11.3 s / 57 MiB, which is below btree’s 59 MiB). It also means a page has to be FINAL before it is written. A directory page is final as soon as the item that does not fit on it arrives, so at most one page per level is open at a time, and a container page as soon as the next container does not fit, so only one of those exists per open key. Block numbers come from a counter: the meta page is block 0, and its image is kept and written last, once the root is known; everything else takes the next block as it is needed. 5. ambuildempty (lion_am.c): the meta page, recording the directory order as a build does, and a single empty leaf that is also the root, in INIT_FORKNUM with log_newpage, as contrib/bloom does.

Historical: the bucket directory (formats 1-3). Before §21 the directory was nbuckets hash buckets, each a rightlink chain of bucket pages starting at a fixed head page, and before §22 a CHAIN entry’s containers were a plain linked chain of pages walked from head. INSERT and VACUUM serialised on the bucket HEAD page, SCAN walked the bucket chain, and ambuild spent a first pass over the sorted data estimating entry bytes to size the directory once and for all (at least 64 buckets, ¾ of a page each, scaled by reltuples), because nothing could grow it: an insert that found its bucket chain longer than four pages warned once per index that the directory had been outgrown and suggested a REINDEX (max_bucket_pages in lion_index_stats() measured the same thing). Measured then: 100k keys inserted into an index created empty kept 64 buckets and 773 bucket pages, and the 100-key IN median went from 0.104 ms (bulk-built, 847 buckets) to 0.321 ms. Online growth by dynamic hashing was rejected as out of scope for v1; the sorted directory replaced the whole arrangement instead, and grows by splitting. nbuckets, the bucket-chain warning and its threshold, and the sizing pass are gone; the buckets reloption is accepted and ignored with a NOTICE (§21 “What goes away”). The bucket-era build kept every bucket page image in memory until the last key had been hashed; the sorted build keeps one open page per level.

6. Handler settings (lion_am.c)

amstrategies = 5 (1 equality, 2 @>, 3 &&, 4 <@, 5 @@ -- see §17)
amsupport = 4 (1 = hash function, same as hash AM; 2 and 3 = GIN's extraction procs, §17;
               4 = btree comparison of the KEY type, optional, §21)
amoptsprocnum = 0                amcanorder = false        amcanorderbyop = false
amcanhash = false                amconsistentequality = true   amconsistentordering = false
amcanbackward = false            amcanunique = false       amcanmulticol = false
amoptionalkey = false            amsearcharray = true      amsearchnulls = true
amstorage = true (§17)           amclusterable = false     ampredlocks = true (§9)
amcanparallel = false            amcanbuildparallel = true (17+, §24)   amcaninclude = false
amusemaintenanceworkmem = false (§18)   amsummarizing = false     amkeytype = InvalidOid
amparallelvacuumoptions = VACUUM_OPTION_PARALLEL_BULKDEL (§11, §18; cleanup stays with the leader)
amgettuple = liongettuple (§29)  amgetbitmap = liongetbitmap    amcanreturn = NULL
ammarkpos/amrestrpos = NULL      parallel scan callbacks = NULL
amcostestimate: genericcostestimate() then indexCorrelation = 0 (as contrib/bloom)
amoptions: reloptions `fillfactor` (int, 10..100, default 90, §21), `posting_fillfactor`
(int, 10..100, default 90, §22), `inline_limit` (int bytes,
64..4096, default 4096), `max_entries` (int, 0 = unlimited, §17), `buckets`, which is
accepted and ignored with a NOTICE since format 4 (§21), `summaries` (enum off | on | auto,
default off, §32) and `summary_tids` (int, 16..2^24, default 4096, §32), via
add_reloption_kind / add_int_reloption / add_enum_reloption / build_reloptions.
ambuild: refuses a table whose table AM is not the heap, by routine (§2).
amvalidate: a scalar opclass must have support proc 1 with signature (T) → int4 and operator
strategy 1; a multi-key one (§17) procs 2 and 3 and strategies within {2,3,4,5}; proc 4 is
optional for both and is the only support function allowed to be cross-type (§21).
Handler follows contrib/bloom in master: `static const IndexAmRoutine amroutine = {...}` returned
with PG_RETURN_POINTER.

Operator classes (in the install script, pg_lion--<version>.sql): one DEFAULT opclass per type, reusing the hash AM’s support-1 functions and the btree AM’s support-1 functions as proc 4 (§21), plus the two multi-key classes of §17. Generate the list from the dev cluster with SELECT ... FROM pg_amproc JOIN pg_opclass ... WHERE amname='hash' AND amprocnum=1 for at least: int2, int4, int8, oid, bool, “char”, text, varchar (via text), bpchar, bytea, uuid, date, time, timestamp, timestamptz, interval, numeric, float4, float8, macaddr, inet, name, jsonb, enum types via anyenum (hashenum). Strategy 1 operator = the type’s =.

7. SQL functions (lion_funcs.c, lion_verify*.c, lion_count_sql.c, lion_facets.c)

lion_index_stats(regclass, OUT attno int2, OUT directory_height int, OUT leaf_pages bigint,
    OUT internal_pages bigint, OUT ordered bool, OUT entries bigint,
    OUT inline_entries bigint, OUT container_pages bigint, OUT containers bigint,
    OUT array_containers bigint, OUT bitset_containers bigint, OUT run_containers bigint,
    OUT ntids bigint, OUT container_bytes bigint, OUT free_bytes bigint,
    OUT sparse_segments bigint, OUT sparse_members bigint, OUT null_tids bigint,
    OUT empty_tids bigint, OUT slack_bytes bigint,
    OUT deleted_pages bigint,
    OUT posting_internal_pages bigint, OUT max_posting_height int,
    OUT inline_slack_bytes bigint, OUT summary_entries bigint, OUT summary_tids bigint,
    OUT summary_bytes bigint, OUT summary_pages bigint, OUT ndistinct bigint,
    OUT narrow_containers bigint, OUT nullelem_tids bigint, OUT position_pages bigint,
    OUT position_internal_pages bigint)
    RETURNS SETOF record
    -- ONE ROW PER KEY COLUMN (§24), in attno order.  Counters that describe an entry or a
    -- posting set - entries, inline_entries, ntids, null_tids, empty_tids, the container and
    -- sparse counts, container_bytes, slack_bytes, container_pages, posting_internal_pages,
    -- position_pages, position_internal_pages - are that column's own; counters that
    -- describe the RELATION - directory_height, leaf_pages, internal_pages, free_bytes,
    -- deleted_pages, max_posting_height - are repeated on every row, because one directory
    -- leaf holds whatever columns' entries land on it.
    -- A container page IS attributed to its column, through the owner_head stamp of §18 and
    -- the attno of the entry that owns that head - for a page of a position tree (§17), the
    -- tree's root, which the entry's positions extension names.
    -- `ordered` is per column, and false means that column's key type has no btree opclass, so
    -- its entries are in a complete but arbitrary order;
    -- container counts include INLINE containers; free_bytes sums directory and container pages;
    -- null_tids is the member count of the reserved NULL entry (§14) and empty_tids that of
    -- the reserved no-key entry (§17); container_pages counts posting-tree LEAVES and
    -- posting_internal_pages the pages above them, with max_posting_height the tallest
    -- posting tree in the index (§22; 0 means every set fits one page);
    -- the SUMMARY entries of §32 are counted apart, in summary_entries, summary_tids (the rows
    -- they hold), summary_bytes (their items) and summary_pages (their posting-tree pages),
    -- and in none of the counters above;
    -- ndistinct is the column's distinct keys as the meta page records them for the planner
    -- (§33) - the last build's, VACUUM's or ANALYZE's count, not this walk's - and NULL for a
    -- column with none (a multi-key one, or an index nothing has counted since before §33);
    -- narrow_containers is the NARROW containers of §38, last so the others keep their places,
    -- and nullelem_tids the member count of the reserved NULLELEMS entry (§17), the rows whose
    -- value had elements and all of them NULL; emptydim_tids and emptyinner_tids, last, those
    -- of the EMPTYDIM and EMPTYINNER entries (§17), the empty arrays with dimensions, which
    -- empty_tids leaves out;
    -- position_pages and position_internal_pages are the leaves and the internal pages of the
    -- column's position trees (§17, "Stored positions"), which hold chunks of word positions
    -- and no posting set: container_pages, posting_internal_pages, container_bytes and
    -- max_posting_height leave them out
lion_index_posting_root(regclass, key anyelement) RETURNS bigint
    -- the ROOT block of one key's posting tree, NULL when the key has no entry or its set is
    -- still INLINE.  For tests only: §22 requires the root block never to move, because it is
    -- the identity every page of the set is stamped with (§18).
    -- A multi-key column (§17) is refused, with the count functions' message: its entries are
    -- extracted keys, not column values, and a whole tsvector used to be hashed as if it were
    -- one lexeme and look up no entry at all.
lion_index_verify(regclass, heapallindexed bool DEFAULT false) RETURNS void
    -- ERRORs on any structural inconsistency: the key column of every entry (in range, and
    -- never below the column of the entry before it, §24), at most one of each reserved entry
    -- (NULL, EMPTY, EMPTYDIM, EMPTYINNER, NULLELEMS) PER COLUMN, and the key-less ones only
    -- in a multi-key column, the EMPTYDIM and EMPTYINNER ones only in an array's, page ids,
    -- page flags (exactly one kind, and no
    -- bit that kind never carries: a page of a posting set is never flagged as one of a
    -- position tree, which every reader of the set refuses, ending the set there), meta
    -- values (every word the format has: its WAL mode, its order flags, the unused bucket
    -- count of 0, the count of directory pages - at least the pages the walk finds, and with
    -- no bound above, since a deletion whose count record a crash or a cancel lost leaves it
    -- high - and the count of pages waiting for reuse, below the index's length), entry flags,
    -- ascending ckeys within pages and across rightlinks, min/max correctness, container_check
    -- on every container and that every member is a heap tuple - an offset 1 ..
    -- MaxHeapTuplesPerPage, a RUN inside one block (§2; a member at offset 0 used to pass,
    -- and every query of its key then failed) - no two items of a page overlapping, the
    -- form of every stored key (§21, "Readers"), ntids/ncontainers sums, an INLINE entry's
    -- positions (one row of them for each TID of its set, and none for a TID it does not
    -- hold) and its items and positions together within the meta page's inline_limit (§4),
    -- the root alone at its level (a root split makes the level above in one record, §21),
    -- and every DELETED page as freeing it left it (§18, §21: a container page empty but for
    -- its safexid, with no right link or min/max; a directory page with its right link, no
    -- owner, no flag but its kind and DELETED, and one MINUS INFINITY high key) with a normal
    -- safexid less than 2^31 past the next transaction id (a crash's restart of the counter
    -- can leave one a little past it; one further on would keep the page from reuse); with
    -- heapallindexed, scans the heap with a fresh snapshot and checks that every visible
    -- tuple's TID is present under its key in EVERY key column (§24) - and, in a column
    -- that stores positions, that each of its keys holds the tuple's own positions for it,
    -- the ones its value gives, position by position and weight by weight (§17) - refusing
    -- (lion_index_usable(), §9) when this transaction's snapshot may not use the index.
    -- The heap scan evaluates the index's expressions and predicate, which are the table
    -- owner's functions and can be replaced after CREATE INDEX with anything at all. They run
    -- AS THE TABLE OWNER, inside a SECURITY_RESTRICTED_OPERATION, under a GUC nest level that
    -- is rolled back when the check ends, and on 17+ with search_path restricted to
    -- pg_catalog, pg_temp - amcheck's rule since CVE-2022-1552, and what the server's own
    -- REINDEX does. Run as the caller, a superuser verifying somebody else's table ran the
    -- owner's code with superuser rights. The table is locked BEFORE the index (and the
    -- index's table looked up again once both are held), the order DROP INDEX takes them in;
    -- the other order deadlocked with `LOCK TABLE t; DROP INDEX t_k` in another session. A
    -- relation that is not a lion index is told by its catalog row before either lock, and
    -- refused under AccessShareLock alone: its table is not locked, nor its owner's identity
    -- taken, to say so (test/isolation/verify_not_lion.spec). Key
    -- values appear in a "not indexed" report only when the CALLER is a superuser, which is
    -- decided before the switch: asked afterwards, superuser() answers for the owner.
    -- LOCKING: ShareUpdateExclusiveLock on the table, then on the index - the lock CREATE
    -- INDEX CONCURRENTLY builds under, and verify() checks the way CIC builds, beside the
    -- writers. INSERT, UPDATE and DELETE go on while it runs; VACUUM, ANALYZE, DDL, CIC and a
    -- second verify() wait for it (and it for them). CREATE INDEX CONCURRENTLY can deadlock
    -- with it, even one on another column of the table: CIC holds the table's
    -- ShareUpdateExclusiveLock while its last phases wait out every older snapshot
    -- (WaitForOlderSnapshots()), and a verify() that arrives meanwhile waits for that lock
    -- under the snapshot of the statement that called it - which the portal took before the
    -- function ran, so nothing verify() does can avoid it. The deadlock detector cancels one
    -- of the two (the CIC, in every round of the review's load run); run them apart. Both
    -- locks are released when it returns
    -- rather than at commit, as amcheck does. With VACUUM locked out nothing leaves the index
    -- while it is walked, so the only changes it meets are an INSERT's, and each of them is
    -- settled rather than reported - "verify() beside writers" below says how, and what it
    -- costs: it may wait for the statements that are writing the index, for as long as
    -- lock_timeout and statement_timeout allow. lion_index_stats() is unaffected: it stays
    -- under AccessShareLock and concurrent. verify() took AccessShareLock and nothing else
    -- until the 2026-09-25 review - a loop of inserts that split the directory made 11 of 20
    -- calls report "the directory points at block 733, but the index has only 732 blocks"
    -- about a sound index - and then ShareLock, bt_index_parent_check()'s, which kept every
    -- writer out for as long as it ran.
    -- DURING RECOVERY no lock above RowExclusiveLock can be taken, and replay takes no relation
    -- locks anyway, so on a hot standby verify() takes AccessShareLock and reads an index that
    -- replay may be changing - VACUUM's records among the rest, which a primary's lock keeps
    -- out. None of the waiting below applies there: it rests on how WRITERS lock pages across
    -- their records, and replay locks each record's pages for that record alone. A standby
    -- tells replay's work from damage by the pages' LSNs instead ("verify() on a standby"
    -- below): replay stamps every page of a record with the record's end, so pages that all
    -- read at or below a replay position noted before the first of them was read were read
    -- as they were at one point of the primary's WAL, where a difference between them is
    -- damage; a difference that involves a page past it is replay's, and is walked again or
    -- left to the next check. A POSTING SET is walked with nothing held and its entry read
    -- again - never with the entry's leaf held, the primary's last resort, because replay
    -- locks a set's pages BEFORE the leaf (§25) and that walk deadlocked replay for good
    -- (2026-09-29 review). With replay paused (pg_wal_replay_pause(), then
    -- pg_wal_replay_resume()) no page changes, and the first walk of everything settles it.
    -- AN UNFINISHED SPLIT IS NOT DAMAGE (§21, §22). A split writes the new right sibling in one
    -- record and its downlink in the next, with the left page flagged
    -- LION_PAGE_INCOMPLETE_SPLIT in between, and a crash or an error there leaves the sibling
    -- reachable by its left neighbour's right link and from nothing above until the next
    -- writer that descends to the left page finishes the split. verify() warns about the
    -- flag, and when it matches a level's downlinks with the pages of the level below it
    -- accepts a page without one if its left neighbour carries the flag - a three-way posting
    -- split flags two pages in a row, and each missing downlink is covered by the page to its
    -- own left. Such a page is bounded above by the next downlink's separator like any other.
    -- The flag with the downlink already in place needs nothing: every split clears the flag
    -- in a record of its own, which a crash or an error can cut off as well, and the next
    -- writer that descends to the page clears it (verify_incomplete_split.spec's second
    -- permutation fails a posting split and a directory split between the two, and verify()
    -- warns, and passes - 2026-10-23 review, R-V4). The flag is only ever on the LEFT half of a split, which
    -- keeps its new right sibling while the flag lasts: neither page deletion nor VACUUM's
    -- removal of an empty posting leaf takes a flagged page or the page right of one, and a
    -- root split or push-down flags nothing. So the flag on the rightmost page of a level - a
    -- root among them - or on a HALF_DEAD directory leaf IS damage, an ERROR: no INSERT can
    -- finish such a split, and the first one that comes to it fails with "has no right
    -- sibling to link" (verify.sql's sixteenth case), where a WARNING would leave the missing
    -- downlink to surface as "level 1 has 4 downlinks but level 0 has 5 pages";
    -- test/isolation/verify_incomplete_split.spec makes a posting split and a directory
    -- split fail at their injection points and verifies both. On a primary the WARNING is
    -- exact even with writers running: a split holds the page it flags EXCLUSIVE from the
    -- record that sets the flag to the one that clears it, so a flag a SHARE lock lets
    -- verify() read is one a crash or an error left behind. On a standby it is not: replay
    -- locks each record's pages for that record alone, and a split the primary is in the
    -- middle of shows its flag to every reader there. So a standby notes each flagged page
    -- with its LSN and the next transaction id, and at the end of the check warns of the
    -- ones still flagged with the same LSN whose split no transaction still running can
    -- finish - a fresh snapshot's xmin is past the id - which a crash or an error abandoned
    -- (lion_verify_flags_recheck(); test/recovery/run.sh phase 3, vsplitdir and vsplitpost).
    -- The WARNING's hint is the next INSERT whose DESCENT passes through the page: an append
    -- that fits a posting set's last leaf never descends, so a flag on a leaf below the
    -- inserts' range outlives any number of inserts into the key (2026-10-23 review).
    -- A DAMAGED PAGE IS AN ERROR, never a crash, a read past the page or a write: every block
    -- number verify() follows - the meta page's root, downlinks, right links, an entry's head,
    -- a posting pivot's child - is checked against the index's length before ReadBuffer(),
    -- which on 16-18 takes InvalidBlockNumber for P_NEW and EXTENDS the relation; every page
    -- header's bounds are checked, then every line pointer (normal, non-empty, inside
    -- [pd_upper, pd_special), MAXALIGNed: amcheck's PageGetItemIdCareful()), then every
    -- directory item's header, key column and key extent and length, before anything
    -- compares, copies or hashes it - the order checks used to hand an entry's column to
    -- lion_column() and compare its key before the entry check had looked at either; and the
    -- meta page's height is bounded by the index's length before the walk sizes arrays by it.
    -- lion_index_stats(), granted to pg_stat_scan_tables, reports damage to nobody, so it
    -- skips the pages and items it cannot read safely instead: a header out of bounds, a line
    -- pointer outside the item space, an entry shorter than its key (whose payload length
    -- would underflow), an item of no known type, and an INLINE payload from its first
    -- damaged item on, whose size no longer says where the next one starts. Every READER of
    -- an INLINE payload - verify(), the counts, the scans, the ordered walks, INSERT and
    -- VACUUM - reports such an item as INDEX_CORRUPTED with the entry's block and offset and
    -- the item's place in the payload, and the copy of a posting set a count keeps in memory
    -- (§10) reports an item out of order with the page it is on. The regression suite writes
    -- each kind of damage into a temporary index's file (verify.sql) and gets an ERROR from
    -- verify() and a count without the damaged item from lion_index_stats(), where the code before
    -- asserted, read past the page, or extended the index. Its sixth case is damage only the
    -- recheck below can tell from a concurrent split: a right link, and the left link that
    -- goes with it, that skip a leaf, so the leaf is reached by its downlink alone. Its
    -- seventh is a posting leaf flagged as a page of a position tree: every reader of the set
    -- stops there, so the counts lose every row from that leaf on, while a lookup through the
    -- tree code, which does not look at the flag, still finds them - so heapallindexed holds
    -- the leaf its lookup lands on to the readers' test. Its eighth is a flag bit no page of
    -- its kind carries, and its ninth a position tree's downlink, separator and leaf min (§17).
    -- Its tenth is an INLINE payload item of no type, which every reader reports with the
    -- entry, and a copied set out of order; its eleventh an INLINE entry's positions for a
    -- TID it does not hold, which a phrase query through the index then misses a row for;
    -- its twelfth INLINE entries past the meta page's inline_limit; its thirteenth the meta
    -- page's other words; its fourteenth a second page at the root's level, each flagged
    -- ROOT; its fifteenth the body and the safexid of a DELETED container page and of a
    -- DELETED directory page, which a VACUUM and a move of the index to another tablespace
    -- put in its file; its sixteenth the unfinished-split flag on the rightmost page of a
    -- directory level and of a posting set's; its seventeenth a position tree without
    -- positions for TIDs of its set; and its eighteenth positions that are not the row's
    -- own, in an INLINE entry and in a position tree, which only heapallindexed finds. Each
    -- case
    -- builds the index it damages and sets every psql variable it uses itself, from the
    -- file or from a twin; the shared tables and helpers come first. The fifteenth's
    -- tablespace, a cluster-wide name, is named after the backend that runs it.
(phase 2) lion_index_count(regclass, key "any") RETURNS bigint

verify() beside writers: suspect, wait, recheck (2026-09-26)

verify() is a parent check: it compares each level of the directory and of every posting tree with the WHOLE level below it, and proves every block reachable exactly once. Pages read at different moments while writers split them disagree, which is why it ran under ShareLock and kept every writer out for as long as it ran. It takes ShareUpdateExclusiveLock now, the lock CREATE INDEX CONCURRENTLY builds under, and checks the way CIC builds: beside the writers, settling afterwards what they may have changed under it.

What it can meet. VACUUM is the only thing that removes anything from a lion index - a TID, an entry, a page - and ShareUpdateExclusiveLock keeps it out, with ANALYZE, DDL, CIC and a second verify(). So nothing leaves the index while it is walked, no page is freed, and no page that is live when the walk reads it is anything else later. What is left is what an INSERT does, and each of those is handled:

new TIDs in a container, an INLINE entry,    a page is read under its SHARE lock and copied: every
  new entries on a leaf                      check of one page is exact, since writers change a
                                             page only in whole records; an INLINE entry's
                                             counters are on the page with its payload
a directory leaf, internal or root split    see "the walk" and "candidates" below
an unfinished split (INCOMPLETE_SPLIT)       exact: a split holds the page it flags EXCLUSIVE from
                                             the record that sets the flag to the one that clears
                                             it, so a flag seen under SHARE is an abandoned one
an INLINE -> CHAIN spill                     the entry was checked INLINE from the leaf's copy; its
                                             new root is found through the entry (candidates)
a posting leaf or internal split, a root     the set is walked again (below)
  push-down, entry counters changing
relation extension                           a link past the block count taken at the start is
                                             checked against the relation's length NOW, and only
                                             one past the end now is reported: a split links its
                                             new page in the record that initialises it
a page taken from the free space map         it is a live page nothing reached (candidates); one
                                             the reachability pass saw DELETED is a free page, as
                                             it always was

The walk reads every level of a tree from the LEAVES UP, as it always did, and that order is what makes a split between two walks harmless. A split puts its new page into its level’s right-link chain in its first record and the downlink into the parent in a later one, holding the flagged left page EXCLUSIVE in between; the walk of the lower level therefore reads that page either before the split or after it has finished. A split made after the lower walk passed its place shows up in the comparison of the two levels as one thing only - a downlink to a page the lower walk never reached - and in the reachability pass as a live page nothing reached. Everything else the comparison checks is exact: a page’s lower bound is the high key its left neighbour had when the walk read it and never changes, a split never takes a page’s first item away, and items move only onto a page the split links immediately right of the page they left. Two things a split changes are settled on the spot, because one more read proves them: the LEFT LINK of the page the walk reaches next names the split’s new page rather than the one the walk came from - the level is walked again from that page, each page held until its right sibling is locked (left to right, the directory’s own order), where one’s left link naming the other is exact - and a page at the height the meta page gave has lost its ROOT flag, which a root split does in the same record that makes the meta page name a taller directory. In a posting set or a position tree each page of the lower level is bounded above by its parent’s high key as well as by the separator of the next downlink: the two are one key while the upper level is sound, and a parent whose high key went down with the separator above it hides the keys over it from every descent while the level, read as one sequence, stays in order (verify.sql’s nineteenth case).

Memory. The comparison of two levels needs the whole lower level, read before the walk of the upper one began - the order that makes a split harmless. What a walk of a posting set or a position tree keeps of a page is 16 bytes (its block, first key, upper bound and flags), in a sequence (LionVerifySeq, lion_funcs.h) that stays in memory up to a quarter of work_mem and goes on in a temporary file past it, read back in order by the walk of the level above; so are the blocks a walk that may be thrown away visited and the unfinished splits it met. A tree of any size is walked in the same memory: verify.sql’s nineteenth case walks a position tree of 1,200 leaves at work_mem = 64kB, the 176 past the 1,024th going to the file, and verify() says at DEBUG1 the most its walks kept in memory and the records they wrote out. The directory’s walk keeps its levels in memory, with each page’s first key and high key, and the reachability pass a byte per block of the index.

Posting sets are walked with nothing held: holding the entry’s directory leaf for every walk would keep the writers of every key on that leaf waiting. The entry is read again afterwards - found by its exact key from the leaf it was on, moving right, since an entry leaves its leaf only with the upper half of a split - and the walk is kept when the entry reads the same (flags, head, tail, ntids, ncontainers) both times. That is exact: every writer of a key holds the entry’s leaf EXCLUSIVE from its first record to its last (§5), and both reads were made under SHARE, so a writer that touched the set between them did all of it between them; and every change an INSERT makes to a set moves the entry - an item added, grown or split off adds its TID to ntids in the record that places it, a push-down moves tail. The one kind of writer that adds no TID - one that finishes an abandoned split on its way down and then finds its TID already there, or fails - adds a downlink and clears a flag, which the walk accepts either way, and any internal split that makes is a downlink the lower walk never saw, which throws the walk away too; so does a root at another level than the descent found it at. A set that keeps changing - a hot key - is walked at most three times like that, and then once more with the leaf held SHARE throughout. That is the lock order every writer uses (directory page before posting page) and keeps waiting only the writers of the keys on that one leaf, for one walk of one set; the 50-call stress run below needed it for 6 of its 70 set walks. It is a PRIMARY’s last resort: replay does not use that lock order (§25), and a standby never makes that walk (below).

Position trees (§17) are walked the same way, after their posting set: with nothing held, the entry read again, the walk kept when the entry reads the same. A writer holds the entry’s leaf from its first record to its last, the row’s positions among them (W1), and every INSERT that gets to its TID changes ntids. What the walk compares the tree with - at least as many members as TIDs, and every TID of the posting set among the members - needs one more step, because an INSERT that fails between its two writes leaves members and no TID, and the entry unchanged: a split it made of a leaf the walk had read can have moved members past the walk. So a kept walk is exact about what it found (the posting set it read is the entry’s, unchanged, and no member leaves a tree while VACUUM is locked out) but not about what it missed: a walk that found every TID among the members is kept, and one that found a TID without positions, or too few members, is a suspicion, which a walk with the entry’s leaf held SHARE settles - as is a tree that changed under three walks in a row. The comparison reads the two trees' leaves in step, a copy of one leaf of each and one member at a time, and keeps nothing else: collecting the members' codes to look the TIDs up in would cost 8 bytes a row, without bound. test/isolation/verify_positions.spec parks the check after each walk (lion-verify-postree-walked) and before each new one (lion-verify-postree-rewalk) while an INSERT changes the key, until the walk with the leaf held (lion-verify-postree-held, ‘notice’).

A standby has no hold to settle a suspicion with, and the entry reading the same settles nothing there: replay applies a writer’s records one at a time, and replays VACUUM beside the check - a TID taken out of the set, its positions after it (V1), and another row’s TID put in leave the entry as it was, while a walk read the set before the first and the tree after the second. What settles a walk there is the pages' LSNs. Replay sets every page a record changes to the end of the record, and GetXLogReplayRecPtr() is the end of the last record replayed: a walk that notes it before it reads anything and then reads no page past it - of the tree, of the posting set, the entry’s leaf - read the index as it was at that one point of the primary’s WAL, where every TID has its positions (W1 and V1 order the records so, and an INSERT or a VACUUM stopped between two of them leaves members without TIDs, nothing else). A difference that walk finds is damage and is reported - the count only when the entry’s leaf was read at that point too. A walk that found no difference is kept whatever its pages' LSNs, as on a primary; one that found a difference after replay changed a page it read is kept for what it checked of the pages, and the comparison alone is made again - the two trees' leaves and one page a level above them, the tree’s members counted - between two reads of the entry, at most eight times (lion_verify_postree_count()). A pass that found no difference settles it, and so does one that read every page at the replay position it began at, which reports what it found; only when none does is the tree kept without the comparison, with a WARNING whose hint is to pause replay, which leaves every page as it is. Walking a set that found a difference again, whole, and warning after three such walks would give the WARNING to one standby check in nine under a steady replay (R-S4). test/recovery/run.sh phase 2 damages a position leaf in the standby’s copy of an index and has the standby report it.

Candidates. What is left - a downlink to a page the lower walk did not reach, a live page nothing reached (directory pages, posting leaves with a live root, internal posting pages) - is recorded, not reported. When the walk and the reachability pass are over and there are candidates, verify() waits for the statements that are writing the index: WaitForLockers() on the INDEX’s lock tag in ShareLock, the mode CIC waits in. An INSERT, UPDATE or COPY takes RowExclusiveLock on each index of its table when it puts its first row into it and lets it go when the statement ends (ExecOpenIndices(), ExecCloseIndices()), and a statement that has ended has finished every split it began, or left it flagged for good. CIC waits on the TABLE’s lock tag because it must outwait every transaction that could still insert without knowing the new index; verify() need not, so a transaction that wrote the table and sits idle, or prepared, holds no lock on the index and is not waited for. WaitForLockers() waits for each holder’s whole transaction, which is more than needed and the only unit it offers. The wait is a lock wait on each writer’s virtual transaction id, so lock_timeout ends it (with “canceling statement due to lock timeout”) as well as statement_timeout; without either, a statement that keeps the index open for writing keeps verify() waiting. An index nobody wrote to during the walk has no candidates and nothing is waited for.

Then each candidate is checked again by itself. A downlink’s page has to be in the lower level’s chain between the reached page whose downlink precedes it and the one whose downlink follows it, which a coupled walk between the two proves or disproves. A page nothing reached has to be where a search for its own first key lands: from the directory’s root, stopped at the page’s level (lion_dir_search_level()), or from the root of its posting set or position tree (§17) - a root that was reached, or that an entry names. In a tree the search takes the route of what sorts just below the key (lion_posting_search_level() with before) and walks the level right from where it lands until it meets the page, or a page whose first key is above the page’s: a position tree’s keys are heap blocks, one block’s rows can fill more than one leaf, and a leaf’s first key is then its left neighbour’s last, or the separator of a run of leaves (lion_postree.c), so a search for the key itself stops left of the page or goes past it. The roots are found through the entries in one walk of the leaves, posting roots and position roots both; that is needed for more than a spill’s root, because either root always comes from the end of the relation (§18) while the pages its tree grows by may come out of the free space map, so a key created during the walk can leave unreached pages under a root nothing reached either (a first version reported “block 7 is not reachable from the meta page” about exactly that). test/isolation/verify_positions.spec has an INSERT make both kinds of unreached position page under a parked check: the tree of a new key, and a leaf an existing key’s tree splits off. A candidate the recheck cannot settle is reported as the walk would have reported it: as a leak, with a WARNING, when it is a kind an interrupted writer leaves and the next VACUUM’s sweep frees - an internal posting page among them - and as corruption otherwise.

What it no longer sees. A block the relation grew by after the check began holds nothing that was in the index then; the reachability pass stops at the count it started with (a link the walk followed into such a block was checked like any other). An entry inserted behind the walk is not checked, and neither is a set created behind it beyond the pages it took. heapallindexed is as sound as it was: its snapshot is taken after the structural check, is registered, and every row it sees was inserted by a transaction that committed before it - whose index inserts had therefore finished - while VACUUM, the one thing that could have taken a visible row’s TID out of the index, is locked out. The lookups are readers' descents: a CHAIN entry’s leaf is let go before its posting tree is descended, and the descent hands back, locked, the leaf that holds the container key, moving right past a split - the only way an item moves - so the answer is the one a held leaf would give. (Until the 2026-09-29 review the lookup held the entry’s leaf for its descent, which replay on a standby deadlocked against, §25.) The owner switch, lion_index_usable() and the indcheckxmin refusal are unchanged. In a column that stores positions each row’s positions under each key are looked up the same way - an INLINE entry’s chunk under the leaf’s lock, a position tree by a reader’s lookup that moves right past a split (lion_postree_fetch()) - and compared with what lion_extract_value_pos() and lion_posmember_from_key() make of the row, the calls the build and the INSERT make. A visible row’s positions cannot move under the check: they were written before its TID (W1), and only VACUUM takes them out. One member and one chunk are kept, for the whole scan, so the comparison adds nothing to the scan’s memory that grows with the index. The TIDs alone would pass a row whose stored positions are not its own, which a phrase query decides from alone (verify.sql’s eighteenth case). A row its key holds without any positions the structural check reports first, and no writer leaves one behind it, so the scan meets one only in an index damaged between the two passes, which is how test/isolation/verify_nopos.spec makes it meet one: parked at lion-verify-heap-scan, the check has the index’s buffers evicted and a position leaf damaged in the file under it.

verify() on a standby (2026-10-23 review) takes AccessShareLock and waits for nothing: the arguments above rest on how writers hold pages across their records, and replay holds each record’s pages for that record alone - and replays VACUUM beside the check, which a primary’s check locks out, so pages are freed, emptied, taken out of their sets and unlinked from their levels under it. What a standby has instead is the pages' LSNs. Replay sets every page of a record to the record’s end and moves GetXLogReplayRecPtr() past it afterwards, so a page read with an LSN at or below a replay position noted before some earlier read was, at that read, as it was at that position: pages that all are, against the position noted before the first of them was read, were read as they were at one point of the primary’s WAL, where the index is sound unless it is damaged. Every check that compares what two reads found is held to that, and what a page past the position breaks is replay’s work:

  • A posting set’s or position tree’s walk (below) reads a page freed, taken out of the set (UNLINKED), stamped as another set’s or at another level than its link promises: damage when the walk has read no page past the position it began at, a reason to walk again when it has (lion_verify_moved()). An entry that is gone from where its key sorts, or has changed its form (VACUUM shrinks a CHAIN entry back to INLINE), on a leaf replay changed since the entry was seen there, is an entry VACUUM’s replay took away, and the check of its set ends there (lion_verify_refind()). test/recovery/run.sh phase 3, vunlink and vfree.
  • The directory: a DELETED page a level’s walk reached by a right link read before it was unlinked - its LSN past the position noted before the page left of it was read - is walked through, as readers do; a left link is compared only between two pages read at that position; a downlink to a page the walk of the level below did not reach is dropped when the page it is on is past the position that walk began at, which a split’s downlink, placed after the walk passed the place, always is; and a page of the level below without a downlink is accepted when either downlink around it is on such a page, or the walk found it dying - page deletion’s first stage takes the downlink out. The meta page’s count of directory pages is compared only when no record has changed the meta page since the check began. Phase 3, vdownlink and vleftlink.
  • The reachability pass reports a page no walk reached only when the page - and the root that tells a leak from a live page - is at or below the position the check began at: then it was what it is when the check began, and the walks, which began after, would have reached it. A page replay made, emptied or took out of a tree since is left to the next check. Phase 3, vreach.
  • The summaries: a row whose inserting transaction is still running may be between its key and its summary for as long as the primary takes, and is no candidate; a transaction’s commit record follows every record of its statements, so a committed row has both (lion_verify_settle()). Phase 3, vsummary.
  • An unfinished split is warned of only when it is abandoned (above, and phase 3’s vsplitdir and vsplitpost).

What it can still falsely report. A WARNING that a block is unused and unreachable for an empty posting page a whole-set free’s first record emptied before the check began and whose second record - which frees the page - replay has not reached yet: the state a crash between the two leaves, and which a standby cannot tell from it. The next VACUUM frees either. No other false report is known.

Posting sets on a standby (2026-09-29 review) are the exception, and the reason is a deadlock, not exactness. Replay takes a record’s blocks in the order they were registered and holds them all to the end of the record, and every record that changes a set registers the set’s pages before the entry’s leaf (§25): a walk with the leaf held - which is how a standby walked every set, and the primary’s last resort above - waits for a posting page the startup process holds while the startup process waits for the leaf. Two buffer content locks: no deadlock detector, and cancel, terminate and pg_ctl stop -m fast do nothing to either side; replay stopped for good within seconds of a write load, in both WAL modes, and only an immediate shutdown got the standby back. So a standby walks a set with nothing held and reads the entry again, as above. Every record that changes a set’s items, counters, last leaf or the level of a leaf root carries the entry, and the records that do not - a downlink, a flag cleared, an internal split or an internal root pushed down - change nothing the entry counts and are what the walk already accepts from a writer between two levels, or throws away. But the entry reading the same both times settles less there than on a primary: replay applies VACUUM, which the primary’s check locks out, and a TID a VACUUM takes out of one leaf and another an INSERT puts into another leave the entry as it was, while a walk that read the first leaf before the one and the second after the other counts a TID more than the entry says (test/recovery/run.sh phase 3 parks a standby’s check between two leaves, at lion-verify-set-leaf-read, for exactly that). So a standby keeps, and compares with the entry, a walk that agrees with an entry that read the same, and a walk that read every page - the entry’s leaf after it among them - as it was at the replay position it began at, which the pages' LSNs show (position trees, above): such a walk read the set and the entry at one point of the primary’s WAL, and a difference there is damage. A walk that got to the end of the set and is neither is kept for what it checked of the pages, and its totals are counted again: a counting pass reads the root, the first page of each level below it and the leaves, nothing else, between two reads of the entry, and settles them when it read every page at the replay position it began at - a difference is damage then - or agrees with an entry that read the same (lion_verify_set_count()). A pass is a fraction of a walk, and far likelier to read one point under a steady replay, which changes the set’s last leaf and its entry with every row; only when eight passes in a row do not settle them are the totals left uncompared, with a WARNING whose hint is to pause replay (three whole walks in its place would warn in one check of nine, R-S4). The entry is read before the first walk as before every other, not taken from the directory walk’s copy of its leaf, which the writers have usually moved on from by then (R-V3) - on a primary too, where that threw the first walk of every busy set away. A root pushed down, an upper level split or a page freed under a walk stops it short of the end, and a set that does that to ten walks in a row is an ERROR asking for replay to be paused. In a 30 s run of 8 clients inserting 15,000 rows a second into three keys of 700,000 rows each (the assert-enabled 18.6, rmgr mode), two clients verifying on the standby made 27,000 calls without an error, 0.5% of the sets they walked were kept uncompared, and replay kept up throughout. test/recovery/run.sh phase 4 is that run, shortened, with a watchdog on replay’s progress: the code before the fix failed it in both modes, replay frozen with the startup process and both verifying backends waiting on BufferContent.

Measured (the assert-enabled PostgreSQL 18.6 of the development slot, 4 CPUs; ratios, not absolute numbers). The stress is a loop inserting 200 new ~140-byte text keys at a time (directory leaf, internal and root splits) and a loop adding 300 rows at a time to one hot key (posting leaf splits), 20 ms apart, against a loop of lion_index_verify(idx, true). The ShareLock code with its lock taken back down to AccessShareLock - which is what it was before the 2026-09-25 review - failed 9 of 20 calls (“the directory points at block 1048, but the index has only 1048 blocks”, “the high key of block 585 sorts after the separator of its right sibling”); this code passed 20 of 20, 50 of 50 and 40 of 40, settling 1,833 candidates after 41 waits in the 50-call run, and throwing away 20 posting-set walks and holding the leaf for 6. With the writers 2 ms apart instead it passed all 36 calls it finished before the table reached 22 million rows, settling 5,728 candidates after 36 waits and throwing away 35 of 72 posting-set walks, holding the leaf for 9.

1M rows; m_k: 20 keys, 280 posting leaves; m_t: 50,000 keys, 1,250 directory leaves
verify() alone, ms     m_k: structural 3-5,   heapallindexed 970-1,050
                       m_t: structural 15-31, heapallindexed 1,520-1,590

INSERTs, 2 pgbench clients,   alone           beside a verify() loop           beside a
20 s each                                     structural      heapallindexed   heap scan loop
this code  avg / p99 ms       0.127 / 0.312   0.139 / 0.408   0.129 / 0.343    0.136 / 0.366
           inserts            312,187         284,563         307,149          291,373
ShareLock  avg / p99 ms       0.123 / 0.286   1.118 / 79.6    34.7 / 1,384     0.132 / 0.329
           inserts            320,758         35,852          1,265            299,710

Beside the inserts a structural verify() of m_t took 111 ms on average (15-31 alone: the index grew by thousands of keys during the phase, and the candidates are waited for and searched), and one with heapallindexed 2.8-3.7 s (the heap grew by 600,000 rows). The inserts beside it lose what they lose to a plain heap scan loop taking the same CPU, and nothing else; under ShareLock they queued behind every verify().

test/isolation/verify_concurrent.spec parks verify() at injection points with nothing held - after the meta page (lion-verify-meta-read), after a directory level (lion-verify-dir-level-walked), on a directory page (lion-verify-dir-page-read), after a posting set’s leaves (lion-verify-set-leaves-walked) - and has INSERTs split leaves and internal pages of the directory behind it, split the page it stands on, split its root, split a posting leaf, push a posting root down, spill an INLINE entry, keep changing a set until it is walked with its leaf held (lion-verify-set-held with ‘notice’; the check parks before each new walk of the set too, at lion-verify-set-rewalk; verify_positions.spec does the same to a position tree), and take pages VACUUM freed for an existing set and for a new one; every time the check passes and the observer shows the change was made. It also shows VACUUM waiting for the check, the check waiting for a statement that holds the index open for writing and not for a transaction whose statement is over, and LOCK TABLE; DROP INDEX in another transaction going through while the check waits for the table. Each change is one the old AccessShareLock code reported as damage or would have; verify.sql’s sixth damage case is real damage that only the recheck can tell from one of them. verify() says at DEBUG1 what writers cost it: candidates, waits, left links and root flags settled on the spot, posting-set and position-tree walks made, thrown away, held and, on a standby, kept without their comparison, a standby’s counting and covers passes, and the flagged pages it looked at again.

Facet counts (lion_facets.c)

lion_index_facets(idx regclass, k integer DEFAULT NULL, filter text DEFAULT NULL,
                  attno int2 DEFAULT 1, OUT key text, OUT count bigint) RETURNS SETOF record

A search page shows, beside its results, how many of them carry each tag, lexeme or category: a GROUP BY of the matching rows' keys, ordered by its count and cut at k. For a multi-key column the query is FROM t, unnest(col) e ... GROUP BY e, which reads and unnests every matching row; the index already holds each key’s rows as a posting set, and the count engine (§9) counts a set against the snapshot without reading the rows. The count functions refuse a multi-key column (an entry’s key is not a column value, so col = key is not what they would count); this one is written for it, and takes a scalar column too.

What it counts. A key’s count is the rows the snapshot sees whose value holds the key, under the filter: count(DISTINCT ctid) over unnest(col), ts_stat()’s ndoc, or GROUP BY col with its NULL group for a scalar column. ROWS, not occurrences: the index files a row once per distinct key of its value (§17), so a count(*) over unnest() of arrays that repeat an element is not what it answers, and that query is not a shape the planner could hand it unless the elements are known to be distinct. NULL elements, NULL and empty values are under no key; the reserved NULL and EMPTY entries, and the length key of a column that stores positions (§17, “Ranking”), are not walked. The keys come out by count descending and then in the order the index keeps them in (the directory’s order for an ordered column, byte order otherwise), and every key tied with the k-th is counted, so the cut is deterministic and equals ORDER BY count DESC, key LIMIT k wherever the key’s order is the column’s.

The filter is text, parsed as SELECT WHERE <filter> (anything but a lone WHERE condition is refused) and analysed against the index’s table as a query’s WHERE is: names resolved, operators chosen, constants folded. It is then taken apart into posting sets (lion_facets_filter_tree()): AND and OR, a constant, col IS NULL, and an operator or = ANY (array) with a Var on one side (commuted when it is on the right) and a value on the other. Each atom is answered by the first valid, non-partial Lion index with the column as a key column whose operator family holds the operator - equality for a scalar column, a multi-key search strategy for a multi-key one - under the clause’s collation, through lion_scankey_sets(), the locate the scans use. An atom it cannot answer exactly - a range, <>, NOT, IS NOT NULL, a lossy or ALL search (a tsquery phrase, weight or negation, jsonpath, <@, an empty containment), a column no Lion index keys, a value that reads the table, a volatile value, a sublink - is an ERROR that deparses the condition. It is never evaluated against the rows: a function whose cost depends on whether its filter happened to be indexable would hide a scan of the table behind a call that looks like an index read. The values are evaluated once, before anything is read.

The reader contract (§9, “Direct readers”). lion_count_open_indexes() opens the faceted index, with a flag that admits a multi-key column for this caller only (its EXECUTE check is then on the element type’s equality, the function GROUP BY e would call), and every index the filter uses goes through lion_reader_vet(): row-level security, an unpopulated materialized view, an index the snapshot may not use and the PostgreSQL 16 old_snapshot_threshold guard; its lookups take the predicate locks of §9, “SERIALIZABLE”. The filter’s columns need SELECT and its operators' functions EXECUTE (lion_check_functions_walker()), as the query’s would. reader_contracts.sql, reader_checkxmin.spec and reader_serializable.spec list both forms, filtered and not.

Counting. The filter’s sets are collected ONCE into a pinless copy (lion_sources_collect(), spilling past hash_mem), as the GROUP BY pushdown does with its WHERE (§10, “The WHERE sets, collected once”); a filter with no row returns nothing. Each key’s set is then located and ANDed with the copy, a batch of keys at a time (lion_count_groups_copy()), and counted against the visibility map and the heap under the caller’s snapshot (lion_count_sources_cached()): the same engine, so the same exactness argument as every count. With no filter each key’s set is counted alone. A key with no visible row under the filter has no row in the output.

The top k is §36’s walk, with the strict stop rule: the candidates with the largest ntids are counted largest bound first until the k-th count is LARGER than the next bound (equal is not enough, since a key tied with the k-th could come before it in key order), and when they run out short of that, the column is walked again counting every key whose bound is not below the k-th count so far. The number of candidates is §36’s 2k/s, s the share of the table’s rows the filter’s copy holds.

A selective filter is cheaper answered from its rows: when the copy holds fewer rows than LION_FACET_KEY_ROWS (5) times the keys that would have to be counted - the candidates whose bound reaches s times the k-th candidate’s bound, or every key without a k - each TID of the copy is fetched under the snapshot (table_index_fetch_tuple(), which follows a HOT chain to its visible member), its value’s keys extracted by the opclass (lion_extract_value()), and counted in a hash table under the column’s own equality. The same rows, so the same counts: a TID is in the filter’s sets only if the version it leads to was filed under the filter’s keys, and a HOT chain’s members share their indexed values. Only for a plain column of a non-partial index: an expression would have to be evaluated, and a partial index’s rows are not all of the filter’s. 5 is the ratio of a key counted in a batch (some 5 µs) to a row fetched and taken apart (some 1 µs), measured on the table below.

Memory. The keys counted are kept until the output is sorted: a top k’s twice k of them at most, and every one of them without a k. Past work_mem - the kept keys' slots and their copies - they go to a tuplesort instead (lion_facets_spill()), which spills to temporary files as any sort does: by the count negated, then by the key - the column’s own < (LionState.ltopr) where the column is ordered, the key’s bytes as a bytea otherwise (lion_facets_key_image(), the order lion_facets_keycmp() keeps), the NULL group last - with the key’s text beside it for the output, which is cut at k there. The selective filter’s hash table holds every distinct key of the filter’s rows, which can be most of the column’s: past hash_mem it is given up, and the keys are counted by their entries instead (lion_facets_heap() returns false) - the same counts, kept as above.

Hashed keys. A key of 2043 bytes of a tsvector column, or of a text, varchar or bytea array column, may be the hashed key of longer lexemes or elements and a lexeme or element spelled like it (§17, “Long lexemes”, “Long elements”), and its rows are all of theirs; ts_stat() and GROUP BY e show each with its own rows. So when such a key is counted (lion_facets_keep_key()), its rows that the filter keeps - its set ANDed with the filter’s copy, or with the filter’s own sets on a standby - are fetched under the snapshot as the selective filter’s are, and each is counted under every lexeme or element of its own value that is filed under the key (lion_ambiguous_values(), which runs GIN’s extraction without the hashing), the value being the index column’s, an expression’s too (FormIndexDatum()). The same rows the key’s count counts, so each count is exact and none is larger than the key’s bound: the top k’s stop rule holds as it is. The selective filter’s own read does the same for a row’s hashed keys. Keys of jsonb_contains_ops past the limit keep no value to show and still print as sha256:<hex>.

DEBUG1 reports each call: entries read, keys counted, whether the column was walked whole, rows read from the heap, keys sorted past work_mem.

A planner shape would let SELECT e, count(*) FROM t, unnest(t.col) e WHERE ... GROUP BY e ORDER BY 2 DESC LIMIT k reach this code without the function. It would need: recognising the lateral function RTE unnest(col) (or ts_stat(), which is a function over a query string and cannot be) joined to one base relation with nothing else in the join tree; mapping col to a lion index whose multi-key key column it is; the WHERE classified into posting sets as §10 does, declining whatever is left over; and proving the count’s semantics - count(DISTINCT t.ctid) or a key unique per row (a tsvector, whose lexemes are distinct; an array only under a constraint that says its elements are) - since count(*) counts repeated elements and the index cannot. The grouped rel’s top k is then §36’s (lion_count_path_topk()), with the keys the node’s output in place of a column’s values. The function exists first because the shape is narrow and the duplicate rule makes the obvious query the wrong one for arrays.

Measured on a synthetic 500,000-row table (three tags drawn from pools of 20, 200 and 2,000, some repeated; five-lexeme tsvectors; 20 groups; lion indexes on the tags, the tsvector and the group; all-visible), PostgreSQL 18, warm cache, 4 shared CPUs:

facets the query, ms lion_index_facets(), ms keys counted
top 10 tags 560-585 0.9-1.4 10 of 2,000
top 10 tags, grp = 3 37-40 5-6 200
top 10 lexemes, grp = 3 (ts_stat()) 40-44 7 202 of 12,002
top 10 tags, tsv @@ 'w7 | w42' (29,573 rows) 45-46 5-6 200
top 10 tags, tsv @@ 'w7 & w42' (133 rows) 1.3 1.6 none: 133 rows read
every tag 545 18 2,000

The queries are count(*) over unnest(tags) and ts_stat(), as the planner runs them (the filtered ones find their rows through the lion indexes); they count occurrences, not rows, which is where the table’s repeated tags make their answers differ. The first call after a restart is 2-3 times slower. With a k the counting stops at the first 10 keys unfiltered and at 200 under a filter keeping 5%: what is left is the walk of the column’s headers and the collection of the filter’s sets.

Tests: test/sql/facets.sql checks every answer against the query it stands for, in order, for 21 filters, five k and seven indexes (array, tsvector storing positions, int[], scalar, jsonb, an expression and a partial index, so both the key and the heap path answer), on the fresh table, after deletes, indexed and HOT updates and inserts without VACUUM, and after VACUUM; with NULL and empty values, NULL and repeated elements, the transaction’s own changes and a rollback; and the errors, a multicolumn index and the privileges. test/isolation/facets_snapshot.spec compares it with the reference query under one snapshot while another session writes, commits and rolls back.

8. Module ownership

src/lion_tid.h            TID ↔ code helpers                          (fixed; written by the architect)
src/lion_container.h/.c   container library + test/unit/container_test.c   (agent "container")
src/lion.h                on-disk structs, LionState, prototypes of the page layer (skeleton by architect)
src/lion_pages.c          meta/bucket/entry/chain primitives, splits, page alloc   (agent "am-core")
                          - since split into lion_pages.c, lion_meta.c, lion_state.c, lion_entry.c
                          and lion_posting_put.c
src/lion_am.c             handler, options, validate, costestimate, buildempty     (agent "am-core")
                          - costestimate since moved to lion_amcost.c
src/lion_build.c          ambuild                                                   (agent "am-core")
src/lion_scan.c           ambeginscan/rescan/endscan/getbitmap                      (agent "am-core")
src/lion_insert.c         aminsert                                                  (wave 2)
src/lion_vacuum.c         ambulkdelete/amvacuumcleanup                              (wave 2)
src/lion_funcs.c          stats/verify (+ count in phase 2)                         (wave 2)
                          - verify since in lion_verify*.c, with lion_funcs.h between them
src/lion_multikey.c       GIN-style extraction and query trees (§17)                (wave 3)
src/lion_ordered.c        LionOrdered: lion-filtered, btree-ordered scans (§30)
src/lion_count.c          the count engine (§9) - since split into lion_set.c, lion_set_copy.c,
                          lion_cursor.c, lion_expr.c, lion_vis.c, lion_count.c,
                          lion_count_groups.c, lion_count_shared.c, lion_rangesrc.c,
                          lion_range.c, lion_count_sql.c and lion_facets.c, which
                          lion_count_int.h joins
src/lion_customscan.c     the LionCount custom scan (§10) - since split into its planner half,
                          lion_plan_*.c, and its executor half, lion_exec_*.c, which
                          lion_customscan.h joins
pg_lion.control, pg_lion--<version>.sql, Makefile, test/                    (am-core, then wave 2)

Coding conventions: PostgreSQL C style (tabs, K&R braces on their own line for functions, / / comments, elog(ERROR, ...) for internal errors, ereport with errcode for user-facing errors), compile clean with -Wall -Wextra -Wno-unused-parameter and cassert enabled. No CRoaring dependency.

9. Phase 2a: heap-skipping count with a visibility-map interlock (the count engine, lion_count.h)

Goal: count(*) WHERE k1 = v1 [AND k2 = v2 ...] computed from containers, visiting the heap only for pages that are not all-visible, with exactly the MVCC semantics of the equivalent SELECT under the caller’s snapshot.

Why this is safe (the argument reviewers will check — keep the code shaped like it) - A TID that is in the index but whose tuple is dead can be counted wrongly only if the heap page is all-visible in the VM at the moment we check. VACUUM sets all-visible only after it has removed the dead TIDs from every index (ambulkdelete) and then from the heap page. (Since PostgreSQL 19 on-access pruning sets all-visible too, so VACUUM is one of two writers. What the argument really rests on is narrower and holds for both: a page holding a dead root line pointer - LP_DEAD - is never marked all-visible by anyone, and only VACUUM, after ambulkdelete, turns LP_DEAD into LP_UNUSED. §11’s last subsection proves it against 19’s pruneheap.c.) - Our ambulkdelete takes LockBufferForCleanup on every container page and on every bucket page whose INLINE entries it rewrites. Cleanup locks wait for all pins to drop. - Therefore: while the counting code holds a pin on the page it took a container from, VACUUM cannot have completed ambulkdelete for that page, so it cannot have set all-visible on any heap page whose dead TIDs are still in that container. Rule: check the VM for a container’s heap blocks before releasing the pin on the page the container came from. The content lock may be dropped (copy the containers out under SHARE lock, keep the pin, consult the VM, then unpin). - Tuples inserted after our snapshot: heap_insert clears the VM bit, so their pages are not all-visible and go through the heap recheck, which applies the snapshot. Tuples deleted but still visible to us: the page is not all-visible (a deleted tuple is not visible to all); recheck sees it. - VM_ALL_VISIBLE may return a slightly stale true only if the bit was cleared concurrently by an insert/update/delete of a tuple that cannot be visible to our snapshot anyway (same reasoning as index-only scans; see visibilitymap_get_status comments). - Serializable isolation: call PredicateLockPage(heap, blk, snapshot) for every heap block counted from the VM without a heap visit, exactly as heapam_indexscan.c does for index-only scans.

Four rules added after the 2026-09-20 adversarial review (all implemented and regression-tested in test/sql/security.sql and test/isolation/count_serializable.spec): - Privileges. The SQL count functions require what the equivalent query requires: SELECT on the table, or SELECT on every column the index reads - its key columns, the columns of its expressions and of its PREDICATE (a partial index’s count is ... WHERE pred AND col = key, so SELECT(id) alone on (id) WHERE secret would reveal secret a row at a time; a whole-row reference needs the table-level grant; an index that reads no column at all, one on a constant, needs SELECT on at least one column, as SELECT count(*) FROM t does); otherwise permission denied (2026-09-23 review). EXECUTE is required on every function the index’s expressions and predicate call, read from pg_index as stored - the relcache’s copies are planner-simplified, with inlinable SQL functions already replaced by their bodies - whatever the table grants. The CustomScan path is covered by the executor’s own ExecCheckPermissions on the range table, and a partial index reaches it only when the query implies the predicate, i.e. references it. EXECUTE on what a count replaces (2026-09-23 review: a pushed-down count(*) WHERE k = 1 answered after REVOKE EXECUTE ON FUNCTION count() or int4eq from PUBLIC, and so did the SQL functions). The rule is the one above - ask exactly what the replaced plan asks, no more, no less - and what core asks was read from nodeAgg.c/execExpr.c and verified against the ordinary plan on 16 and 20: ExecInitAgg checks EXECUTE on each aggregate for the current user and on its transition and final functions for the AGGREGATE’S OWNER; ExecInitFunc checks every function and operator of an initialised expression - the scan’s quals, including a hashed IN list’s hash function and a hash or nested-loop join’s clause - for the current user; the grouping equality of a GROUP BY is checked (ExecBuildGroupingEqual), its hash and sort support functions are not, and count(DISTINCT k) compares k through an unchecked FmgrInfo, so it runs with int4eq revoked. Each check fires InvokeFunctionExecuteHook. So the planner records, in custom_private member LION_PRIV_EXECUTE (shape 10), the aggregates of the target list and the HAVING (setrefs.c turns the HAVING’s into references to the node’s own columns, so core never sees them), the functions of every WHERE clause - OR trees and IN lists included, from the parent’s quals for a partitioned table - and of an FK-side join’s clause (§27), and the GROUP BY’s equality functions; lion_begin_custom_scan() checks them for GetUserId() with core’s errors (permission denied for function int4eq, ... for aggregate count) before it opens anything, every time the plan starts, so a cached plan answers a REVOKE or a SET ROLE as the ordinary one does. Plain EXPLAIN initialises the plan too and core then checks the quals and the aggregates but not a HashAggregate’s grouping equality (its hash table is not built), so the node skips that one under EXEC_FLAG_EXPLAIN_ONLY. One deliberate difference: a merge join compares through the btree support function unchecked, so a join the planner would have merged runs with its operator revoked while the node refuses it - the node keeps the SQL meaning, the query calls =. The SQL functions stand for SELECT count(*) FROM t WHERE col = key (or = ANY (keys), or GROUP BY col), so they require EXECUTE on count() - checked as an aggregate, as above - and on the equality function they look up with: strategy 1 of the key column’s opfamily for (opcintype, the key’s type as “SQL surface” below resolves it), which is int48eq for an int8 key on an int4 column exactly as in the query, and (opcintype, opcintype) for the grouped form. These checks follow the exact SELECT check, under the lock (test/sql/security_exec.sql). One case follows core rather than the rule: a clause implied by a PARTIAL index’s predicate is dropped by the planner, so a keyless plain or index-only scan of that index answers without calling the clause’s functions, and without checking EXECUTE on them. Whether the clause is implied can depend on who built the index’s relcache entry (a role without EXECUTE sees an inlinable SQL function un-inlined, which then matches the query’s clause exactly). A B-tree behaves identically; the predicate was evaluated by whoever built the index, as core intends. The diagnostic functions check neither privileges nor RLS - lion_index_posting_root() answers key membership, lion_index_stats() gives counts, lion_index_verify() reads the heap, lion_index_wal_mode() locks whatever it is handed - so they are revoked from PUBLIC like pageinspect’s and amcheck’s (lion_index_stats() is granted to pg_stat_scan_tables, as pgstattuple’s functions are). lion_index_verify() names a missing row’s key value only to a superuser; a role it is granted to gets the TID (test/sql/hardening.sql). The cheap half of the count functions' check - SELECT on the table or on at least one column - comes BEFORE any lock is taken, so a role with no privilege cannot queue behind (and hold up) locks on a table it cannot read; the exact check follows under the lock, where the index definition can be trusted (third round of the 2026-09-23 review, test/isolation/count_lock_privilege.spec). - Install scripts. Every support function a script names is found in pg_catalog or, for citext, through @extschema:citext@ - never in the target schema, where a role with CREATE could plant one that then runs as whoever inserts into the index (test/sql/hardening.sql). - Row-level security. The SQL functions refuse a table on which RLS applies to the caller (policies would have to be evaluated per row); the CustomScan declines relations with security quals, so the ordinary plan applies the policies. - Table access method. Everything here reads the heap’s visibility map; the SQL functions refuse, and the CustomScan declines, a table whose table AM is not the heap, which ambuild has already refused to index (§2). - SERIALIZABLE. The count’s lookups and walks predicate-lock the index pages that stand for the keys they read, absent ones included (“SERIALIZABLE” below), and the blocks it counts from the visibility map are page-locked in the heap, as an index-only scan locks them. - Hot standby. The pin interlock relies on ambulkdelete taking cleanup locks, and what replay takes depends on which resource manager wrote the record (§25). Generic WAL: replay takes only exclusive locks, so on a standby a reader’s pin does not stop replay from removing TIDs and the following heap records from setting all-visible. In recovery a count over such an index treats every heap block as not all-visible and rechecks every candidate TID (still correct, no longer O(1) per container). The custom resource manager (§25, implemented): every record that removes a TID or deletes an item declares the page it removes them from, and lion_redo() takes that block with XLogReadBufferForRedoExtended(..., get_cleanup_lock = true), the way btree_xlog_vacuum does. The argument above then reads on the standby word for word with “VACUUM” replaced by “the startup process”: while the count holds a pin on the page a container came from, replay cannot have removed a TID from that page, so it cannot have replayed the heap record that set any of that container’s heap pages all-visible. The count uses the visibility map there exactly as on a primary, and the standby reader is paid for the way a primary reader is - replay WAITS, which max_standby_streaming_delay turns into a recovery conflict rather than a wrong answer. That sentence is only half of it (2026-09-22 review): the TIDs a reader copied can MOVE - a split to the right sibling, a root push-down to a new child (§22), an INLINE spill off the directory leaf - and be removed from the page they moved to, which the reader does not pin. On the primary that removal waits anyway, because VACUUM cleanup-locks every page in chain order (§11), but a page VACUUM locks and leaves unchanged writes no record, so replay used to take no lock on it at all. VACUUM therefore hands every such page to the next record that takes a cleanup lock at redo, as a BARRIER that redo cleanup-locks first (§25, xl_lion_visit), and the argument holds on the standby for moved TIDs as well. test/recovery/run.sh phase 3 parks a standby reader with its page pinned and has the primary split, push down or spill that page and then VACUUM: before the barrier all three answered the dead rows (50000 for 48548, 1000 for 500, 200 for 100); with it replay blocks on the reader’s pin and every answer is right. The decision is per COUNT and not per index (lion_sources_all_rmgr()): a container of an intersection carries the dead TIDs of every source it came from, so ONE generic-mode source puts the whole count back on rechecking everything. The index-only scan (§29.9) trusts the map for every TID it returns, and its pin is the same interlock, so it makes the same decision per index (2026-09-27 review; liongettuple() never asked whether it was in recovery, and returned the dead rows of a generic-mode index on a standby): in recovery, over a generic-mode index, it looks every TID up in the heap under its snapshot, as its UNION shape always does, and returns only the visible ones (test/recovery/run.sh phase 2 compares a standby’s index-only scan with its sequential scan, before and after a VACUUM on the primary, in both modes). A bitmap scan needs nothing of the kind: the bitmap heap scan fetches every page and applies the snapshot itself (its skip-fetch shortcut, which trusted the map, was unsafe for every index AM and is disabled in core’s current minor releases - 16.15’s can_skip_fetch = false - and gone from master), and a plain index scan fetches every tuple. - old_snapshot_threshold (PostgreSQL 16 only). With the threshold set, VACUUM and pruning use a horizon that may have passed an old snapshot’s xmin: they remove TIDs that snapshot still sees and mark their pages all-visible. Core’s access methods compare every page they read with the snapshot (TestForOldSnapshot()) and raise “snapshot too old”; lion had not one such test, so a count, a bitmap scan and an index-only scan answered silently differently inside one snapshot (2026-09-27 review) - a TID gone is a row missed, and nothing in the heap raises for a page it is never asked to read. Rather than test every page of every path, lion is not READ at all while the threshold is set (lion_check_old_snapshot(), the test in lion_compat.h, constant false from 17, which removed the setting): lioncostestimate() adds disable_cost to every path on a lion index, the LionCount (count, GROUP BY, FK join) and LionOrdered hooks offer nothing, and what reaches a lion index under an MVCC snapshot anyway - a direct SQL count, a plan with every other path disabled - is an ERROR (ERRCODE_FEATURE_NOT_SUPPORTED) in amgettuple, amgetbitmap and the count functions. A lionvec index (§44) is not read either (2026-10-18; until then its scans ran under the threshold, and only its reads of vectors by their TOAST places, §44.13, were off). It has no such test: VACUUM’s early horizon removes a dead row’s CODE items as any other’s, a scan never meets the row again, and 16’s index fetch (heapam_index_fetch_tuple(), heap_hot_search_buffer()) tests no heap page, so an old snapshot’s nearest neighbours came back short without a word, on the exact path as on the probing one. lionveccostestimate() adds disable_cost to every order on one - a filtered order is then planned exactly, not refused under pg_lion.vector_postfilter = error, since no lionvec order is read for a Filter to follow - and amgettuple, its only read (it has no amgetbitmap), errors before it reads a page (lv_start()), in each process of a parallel scan. The snapshot is not known yet in ambeginscan, which is why the refusal is in the calls that read. Inserts and VACUUM are not affected; neither is lion_index_verify(), whose heap scan raises “snapshot too old” itself on any page early pruning changed, and whose structural checks use no snapshot. The setting is a postmaster one, which a regression file cannot set, so the test is make hookcheck’s: on 16 alone it restarts the dev cluster with old_snapshot_threshold = 60 for test/modules/lion_hooktest/sql/oldsnapshot.sql, which checks that a count, a GROUP BY, a plain scan’s query, a LionOrdered and a LionBm25 query are all planned without lion, that a bitmap and a plain scan forced onto a lion index, and every direct SQL count and lion_bm25(), stop at the ERROR, and that inserts, updates, deletes, VACUUM and lion_index_verify() still work (2026-10-18 review: the refusals had no test at all).

Algorithm lion_count_keys(Relation heap, int nkeys, Relation *indexes, Datum *keys, Snapshot snap) 1. For each (index, key): locate the entry (bucket head SHARE lock; copy the entry header; for INLINE entries copy the payload while holding the pin; for CHAIN entries note head). Pins are budgeted: an INLINE set located past the budget holds none, and §15’s “The pin budget” says how the count keeps this section’s rule for it. 2. Merge-iterate the k posting sets by ckey (they are sorted). For the AND of k containers with the same ckey use lion_container_and into a work buffer (k-1 times). Containers whose ckey is missing from any set contribute nothing. Pin discipline: at any time hold at most one pinned page per index (the page whose containers you are currently consuming). Advance a set’s cursor page only after its containers have been fully consumed, and consume means: for the resulting AND container, do the VM checks (step 3) before any source page pin is released. 3. For the result container: for each of the ≤ LION_BLOCKS_PER_CONTAINER heap blocks with members (compute member counts per block via lion_container_range_cardinality over the block’s lo range; skip blocks with 0), visibilitymap_get_status(heap, blk, &vmbuf); if all-visible: count += members, PredicateLockPage; else: append the block’s TIDs (lion_code_to_tid) to a recheck list. 4. After the merge completes, recheck: sort the TID list (it is already in TID order if produced in ckey order), and for each TID call table_index_fetch_tuple(fetch, &tid, snap, slot, &call_again, &all_dead) in a loop over call_again (HOT chains); count each visible tuple found. Use table_index_fetch_begin/end. Tuples for which the fetch finds nothing are not counted. (In PostgreSQL 20devel that API is gone and table_fetch_tid() replaces it; the heap AM is served by one heap_hot_search_buffer() loop per heap BLOCK instead, under one share lock.) 5. Return the count. Memory: the recheck list lives in a per-call context and is BOUNDED - it holds at most work_mem worth of TIDs (floor: a few thousand). When it fills up, the batch it holds is rechecked right there, its visible rows are added to the running total and the list starts over; the flush waits for a heap-block boundary, so no block is pinned twice and blocks_rechecked stays exact - the first boundary past which the next block might not fit (a block puts at most LION_RECHECK_BLOCK_TIDS, 512 at 8 kB pages, on the list), so the list never outgrows its budget. That matters to every queue of a grouped walk (§10): a flush that waited for the end of the block it was on would take each one a step of 1024 TIDs past its share. Without a bound, 20M candidates - which a standby produces for any 20M-row posting set, since it never trusts the VM - meant a 192 MiB array, and 134M candidates an impossible allocation. That holds for ONE merge, whose list comes out in block order. A count taken in PASSES - the disjoint sum’s entries (§15), a long list’s batches (lion_run_batches(), §15 “Bounded cursors”) - yields every container key from the first again in each pass, so the dirty blocks one pass queues are the next pass’s too, and a flush would read them again in every pass after it: flushed so, at 64 kB of work_mem x IN (1, 2) over a fully dirty heap of 4,286 pages would recheck 8,573 blocks (the regression is test/sql/countbudget.sql’s). Such a count’s list (LionCountCtx.passes) is not flushed while a pass is still to come: once it is full, it and every TID after it go to a sort of their codes (lion_recheck_spill()) that is given work_mem and writes sorted runs to temporary files past it, as a Sort node does, and the flush at the end reads it back in order into a list of LION_RECHECK_MIN_BATCH, flushed at block boundaries like one merge’s, so every dirty block is read once (EXPLAIN ANALYZE: Heap TIDs Spilled). Each TID waits in the sort for the recheck an unbounded list would have made at the end of the count, and a recheck needs no pin (below). An existence test (§26) is not taken in passes: it rechecks after every container until it is settled, and may read a block of a later entry again. Counts of their own - a GROUP BY’s groups, the batches a node locates a long list in (“A list too long to locate at once”), a range walk’s entries - share the visibility cache of step 6 instead. Flushing mid-merge does not weaken the interlock above: the ordering rule is that the VM question about a container’s blocks is asked before the pin on the page that container came from is released, and a flush asks no VM question - those are what put the TIDs on the list. Rechecking a TID needs no index pin at all: if VACUUM removed that TID meanwhile, the tuple was dead to every snapshot including ours (nothing to count, and nothing is found); if the line pointer was reused, the new tuple’s xmin is later than our snapshot and is invisible to it. 6. Per-query visibility cache (LionVisCache, lion_count.h). A grouped count asks the recheck about the same dirty heap pages once per group: at 5M rows with 200 groups and 5% of the rows updated, 9,062 dirty pages cost 415,884 heap block visits (§12’s grouped-count finding). The answer cannot change between those visits, so it is resolved once per (page, snapshot) and reused. The cache maps a heap block to a bitmap of MaxHeapTuplesPerPage bits - which of its ROOT line pointers hold a tuple visible to this snapshot - in a hash table in a per-node memory context, bounded by work_mem in entries; when the budget is reached nothing more is inserted and the blocks that did not get in are fetched per batch exactly as before (cache_full counts those visits). - Where: in the flush only, i.e. after the VM check, exactly where rechecks happen today. The pin rule of the argument above is untouched; the cache changes nothing about which TIDs are rechecked, only whether the page has to be read for them. - Why an answer keeps: heap_hot_search_buffer(root TID, snapshot) is a function of the snapshot. A tuple visible to us is not dead to all, so nothing may remove it; HOT pruning may rewrite the chain, but it only drops versions dead to every snapshot and leaves the survivors reachable from the root in order, so the version found from a given root does not change (the same guarantee a bitmap heap scan relies on between building its TID list and visiting the heap); a “not visible” answer cannot become visible, because every tuple written after we looked belongs to a transaction that is in progress at our snapshot, or began after it, or is ours with a command id at or above the snapshot’s curcid; and line pointer numbers never move. - Serializable isolation: a cache hit calls neither HeapCheckForSerializableConflictOut() nor PredicateLockTID(), so filling an entry takes PredicateLockPage(heap, blk, snapshot) once. Both directions of rw-conflict detection are then closed: a write that happened BEFORE the page was resolved is caught by the conflict-out check inside the sweep (which walks every chain on the page, a superset of what any one recheck tests), and a write after it by the writer’s own CheckForSerializableConflictIn() against that page lock. Page locks may be coarsened to a relation lock by SSI, which is conservative. - When it fills: the sweep costs one visibility test per line pointer instead of one per wanted TID, so it is only done once reuse is evident - the first count records nothing, the second records the blocks it visits, and a block is resolved on its second visit. A one-shot count therefore behaves exactly as it did before the cache existed, down to the last buffer visit. - Lifetime: one handle per count node execution, created by the caller (lion_vis_cache_create/_destroy) and passed to lion_count_sources_cached(); NULL means no cache. The handle empties itself when it is handed a different relation or a different snapshot, so the partitioned counts of §16 may share one handle and get one cache per partition. lion_index_count_group_stats(idx, use_cache) is the SQL image of that driver, for tests: same entry scan, one count per group, one cache.

SQL surface for tests: lion_index_count(idx regclass, key "any") RETURNS bigint and lion_index_count(idx1 regclass, key1 "any", idx2 regclass, key2 "any") RETURNS bigint, whose keys are each compared with its own index’s column, and the two columns need not share a type. The keys are "any", not polymorphic, so that an untyped literal arrives as one (unknown, which a polymorphic argument refuses), as it would in col = 'abc'. Both open the heap via IndexGetRelation with AccessShareLock, use GetActiveSnapshot(), and must return exactly count(*) of the equivalent SELECT. The key is taken as col = key would take it. A key whose own type the column’s opclass takes is looked up as that type (lion_count_key_type(), below). Any other key is first CONVERTED as the query converts it (lion_count_coerce_key()): the parser’s own make_op() (make_scalar_array_op() for lion_index_count_any()’s col = ANY (keys)) resolves the operator named as the index’s equality is named - =, or a class’s own ==== - for (the column’s type, the key’s type) and coerces the key to its input, an untyped literal through the column’s input function, 1 to numeric on a numeric column, a numeric 0.1 to float8 on a float4 one; the coercion is evaluated once, its cast functions checked for EXECUTE as the executor checks the query’s. The count is made only where the operator the parser picked is the index’s equality - strategy 1 of the key column’s family with the column’s type on its left - and the column reaches it unconverted, relabelled at most. Then the count is the query’s: f4 = 0.1 is float4 = float8, which no float4 value satisfies, and the count is 0 as the query is, where rounding the key to float4 would have counted the rows of 0.1::float4. Otherwise the query compares something the index does not hold - intcol = 1.5 is numeric = numeric on intcol::numeric, bpcharcol = 'x'::text is text = text on the column cast by rtrim1() - and the count is an ERROR whose hint says to cast the key to the column’s type, never a number made another way (1.5 cast to integer would count the rows of 2). A key no operator takes is refused as the query would refuse it. The key’s own type is resolved by lion_count_key_type() (2026-09-25 review; before it the key had to BE the index’s opcintype or have a cross-type member, so enum_ops, a DEFAULT class, could not be counted at all and neither could a varchar key on the varchar column it indexes). A domain is its base type. For a class on a POLYMORPHIC type (enum_ops) the key must be of the column’s actual type and nothing else: a different enum would pass for “an enum”, and its OIDs mean nothing to this column (the DETAIL names the column’s type, not anyenum). Otherwise the key is of the class’s own type, or of a type the family has a strategy-1 member for with it (int8 on int4: int48eq, which §21’s probe resolves further), or - lacking one - BINARY-coercible to the class’s type where the parser makes that coercion itself: its = for (column type, key type) must be the class’s own (opcintype, opcintype) operator reached without a cast function (compatible_oper()), as for varchar on text_ops, which the parser relabels to call texteq. A coercion merely existing is not enough: text is binary-coercible to bpchar, but bpcharcol = 'x '::text is text = text on the column cast by rtrim1(), and matches none of the rows bpchar’s own equality would count. A key of any other type goes through the conversion above. The resolved type is what the lookup is made as and what the EXECUTE check names the equality by (enum_eq, texteq; test/sql/security_exec.sql). lion_index_count_any() resolves its array’s element type the same way. So is its collation (lion_count_check_collation(), 2026-09-28 review; the functions used to look every key up under the index’s collation whatever the call said). The index hashed and compared its keys under its own collation; col = key compares under the key’s when the key brings one of its own - an explicit COLLATE, or a column of another collation, read from the argument’s own expression (lion_count_arg_collation(), so that the two-key form’s keys are taken one by one, as the two clauses of its query are) - and otherwise under the column’s: a literal or a parameter brings the default collation, which gives way to the column’s in the parser’s rule for an operator’s inputs. The column’s collation is the table column’s, or the expression’s for an expression column, and need not be the index’s: (c COLLATE "x") keeps c and compares under x. Two deterministic collations agree on which values are equal - each calls two values equal when their bytes are - so a count under another deterministic collation than the index’s is made, and is the query’s. A nondeterministic one agrees with no other: a case-insensitive index counts ‘ABC’ under the key ‘abc’, and c = 'abc' under the column’s own collation does not; so where either collation is nondeterministic and the two differ the count is an ERROR (collation_mismatch), never a different number. lion_index_count_group_stats() stands for GROUP BY col, which groups under the column’s collation, and is refused the same way. A key of the default collation (2026-09-29 review) is the one the call cannot read. An explicit COLLATE "default" does not give way to the column’s collation - c = 'abc' COLLATE "default" on a case-insensitive column counts ‘abc’ alone - but the planner folds a COLLATE into the literal or the parameter under it (eval_const_expressions() turns the CollateExpr into the Const’s own constcollid), and the function’s inputcollid is “default” either way, so the FuncExpr the function is handed is the same with it and without it. Taken for no COLLATE at all, as it was, it counted ‘ABC’ too: 600 rows where the query counts 200. Both readings are taken instead, and the count is made only where both would make it. They part on one case alone, a column of a nondeterministic collation counted through an index under that collation: the literal’s reading counts, the explicit one is refused. That case is an ERROR (indeterminate_collation) whose hint says to name the column’s collation on the key ('abc'::text COLLATE case_insensitive), which is then counted as the column’s queries count; everywhere else - every column of a deterministic collation, the database’s default among them, since a default collation is always deterministic - nothing changed. A key taken from a column brings that column’s collation and is read as such. test/sql/countcoll.sql covers both halves; the nondeterministic one needs ICU and a UTF8 database and is skipped without them (countcoll_1.out), as it is on CI’s source builds; test/sql/countcoll_default.sql the default collation’s key, likewise. A NULL key counts 0, as count(*) WHERE col = NULL does: the functions are not STRICT. The key’s type is resolved and every check a key gets is made - the index, the type, the collation, the privileges - and then no lookup: col = NULL selects no row. A NULL index names no index and answers NULL. A NULL ELEMENT of lion_index_count_any()’s array selects nothing, as in = ANY (...), and counts 0, and so does a NULL array. They, lion_index_count_any() and lion_index_count_group_stats() refuse a MULTI-KEY column (§17): its entries are extracted keys, not column values, so a whole tsvector as the search key matched no entry’s meaning and used to be hashed and compared as if it did. All of them refuse a materialized view created WITH NO DATA with core’s error (“has not been populated”), as ExecOpenScanRelation() refuses the query, between the privilege checks and the EXECUTE checks where the executor raises it; its heap and indexes are empty, so the count used to answer 0 (2026-09-25 review; the pushdown node already refused it). Nobody vetted the index they were handed, so they also make the decision the planner makes in get_relation_info() before looking anything up: lion_index_usable(index, snapshot, &why) (lion.h, implemented in lion_state.c, shared with lion_index_verify’s heapallindexed pass) requires indisvalid and indisready and applies the indcheckxmin rule against TransactionXmin exactly as plancat.c does; an unusable index is an ERROR naming the reason, never a count. This is not an optimisation: an index built from a broken HOT chain holds only the latest version of that chain, so an older snapshot’s row is not in the posting set and no amount of heap rechecking can put it back (the 2026-09-20 review reproduced count(*) WHERE k = 1 = 1 against lion_index_count() = 0; test/isolation/count_checkxmin.spec is that case). Isolation tests must cover: concurrent uncommitted insert (not counted), committed insert (counted in a new snapshot, not in an old REPEATABLE READ one), delete + VACUUM racing with the count (a cursor or pg_sleep-free spec using isolationtester steps: s1 opens a REPEATABLE READ transaction and counts once, s2 deletes rows and commits, s3 runs VACUUM, s1 counts again and must still see the old value; then in READ COMMITTED s1 must see the new value), and an index the caller’s snapshot may not use at all (count_checkxmin.spec: a HOT update before CREATE INDEX makes the build set indcheckxmin, and the old REPEATABLE READ reader must get the eligibility error rather than a count that is missing the row it still sees).

Direct readers (2026-10-08)

An index scan the executor starts through index_beginscan() gets four guarantees from PostgreSQL: the planner offers only an index the snapshot may use, row-level security is applied above it, and the executor refuses an unpopulated materialized view (the fourth, the predicate locks, every reader takes the same way: “SERIALIZABLE” below). lion reads its indexes without a scan in two kinds of places, and lion_reader.c is where each gets what is missing:

  • Functions handed an index by name (the count functions, lion_bm25()) call lion_reader_vet(), or lion_reader_open(), which opens the index and table and then vets them. Vetting refuses an unpopulated materialized view, a table where row-level security applies to the caller (the policies cannot be evaluated without reading the rows; the same query through the planner applies them, since lion’s custom paths decline relations with security quals), and an index lion_index_usable() rejects for the snapshot. It also applies the PostgreSQL 16 old_snapshot_threshold guard. lion_reader_open() takes a policy: LION_READ_ROWS for anything that returns or counts rows, which vets, and LION_READ_STATS for lion_bm25_score(), which reads only N, df and avgdl, and checks privileges and row-level security only. The statistics count every row, hidden ones included, so under a policy a score - even of a made-up document, even per row of a query that applies the policies - would tell whether hidden rows hold a lexeme, and how many. It also refuses a partial index: both its callers rank over the whole table, and a partial index’s walk would miss the rows its predicate leaves out while its N, df and avgdl described only the rest.
  • Custom scan nodes, whose index the planner vetted, need none of it.

Privileges beyond SELECT on the table (per-column grants, EXECUTE on the functions a count stands for) depend on what the call stands for and stay with the caller (lion_count_open_indexes()).

The ranking functions first opened their index on their own and missed three of these. So the contract is now tested across every reader rather than per feature: test/sql/reader_contracts.sql (row-level security, an unpopulated materialized view, EXECUTE revoked on a key type’s equality, an index that is not valid) and the reader_checkxmin and reader_serializable specs each list every reader. A new reader adds a line to each.

The custom scan nodes keep the executor’s side of the contract for what they stand in for (a clause’s operator answered from a posting set, an aggregate, the BM25 score), and that is tested across every node the same way: test/sql/node_contracts.sql runs one query per node with the function revoked from PUBLIC (a role without it, plain EXPLAIN, a generic plan made by a role with it, SECURITY DEFINER functions) against the ordinary plan, and test/modules/lion_hooktest/sql/exec_hook.sql runs the same list under an object access hook that refuses the function (make hookcheck). A new node adds a line to both.

SERIALIZABLE

The handler sets ampredlocks, and the locks are page locks, as GIN’s are: what a read depends on is where it found each key it looked for, or where the key would have been, and the reader locks the page that stands for that (lion_reader.c, which every reader goes through, the AM’s own scans included):

  • an INLINE entry, or no entry at all: the directory leaf that holds it, or would;
  • a CHAIN entry: its posting tree’s root, which stays at its block for the life of the set (a root split pushes down, §22), so a lock on it stands for the key alone;
  • a walk (a range, a GROUP BY, IS NOT NULL, an ordered walk, the fk join’s groups): every leaf it reads, which covers every key an insert could put between the ones it saw, and the root of every CHAIN entry of its column on them.

Each lock is taken while the leaf the entry or the gap was read from is share-locked, before the reader believes anything the entry’s header says (a set with no rows is skipped, a top-k is cut by the counts). An insert checks the one page that stands for its key - the leaf for a new or an INLINE entry, the root for a CHAIN one; a summary insert (§32) its bucket’s, and the leaf too when it rekeys or adds a bucket - with CheckForSerializableConflictIn(), while it holds that leaf exclusive and before it changes anything (lion_writer_check()). It keeps the leaf until its TID is in the set. So either the insert comes first and the reader meets its TID, whose heap visit tells SSI of the conflict (a block counted from the visibility map is never one an uncommitted insert has touched), or the reader’s lock is there when the insert looks.

The locks follow the keys. A directory leaf split copies the leaf’s locks to its new right sibling (lion_dir_split(), PredicateLockPageSplit()); the spill of an INLINE set copies its leaf’s locks to the new root (lion_entry_spill_pos()), as GIN’s createPostingTree() does; and VACUUM, deleting an empty CHAIN entry, moves its root’s locks to the leaf (lion_vacuum_delete_entries(), PredicateLockPageCombine()), where an insert of the key will now look; deleting an empty directory leaf moves its locks to the right sibling that takes its range over (lion_dir_delete.c, §21), as nbtree does. Posting-tree pages other than the root carry no locks.

Two cases lock the whole index: a lookup of an absent key that followed a run of colliding hashes onto the next leaf (an insert of the key starts from a leaf to the left of where the lookup ended), and the cross-type fallback of lion_dir_find(), which reads every leaf. The aggregates over keys (§37) lock the heap instead, as the sequential scan they stand for does; that also covers every insert, and is one lock where the index’s would be every leaf and root. BM25’s N and df are read under the same locks as the entries they come from (the empty key’s entry, the lexemes'), so a ranking conflicts with every insert of a document, as its scores depend on N and avgdl. A lionvec index (§44) has no ampredlocks and core locks it whole.

Locks the planner’s estimates or lion_verify() take in a SERIALIZABLE transaction can only add false conflicts. test/isolation/serializable_keys.spec is the write skew through every reader, which must fail, against inserts of keys the reads did not touch, which must commit, for CHAIN keys that share one leaf and INLINE keys on leaves of their own; serializable_moves.spec moves the key between the read and the insert by each of the three transfers above; reader_serializable.spec has every reader’s absent key.

10. Phase 2b: CustomScan for count(*) [GROUP BY k] FROM t WHERE k1 = c1 AND ... (lion_plan_*.c, lion_exec_*.c)

Planner integration - Install create_upper_paths_hook (chaining to any previous hook) at _PG_init; act only for stage UPPERREL_GROUP_AGG (and UPPERREL_DISTINCT over one base relation, end of this section). GUC pg_lion.enable_count_pushdown (bool, default on). - Applicability, all required (bail out silently otherwise): - input_rel is a single base relation (RELOPT_BASEREL, RTE_RELATION, relkind ordinary table or materialized view) — no joins, no subqueries, no old-style inheritance parents. §16 added partitioned parents, which are counted one live leaf partition at a time; everything this section says about “the relation” then means “the partition being counted”. §27 added one join shape - a fact table joined to a dimension on a lion-indexed fk - where “the relation” is the fact table and the dimension side is a child plan. - The relation’s table AM is the heap, by routine (§2). ambuild already refuses any other, so this only declines an index that got past it; the executor checks again when it opens the relation. - The query has no DISTINCT (but a SELECT DISTINCT over the table alone is a GROUP BY of its keys, at end of this section), grouping sets (but those are counted as their finest grouping, “Grouping sets” at end of this section), ORDER BY inside aggregates, or aggregates other than count(*) (or count(1) of any non-NULL constant) and the count(col) cases of §14 - and, without a GROUP BY, count(col) of any one column whose index answers col IS NOT NULL. A FILTER, and a sum or count of a one-armed CASE, is taken when it is a filtered count (“Filtered counts” at the end of this section). The target list holds those counts, the columns the rules below let it print, and any expression over them; window functions over them are computed above the node (“Shapes rewritten to the node’s counts” at the end of this section). - HAVING is accepted when it is a filter the node can apply itself. By the time the hook runs, havingQual is an implicit-AND list of only those clauses that mention an aggregate (or are volatile, or contain a subquery): subquery_planner() has moved every other clause into WHERE, so HAVING a > 3 reaches us as a WHERE inequality and is refused like any other. Of what is left, every Var must satisfy the target-list rule below (a group column, or a column a clause pins to one value, whose printed value is the stored key) and every aggregate must pass the count test of §14; a SubPlan (a correlated subquery, evaluated per group) is refused, while an uncorrelated one is an InitPlan by then and arrives as a Param, which is a plain value. The accepted list becomes the CustomScan’s plan.qual; setrefs.c rewrites its Aggrefs and Vars into INDEX_VAR references against custom_scan_tlist, which is why the plan step adds every column and count the HAVING mentions to that list even when the target list does not print it (an Aggref left unmatched there would reach ExecInitExpr, which only an Agg node may do). lion_emit_tuple() evaluates the qual on the finished group’s scan tuple; a group that fails is consumed like any other and the outer fetch loop asks for the next one, counting it in Rows Removed by Filter. The cost adds the qual’s evaluation per group and scales the row estimate by clauselist_selectivity(), as cost_agg() does for an Agg’s quals. A partitioned table’s node emits partial counts, so its HAVING goes to the Finalize Agg above it instead (§16), and the partial target carries the HAVING’s counts as well - core’s make_partial_grouping_target() does the same with the havingQual. - Every baserestrictinfo clause is Var opeq Const or Const opeq Var where Var is a plain column of the rel with a valid lion index whose opfamily contains that operator as strategy 1 (use the index’s opfamily and the operator OID; cross-type integer equality is fine because the integer opfamily contains it), and the compared value is not a literal NULL. §15 adds Var = ANY (array) and §14 the two null tests to the shapes accepted here. §19 adds a top-level OR of such clauses, whose leaves enter none of the per-column bookkeeping below, because they constrain no column of the result; §35 adds <>, and NOT IN of a literal list is its <> clauses (end of this section). The “Const” is anything the node can evaluate once per scan - a literal, a Param, a stable expression over them - and a boolean column tested by itself is the equality it stands for (both below). - Several clauses may constrain one column, and each is a source of its own (2026-09-25 review). A second positive clause on a column is ANDed with the first exactly as a clause on another column is: it is matched to an index for ITS operator under ITS input collation, and the query is declined when there is none - the AND of exact sources is exact. Only the very same clause again is dropped as a duplicate: the same kind, the same operator, the same input collation and an equal() value (so v === 'A' AND 'A' === v is one clause, and v === $1 AND v === $2 is two), because the first one’s lookup is then its answer. The rule used to be “at most one clause per column”, with a duplicate recognised by its kind and value alone, and that dropped clauses no index had answered: over an index whose case-insensitive opclass has === as strategy 1, v === 'A' AND v = 'A' counted the entry of === - both spellings, 10000 rows where the query selects 5000 - and never asked anything about =. A nondeterministic collation does the same with one operator, s COLLATE ci = 'a' AND s = 'a'. The generic plan (v === $1 AND v = $1), the IN-list forms, a GROUP BY and the FK-side join’s fact filters (§27) all went through that one test. Nothing downstream needs one clause per column: the key a target list prints for a pinned column is its FIRST pinning clause’s, and every equality on that column must then pass the value-representation rule below; an IN list that drives the groups (§15) is the FIRST list on the driving index’s key column, by one rule in the planner and the executor (see the executor’s GROUP BY paragraph). Two clauses that differ only in value used to be declined as well - they select little or nothing - and are answered now: under a coarse equality they need not even disagree (v === 'a' AND v === 'A' is one entry, looked up twice). test/sql/samecolumn.sql covers every form against a sequential scan, with the other plans disabled so that a node that is built at all is the plan. IS NOT NULL, a multi-key clause (§17) and an OR leaf never took part in the old rule and are unchanged. Not done: a clause is matched against the FIRST lion index on its column only (lion_find_roaring_index()), so two clauses that two different indexes of one column would answer - === from a case-insensitive one and = from a plain one - are declined, not answered; that is a plan left on the table and not a wrong answer. - An index is an (index, KEY COLUMN) pair since §24. A multicolumn lion index holds each of its columns' keys as an independent set of entries, so lion_find_roaring_index() accepts a match on ANY key column and returns its number i beside the index; every opclass question in this section is then asked of THAT column - opfamily[i], opcintype[i], indexcollations[i] - and every lookup, entry scan and datumCopy() of that clause names the column (lion_posting_set_lookup_col(), lion_entry_scan_begin_col(), lion_index_column_state()). Two clauses of one query may name the same index; a query constraining three columns of one index is three posting-set sources located through one relcache entry, ANDed exactly as three separate indexes would be. The column is NOT carried in custom_private: the executor derives it in lion_open_relation() from the index it really opened and the clause’s heap attnum, because a partition’s index may put the same heap column at a different position from the parent’s (§16) - and may number the heap column differently too, which is resolved by name. - A Param stands wherever a Const may (2026-09-21 follow-up review). A prepared statement’s GENERIC plan keeps k = $1 as a Param - that is what a generic plan IS - and accepting only literals meant the node was never used by one: a dense count fell back to the ordinary plan and measured 5.6 ms against 0.025 ms on 100k rows, and 207 ms against 3.1 ms on five million. Both parameter kinds are accepted, PARAM_EXTERN for a statement’s own parameters and PARAM_EXEC for the ones a nested loop or a LATERAL reference supplies, for equality and for the array of an IN list; k IN ($1, $2) keeps an ArrayExpr over them in a generic plan, and that is accepted too, as long as its elements are themselves literals or parameters (the node evaluates the array ONCE per scan, and an element that could be volatile does not mean the same thing evaluated once as it does evaluated per row). A multi-key clause (§17) required a Const until 2026-09-27, because there the query’s SHAPE and not just its value decides whether the posting sets can answer it at all; it takes a Param now, and answers the values its posting sets cannot answer exactly from a superset rechecked in the heap (§17, “A query known only at run time”). At plan time the value is unknown, so clause_selectivity() gives the estimate it gives any non-Const comparison; at run time a NULL value selects no rows, which every operator involved agrees with by being strict. - So does any expression an index scan would take as a run-time key (2026-09-27). A time window is written ts >= now() - interval '90 days' or d >= current_date - 30, and neither is a Const or a Param: eval_const_expressions() folds only immutable functions, and now(), current_date and timestamptz arithmetic (which depends on the time zone) are stable. Such a clause was declined before any cost was asked, on the filter nearly every time-windowed count carries. lion_is_value_expr() now takes core’s rule for an index scan’s run-time keys (ExecIndexBuildScanKeys()): no Var of any level, no volatile function, no SubLink or SubPlan, no aggregate or window function, no set-returning function, and a Param only of the two kinds above. A stable expression has one value for the whole statement, so evaluating it once per scan - through the same ExprState, at the same moment and again after every ReScan as a Param (lion_eval_clause_values()) - means what evaluating it per row means; k = random() does not, and is declined. It travels in custom_exprs like a Param, so setrefs.c records its functions as plan dependencies, an exec Param inside it (pc = v.k + 1 under a LATERAL nested loop) rescans the node, and a cached generic plan evaluates it anew at every execution: d = $1::date + 1 under force_generic_plan, and a STABLE function whose answer changes between two EXECUTEs of one cached plan. At plan time it is only ESTIMATED. Core’s selectivity functions already reduce each side of a comparison with estimate_expression_value(), so the clause is priced as its plan-time value would be - the same cost to the unit as the literal, in the benchmark’s reproduction - and an IN list whose array is an expression other than a literal or an ARRAY[] (k = ANY (string_to_array(current_setting( ...), ','))) is estimated by the clause analysis itself, so that §15’s length cap applies to the estimate and the cost model prices the list’s real length rather than estimate_array_length()’s default for an expression. An ARRAY[] over such expressions is taken element by element, as a generic plan’s ARRAY[$1, $2] is. EXPLAIN deparses the expression like any clause value, ts >= (now() - '30 days'::interval). A multi-key clause (§17) takes one as its query too - tsv @@ to_tsquery(current_setting(...)) - under the same rule as a Param. - A boolean column tested by itself is the equality it stands for (2026-09-27). eval_const_expressions() folds flag = true into the bare Var flag, and flag = false and flag <> true into NOT flag, before any path is built, so no boolean equality ever reaches the pushdown as an OpExpr - and flag IS TRUE never was one. Core’s index matching takes all of these through match_boolean_index_clause(), which is why a lion index scan printed Index Cond: (flag = false) for NOT flag while a count of the same rows could not be pushed down. The clause analysis now does the same (lion_boolean_eq_test()): flag and flag IS TRUE are flag = true, NOT flag and flag IS FALSE are flag = false, with BooleanEqualOperator (bool_ops' strategy 1) and a boolean Const as the clause’s value, so that the executor looks it up like any literal and EXPLAIN prints flag = true as core prints the index condition; the cost model is handed that OpExpr in place of the bare column. All four are false for NULL, as = is. flag IS NOT TRUE and flag IS NOT FALSE are true for NULL as well and so are no equality: they are the OR of the other value and the column’s NULL entry, flag = false OR flag IS NULL, which §19 takes apart like any other OR - as a whole restriction, flattened into the arms of one, or inside an AND arm distributed into two arms (lion_boolean_not_test(), and §19’s “Nested ORs, distributed”; until that was written it was declined there). flag IS UNKNOWN and flag IS NOT UNKNOWN are §14’s two null tests. A domain over boolean is relabelled to boolean by the WHERE clause and answered by bool_ops unchanged. - Enum keys: the class’s own type, for every clause kind (2026-09-27). enum_ops is FOR TYPE anyenum, so its members are (anyenum, anyenum) and so is its hash proc, while the constant of k = 'x' and the elements of k IN (...) are of the column’s own enum. lion_match_index() resolved the value’s type to the class’s for a range bound (§28) and for nothing else: for an equality or an IN list it looked up an (anyenum, mood) member, found none and declined the index, so an enum column was never counted by the node at all - with every other scan disabled the plan was a disabled sequential scan, on the very low-cardinality columns the node is best at. The resolution is now made before any member is looked up, for every clause kind, by the rule lion_probe_init() applies at run time (§22’s addendum on key types): the same BASE type as the key column’s own, so that a domain over the enum - which the parser only takes cast to its enum, dstatus::mood = 'x' - is its enum, and another enum, whose OIDs mean nothing to this column, is not. GROUP BY an enum, the value-representation rule, the FK-side join (§27) on an enum key and EXPLAIN needed nothing more. lion_index_posting_root() made the same mistake (keytype != opcintype) and refused every enum key; it takes the column’s enum, or a domain over it, now - and, since the 2026-10-18 review, a domain over any key column’s type and a value of the column’s own type (a varchar on a varchar column of text_ops), the keys the count functions take that have the column’s representation; a cross-type key (int8 on int4), which the counts resolve through a probe, it still refuses. - Collations follow the planner’s IndexCollMatchesExprColl() rule: a collation-sensitive clause (OpExpr/ScalarArrayOpExpr inputcollid valid) or grouping column may only use an index whose indexcollations[i] - of the KEY COLUMN that indexes the clause’s column (§24) - equals that collation, because the index hashed and compared keys under its own collation and the count never rechecks the predicate. Checked per partition. - Partial indexes (2026-10-19, §45): the lookups read whole indexes only, except in an attempt made for a partial index whose predicate the restrictions imply, which they then try first; the restrictions the predicate implies are left out, and the count is held to the predicate’s rows (§45.1) or the attempt declined. - GROUP BY is empty, or exactly one plain Var of the rel with a lion index - or two of them, which §20 answers as a nested loop over the two indexes' entries, applying every rule of this section per column. Since §24 the two may be two KEY COLUMNS of ONE multicolumn index, which is that feature’s natural use: the relation is opened twice, each side walks its own column’s entries, and the outer/inner choice is made from the columns' cardinalities exactly as it is between two indexes. A grouped column may also appear in the WHERE clause (then it is a single group). §14 removed the attnotnull requirement: the NULL group comes out of the reserved NULL entry. - GROUP BY coalesce(col, c) (2026-09-28, from a benchmark). A report that labels the rows with no value writes the group as coalesce(col, c), and only a bare column used to be accepted. Its groups are col’s with ONE change, which is where an inexact answer would come from: the rows where col is NULL are counted in the group of c - a group of their own, printed as c, when no row has col = c, and MERGED into that key’s group when one does, so that the key is emitted once with the two counts added. The node does exactly that. lion_next_group() walks col’s entries as for GROUP BY col and takes two of them out of the stream: the reserved NULL entry, and the entry whose key the grouping equality finds equal to c (the equality operator’s function called on each key under the grouping collation, one call per entry). Their counts are added up instead of emitted, and the one group they make comes out as c after the last entry, when it has a visible row. That is exact whatever order the two come in and whichever of them exists: the classes of one index are disjoint, so no second entry can equal c; an entry of c that VACUUM deletes before the walk reaches it held no row the snapshot sees (§18), and one inserted meanwhile holds none either, so the group is the same without it; and the decision is made in the walk itself, never by a lookup of c at another moment. The value printed for that group is the Const - the constant as the query wrote it, which is what core prints for NULL rows - and for c’s own group it is the value those rows hold as well, because a key the target list prints is only produced by an index whose equality implies an identical representation (the value gate below). What lion_group_coalesce() requires, and why: - exactly one GROUP BY expression, coalesce(col, c) as eval_const_expressions() leaves it: a plain column of the relation and a non-NULL Const (NULL arguments and everything after the first non-NULL constant are gone by then, and a constant expression is folded). A Param - coalesce(col, $1) under a generic plan - another column or a stable function is declined; so is anything wrapped around the expression; - no coercion: the column, the Const and the expression all of the column’s own type, so that the grouping equality (SortGroupClause.eqop, which the next rule makes the driving index’s strategy 1, as for a bare column) compares c with the stored keys as it compares the rows. A cast of the column - coalesce(i4::int8, 0) - is a function call and declines, even where it would keep equality, and a relabelled one declines too; the constant is coerced to the column’s type by the parser, which is what a written coalesce(col, 0) is; - the expression’s collation the column’s, which the driving index’s collation has to match per relation as for a bare column, so that c is compared under the collation the GROUP BY groups under (coalesce(t, 'x' COLLATE "C") over a default-collation index declines).

The target list and the HAVING may use the expression itself, which the node prints as the
group key, and the counts; not the bare column (a HAVING that takes the expression apart) and
no other expression over it. `count(col)` of the coalesced column is the one count the merge
changes - in the merged group it is c's rows alone - so it, and a `count(DISTINCT col)` proved
to be it (§26), is answered only where no counted row has col NULL: col declared NOT NULL, or a
clause of the WHERE that says so, and then nothing is merged. `count(DISTINCT k)` of §26 beside
it and a second GROUP BY column decline. An IN list on col drives the groups as in §15 and
never meets the NULL entry, and neither does a range on col (§28) or `col IS NOT NULL`. The
path claims no pathkeys - the group of c comes last, wherever c sorts - so an `ORDER BY` puts
a Sort above the node. A partitioned table is §16's shape unchanged: each partition emits its
own partial group of c, and the Finalize Aggregate adds the partitions' groups of c up like
any other key's. EXPLAIN prints `Group Key: COALESCE(g, 0)`. The constant travels in the new
`LION_PRIV_COALESCE` member of `custom_private` with the equality and its collation (shape 14).
`test/sql/group_coalesce.sql` compares each form against a sequential scan: a constant that is
a key and one that is not, for int, text and boolean columns, no NULL rows (a NOT NULL column,
`IS NOT NULL`, an IN list, a range), only NULL rows, a key the WHERE leaves no row of beside
NULL rows that it does, HAVING that keeps or drops the merged group alone, parameters, a
LATERAL rescan, a dirty heap in which c's key has lost every row, after VACUUM has deleted that
entry, a partitioned table whose NULLs and whose c rows are in different partitions, and the
declines. Not done: another spelling of the same grouping (`CASE WHEN col IS NULL THEN c ELSE
col END`), and a c known only at run time, which the node could evaluate once per scan like a
clause value (§10) but does not.
  • The driving index’s equality is the grouping equality (2026-09-20 review, finding 3). An index’s entries are the classes of strategy 1 of ITS opfamily on its own key type, and an operator class is free to define a coarser equality than the type’s (a text opclass over lower() is a valid opclass, and the 2026-09-20 review built one: 50k 'A' and 50k 'a' came out as one group of 100k). So the equality operator the planner chose for the grouping column - SortGroupClause.eqop, which is the type’s own - must be exactly the operator get_opfamily_member(idx->opfamily[i], opcintype[i], opcintype[i], 1) returns for the driving KEY COLUMN i (§24), compared by Oid; cross-type equality does not arise for a group column. Matching collation is NOT enough. The check is per relation, so every partition’s own index is checked (§16), and it covers the count(col) cases of §14 that read the group column’s entries, since those are only reached through a grouping index. A sum-over-all driver (§14) is exempt: the entries of one index are disjoint and cover every row however they partition it, so their total does not depend on the equality. Without GROUP BY nothing else does either - counting a class needs no assumption about how the class was formed, only that the clause’s operator is the index’s strategy 1.
  • The planner’s grouped_rel target contains only the GROUP BY column(s) and the aggregate(s).
  • Value-producing pushdown has a stricter contract than counting (2026-09-20 review, finding 4). A posting set stores ONE representative key per equality class, chosen when the entry was created, and nothing keeps a visible row with that exact representation in it: indexing citext 'SecretOldSpelling', deleting it, inserting 10,000 'secretoldspelling' and vacuuming leaves an entry whose key is a spelling the table no longer contains. So a pushdown that PRODUCES a value - the GROUP BY column in the output target, or a column a = clause pins whose value the target list prints from the entry’s stored key - is only built when equality on that index implies an identical binary representation. Both halves are checked, per relation, in lion_collect_targets():
    • the index’s own equality (strategy 1 for (opcintype, opcintype)) IS the type’s equality, i.e. the equality of the type’s default btree opclass. Otherwise “same entry” is weaker than “equal” and the question below is not even the right one;
    • and that equality is representation-preserving, decided exactly as btree deduplication decides it in _bt_allequalimage(): the type’s default btree opfamily (lookup_type_cache with TYPECACHE_BTREE_OPFAMILY), its BTEQUALIMAGE_PROC (support function 4) for (type, type), called under the INDEX’s collation - which is how btequalimage/btvarstrequalimage decide determinism. A missing support function or a false answer (citext, numeric 1.0/1.00, nondeterministic collations) means no.
    • and, before either is asked, the type is on an explicit list of core types whose equality compares every byte the output function prints (2026-09-23 security review): lion_type_equalimage() names them and argues each. equalimage only promises that equal values are interchangeable for deduplication, which is weaker: bpchar registers btvarstrequalimage, yet 'a ' and 'a' are equal and print differently, so a GROUP BY printed a deleted row’s padding. bpchar and every extension type are refused.

    A count-only pushdown - no group column in the output and no pinned column echoed - is unaffected: every member of the class is counted whatever it is spelled like. IS NULL is exempt as well: NULL has one representation. Regression coverage is in test/sql/citext.sql (the review’s representative case) and test/sql/pushdown.sql (numeric, and a text GROUP BY that still pushes down).

  • Add a CustomPath with flags = 0 (no parallel, no backward, no mark/restore), pathtarget = grouped_rel->reltarget (a partitioned GROUP BY uses a partially-grouped target instead and is wrapped in a Finalize Agg: §16), rows = estimated groups (from estimate_num_groups, or 1 without GROUP BY), and a cost (lion_cost_count_rel(), per relation, summed over the leaves of §16) that is the sum of what the node really reads:
    • one bucket page per looked-up key at random_page_cost, and that key’s own container chain - written sequentially, so its share of the index’s container pages, at least one page - at seq_page_cost. An IN list is one such lookup per element (§15) and the bucket pages are shared once the list is longer than the index has buckets, so the pages are Min(nelems, nbuckets). Since §22 only the source that DRIVES the merge pays for its whole chain: the others are SOUGHT to the driver’s container keys and pay for the leaves those probes touch, which is the formula in §22’s cost model and never more than the chain itself;
    • every page of the group index for a GROUP BY, at seq_page_cost: its entries are all walked;
    • and all of those page terms are scaled to ONE KEY COLUMN of a multicolumn index (§24). idx->pages and lion_index_dir_pages() are per RELATION, and a multicolumn index is one relation holding n independent sets of entries, so charging a column the whole directory and the whole page count prices WHERE b = 1 on (a, b, c) at three times what the same query on a single-column index of b costs - and the node is then refused for a query it answers exactly as fast. The share is that column’s estimated fraction of the relation’s entries, n_distinct(col) / sum of n_distinct over the index's key columns, taken through examine_variable()/get_variable_numdistinct() on a Var of this relation’s own attribute numbers (the partition’s, for a partition). It scales the clause’s directory and container page terms and the entry scan of a driving column, and nothing else: the directory HEIGHT is not scaled, because a descent passes through the upper levels the columns share, and neither is the index’s own size where lion_heap_page_cost() uses it to decide how much of it is cached, which is a property of the relation. The limitation is that n_distinct is not an entry count: a multi-key column (§17) has one entry per LEXEME rather than per row value, so its share is understated and the scalar columns beside it are charged for its directory; a column with no statistics falls back to DEFAULT_NUM_DISTINCT for itself alone, which makes the split equal when NO column has statistics and biased when only some do. Both errors are bounded by the number of columns, which is exactly the error the correction removes - without it, the factor is the column count, always, and always against the node. test/sql/multicolumn.sql pins it by asking the same question of one multicolumn index and of n single-column ones and requiring the same plan choice; the five-clause case there is refused over the multicolumn index with the correction disabled and accepted with it (2026-09-22). (Since 2026-09-27 that split applies to the DIRECTORY only. A column’s CONTAINER pages follow its rows - every row is under one entry of each scalar column - so they are split by what each column’s sets take to store: n_distinct entries of N / n_distinct rows, a header a container and two bytes a member up to a bitset’s 4 kB, three bytes a row where the rows are too few a container for one (lion_index_column_posting_share()). Split by entries, the three values of a status column beside a 2,000-value column got 0.14% of a 110 MB index’s pages where they hold a tenth of it, and status = 'val2' - a million rows at every container key - was priced at 16 for 1 ms. A dense set is weighed as the NARROW the index writes, a set’s pages are its share of its column’s by the same bytes, and a walk’s are the column’s share of the directory by entries and of the container pages by bytes - “Dense sources” below.)
    • the merge’s CPU (lion_merge_cpu_cost(), “The units” below, since 2026-09-27): each container the driving source reads and counts, LION_CONTAINER_COST and LION_MEMBER_COST a member up to a bitset’s worth; each probe of another source at the driver’s container keys, LION_PROBE_COST (LION_MEMORY_PROBE_COST into a copy in memory) and LION_AND_MEMBER_COST a member of the running intersection; the k'th source sought only at the keys the first k - 1 left a row at. A source’s containers are Min(heap_pages / LION_BLOCKS_PER_CONTAINER, its members) when its column’s values are scattered, and its rows' share of the container keys when it is stored in value order, interpolated by the correlation’s square (lion_key_containers()). A GROUP BY runs one merge per ENTRY of its column - every entry is counted, whatever the WHERE leaves of it - with the entry’s own set as a source, and one per pair of two columns (§20), plus each count’s set-up, LION_ENTRY_COUNT_COST (LION_LIST_GROUP_COST for a group of an IN list that drives them), and each set of a union source rebuilt at every count, LION_UNION_SET_COST. Before, every container was two cpu_operator_cost whatever it held, members and probes were free, and a group was charged the containers of its share of the WHERE’s rows rather than of its own set. Since 2026-10-20 a bitmap container is priced by its words, an ARRAY looked up in one by a bit a member, and a seek that stays on its leaf as a step (“Dense sources” below);
    • the union of an IN list, cpu_operator_cost × members × log2(nelems): a merge of k sets costs that per member however it is organised (§15 builds the k-way one), and a single-key clause with k = 1 pays nothing for a merge it does not make;
    • the heap the visibility map cannot vouch for: recheck_tids = rows × dirtyfrac from pg_class.relallvisible/relpages, LION_RECHECK_TID_COST (1.5 cpu_tuple_cost; 2 for a GROUP BY’s, LION_RECHECK_GROUP_TID_COST, “The units”) each - that is a visibility-bit lookup in the per-query cache of §9 - on Min(recheck_tids, heap_pages × dirtyfrac) DISTINCT pages, fetched ONCE per query whatever brings the count back to them.

    That last bound is the whole of the 2026-09-21 follow-up review’s grouped-count finding. The node used to be charged Min(numgroups × heap_pages × dirtyfrac, recheck_tids) page visits at random_page_cost, on the grounds that each group’s merge returns to a dirty block separately. It does, but §9’s visibility cache resolves a dirty page once per snapshot and answers every later group out of memory, so those are repeated CPU and not repeated I/O. At 5M rows, 200 groups and 5% of the rows updated (91,345 of 100,407 pages all-visible) the old model asked 1,815,739 cost units against the sequential aggregate’s 175,511 - for a node that ran in 148 ms against 1,174 ms, with 418,819 buffer hits and zero physical reads over a 71 MiB working set. The same shape reproduced here: 15,661 against 155,840, and 141 ms against 850 ms.

    The page cost respects caching for the same reason (lion_heap_page_cost()): it interpolates between seq_page_cost and random_page_cost on whichever of two ratios argues more strongly for sequential access - the working set’s share of this relation’s prorated effective_cache_size, as index_pages_fetched() prorates it, and the fraction of the relation the set covers, as cost_bitmap_heap_scan() interpolates it. A dirty working set much larger than the cache is still charged as random I/O, which is the failure a blanket preference for this node would hide.

    The node streams: every row is emitted as it is counted, so the startup cost is 0 except for the forms that produce a single row (a plain count, or a GROUP BY the planner folded to one group), which cannot emit anything before the whole scan is done. There is no per-group memory to charge beyond one group’s iteration state, and nothing above the node is priced here - a partitioned GROUP BY puts core’s Finalize Agg on top and core costs that (§16).

    This is deliberately optimistic but proportional; document it. test/sql/pushdown.sql pins every end of it with nothing disabled, so that which plan the cost model picks IS the test:

    • a 200k-row table vacuumed and then slightly extended (relallvisible just under relpages) must choose the node for a single count and for a small GROUP BY;
    • a 100k-row table vacuumed and then 5% of its rows updated - about a tenth of its pages dirty - must choose it for a 20-group GROUP BY (2.0 ms against 15.8 ms);
    • a 1000-group never-vacuumed table must choose it as well, which is the answer that CHANGED with the visibility cache: one pass over the dirty pages plus 100,000 bit lookups is 17.9 ms against the sequential aggregate’s 26.8 ms;
    • and a 20000-group never-vacuumed table must still lose to it, because there the per-group work and not the recheck is what costs: 40.8 ms against 28.0 ms. Higher-cardinality groupings lose by more (at 5M rows, GROUP BY over 20000 keys is 1,541 ms against 1,114 ms and over a million keys 4,654 ms against 2,047 ms), and the model refuses all of them.
  • PlanCustomPath produces a CustomScan with scan.scanrelid = 0 (upper-level node), custom_scan_tlist = the output columns (group key Var(s) with their original varno/varattno, and the aggregate as a Const-shaped placeholder replaced at execution), and custom_private holding: the heap relid, the list of (index oid, attnum) for WHERE, the group-by (index oid, attnum) pairs
    • two of them since §20 - or none, and the aggregate kinds. The clause VALUES travel in custom_exprs rather than in custom_private, because that is the only field of a CustomScan the planner’s later passes look inside: set_customscan_references() fixes its expressions up and SS_finalize_plan() collects the Param ids it finds there into the plan’s extParam/allParam, which is what makes the executor rescan this node when an exec Param changes. A value left only in custom_private would be invisible to both, and a parameterised inner side would then count the first outer row’s key again. Use build_path_tlist-style handling with INDEX_VAR references in the plan’s targetlist as pg_strom/TimescaleDB do. custom_private is a POSITIONAL list (the LION_PRIV_* indexes), so its first member is a shape marker - an IntList of LION_PRIV_MAGIC and the number of members - which BeginCustomScan checks before reading any offset and ERRORs on. §16 added LION_PRIV_PARTS and §19 LION_PRIV_ORS, the structure of each OR restriction over the flattened clause array. Planner/executor drift, or a plan built by a differently shaped build of the library, is then a message and not a misread Oid. The executor reads the list through one decoder, lion_count_priv_decode() (src/lion_plan_private.c), into a struct: it checks every member’s node type, each per-clause and per-partition list’s length against the number of clauses, and the kinds, and lion_count_priv_check() holds the rules between members.

Executor - BeginCustomScan: open heap with NoLock (the executor already locked every RTE); index_open each index with AccessShareLock; register the executor snapshot (estate->es_snapshot); allocate work buffers, a per-group memory context and one visibility cache for the whole execution (§9); and take each clause’s value from custom_exprs - a Const’s is read straight out of it, anything else gets an ExecInitExpr() ExprState. - The parameters are evaluated at the START of each scan, not in BeginCustomScan: an exec Param is set by the nested loop after the node has been initialised and again before every rescan, so the values are computed on the first ExecCustomScan call after each ReScan (ExecEvalExprSwitchContext() through the node’s own ExprContext, then datumCopy() into a context of the node’s that lives exactly as long as the scan - the per-tuple memory the evaluation leaves its result in belongs to nobody here). ReScan throws them away with everything else. - ExecCustomScan without GROUP BY: on the first call compute lion_count_keys for the WHERE keys and return one tuple (count); subsequent calls return NULL. (What the node holds between two rows and after its last one: §15, “Paused and finished counts”.) - ExecCustomScan with GROUP BY: iterate all entries of the group-by index in bucket order (walk bucket pages; copy each entry’s key and head/inline payload while pinned as in §9). For each entry, count the AND of its posting set with the WHERE posting sets (§9). Emit (key, count) only when count > 0 (a group exists only if at least one row is visible). Output order is arbitrary; the planner must not assume sortedness (pathkeys = NIL). This is one function, lion_next_group(), and it is the whole of the GROUP BY executor: a partitioned scan runs it once per partition (§16).

(Since written: §21 ordered the entries and gave an entry walk pathkeys, and §15 let an IN list on the group column drive the groups instead - lion_next_group_inlist(), which emits them in the order the list’s sets were located in and so claims no order at all. WHICH clause drives is one rule, applied by the planner (lion_inlist_shape(), for the cost and for the pathkeys) and by the executor (lion_locate_where()) alike: the FIRST list, in clause order and outside any OR, on the driving index’s own key column - whatever other lists the WHERE holds. The planner used to give up at the second list anywhere in the WHERE, so g IN (...) AND h IN (...) GROUP BY g was priced as a walk of every entry of g and claimed ORDER BY g, while the executor drove the groups from the list on g all the same; for an opfamily whose cross-type equality has no ordering to sort the list with, the list is located in hash order, and the query returned its groups unsorted under ORDER BY g (the 2026-09-25 review). test/sql/pushdown.sql pins both halves: that opfamily’s ordered output with a Sort above the node, and the plan choice the corrected estimate makes on the shipped one - eight listed entries against a walk of twenty thousand.) - ReScanCustomScan: reset iteration state. EndCustomScan: close indexes, free. - ExplainCustomScan: print “Indexes: idx1 (col = const), …” - with the index’s KEY COLUMN appended as idx1.col when, and only when, the index is a MULTICOLUMN one (§24), so that two clauses answered by one index can be told apart and every plan written before §24 is unchanged - and “Group Key: col” and, with ANALYZE, the number of TIDs rechecked in the heap, the heap blocks skipped via the visibility map, the containers visited, and the block visits the visibility cache answered or had to let past its budget (“Heap Blocks From Cache” / “Heap Blocks Past Cache Budget”, §9). Every clause is printed with its OWN operator’s name and with its value as core’s EXPLAIN prints a qual’s - through deparse_expression() against the plan’s own deparse context, whatever the value is: a literal with its quotes and its type (idx (v === 'A'::text), idx (k = ANY ('{1,2}'::integer[])), idx (tags @> '{a}'::text[])), a prepared statement’s parameter as $1, and the FK-side join’s key as the other table’s column (§27). Until the 2026-09-25 review an equality or a list was always printed with =, and a literal through its type’s output function alone - (v = A), ANY ({90,5,50,1}), as §15 still quotes it - which made the === clause of that review’s wrong answer look exactly like the = the planner had dropped beside it.

Tests (pg_regress): the pushdown produces identical results to the plain plan for: no rows; all rows; WHERE constants that match no key; cross-type constants; GROUP BY with and without WHERE; after DELETE without VACUUM (recheck path) and after VACUUM (VM path); EXPLAIN shows the custom node when pg_lion.enable_count_pushdown = on and the normal plan when off. Parameters get the same treatment under plan_cache_mode = force_generic_plan, which is what keeps a $n a Param: a count, a count that matches nothing, a NULL parameter, = ANY ($1), IN ($1, $2), a GROUP BY with a parameterised WHERE clause, a parameterised clause whose column the target list prints, and a LATERAL nested loop whose inner side is rescanned with a new exec Param for every outer row - each compared against the same query with the pushdown switched off, as a multiset both ways round. test/sql/countclauses.sql does the same for the enum, boolean and stable-value shapes above - an enum =, IN list, GROUP BY, domain and FK-side join, every boolean form beside other columns and under an OR, now() and current_date windows, a STABLE function, generic plans over $1::date + 1, an exec Param inside an expression, and the volatile values that must be declined - over one single-column index per column and again over one multicolumn index, with every other scan disabled so that a LionCount path that is built at all is the plan; and the ORs nested inside AND arms of §19, flag IS NOT TRUE among them, up to the distribution that is too large and declined. test/sql/countmultikey.sql does it for the multi-key queries only the executor sees (§17). test/sql/null.sql and test/sql/inlist.sql do the same for the clause kinds of §14 and §15, comparing every query against a forced sequential scan rather than against the pushdown-off plan, so that the access method’s own answers are checked too. Section 11 of test/sql/multicolumn.sql runs the whole of this section over ONE multicolumn index (§24): a count and a GROUP BY per column, several columns ANDed, a two-column GROUP BY out of one index, IN lists on the group column and on another column of the same index, the null tests including two IS NOT NULLs over one index, an OR across two of its columns, parameters, a partitioned table whose partitions order their index columns differently and number their heap columns differently again, a dirty heap and a clean one - each against the same query with the pushdown off, as a multiset both ways round - and it pins the plan CHOICE against a twin table carrying one single-column index per column.

The WHERE sets, collected once (2026-09-28)

A GROUP BY is one count per entry of its column - per PAIR of entries with two columns (§20) - and each count is the group’s own set ANDed with the WHERE items. So every count intersected the WHERE again: it probed each item at the group’s container keys, built the union of an IN list, an OR (§19) or a multi-key query (§17) again from its sets, and read the pages of every set it keeps no private copy of. §9’s materialization copies a WHERE set on its second use, but only a CHAIN set small enough (LION_MATERIALIZE_MAX_CONTAINERS or _BYTES), only while the copies of one count stay within work_mem (§15, “Bounded cursors”), and never a leaf of an OR, whose interlock is that every leaf holds a pin. A benchmark measured the rest: over a large table with several filters, the buffers a GROUP BY read grew with its groups - about the buffers of the same count without the GROUP BY, once per group.

Now the WHERE items are COLLECTED once per relation into one private posting set - their intersection, less what the negated ones subtract, made by lion_sources_collect() exactly as the FK-side join’s fact filters are (§27) - and every count after that is the group’s set ANDed with that one set (lion_group_count(), lion_exec_count.c). Three drivers take it: the entry walk (lion_next_group()), an IN list that drives its own groups (lion_next_group_inlist(), §15, whose list is no item of the WHERE), and the pairs of two columns (lion_next_group2(), §20).

  • What is collected. Every WHERE item of the relation except a range taken as a source that the plan walks at every count (§32, LION_FLAG_RANGEWALK): each count still walks that one beside the collected set, which is how the planner priced it. A range collected in windows is a set like any other, and is collected with the rest, from its file. A negated item with nothing to subtract (IS NOT NULL over a column with no NULLs, a multi-key query no key narrows) is no item, and a WHERE with no positive item has nothing to intersect and is never collected.
  • Memory. What the ranges taken as sources have left of a hash table’s memory (get_hash_memory_limit()), and past that the set SPILLS to a temporary file (LionSpill, §27): sixteen bytes a container key in memory, a container read back at a time. It lives in the WHERE sets' context and goes with them - at the end of the relation (of each partition’s turn, §16), on a rescan, which collects again with the new parameters, and when the node is done. Nothing collects on a hot standby, where whether a count may trust the map depends on the WAL mode of every index it reads (§25) and a collected set carries the TIDs of all of them under one index’s name; the counts read the items there, as before.
  • When. Collecting reads the items once, a merge over all of their containers. A count reads them again at the container keys of its own set. Neither is known in advance - the walk only finds out how many groups there are, and how many rows each has, as it goes - so the node decides as it goes: the counts read the items until what they have read of them, WITH what the next count would read, reaches what the collection reads, and the items are collected before that count. Both are counted in container keys: a set’s reach is its rows, up to the relation’s container keys; the collection reads the reach of its smallest positive item (the one that drives its merge), and a count the reach of its own smallest set, but no more than the collection would. Collecting at that point costs at most what the counts so far and the next one would have read, so never more than twice the better of never collecting and collecting before the first group. A count is preceded by a collection only when another count of the relation is sure to follow - another entry in hand, another listed value, another inner key - so a GROUP BY with ONE group is the count it always was. Over groups that span the heap the first count already reaches it, and the WHERE is read once, before the first group - or, when the walk’s first leaf hands out one entry, after that entry’s count; a handful of one-row groups (a range over a unique column, §28) never do. And a WHERE that is one set a count keeps a copy of anyway - an INLINE set, a CHAIN set the count materializes, a collected range - is never collected; one it does not keep (lion_posting_set_rewalked()) is.
  • Why the counts stay exact (§9). The collected set is a copy, pinless and possibly stale by the time it is counted, and it is safe on lion_posting_set_materialize()’s terms: it is only ever counted ANDed with the group’s own set, which each count locates afresh under a pin of its own - the walk’s entry, the listed value located again (§15, “Paused and finished counts”), the inner set of a pair. A dead TID the copy still lists is either gone from the group’s container, and out of the intersection, or in it, which means the group’s page was read before VACUUM’s ambulkdelete got past it: VACUUM has not finished ambulkdelete on that index, so it has not set the TID’s heap page all-visible, and the TID goes to the recheck, where the snapshot decides. The copy cannot lack a row the snapshot sees: it is made after the snapshot was taken, and a visible row was in every index before its transaction committed. What a negated item’s stale copy subtracts beyond the truth is dead rows and rows inserted after the snapshot, the argument that lets a negated set always be copied. A count with no pinned positive source trusts no map (cx.novm), as any count does; none of the three drivers leaves one without, because the group’s set always holds its pin. A multi-key query’s row filter (§17, “A query known only at run time”) is not applied to the copy but to every count of it, which sends each candidate to the heap as before.
  • Pauses. The copy holds no pin, so it is kept across rows (§15, “Paused and finished counts”) like the WHERE sets' own copies, and collecting needs no pin either: a count after a pause collects the NOPIN copies as it would the pinned sets.
  • Not taken. A count(DISTINCT) walk (§26): its tests stop at the first visible row and read a share of each intersection, so a collection of all of it could read more than every test together. The sums of §14 and §28, which make many counts for one row, re-read their WHERE sets the same way and are left for later. The FK-side join has had its own copy since §27.

The cost model is unchanged, and so is every plan. It priced a grouped count’s WHERE sets as read once and probed in memory (the comment in lion_cost_count_rel(): “the sets a GROUP BY intersects with every group are copied out on their second use”), which was true only of the sets small enough to copy; the collection makes it true of the rest. What it now overstates is per count: every item probed where one collected set is, and each union’s sets set up again (LION_UNION_SET_COST). A later calibration can price the collection itself, as lion_cost_fkjoin_rel() prices the fact filters' copy; until then a grouped count is priced as it was, which errs high where the WHERE is collected. A range too large to collect is priced as §32 prices it. (Superseded by the next subsection, which prices the collection and the batched walk that follows it.)

EXPLAIN ANALYZE prints WHERE Sets Collected - the relations whose WHERE was collected, one per partition, summed over rescans - and WHERE Sets Spilled, each only when there were any. test/sql/groupwhere.sql checks every shape against a sequential scan with the pushdown off - plain filters and a NULL group, IS NULL and IS NOT NULL beside them, IN lists on the group column and on another, ORs, multi-key queries, a range as a source, two GROUP BY columns, a partitioned table, a generic plan, a LATERAL rescan, a cursor paused between groups (which pins no page of the indexes while it waits), a dirty heap and a collection that spills - and says which of them collected: none of a GROUP BY with one group, without a WHERE, over one set, over only IS NOT NULL, over a collected range alone, of a count(DISTINCT), or of a handful of one-row groups. And it compares the shared buffers a grouped count over an OR of two CHAIN sets reads with the same count ungrouped: beyond the walk of the groups, at most three times the ungrouped count’s, where each group used to read the OR’s sets again.

The groups of a walk, counted together (2026-09-28)

A benchmark’s GROUP BY over a column of many values on nearly every row, under a WHERE of several clauses, took several times what the same WHERE took ungrouped. The WHERE was collected once (the subsection above), so what remained was the counts: each group’s set ANDed with the collected set, at each of the group’s container keys - which read the whole collected set again for every group.

The repro. Every measurement of this subsection and of §15’s “Unions and intersections unoptimized” is from a synthetic table built to that shape, scaled to a 4-core, 15 GB VM: 8.5M rows in arrival order (every value drawn at random per row, so no column follows the heap), 1.9 GB of heap (249,321 pages, 34 rows a page, 3,896 container keys), all-visible, and one lion index of 246 MB over seven columns - grp, a grouping column of a hundred-odd values on nearly every row; f1 and f2, booleans, one dense and one not; lvl, a few values, one of them dense; c20, 20 values; and tags int[] and tags2 int[], each with a few dense elements beside many rare ones. PostgreSQL 18.6 as packaged (-O2), shared_buffers 1 GB with the index resident, work_mem 64 MB, max_parallel_workers_per_gather 0; times are medians of EXPLAIN ANALYZE’s Execution Time over 18 runs, the libraries before and after alternated round by round on a machine shared with other work. A key holds about 2,200 rows here, so a group’s container has some fifteen members; on a wider row a key holds fewer, and a group’s container fewer still.

The grouped count, grp, count(*) ... WHERE f1 AND NOT f2 AND lvl = 1 AND tags && '{4}' GROUP BY grp, took 394 ms against 72 ungrouped. Its profile:

function share
array_gallop() 62.6%
lion_container_and() 20.2%
lion_container_optimize() 4.7%
lion_count_container_vm() 3.3%

Each group’s container of fifteen members was galloped through the collected ARRAY of 350 at its key - 584,000 ANDs, half a microsecond each in mispredicted branches - and each non-empty result asked the visibility map again about blocks the previous group had just asked about. §15’s “Unions and intersections unoptimized” made the collection four times faster and left the walk as it was: 266 ms, 61% of it still array_gallop().

The walk. Once the WHERE is collected, lion_next_group() takes the rest of the entry walk a batch at a time - up to LION_GROUP_BATCH_MAX (256) entries, fewer when the open budget of one count’s cursors (work_mem, and the pins the lists leave; §15, “Bounded cursors”) holds fewer, since each group holds a cursor, a page image for a CHAIN set, and a pin - and counts the batch in ONE walk of container keys (lion_count_groups_copy(), lion_count_groups.c):

  • the groups' cursors stand on a binary heap by container key. At each key the copy has and a group stands at, the copy’s container is made a bitset image once, and every group standing there is tested against it: a bit test per member of the group’s ARRAY, which gives the AND’s count and the heap blocks it lies on without building it (lion_container_and_image_count()); a group’s RUN or BITSET is ANDed with the image (lion_container_and_raw()). Where the copy’s container has few members for the groups standing at it (fewer than LION_GROUP_IMAGE_MIN, 128, members times groups) there is no image, and each group ANDs its container with the copy’s directly - a lookup of those few members;
  • the visibility map is asked ONCE per key, about every block the copy’s container has a member on - every group’s intersection there lies on those. A group whose intersection lies on all-visible blocks is counted without the intersection being built; otherwise it is built and counted as every container is (lion_count_container_masks(), the second half of lion_count_container_vm()), its dirty blocks' TIDs queued for the heap in the group’s own recheck queue, which takes a share of one count’s budget: the shares add up to it, each a heap block’s TIDs at the least (§9, step 5), a floor that binds only below the open budget’s own floor of 256 kB. The floor used to be one count’s, 4096 TIDs a group - 256 groups held 6 MB of queues at the default work_mem of 4 MB (2026-10 review). A queue that fills is flushed where it is, and the queues are flushed group by group after the walk; they share the visibility cache as the groups' counts did, each a count begun;
  • where the copy has no container, the groups below its next key are sought to it, and after a key each group standing there goes straight to the copy’s next key - a step where that is the next key, a seek (§22) past the keys between otherwise.

The rows come out in the walk’s order, one a call, as before - so the pathkeys of a GROUP BY driven by its entry walk (§21) still hold - and between them the node holds the batch’s keys and counts and nothing pinned: the sets are released as soon as the batch is counted (§15, “Paused and finished counts”). HAVING and LIMIT apply per row as always.

Why it is exact (§9). Each group’s set comes from the walk as its own count took it - an INLINE one with its leaf’s pin, a CHAIN one read by a cursor that pins the page each container came from - and the copy is the same pinless copy those counts were ANDed with, on the same argument (lion_sources_collect()). At a key the map is asked after every group standing there has copied its container under its pin, and before any of them moves past it: lion_count_container()’s order, for all of them at once. A key the copy has nothing at, and a group whose intersection there is empty, ask the map nothing, and the pins let go of past them had nothing counted from them. A group whose set holds no pin - located past the pin budget, or a private copy - trusts no map (its own novm), as its count alone would; every group does while a row filter applies (§17, “A query known only at run time”), whose rows all go to the heap. Under SERIALIZABLE a group counted from the map takes its blocks' predicate locks through the ordinary path.

When. A single GROUP BY column walked by its index’s entries, once its WHERE is collected, when every count is that one set ANDed with the group’s own: not beside a range walked with it (§32, a range too large to collect that the plan walks), and not for GROUP BY coalesce(g, c), which adds two entries into one group. The group counted before the collection - the first, over groups that span the heap - is counted as it always was. Not taken: an IN list that drives its own groups (§15), whose sets are the clause’s; two GROUP BY columns (§20), whose pairs each AND two groups' sets; count(DISTINCT) (§26), whose tests stop at a row. They count against the collected copy one count at a time, as before.

Parallel. As first written the walk was serial, as every LionCount path but the FK-side join of §27 was: parallel_safe false, no workers. With max_parallel_workers_per_gather 4 the repro’s grouped count still planned as the serial node, and the plan without the pushdown - a Parallel Bitmap Heap Scan of four workers under a partial HashAggregate - took 1,138 ms. The walk and the collection split by ranges of container keys, and do now: “A GROUP BY in parallel”, below. An ungrouped count is still serial.

Cost. A grouped count whose WHERE the executor will collect (lion_cost_where_collected(), the rule of lion_where_describe() from the planner’s sources: two or more, or one of several sets - a union, or a multi-key query of several keys, which the merge prices as one set) is priced as the collection - the WHERE sources' merge as an ungrouped count’s (lion_merge_cpu_cost()), and a container of the copy written at each key it keeps (LION_FKJOIN_COPY_CONTAINER_COST, the same copy the FK-side join makes) - plus, per group, its containers at the keys the copy has, each LION_CONTAINER_COST, LION_MEMBER_COST a member and a heap sift of log2(batch) cpu_operator_cost, plus, per batch, the copy’s containers made images, and LION_ENTRY_COUNT_COST a group; the WHERE’s pages are read once. It was priced as every group’s merge with each WHERE set probed in memory (LION_MEMORY_PROBE_COST a set a key) and each union’s sets set up again (LION_UNION_SET_COST), which the batched walk does neither of.

Measured, the node before (the library before this subsection and before §15’s, base), with only §15’s change (c1), and now:

grouped count base c1 now cost, before and now units a ms, base and now
grp ... WHERE f1 AND NOT f2 AND lvl = 1 AND tags && '{4}' 394.4 ms 281.0 68.4 217,468; 44,857 551, 656
c20 ... WHERE tags && '{1,2}' AND NOT f1 116.2 38.3 24.0 36,747; 10,068 316, 419
lvl ... WHERE tags2 && '{1,5,9}' AND c20 IN (1, 2) 92.7 30.4 26.1 20,359; 11,198 220, 428
grp ... WHERE f2 AND lvl = 1 355.8 260.2 52.5 121,973; 32,293 343, 615
grp ... WHERE c20 = 3 AND tags && '{2}' 124.0 85.6 43.5 99,621; 28,222 803, 648
grp ... WHERE tags && '{1}' (one set, too large to copy) 348.8 256.7 47.5 76,520; 27,694 219, 583
c20 ... WHERE tags && '{1000,1001,1002}' AND f1 (a WHERE of 1,500 rows) 6.5 6.3 6.2 7,758; 5,097 1,201, 823

The first of them went from 5.5 times its ungrouped count (72 ms) to 3.8 times (18 ms, §15), most of the difference a group’s own set read from its pages: some 90 ns a group’s container, against half a microsecond before. Its profile now: the walk itself (inlined, 32%), lion_page_item_fetch() and lion_cursor_take_page() (the groups' posting pages, 21%), the images (container_or_bitset(), 13%, the collection’s too), the collection’s ANDs and copies (11%), buffer lookups (5%). Priced as before, the batched counts ran at 860 to 2,900 units a millisecond, overpriced up to six times; they are at 419 to 823 now.

One plan the regression suite pins moved with it: pushdown.sql’s SELECT a, count(*) FROM lion_pdpo WHERE a = 17 OR b = 3 GROUP BY a, 200 groups over two partitions of 50,000 rows, is the node now (a Finalize HashAggregate over it) where it was a HashAggregate over each partition’s BitmapOr: 0.63 ms against 1.15, the node before this change 1.17.

EXPLAIN ANALYZE prints Group Batches and Groups Counted in Batches when there were any. test/sql/groupwhere.sql reports the batches of every shape it checks - one per relation that collected, none where the groups are an IN list’s or pairs - and adds a table of 600 groups and a NULL one, counted in three batches and, at a work_mem of 64 kB, in twenty; groups whose containers are RUNs; a WHERE that is a BITSET at every key, and one of a row or two a key, ANDed directly; a HAVING, a LIMIT inside a batch, SERIALIZABLE, the order an ORDER BY relies on across two batches with no Sort, and a dirty heap - each against a sequential scan with the pushdown off.

A GROUP BY in parallel (2026-09-28)

The walk above divides by container key: a group’s count is the sum over any cut of the heap’s keys of its rows at the keys of each piece, and the collected WHERE of a piece is the whole copy’s containers at its keys. So the grouped count is offered, besides the serial node, as a parallel-aware one below a Gather and core’s Finalize HashAggregate:

Finalize HashAggregate
  Group Key: grp
  ->  Gather
        Workers Planned: 3
        ->  Parallel Custom Scan (LionCount)
              Lion Indexes: rp_lion.grp (grp), rp_lion.f1 (f1 = true), ...
              Group Key: grp
  • Ranges. When the Gather sets up its shared memory the heap’s container keys are cut into ranges, a few for each participant the plan was made for (lion_key_ranges(): four each, at most 64, no fewer than 16 keys a range - pg_lion.parallel_range_keys at run time, a testing knob; the planner prices the default), as evenly as whole keys go, the last one open-ended for a heap that has grown since. The shared copy of §27’s FK-side join is cut the same way, and no longer leaves its last chunks empty when the keys do not divide. Each participant claims the next range nobody has from a shared counter (LionJoinShared.nextchunk, the struct the FK-side join keeps in the Gather’s DSM; its sums carry the workers' counters to EXPLAIN), collects the WHERE of that range alone (lion_sources_collect_range(): the sources sought to the range’s first key, the merge stopped at its end, LionCollect.ranged), walks the entries from the first and counts every group against the range’s copy a batch at a time, exactly as above - a partial row per group with rows in the range - and claims the next. A range whose copy is empty walks nothing.
  • Why it is exact (§9). Each range is the walk above over a copy of its keys: every group’s set located afresh by the participant’s own walk under its own pins, the map asked once a key after every group there has copied its container, the copy pinless and made after the snapshot was taken (workers run under the leader’s). The ranges cover every key once, so every row is counted in one range by one participant, and the Finalize Agg adds a group’s partial counts up - and applies the HAVING, which the node’s partial target (lion_make_partial_target(), as for a partitioned table, §16) leaves to it.
  • When. One GROUP BY column walked whole over one table, whose WHERE the planner prices as collected (lion_cost_where_collected()), with a positive clause that is positive whatever its value - an equality, an IN list, IS NULL, a multi-key literal without a prefix lexeme; a multi-key parameter, a literal tsquery whose prefix the run finds too common to expand (§17, “Prefix lexemes”) and a jsonpath, whose keys are all prefixes (§43.2), may narrow nothing at run time and turn into a negated source, and a WHERE of negated sources alone has nothing to collect by range - and no range taken as a source; a grouping core can hash and aggregates it can split, and the table, the target and the clause values parallel-safe. Not a partitioned table, two GROUP BY columns, count(DISTINCT), an IN list that drives the groups, and not during recovery, where the WHERE is never collected. GROUP BY coalesce(col, c) divides too (2026-10-19): its group of c is the NULL entry’s rows and those of the entry the grouping equality finds equal to c, which the serial walk adds up as it meets them (LionCoalState); in a range each of the two goes up as a partial row of the value c itself (lion_next_group_ranged()), and the Finalize Agg, which groups by coalesce(col, c), adds them to each other and to the group’s rows of other ranges like any partial rows of one key - the classes of one index being disjoint, no other entry is c’s. A group of c with no row visible in any range has no partial row, and so no row, as serially. A parallel-aware node run without the Gather’s shared memory (a Gather that cannot use parallel mode sets none up) is the only participant and counts every group whole, the serial walk.
  • Workers. What a parallel scan of the grouping column’s posting pages would get (compute_parallel_worker() over them, lion_index_column_posting_share() of the index’s pages), capped by max_parallel_workers_per_gather; the table’s parallel_workers decides alone.
  • Cost, one participant’s: the serial node’s price before its HAVING divided among the participants (get_parallel_divisor()’s rule), since the collection, the groups' containers and the pages they lie on all divide by key; plus, for its share of the ranges, what each range pays again - the entry walk, the column’s share of the directory and not its sets, whose pages each range reads its own share of (“Dense sources”), and for each group and each WHERE source a descent to the range’s first key: LION_ENTRY_COUNT_COST and LION_PROBE_COST a group, LION_PROBE_COST a source, a page each (LION_RANGE_DESCENT_PAGES); plus a cpu_tuple_cost a partial row, a group’s per range it has rows in. The Gather prices its rows and parallel_setup_cost, core’s Finalize Agg its own work.

Two things the ranges made worth doing in the walk itself, serial or not. A group’s cursor now begins at the copy’s first key (lion_cursor_init_at(): a CHAIN set’s first leaf comes from a descent for that key, an INLINE set skips to it, a copy is sought) where it began at the group’s first container and was sought from there; and the walk’s cursors copy their leaves into page images the node allocates once (gbimages), where each batch allocated and freed a page per group: one participant counting the repro’s first GROUP BY below as 16 ranges took 18 ms more than the serial walk (65 ms) while each batch allocated its pages, much of it page faults and brk(), and takes 7 ms more now.

Measured on the repro above, the library of this change with max_parallel_workers_per_gather at 0, 2 and 3 - medians of 24 runs each, the settings' order rotated round by round, on four cores that other work shared (the one-minute load rose from 1 to 3 while it ran); plans and costs are the planner’s own, every parallel one the node below a Finalize HashAggregate:

grouped count serial 2 workers 3 workers cost: serial, 2, 3
grp ... WHERE f1 AND NOT f2 AND lvl = 1 AND tags && '{4}' 69.3 ms 44.1 36.0 44,857; 21,115; 16,998
c20 ... WHERE tags && '{1,2}' AND NOT f1 22.2 19.8 19.2 10,068; 5,397; 4,464
lvl ... WHERE tags2 && '{1,5,9}' AND c20 IN (1, 2) 25.6 23.9 21.8 11,198; 5,728; 4,679
grp ... WHERE f2 AND lvl = 1 55.6 39.8 33.4 32,293; 15,869; 12,933
grp ... WHERE c20 = 3 AND tags && '{2}' 43.8 32.0 27.4 28,222; 14,173; 11,620
grp ... WHERE tags && '{1}' 51.1 32.4 28.2 27,694; 13,948; 11,444
c20 ... WHERE tags && '{1000,1001,1002}' AND f1 6.9 11.1 12.6 5,097; 3,326; 2,861

Every parallel plan here starts its workers first, and on this machine that takes about 6 ms whatever the plan: core’s own Parallel Seq Scan of a table of 1,000 rows took 6 ms with one worker or three, against 0.05 ms serially, with the library loaded or not. parallel_setup_cost prices it at 1,000 - 2 ms at the 500 units a millisecond the model is fitted to - so the counts of 20 to 25 ms gain a little, and the last one, 7 ms, loses: it goes parallel as core’s plans of its price would, and takes 11. At a parallel_setup_cost of 3,000, what the start is worth here, it stays serial with two workers. The larger counts gain 1.6 to 1.9 times with three workers on the shared cores; over a larger table the walk grows and the start does not. Serially the library is the one before this change within the noise (12 rounds of the same seven, the two alternated: -3% to +1%).

The per-range price was fitted with the leader alone (max_parallel_workers = 0, so the Gather launches nothing and its node counts all 16 ranges): the four-clause GROUP BY above took 73 ms against the serial 65 (perf’s task clock, 60 runs), 500 us a range for the repro’s groups, priced at 245 units; the 1,500-row WHERE took 6.9 ms against 5.6, 83 us a range for 20 groups, priced at 35.

test/sql/groupparallel.sql checks the plans - the node below a Gather with two workers and one, the serial node with none or with the table’s parallel_workers at 0 - and, at ranges of one key (a table of fourteen keys, cut into twelve ranges for three participants), the answers of filters, a negated filter and the NULL entry as one, an IN list, an OR across columns, multi-key queries, a filter in two ranges of the twelve and groups that lie in one range or two, the NULL group, counts of columns, a printed filter column, a HAVING, an ORDER BY with LIMIT, GROUP BY coalesce(col, c) with c a key of the column or none, the leader alone with no worker started, SERIALIZABLE, a generic plan’s parameters and a heap the map cannot vouch for - each against the same query run serially and with the pushdown off - and the ranges, collections and batches the participants counted; and the shapes that stay serial, a WHERE whose positive clauses are literal prefixes too common to expand, a tsquery’s or a jsonpath’s, among them.

A count in parallel (2026-10-19)

The GROUP BY above divides by container key because a count is a sum over any cut of the heap’s keys; so does the node of ONE row - the count of a WHERE, or a sum over a column’s entries (§14), with filtered counts, count(col)s, residual clauses, OR lists and ranges beside it. It is offered, besides the serial node, as a parallel-aware one below a Gather and core’s Finalize Aggregate, which adds the participants' partial counts up and applies the HAVING:

Finalize Aggregate
  ->  Gather
        Workers Planned: 2
        ->  Parallel Custom Scan (LionCount)
              Lion Indexes: t_f.f1 (f1 = 1), t_f.f2 (f2 = 0), ...
              Count Filter 1: t_s (s = 3)
  • Chunks (lion_count_ranged()). The heap’s container keys are cut into ranges when the Gather sets up its shared memory, exactly as for a GROUP BY (lion_key_ranges(): four for each participant, at most 64, no fewer than pg_lion.parallel_range_keys keys a range), and the work into one chunk for each range of each RUN - the WHERE’s count and each filtered count’s (nrun, “Filtered counts”). Each participant claims the next chunk nobody has from the shared counter, chunk c being range c % nranges of run c / nranges, and counts that run at the keys of that range alone (lion_count_range()). The counter only grows, so a participant meets the runs in order: it locates a run’s WHERE once, at its first chunk of the run, and releases it before it locates the next, as the serial node does (lion_count_runs()). Its counts go into runcount - the row’s filtered counts are read from there as they are serially - and the partial row is emitted once the counter has run out.
  • A window of keys. Each count of a chunk is the serial count, its cursors confined to the range [lo, hi): the range is set on the visibility cache every count of the node is handed (lion_vis_cache_set_window(), cleared after the chunk and by a rescan), and the count context reads it from there (LionCountCtx.windowed), so no caller of the count changes. The merge seeks its positive cursors to lo - nothing passed over is counted - and stops at hi (lion_count_one_set(), lion_run_merge(), the copy’s lion_run_merge_copy(), the multi-key evaluator’s lion_wide_fill()); a range walked at every count (§32) collects the keys of the window alone (lion_range_probe_begin() over lion_sources_collect_range()); and the heap scan of a WHERE nothing narrows (§17, “A query known only at run time”) reads the window’s blocks (heap_setscanlimits(), no synchronized scan). A grouped or decoded walk refuses a window; it is never handed one. A run with an IN list too long to locate at once (“A list too long to locate at once”) is cut by its batches instead: range sub counts the batches b with b % nranges == sub, at every key - a window would locate every batch again for each range - and the list’s count is the sum of its batches' for the reason that section gives.
  • Why it is exact (§9). The ranges of a cut cover every container key once - the last has no end, for a heap that has grown since - and a row lies at exactly one key, its block’s. Every count the node makes is a count of rows: the merge of its sources, the disjoint sum of a list (§15), a walk’s sum over entries (§14), a complement OR’s two counts (§35), a range walked at every count, a probed walk’s final count (§28), the heap scan of a WHERE nothing narrows; each identity they rest on - the entries of a scalar column are disjoint, `|F ∧ (P ∨ ¬M)| = |F ∧ ¬M|
    • |F ∧ M ∧ P|` - holds of the rows at any set of keys as it holds of all of them, so the ranges' counts add up to the serial node’s. Each is an ordinary count of the containers at its keys: the containers in between are put to the visibility map before the page they came from is let go, under the pins the serial count holds; what the map does not vouch for - and every row a residual clause tests - goes to the heap under the leader’s snapshot, which every worker runs under, the row filter (§17, residual clauses) tested there as serially; and SERIALIZABLE’s predicate locks are taken as the serial count takes them, in whichever participant. A participant’s sets are its own, located under its own pins. Run without the Gather’s shared memory, the node is the only participant and counts every run whole.
  • When (lion_count_path_add(), and lion_count_priv_check()’s “malformed parallel count”): no GROUP BY, or one folded away (a sum over the column’s entries, §14); one table, not a partitioned one, and not the FK-side join, which is parallel its own way (§27); not a count(DISTINCT), whose values may have rows in two ranges and which core cannot split anyway; not the aggregates over keys (sum(k), min(k), §37), which have no partial form here; not a GROUP BY a WHERE equality folds to a constant (LION_FLAG_SINGLEGROUP), whose row is missing when no row matches, which a plain Finalize Aggregate cannot leave out; not during recovery; and aggregates core can split, the table, the target and the clause values parallel-safe. Everything else the serial node takes is taken: filtered counts, residual clauses, complement ORs, ORs across columns, IN lists, multi-key queries literal or known at run time, ranges as sources or bounding the sum, count(col), a HAVING and expressions over the counts.
  • Workers. What a parallel scan of the pages the count reads would get (compute_parallel_worker() over LionRangeCost.pages: the WHERE’s posting and recheck pages and the walk’s, as the serial price has them), capped by max_parallel_workers_per_gather; the table’s parallel_workers decides alone. A count that reads a page or two gets none. A range is counted by one participant, so a heap the executor would cut into one range - fewer than 32 keys, about 2,000 pages, at the default pg_lion.parallel_range_keys, read here as it is set - gets no parallel count at all, and the divisor below is never more than the ranges.
  • Cost, one participant’s (lion_add_parallel_count_path()): its share (core’s get_parallel_divisor() rule) of the serial node’s price before its HAVING, but for what every participant pays whole (LionRangeCost.whole: the lookups that locate the WHERE, and the ranges taken as sources it collects); that whole; and what each of its share of the chunks pays again (countrange, every run’s, summed by lion_count_rel_range_price()): every set of the WHERE sought to the range’s first key, a descent of LION_RANGE_DESCENT_PAGES and a LION_PROBE_COST a set (a range source’s copy at LION_MEMORY_PROBE_COST), and a walk’s entries counted again, each sought there, with the walk’s pages read again; plus a cpu_tuple_cost for its row. The Gather prices its rows and parallel_setup_cost, core’s Finalize Aggregate its own work. Both terms are in the node’s units (lion_units_price()).
  • EXPLAIN ANALYZE prints Key Ranges, the chunks counted - the ranges times the runs - and every counter of the scan a worker moves is summed into the leader’s (LionScanWorkerSums in LionJoinShared), so that none of them prints the leader’s share alone, or nothing where the leader takes no part. A range source is collected by each participant that counts a range of its run, but Range Sources Collected counts the sources of the query, not the collections: each clause marks how it was taken (LionClauseState.counted), and a worker ORs its marks into an array after LionJoinShared that the leader reads at shutdown (LION_JOIN_SHARED_COUNTED()). It counted collections before the 2026-10-23 review: 2 over two partitions, 3 over three rescans, 2 or 3 in a parallel plan, for the one range.

Measured on the release build (PostgreSQL 18.6 at -O2 without assertions, the library at -O2), on four cores other sessions' work shared (a one-minute load of 1.6 to 2.0 throughout), shared_buffers 2 GB with everything resident, pg_lion.parallel_range_keys at its default: a synthetic table of 4,000,000 rows on 58,788 pages (919 container keys) - nine filters of two to four values in one multicolumn lion index, an int[] of three keys of 2,000, a tsvector of four words of 120. Medians of 21 runs (15 for the dirty heap), the settings' order rotated round by round, EXPLAIN (ANALYZE, TIMING OFF)’s execution time; plans and costs the planner’s own, every parallel one the node below a Finalize Aggregate:

count, heap all-visible serial 2 workers 3 workers cost: serial, 2, 3
9-source AND, f1 = 1 AND ... AND f9 = 1 (870 rows) 2.2 ms 10.9 7.7 4,471; 2,885; 2,466
… three of the nine IN lists (20,800 rows) 4.4 7.5 10.9 8,181; 4,442; 3,674
tags && '{0, ..., 199}', 200 keys (1.08M rows) 30.7 15.0 13.7 5,702; 3,779; 3,256
the same array from a subquery (an InitPlan) 5,898 (seq scan) 19.9 15.6 (two planned) 108,892; 46,431; 46,431
tsquery, eight ORs of ANDs (§17) 4.3 serial serial 245
count, 1% of rows updated, 69% of pages not all-visible serial 2 workers 3 workers cost: serial, 2, 3
9-source AND 3.7 ms 6.4 9.1 5,726; 3,408; 2,871
… three of the nine IN lists 13.0 13.3 16.0 22,711; 10,496; 8,361
tags && '{0, ..., 199}' 89.2 54.5 56.7 56,281; 24,853; 19,572
tsquery, eight ORs of ANDs 15.8 14.2 16.3 21,976; 10,162; 8,095

The counts of tens of milliseconds gain - the wide overlap 2.0 to 2.2 times on the clean heap, 1.6 on the dirty one, where 800,000 TIDs go to the heap - and the third worker adds little on four shared cores. The leader alone (max_parallel_workers = 0) counted the overlap’s twelve ranges in 96 ms against the serial node’s 97: a range costs next to nothing over its share there. The InitPlan’s array is a value the serial price cannot estimate (§17), priced as the ordinary plan’s work, so the serial node lost to the sequential scan by a hair and the parallel one won by its divisor - an accident of that price. The counts of a few milliseconds lost in these runs: the 9-source AND went parallel and took 7.7 to 10.9 ms for 2.2, because the serial node was priced at 4,471 units for 2.2 ms - 2,000 units a millisecond against the 500 the model is fitted to (“The units”) - and a worker’s start, about 2 ms here and more under load, is more than the whole count: with parallel_setup_cost and parallel_tuple_cost at 0 it took 3.6 ms with two workers and 3.8 with three, against 1.6 serially in the same runs. The tsquery’s serial price, 245, kept it serial; made parallel the same way it took 5.3 and 5.6 ms against 4.3. The parallel GROUP BY had the same exposure (“A GROUP BY in parallel”): a serial price that overstates a short count by more than parallel_setup_cost sends it parallel. The serial price of an AND of dense sources was what was wrong, not this path’s, and is fixed (“Dense sources” below): the 9-source AND is priced at 620 units for 1.2 ms and the three IN lists at 1,024 for 3.0 ms, both serial at every worker count, and over 40M rows they and the GROUP BYs of 9 to 33 ms still go parallel and gain 1.4 to 2.6 times.

test/sql/countparallel.sql checks the plans - the node below a Gather with two workers and one, the serial node with none or with the table’s parallel_workers at 0, a HAVING the Finalize Aggregate’s - and, at ranges of one key (fourteen keys cut into twelve ranges), the answers of ANDs, the NULL entry, negated sources, IN lists alone and ANDed, ORs across columns and with a complement arm, multi-key queries, filtered counts, count(col), a residual clause, a filter in one range of the twelve, sums over every entry, over every non-NULL one and bounded by a range, ranges as sources, a HAVING, the leader alone, SERIALIZABLE, a generic plan’s parameters (a runtime tsquery among them, and one no key narrows, read from the heap a range at a time), an InitPlan’s value, a list of 5,000 values located in batches, and a heap the map cannot vouch for - each against the same query run serially and with the pushdown off - and the chunks, batches and skipped heap blocks the participants counted. test/isolation/count_parallel.spec runs a REPEATABLE READ snapshot’s parallel count against updates, deletes, inserts and VACUUM, and two SERIALIZABLE parallel counts whose inserts conflict.

Not done: count(DISTINCT k) in parallel. Investigated, and left serial. Its count is not a sum over the heap’s keys: a value with rows in two ranges is one value, so the ranges' partial answers do not add up, and core will not split a DISTINCT aggregate either (no GROUPING_CAN_PARTIAL_AGG). Two divisions were weighed. By key ranges, as here, each range would emit the values with a visible row there and core would count the distinct ones over the Gather: every entry is then tested once per range, up to nranges times the serial walk’s tests, and as many rows go through the Gather - worst exactly where the walk is long, with many values. By ENTRIES, each participant would take a share of k’s entry walk and test each of its entries over every key, exactly the serial test (§26, so §9 unchanged), the shares' counts then adding up since the entries of a scalar column are disjoint; but the entry walk has no parallel division yet (a shared cursor over the directory’s leaves, or key bounds cut from it when the Gather sets up), and the plan would have to turn the query’s count(DISTINCT k) into a partial count the Finalize Aggregate adds up, which core does not do for a DISTINCT aggregate. That is the shape to build if it is wanted: the existence tests, the IN-list driver and the grouped form (§26 shape 2, by the outer column’s entries) all divide by entries.

The units (2026-09-27, release build)

Every lion constant used to be fitted on the assert-enabled development build (-O1, cassert), whose assertions inflate lion’s own code far more than core’s - so each of them was off, against core’s units, by a factor nobody knew. This pass fitted them again on a RELEASE build: PostgreSQL 18.6 at -O2 without assertions, on a 4-core, 15 GB VM, shared_buffers 5 GB with every relation of the test database resident (the case the model is fitted for is warm), work_mem 64 MB, and every core cost GUC at its default (random_page_cost 4, seq_page_cost 1, the cpu_* and parallel costs as shipped). Numbers in this section and in the cost sections that cite it are from that build and machine; §31 has the data sets and the decision matrix. The first measurements are medians of five EXPLAIN (ANALYZE, TIMING OFF) runs on a quiet machine; once other work shared it, the backend’s own CPU time (/proc/<pid>/schedstat) over a DO loop that plans a query once and runs it many times, alternatives interleaved - serial plans only, which is what it can measure.

Core’s units on this machine. Cost units per millisecond of core’s own plans over 5M-row tables (s: 66,667 narrow pages; w: 102,639 wide ones; §31):

core plan cost ms units/ms
seq scan, count(*), narrow rows 129,189 210 617
… one int qual / two ANDed / three ORed 129,252 / 141,693 / 154,860 263 / 265 / 341 491 / 534 / 454
… sum() 129,189 313 412
seq scan, count(*), wide rows / one text qual / four quals 165,157 / 165,400 / 190,262 238 / 394 / 474 695 / 419 / 401
btree index-only scan, 25k / 250k / 2.5M rows 614 / 6,609 / 77,410 1.28 / 12.3 / 187 480 / 536 / 415
btree bitmap heap scan, 500 / 25k / 250k rows scattered 1,834 / 49,218 / 74,525 0.41 / 28.9 / 137 4,419 / 1,702 / 544
hash aggregate over a seq scan, 200 / 20,000 groups 141,692 / 141,888 680 / 905 208 / 157
sort and group aggregate, 200 groups 778,903 1,115 699
hash join 2M x 1,000 / 5M x 200k (4.4k and 140k rows hashed) / 5M x 2M (1.4M) 45,201 / 110,063 / 122,632 / 178,551 161 / 390 / 664 / 3,148 280 / 283 / 185 / 57
nested loop into a btree, 4.4k outer rows 14,154 13.4 1,057
parallel (2 workers) seq scan count(*) / one qual / wide, four quals 93,730 / 93,756 / 140,159 102 / 118 / 210 915 / 797 / 667
parallel hash join / hash aggregate 80,029 / 104,341 365 / 337 219 / 309

Core is not consistent with itself, and three things account for most of it. A page is charged as I/O whether or not it is in memory: a warm sequential page costs 0.6 us against the 2 us a seq_page_cost stands for at 500 units a millisecond, and a warm random one about 1 us against 8, which is why plans that read scattered pages run at 1,700 to 6,300 units a millisecond. Operators are charged by count, not by type: an int = costs 10.6 ns a row, a text one 31 ns, both one cpu_operator_cost (5 ns at 500). And hashing is charged far below what it costs: 94 ns a row into a hash aggregate against one or two cpu_operator_cost, which puts every plan with a hash aggregate or a large hash join at 57 to 310 units a millisecond.

The reference. Lion’s CPU constants are fitted at 500 units a millisecond - the middle of core’s CPU-bound scans (sequential scans 400 to 700, index-only scans 415 to 535) - so a microsecond is 0.5 units, a cpu_tuple_cost 20 ns and a cpu_operator_cost 5 ns. Lion’s I/O terms keep core’s convention: a page read in order at seq_page_cost, a random one at random_page_cost, interpolated by lion_heap_page_cost() where core would interpolate, so that lion and core overcharge a warm page alike and random_page_cost moves both the same way. Against core’s hash aggregates and hash joins that rate makes lion honest and them cheap: where a grouped count competes with a hash aggregate on close to equal terms, the aggregate wins in the model and loses on the machine (§31 lists the cases). Pricing lion at the hash aggregate’s rate instead would make it beat sequential and index-only scans it is up to twice as slow as. (Since 2026-10-01 a lion custom path is priced at the rate of the plan it competes with - its price, as fitted here, times that kind of plan’s rate over this reference - and a site can ask it to beat that plan by a margin (§39).)

The merge (lion_merge_cpu_cost(); every count, every group and pair of a GROUP BY, the AND of a lion index scan’s sets in lioncostestimate() - since 2026-09-28 with the lookups, unions and pages around it, one price for both, lion_cost_set_and(), §29.11). Fitted to the backend CPU time of 35 counts over the 5M-row s - eleven single sets holding 1 to 4,500 rows a container (sparse arrays, dense arrays, bitsets, RUN containers), eighteen pairs, five triples and a quadruple of them ANDed - by non-negative least squares on the features the planner can compute (the driver’s containers and members, the probes after the early exit, the running intersection’s members, the probed pages):

constant value fitted what
LION_CONTAINER_COST 8 cpu_operator_cost 32 to 39 ns a container the driver reads and counts
LION_MEMBER_COST 0.15 cpu_operator_cost, up to LION_MEMBER_CAP (1,024) a container 0.74 ns a member of it: the visibility-map mask is built from them; a bitset costs what an array of a thousand does (800 ns)
LION_PROBE_COST 40 cpu_operator_cost 210 ns a seek of another source’s posting tree to a driver key, and the container found; the leaves the seeks cross fit at 0.7 us a page and are charged as I/O (lion_probed_pages())
LION_MEMORY_PROBE_COST 30 cpu_operator_cost 100 to 200 ns the same into a set copied into memory (a GROUP BY’s WHERE sets after their first use, §9; a two-column GROUP BY’s outer set); 20,000 groups of 219 containers and 1,000 of 38
LION_AND_MEMBER_COST 0.8 cpu_operator_cost 4 to 5 ns a member of the running intersection ANDed with what a probe found
LION_UNION_KEY_COST (2026-09-28) 30 cpu_operator_cost 150 ns the union of a list’s containers built at a key: a bitset image cleared and counted (§29.11, “Unions probed”)
LION_UNION_MEMBER_COST (2026-09-28) 0.25 cpu_operator_cost 1.2 to 1.5 ns a member set in that image, or of the intersection tested against it
LION_BITMAP_WORD_COST (2026-10-20) 0.07 cpu_operator_cost 0.27 to 0.36 ns a word of a bitmap container passed over, ANDed, united, subtracted or counted (“Dense sources”)
LION_BITMAP_MEMBER_COST (2026-10-20) 0.2 cpu_operator_cost 0.86 to 0.93 ns a member of an ARRAY looked up in a bitmap container (“Dense sources”)

The k'th source is sought only at the keys the first k - 1 left a row at, 1 - exp(-lambda) of them (§25’s early exit, which was never priced). Median residual 27%. Re-costing the 35 counts with the model moved them from 87 to 46,239 units a millisecond (a ratio of 530 between the cheapest and the dearest) to 283 to 3,724 (13); the high end is counts of a few containers, whose time is a directory lookup priced as a random read, as btree’s leaf is. The per-container charge it replaces - two cpu_operator_cost whatever the container held, members and probes free - was calibrated in §10’s first version, before the posting tree, and never again.

The counts around the merge.

constant before now fitted on the release build
LION_DESCENT_COST (a directory level of a lookup: a count’s, an IN list value’s, an FK join’s) 50 cpu_operator_cost (25 in the FK join) 120 0.6 us a directory page read, from the FK join’s 23 shapes (§27), median residual 9%
an IN list value’s search nothing but its leaf’s page a descent (§21: a leaf’s binary search, the levels above it too when the values are sparser than the leaves) 1,000 values over 1M keys: 7.6 ms, chosen over a 1.8 ms btree index-only scan at 1,032
LION_ENTRY_COUNT_COST (each count of a GROUP BY: one per entry of its column, per pair of two) cpu_tuple_cost 50 cpu_tuple_cost 1.0 us a test of a walk of 20,000 entries; 1.3 us a group of 200 of six RUN containers
LION_LIST_GROUP_COST (the same, a group of an IN list that drives them) cpu_tuple_cost 18 cpu_tuple_cost 0.35 us a group over 1,000 of them
LION_UNION_SET_COST (a set of a union source, rebuilt by every count of a GROUP BY; LION_FKJOIN_SET_COST is it) 0 (80 cpu_tuple_cost in the FK join) 100 cpu_tuple_cost 2 us a set for 4 groups ANDed with a 1,000-value list; 2.7 and 6 us for 200 groups ANDed with a two-arm OR and a two-value list
LION_RECHECK_TID_COST / LION_RECHECK_GROUP_TID_COST (a candidate TID of the heap recheck) cpu_tuple_cost 1.5 / 6 cpu_tuple_cost 24 to 29 ns a TID on pages holding 68 to 150 candidates, up to 70 ns at 34; 130 ns in a 200-group count of a wholly dirty table, 1.6M of 2M pages answered from the visibility cache. (The grouped one is 2 since 2026-11-01, below what it fits: see below.)
LION_DISTINCT_TEST_COST 50 cpu_tuple_cost unchanged 1.0 us (§26)

The grouped recheck below its fit (2026-11-01). LION_RECHECK_GROUP_TID_COST went from 6 to 2 cpu_tuple_cost, for a TID that takes 87 to 130 ns, on purpose. “Without the rates” (§39) left the whole-table GROUP BYs of a dirty heap to a hash aggregate core prices at about 190 units a millisecond, to be won back by pricing the node’s own work; this constant is the only term of that work that decides them, and it decides them only: a clean heap rechecks no TID, and a count of one group summed over a column’s entries keeps 6 as LION_RECHECK_SUM_TID_COST, since a partial index’s sum at 1 beat a whole index’s AND it runs twice as slow as (0.37 ms against 0.17, test/sql/countpartial.sql). At 5M rows, 1% of rows updated (48% of pages all-visible), 200 groups: a hash aggregate of 896 ms at 171,204 against the node’s 326 ms at 214,714, and at 110,656 now. On bench/calib/matrix.py’s ten grouped queries, 2M rows:

rows updated mispicks, time lost at 6 at 3 at 2 at 1
1% 7, 1,236 ms 6, 1,119 ms 3, 484 ms 3, 484 ms
2% 7, 1,194 ms 6, 1,070 ms 3, 451 ms
5% 7, 1,066 ms 6, 956 ms 3, 425 ms 3, 425 ms
10% 7, 1,067 ms 7, 1,067 ms 3, 402 ms 3, 402 ms
50% 6, 621 ms 6, 621 ms 6, 621 ms 2, 264 ms
100% 6, 494 ms 6, 494 ms 6, 494 ms 3, 305 ms

Every plan a lower value moved to was the faster one, wholly dirty included. 2 is the whole number nearest what the gap between the units explains (6 x 190 / 500, 2.3). On these tables it does all that 1 does up to 10% of rows updated, and leaves half and wholly updated tables to a hash aggregate 1.2 to 1.5 times slower, tables a VACUUM is due on. It also leaves narrow rows there, since a seq scan’s price falls with the row and the recheck’s does not: with 1% of rows updated and 200 groups, a 40-byte column keeps the hash aggregate at 2 at 200,000 and at 1M rows, 2.2 to 3.3 times slower than the node (31 ms against 14, 157 against 77), and 1 moves it; at 64 bytes 2 moves it. It is the rate in another form, the honest number being 6; what sets it apart is that a clean count and every other kind of competitor keep their prices, and that it is a constant: nothing here reads a competing path.

A GROUP BY is charged one merge per ENTRY of its column - the executor counts every entry and skips the empty ones after - with the entry’s own set a source. Before, each group was charged the containers of its share of the WHERE’s rows: c20k, count(*) FROM s WHERE c20 = 3 GROUP BY c20k cost 6,550 for 1,091 ms (6 units a millisecond, against the hash aggregate’s 142,000 for 905); it now costs 502,000. How many containers one key’s rows lie in follows its column’s correlation (lion_key_containers(): one a row when scattered, their share of the heap when stored in value order, interpolated by the correlation’s square): a GROUP BY over 365 days of a table loaded in date order cost 4,650 for 0.6 ms when every day was taken to be scattered, and costs 317.

Dense sources (2026-10-20)

The serial price of an AND of many dense sources was about four times its time. That sent short counts parallel (“A count in parallel”: the 9-source AND, priced 4,471 for 2.2 ms, took 7.7 to 10.9 ms there) and chose a partial index’s sum over a whole index’s merge that ran faster (§45.6). On a table of that section’s shape - 4,000,000 rows on 68,966 pages (58 a block), 1,078 container keys, the release build, warm - f1 = 1 AND ... AND f9 = 1 was priced at 4,266 units for 1.21 ms, 3,519 units a millisecond. About 4,000 of them were the merge’s (lion_merge_cpu_cost_sets()): 397 for the driver’s containers (f8 = 1, 928 rows a key), 836 for the other eight sought at every key at a LION_PROBE_COST each, and 2,773 for LION_AND_MEMBER_COST a member of the running intersection at each of those seeks - 928 members, then 232, 77, 26 and on down. Both terms are an ARRAY’s: two sorted lists of members merged, and a seek that descends to its key. Neither is what these sets are.

  • What a container is. A build, an insert that writes an item and a VACUUM write each container as the smallest that holds it (lion_container_optimize(), §38): an ARRAY of up to 256 k members, and past them a NARROW of width k, 512 k bytes - k the words a heap block takes, one at 58 rows a block; an ARRAY added to in place grows past that only by its slack, a few dozen members. The merge ANDs two bitmaps in one pass of the narrower’s words whatever they hold (lion_container_and_raw()), and looks an ARRAY’s members up in a bitmap a bit each: the nine sets take 1.2 ms as NARROWs and 8.8 ms as ARRAYs of up to 2,048 members and BITSETs past them. The price follows the containers (lion_set_bitmap_words()): a set of more than 256 k members a container is taken as NARROWs, one of fewer as ARRAYs. Two new constants, from the containers of heaps of 58, 120 and 226 rows a block:

    constant value measured what
    LION_BITMAP_WORD_COST (pg_lion.bitmap_word_cost) 0.07 cpu_operator_cost 0.27 to 0.36 ns a word of a bitmap passed over: two NARROWs ANDed in 23, 38 to 43 and 69 ns at one, two and four words a block; a bitmap counted into the visibility mask, 0.34 ns a word
    LION_BITMAP_MEMBER_COST (pg_lion.bitmap_member_cost) 0.2 cpu_operator_cost 0.86 to 0.93 ns a member of an ARRAY looked up in a bitmap, to AND or subtract the two; two ARRAYs merged take 1.5 ns a member, LION_AND_MEMBER_COST

    The driver’s container is read and counted at LION_CONTAINER_COST and a pass over its words (lion_container_members_cost()); a source sought ANDs the running intersection with what it finds in a pass over the narrower bitmap’s words, a bit tested a member of whichever of the two is an ARRAY where the other is a bitmap, or LION_AND_MEMBER_COST a member where both are ARRAYs - and the intersection stays a bitmap as long as both are.

  • A seek that stays on its leaf (§22). A source’s seeks move it forward through its posting leaves, never back, so no more of them than the leaves it has can find their key on another leaf. The rest step to the item that holds it on the leaf the cursor stands on, LION_PROBE_STEP_COST (10 cpu_operator_cost; 60 to 85 ns in this AND). Priced as descents, LION_PROBE_COST, each was charged 210 ns.
  • Pages by bytes. A set’s posting pages are its share of its column’s container pages by the bytes it takes (lion_set_bytes(), lion_column_set_pages()), not by its rows: a bitmap container takes its bytes however many members it holds, and by rows the tenth of a two-value column was given a ninth of the pages its other value was where the two have as many. A column’s own share of the index weighs a dense set as the NARROW it is (lion_index_column_posting_share()). The pages a walk of a column reads - a GROUP BY’s, a sum’s over its entries (§14), the inner column of two (§20) - are its share of the directory by entries and its share of the container pages by bytes (lion_column_walk_pages()). Split by entries alone, the five dense entries of a partial index’s column beside a column of a thousand were given half a percent of the index’s pages, where they fill a ninth of them (§45.6). A range of a parallel GROUP BY (“A GROUP BY in parallel”) walks the entries again but reads only its own share of their sets, so the directory’s share is what each range pays again. Charged the column’s whole sets again each range, the 40M-row GROUP BYs below were priced higher in parallel than serially, though they ran in half the time.
  • A driving union - an IN list or a multi-key &&, the source with the fewest rows - is built into a bitset image at each key its sets meet at (lion_or_hot_raw()), unless it is a few ARRAYs' members folded pairwise (LION_OR_FOLD_MEMBERS, 128, now in lion_count.h beside LION_OR_BITSET_MIN), and the AND takes the image as the bitmap it is. ANDed member by member instead, an IN list of three values ANDed with three dense sets was priced at 3,082 units a millisecond.
  • Negated sources - IS NOT NULL, <>, IS DISTINCT FROM, a multi-key NOT (§14, §35) - are located as any clause is and priced where the merges subtract them (lion_count_rel_negated(), lion_count_rel_subtract()). Each is sought at the keys the AND leaves, a step or a probe, and taken out of the intersection by what the two are: a pass over the intersection’s words where both are bitmaps, a bit cleared for each of the source’s members where only the intersection is, a bit tested for each of the intersection’s members where only the source is. Unpriced, count(*) WHERE c1 <> 3 cost what count(*) cost, and a partial index that subtracts per entry was chosen at 1.7 ms over the whole index’s one merge at 0.38 (§45.6).
  • A filtered run over a sum of every row (“Filtered counts”, §35) that has a source of its own that selects rows - an equality, a list, IS NULL, an array column’s reserved entries (§17), an OR with no range and no complement arm - is counted as that source’s intersection with the rest (lion_run_count()), and is priced as one (lion_count_run_plain()), not as the sum.
  • Residual rows over a partial index (§45). The rows a residual clause is tested on are those the restrictions select within the index’s predicate: each predicate clause the restrictions do not imply restricts the rows fetched (lion_residual_rows()). Without it, an index WHERE w > 5 read under w > 7 AND b = 4 was priced as fetching every row with b = 4, as the whole index does, where it fetches 40% of them in half the time.

Measured on the release build, warm, on four cores other work shared; medians of 21 runs (11 for the 200-key overlap and the GROUP BYs over 4M rows) of EXPLAIN (ANALYZE, TIMING OFF), the plans and prices the planner’s own with max_parallel_workers_per_gather at 0, 2 and 3, “before” the library without this change. The 4,000,000 rows above:

4M rows before: price; serial; 2 and 3 workers now
9-source AND, f1 = 1 AND ... AND f9 = 1 4,266; 1.21 ms; parallel, 3.59 and 3.11 ms 620; 1.20 ms; serial
… three of the nine IN lists 5,826; 2.97 ms; parallel, 3.89 and 3.91 ms 1,024; 3.08 ms; serial
tags && '{0, ..., 199}' 5,780; 13.3 ms; parallel, 11.6 and 11.0 ms 5,552; 14.5 ms; parallel, 11.1 and 9.5 ms
tsquery, eight ORs of ANDs 29; 2.5 ms; serial the same
f8, count(*) ... WHERE f1 = 1 AND ... AND f5 = 1 GROUP BY f8 7,997; 3.05 ms; parallel, 3.82 and 3.82 ms 644; 3.02 ms; serial
f9, count(*) ... WHERE f1 = 1 AND f6 = 1 GROUP BY f9 3,618; 1.54 ms; parallel, 3.21 and 3.29 ms 356; 1.50 ms; serial
f8, count(*) ... WHERE tags && '{1,2,3}' AND f1 = 1 GROUP BY f8 1,993; 0.96 ms; serial 448; 0.94 ms; serial

The same table at 40,000,000 rows, 689,656 pages (5.4 GB), 10,776 container keys, its nine filters' index 156 MB:

40M rows before: price; serial; 2 and 3 workers now
9-source AND 44,184; 11.9 ms; parallel, 7.55 and 6.44 ms 7,489; 11.8 ms; parallel, 7.38 and 6.65 ms
… three of the nine IN lists 61,533; 32.1 ms; parallel, 16.3 and 15.2 ms 12,581; 33.3 ms; parallel, 17.0 and 14.0 ms
tags && '{0, ..., 199}' 62,082; 133 ms; parallel, 53.7 and 41.4 ms 59,672; 135 ms; parallel, 53.2 and 47.0 ms
tsquery, eight ORs of ANDs 269; 25 ms; serial the same
f8 ... GROUP BY f8, five filters 81,687; 33.1 ms; parallel, 16.6 and 13.1 ms 7,611; 33.2 ms; parallel, 16.3 and 13.0 ms
f9 ... GROUP BY f9, two filters 36,815; 15.5 ms; parallel, 9.30 and 8.37 ms 4,173; 15.6 ms; parallel, 8.54 and 7.40 ms
f8 ... GROUP BY f8, the overlap and a filter 20,778; 9.87 ms; parallel, 7.15 and 6.00 ms 5,245; 9.44 ms; parallel, 7.22 and 6.52 ms

The counts of a millisecond or a few stay serial now, where they went parallel and took two to three times as long; the counts and GROUP BYs of 10 ms and more over 40M rows still go parallel, and gain 1.4 to 2.6 times. The AND is priced at 518 to 633 units a millisecond at both scales. ARRAY sets ANDed with dense ones - the whole index of §45.6, 4M rows of 157 a block, its g and c2 sets ARRAYs and its three dense ones NARROWs of three words a block:

count(*) FROM pp WHERE ... before: price; ms; units a ms now
g = 5 AND f1 351; 0.216; 1,626 139; 0.208; 666
g = 5 AND f1 AND f2 = 'a' AND f3 = 'n' 910; 0.582; 1,564 306; 0.576; 531
c2 = 7 AND f1 AND f2 = 'a' AND f3 = 'n' 467; 0.295; 1,583 168; 0.299; 562
g IN (1, 2, 3) AND f1 AND f2 = 'a' AND f3 = 'n' 2,093; 0.679; 3,082 457; 0.684; 668
c2 < 20 AND f1 AND f2 = 'a' AND f3 = 'n' (a summed range) 3,443; 4.61; 746 3,229; 4.74; 682

Not done. A GROUP BY’s own work is priced below its time: the GROUP BYs with dense filters above at 210 to 280 units a millisecond. The WHERE collected for the groups (“The WHERE sets, collected once”) costs about twice its count, the first group is counted by the merge that collects it, and each group after it takes 450 to 550 ns a key against the 70 it is priced at: the batch’s own work - a step, the copy’s image narrowed to the group’s width, the block mask of the result - is not in the model. A summed range’s counts and a range taken as a source (§32), and the FK-side join’s sets (§27), still price a container’s members at LION_MEMBER_COST, whatever the container; complement ORs and summed ranges leave their negations unpriced; and a tsquery of many keys is priced far below its time (269 units for 25 ms above, §17).

test/sql/costdense.sql checks it on 60,000 rows indexed twice, built and grown by inserts: the NARROWs each has, and the same AND over both priced in the same units - a member ANDed priced ten times as dear moves neither price, a word of a bitmap both, a step on the leaf both - an IN list that drives the AND priced as the bitmap image it is built into, and a <> beside the AND subtracted, priced, and dearer where a seek leaves its page; every count against a sequential scan with the pushdown off.

Shapes rewritten to the node’s counts (2026-10-11)

Each of these is a query the node answers with the counts it already makes; nothing in the count engine or its visibility rules (§9, §11) changes, so every one is exact wherever the plain count is. test/sql/countshapes.sql checks each against a sequential scan, over NULLs and over dead and updated rows the visibility map no longer covers.

Expressions over the counts. SELECT count(*)::int, 100.0 * count(*) / n, g, round(100.0 * count(*) / n, 2) ... GROUP BY g or g || '!' of a group column: the grouped target holds the expression, and the checks used to refuse anything in it but a column or an aggregate. Now an expression is taken apart into what it reads - its aggregates whole and its columns, an expression column (§41) as one - and each must pass the rules of a target entry of its own (lion_count_path_having()); a SubPlan or a window function in it declines. The plan keeps the expression in the CustomScan’s target list and puts only what it reads in custom_scan_tlist, so setrefs.c makes it the node’s projection, evaluated once per row the node emits - once a group, as the Agg’s projection would; a volatile function in it is called as often. Under a Finalize Agg (a partitioned table, a parallel or decoded GROUP BY, the FK-side join and the semi join) the partial target already held only the aggregates and columns, and the Finalize Agg computes the expression. The grouping coalesce(col, c) is still the one expression the node emits itself, and only that exact expression: coalesce(count(*), 0) is an expression over a count. 1M rows: count(*)::int with two filters 23 ms to 0.3 ms; the grouped round(...) 97 ms to 3 ms.

count(1). count(c) of a constant that is not NULL counts every row, which is count(*) (lion_agg_is_count()), as the FK-side join already took it: count(1), count('x'), and a constant expression eval_const_expressions() has folded. count(NULL) and a Param or stable argument are not taken. The plan classifies it as count(*), and the top k accepts it as one. 1M rows: 23 ms to 0.2 ms.

count(col) of a nullable column. Without a GROUP BY, where every aggregate is count(col) of one column that nothing proves non-NULL (§14), the count is count(*) of the rows the WHERE selects AND col IS NOT NULL, which the posting sets answer (§14, §35: the NULL entry subtracted, or skipped when the column drives a count of every row). lion_count_path_countcol() adds the clause as if the query had written it, and EXPLAIN lists it (cs_n (n IS NOT NULL)); a column whose index cannot answer it declines as such a WHERE would. With a GROUP BY it is not taken: a group whose rows all have col NULL is a row of 0 in the query and no row at all under the clause. Beside count(*) or a count of another column it is not taken either, since the clause would change those. 1M rows, count(nullable) WHERE c20 = 3: 23 ms to 0.3 ms.

NOT IN (list). col NOT IN (a, b), which the parser makes col <> ALL ('{a,b}'), is the conjunction col <> a AND col <> b (lion_all_as_and()), and each conjunct is a §35 <> clause - a negated source, the entry of its value and the NULL entry subtracted. Both forms are false or unknown where col is NULL, and agree element by element everywhere else, so the rewrite is exact. A NULL element makes the ALL unknown for every row and an empty array makes it true for every row; both are left to the ordinary plan, as is an array known only at run time (a generic plan’s NOT IN ($1, $2)), whose elements cannot be taken apart at plan time, and a list longer than §15’s 1000. The conjuncts share the restriction’s position, so a partition whose bounds imply the whole restriction leaves all of them out (§16). The FK-side join’s and the semi join’s fact filters go through the same code. 1M rows, count(*) WHERE c20 NOT IN (3, 4, 5): 91 ms to 1.1 ms.

Window functions over the counts. A window function is evaluated by a WindowAgg above the grouping, over the grouped rel’s rows. With windows in the query the grouped rel’s target is make_window_input_target()’s: the grouping columns and every aggregate and column a window’s arguments, PARTITION BY or ORDER BY read, flattened out of the window expressions. That target goes through the same checks as any other, so a window that reads something the node cannot produce - sum(count(u)) OVER () of a count it cannot answer - declines through them, and the query-level refusal of window functions is gone. The top k (§36) still declines a query with windows, whose rows it would cut before the window saw them. 1M rows, c20, round(100.0 * count(*) / sum(count(*)) OVER (), 2) ... GROUP BY c20: 109 ms to 1.3 ms.

GROUP BY or DISTINCT with no aggregate. SELECT c20, c200 FROM fact GROUP BY 1, 2 and SELECT DISTINCT c200 FROM fact ask for the keys that have a row, each once - which is what a GROUP BY walk emits already: a group is emitted only when its count of visible rows is not 0, under the same visibility rules as any count (§9, §11), so a key whose rows are all dead, or not yet visible to the snapshot, is not emitted, as a sequential scan would not see it. The checks used to require an aggregate; a GROUP BY with none is taken now, its target holding only grouping columns and expressions of them. A DISTINCT is planned at UPPERREL_DISTINCT, where the hook now acts too when the stage’s input is one base relation - no GROUP BY, aggregate, window, set-returning function or DISTINCT ON below it (lion_try_distinct_path()). Its keys are the groups of a GROUP BY of the same columns: for the build, processed_groupClause and parse->groupClause are the DISTINCT’s (core’s processed_distinctClause, without the columns a WHERE equality pins, which makes the query one group as a GROUP BY’s do), and the target is the input rel’s - core leaves the distinct rel’s own target empty and its paths carry their input’s. The grouping extra is the one a partially aggregatable GROUP BY gets, so the parallel, decoded and partitioned forms work as for counts, with a Finalize HashAggregate of no aggregates above them. The claimed sort order is the GROUP BY’s (§21): SELECT DISTINCT ... ORDER BY gets a Sort above the node when it is not that order. PostgreSQL 18’s B-tree skip scan does not compete here: core’s DISTINCT over a B-tree is a Unique over a full index-only scan, priced as such, and the cost model decides. 1M rows: SELECT DISTINCT c200 128 ms to 4.6 ms (51 ms with a B-tree on c200); c20, c200 ... GROUP BY 1, 2 127 ms to 15 ms (decoded walk).

Grouping sets (2026-10-11)

GROUP BY ROLLUP (...), CUBE (...) and GROUPING SETS (...) over one table (lion_plan_gsets.c). Every grouping set is a subset of the union of the sets - the finest grouping, which is parse->groupClause - so each group of a set is the union of some groups of the finest grouping, and an aggregate of a set’s group is an aggregate over theirs. The node counts the finest grouping as the plain GROUP BY it is, one row a group with its count n, and core’s own grouping sets Agg forms every set’s groups over those rows, with each aggregate rewritten over them as the FK-side join’s rows mode rewrites it (§27, “Every aggregate over the node’s rows”):

the query’s aggregate over the finest grouping’s rows
count(*) lion_join_count(n)
count(x), x of grouping columns lion_join_count(n) FILTER (WHERE x IS NOT NULL)
sum(x), x an int2 or int4 of grouping columns lion_join_sum(x * n)
min, max, bool_and, bool_or of grouping columns as it stands
any aggregate over DISTINCT grouping columns as it stands
GROUPING(...) as it stands

A FILTER over grouping columns is kept, ANDed with the IS NOT NULL of a count(x). The aggregates left as they stand are the ones a group’s repeated rows do not change: those with a sort operator in pg_aggregate (the aggregate is the first of its input in that order) and those over DISTINCT arguments, whose distinct inputs are the same over the finest grouping’s rows as over the table’s. lion_join_count is 0 over no rows, so the empty grouping set of an empty input is one row of count 0, as count(*) gives.

The plan is a LionJoinAgg over a GroupAggregate over the node, sorted for the first rollup:

Custom Scan (LionJoinAgg)
  ->  GroupAggregate                 core's, the aggregates rewritten
        Group Key: a, b
        Group Key: a
        Group Key: ()
        Sort Key: b                  a CUBE's second rollup, sorted by the Agg
          Group Key: b
        ->  Sort
              Sort Key: a, b
              ->  Custom Scan (LionCount)   GROUP BY a, b, each group with count(*)

The LionJoinAgg is §27’s: the grouped rel’s target is the query’s own, which what core puts above (a Sort for ORDER BY count(*), a window, a projection) finds its aggregates in by equal(), and the Agg below computes the rewritten ones.

The finest grouping. lion_try_gsets_path() builds a rel of its own for it, a copy of the grouped rel with an empty path list and a target of the grouping columns - with their sortgrouprefs, as they are below the grouping step: from PostgreSQL 18 on, a grouping column above it is nulled by the grouping step’s RT index, which remove_nulling_relids() takes off - and a count(*). lion_try_count_path() then plans the plain GROUP BY into that rel, with the query’s groupingSets and limitCount cleared and processed_tlist below the grouping for the call: the top k (§36) would cut the finest grouping to the query’s LIMIT, which is the sets'. Every form of the plain GROUP BY is open to it - the nested loop of §20, the decoded walk of §34, a partitioned table’s Finalize Agg, the parallel GROUP BY - and its paths are priced in the units pinned for the grouped rel (§39, lion_units_pin_move()), whose competitor is core’s grouping sets plan.

Rollups. The sets are chained greedily from the largest down, each to the first chain whose last set holds it; a chain is one sorted pass of the Agg, its grouping clause the columns of its smallest set, then those its next set adds, so that every set of it is a prefix. The first chain holds the largest set and reads the node’s rows sorted by its clause; core sorts them again for each other chain (create_groupingsets_plan()). Every rollup is sorted, never hashed: the finest grouping’s rows are few, which is what makes the node worth using. A set’s group count is estimate_num_groups() over the node’s rows. The Agg claims no order, since its first rollup need not be the one core’s group_pathkeys were made for, and an ORDER BY gets a Sort above the LionJoinAgg. The cost is core’s cost_sort() and cost_agg() over the node’s path, and add_path() chooses between the plan and core’s own. The Agg’s aggregate costs are the query’s, computed for it (get_agg_clause_costs(), AGGSPLIT_SIMPLE, as core prices its own Agg over the table), not read from the GroupPathExtraData: core sets that struct’s agg_final_costs only when it builds partially aggregated paths (partial_costs_set), which grouping sets and a DISTINCT aggregate never allow, and leaves it otherwise as its stack held it. Read there, the first version’s Agg was priced at whatever a backend’s stack held - its real cost in one session, 1e108 in a fresh backend on PostgreSQL 16, where core’s plan then won - and countgsets.sql checks the price in a fresh backend (\c).

Privileges. The Agg below asks EXECUTE on what it runs - the extension’s aggregates and those left as they stand - and the node on count(); the query’s rewritten aggregates (sum(int4), count("any")) would be asked by no one, so the LionJoinAgg path carries them in its custom_private and the node asks EXECUTE on each when it starts, as ExecInitAgg would (§9, “Privileges”; test/sql/node_contracts.sql has the line). The plan asks one privilege more than the ordinary plan where the query has no count(*): count(), of the node’s own count, and the extension’s lion_join_count and lion_join_sum, which are looked up at plan time.

Declined, to the ordinary plan: an aggregate that reads a column outside the grouping, or one not in the table above (avg, array_agg without DISTINCT, bit_xor, a sum of a bigint); an aggregate that reads a column whose equality is not its value’s identity (a numeric, a float; “DISTINCT aggregates over a finer grouping”, below); an aggregate with ORDER BY other than those left as they stand; a volatile function or a subquery inside an aggregate, which would run once a group of the finest grouping and not once a row; an ordered-set aggregate; a grouping expression that is not a plain column of the table; a set that names a column twice; DISTINCT, set-returning functions, joins; and a finest grouping the node does not count (a column without a lion index, a WHERE it cannot answer).

5M rows (two columns of 20 and 2 values, one of 200): ROLLUP (c20, c2) 885 ms to 46 ms, CUBE (c20, c2) 1,237 ms to 46 ms, GROUPING SETS ((c20), (c200), ()) WHERE c2 = 1 577 ms to 52 ms. ROLLUP (c20k, c200) and ROLLUP (c1m), whose finest groupings run to millions of groups, stay core’s MixedAggregate, at 4.8 s and 2.4 s. test/sql/countgsets.sql compares every shape with the sequential plan, over NULL grouping keys and dead and updated rows VACUUM has not seen, and of a partitioned table, and checks that the declined shapes are declined.

DISTINCT aggregates over a finer grouping (2026-10-11)

SELECT a, b, count(DISTINCT c) FROM t GROUP BY a, b is the number of groups of GROUP BY a, b, c with c not NULL within each group of (a, b), and in general a DISTINCT aggregate of a group is the same aggregate over the distinct values its rows hold - which the groups of the GROUP BY with the aggregate’s columns added are, one row each. §26’s distinct walk takes one grouping column at most and one distinct column; lion_try_distinct_agg_path() (lion_plan_gsets.c) takes the rest the way grouping sets are taken (above): the GROUP BY with every column of every DISTINCT aggregate appended - under fresh sortgrouprefs, as resjunk entries of a copy of the target list - is the finer grouping, counted by the node into a rel of its own, and core’s sorted Agg forms the query’s groups over its rows below a LionJoinAgg, every aggregate rewritten over the counts by the same rules (lion_gsets_mutator()): the DISTINCT aggregates stand as they are, count(*), count(x) and sum(x) of the finer grouping’s columns become lion_join_count and lion_join_sum of the counts, min, max, bool_and and bool_or stand as they are. An aggregate over any other column declines, so max(u) beside count(DISTINCT c) leaves the query to core.

Custom Scan (LionJoinAgg)
  ->  GroupAggregate
        Group Key: a, b
        ->  Sort
              Sort Key: a, b, c
              ->  Finalize HashAggregate
                    Group Key: a, b, c
                    ->  Custom Scan (LionCount)   decoded walk of a, b, c (§34)

The Agg’s input is sorted by root->group_pathkeys. When an aggregate’s DISTINCT can use a sorted input, core appends its keys to those pathkeys and marks the aggregate presorted (adjust_group_pathkeys_for_groupagg()), and nodeAgg then only skips consecutive duplicates instead of sorting each group’s values; the Agg relies on that order over the node’s rows as it would over the table’s, and the sort supplies it. With enable_presorted_aggregate off the pathkeys are the GROUP BY’s alone and the Agg sorts each group’s values itself.

It is tried only where the node has not counted the GROUP BY as it stands: §26’s walk of one grouping column and one distinct column keeps its plan, and a second form of the same query would only compete with it. Where §26 declines - two or more grouping columns, two distinct columns, a DISTINCT aggregate other than count, a partitioned table (whose (g, k) groups the Finalize Agg under the node merges, so the distinct counts above it are exact) - or loses to core on price, the finer grouping is offered. The cost is core’s sort and Agg over the node’s path, its aggregate costs computed as for grouping sets, above, and add_path() chooses.

A column the finer grouping adds is grouped by its type’s default equality, so one value of each equality class reaches the Agg; every column an aggregate reads must therefore have an equality that is its value’s identity (lion_type_equalimage()), or count(DISTINCT n::text) of a numeric n, whose 1.0 and 1.00 are one group, would count one text where the table has two. The same rule holds for grouping sets. Its grouping equality is one privilege more than the ordinary plan asks: nodeAgg compares a single DISTINCT column through an unchecked function (§26), while the Finalize Agg under the node checks the equality of every column it groups by.

5M rows (two grouping columns of 20 and 2 values, a distinct column of 200): 3,252 ms to 92 ms. 100k rows, g200, count(DISTINCT k50) ... GROUP BY g200, which §26’s walk loses to core: 19 ms to 2.2 ms. test/sql/countgsets.sql compares each shape with the sequential plan over the same table, with and without enable_presorted_aggregate, and checks the declines.

Filtered counts

count(*) FILTER (WHERE c), count(CASE WHEN c THEN 1 END), sum(CASE WHEN c THEN 1 ELSE 0 END) and sum(CASE WHEN c THEN 1 END) each count the rows the WHERE selects - or the group holds - for which c is true. A row where c is NULL is one where it is not true, in a FILTER and in a CASE alike, and a clause the posting sets answer selects exactly the rows where it is true, so a condition the WHERE could take is one more set to intersect with the WHERE’s. lion_agg_filtered() (lion_plan_target.c) recognises the forms and the way each reports an empty answer:

form LION_FCOUNT_* no row has c true no row at all
count(*) FILTER (WHERE c), count(k) FILTER (WHERE c) of a non-NULL constant, count(CASE WHEN c THEN k END) COUNT 0 0
sum(CASE WHEN c THEN 1 ELSE 0 END) of int4 or int2 SUM 0 NULL
sum(CASE WHEN c THEN 1 END), sum(1) FILTER (WHERE c) SUMNULL NULL NULL

count(col) FILTER (WHERE c) is the COUNT form where every row it counts has col non-NULL: proved by a clause of the WHERE or of c itself that implies it, or by a NOT NULL column (§14); otherwise it declines. Anything else - another constant, a second arm, an ELSE but 0, a CASE inside a FILTER, DISTINCT or ORDER BY - is not a filtered count, and with the generic checks of this section it leaves the query to the ordinary plan.

The planner. Each distinct condition (by equal()) is a filter, numbered from 1 in the order the target list and HAVING meet it; two forms of one condition share it. After the strategy is settled, lion_count_path_filters() takes each condition through canonicalize_qual() and analyses each conjunct as a WHERE restriction is (lion_count_path_filter_clause()): the same rewrites, an OR as one source (§19), NOT IN as its <> clauses, IS NOT TRUE, a multi-key clause (§17), a range. A conjunct no posting set answers, or a condition that is a constant, declines the whole query: what the node would count without it is not the query’s answer. The clauses are appended to the clause lists after the WHERE’s, a filter’s together, each LionClauseInfo with its filter’s number; they constrain no column of the result, so what they say of their columns (positive, pinned, non-NULL) is the filter’s alone and is restored for the WHERE after each filter. A duplicate is dropped only within one filter. A filter’s clause is no restriction of the relation, so a partition’s bounds never imply it away (§16), and a range in a filter bounds no walk: it is a range taken as a source (§32), which needs a pinned source in the same run - the walk, or a positive clause of the WHERE or of the filter - as the WHERE alone does.

Runs. The node counts the WHERE alone (run 0) and then the WHERE with each filter (run f): run f’s clauses are the WHERE’s followed by filter f’s, and each run is an ordinary count with its own sets, located, counted and released before the next run’s are located (lion_select_run(), lion_count_runs()). Every rule of §9 holds within a run; the runs share the executor’s snapshot, so each counts what that snapshot sees whatever happens to the table between them. A run touches only its own clauses: lion_clause_in_run() keeps the others out of the row filter, and a range source’s other bounds are looked for in its own filter’s clauses only (lion_clause_window()). EXPLAIN prints run 0’s clauses as Lion Indexes and each filter as Count Filter N: index (clauses).

  • Without a GROUP BY (the count of the WHERE, its sum over every row of §35, a partitioned table’s sum over its partitions) the runs' totals make the one row: a COUNT form prints run f’s total, SUM prints NULL when run 0’s is 0, SUMNULL when run f’s is. Where the WHERE alone is summed over every row (§35), a filter’s run with a pinned source of its own - an equality, a list, a null test, an array column’s reserved entries (§17), or an OR with no range and no complement arm - is counted as a plain count instead. An OR with a complement arm selects nothing by itself (§35, “Complements in an OR”), and a run whose only source it is is summed over every row as the WHERE’s run is.
  • With a GROUP BY - one column, two (§20), the decoded walk (§34), on a table or a partitioned one - every run emits its groups as partial rows below core’s Finalize HashAggregate: a row of run r holds run r’s count in the columns of filter r, and in the others what adds nothing, 0 for a COUNT form or the base count(*), NULL for SUMNULL, and for SUM 0 in run 0’s rows and NULL in the rest, so the Finalize Agg’s sum of a group is NULL only when run 0 had no such group. A group is emitted only when its count is not 0, as without filters. The query must allow partial aggregation (GROUPING_CAN_PARTIAL_AGG) with a hashable grouping, and the path claims no sort order. The decoded walk is taken only when every filter’s run has a source positive whatever its value, in the WHERE or in the filter - an equality, a list, a null test, a literal multi-key clause other than a jsonpath and without a prefix lexeme, or an OR of those - and no range source.

Declined with filters: the FK-side join (§27), count(DISTINCT) (§26), the aggregates over keys (§37), a coalesce group, expressions of the grouping columns that are not grouping columns, the top k (§36) and the parallel GROUP BY (“A GROUP BY in parallel” above). They leave the query to the ordinary plan.

Price. The runs' prices added up: each run is priced by the cost functions of its strategy with the other filters' clauses left out (lion_count_run_targets()), and a grouped path’s rows are its groups once per run. Where the WHERE selects most of a small table one pass over the rows can be cheaper than a count per filter, and the cost model then keeps the ordinary plan.

custom_private. LION_PRIV_FILTERS (member 20, shape 24): an IntList of the number of filters and each clause’s filter, and at the path stage the conditions themselves; the residual clauses (“Residual clauses” below, member 21) and the target-list kinds (member 22) follow it. LION_TL_FCOUNT(f, form) is a filtered count’s target-list kind. lion_count_priv_check() requires the clauses' filter numbers to rise by at most one and end at the number of filters, no RANGE clause in a filter, an OR’s leaves inside one filter, and no join, DISTINCT, aggregate over keys, top k, parallel walk or coalesce group beside filters.

Privileges. A filter’s operators are clauses of the node like the WHERE’s, and the operators of every aggregate argument and FILTER the node stands in for are recorded for the EXECUTE check (§9, “Privileges”): revoking EXECUTE on a filter’s operator refuses the query as it refuses one on a WHERE operator. test/sql/countfilter.sql checks every form against a sequential scan, on a table with NULLs and with deleted and updated rows not vacuumed, through a generic plan with a multi-key condition only the run knows, and through a cursor whose rows are read one at a time with rows deleted between them; test/sql/node_contracts.sql checks the privilege.

Measured on a synthetic table of 5M rows (columns of 100, 10, 5 and 13 values, a lion index on each), with the ordinary plan against the node:

query ordinary plan LionCount
count(*) FILTER (WHERE b = 1), count(*) FILTER (WHERE c = 3), count(*) ... WHERE a = 2 94 ms 7.6 ms
sum(CASE WHEN b = 1 THEN 1 ELSE 0 END), sum(CASE WHEN c = 3 THEN 1 END), count(*) 373 ms 28 ms
count(*) FILTER (WHERE b = 1), count(*) ... WHERE a = 2 AND g = 7 15 ms 0.7 ms
g, sum(CASE WHEN b = 1 THEN 1 ELSE 0 END), count(*) ... GROUP BY g 692 ms 68 ms
a, b, count(*) FILTER (WHERE c = 3), count(*) ... GROUP BY a, b (decoded) 628 ms 100 ms
g, a, b, count(*) FILTER (WHERE c = 3) ... GROUP BY g, a, b (decoded) 636 ms 158 ms

Residual clauses (2026-10-11)

A WHERE clause no posting set answers - payload LIKE 'ab%', id > 2500000 on a column with no lion index, length(t) = 3, n % 2 = 0 on one with - used to leave the whole count to the ordinary plan, which then fetched every row the other clauses select from the heap anyway, through a bitmap heap scan. Now such a clause is RESIDUAL: the node counts as before, and every row its counts are handed is fetched and tested.

Which clauses. lion_count_path_where() tries each restriction as it always did; one the posting sets do not answer - or that only an index the table does not have would answer (lion_where_indexed(), asked here rather than when the indexes are matched, which declined the whole query) - is residual when lion_residual_ok() takes it: a table’s count, or a partitioned table’s (§16, “Residual clauses over a partitioned table”), not an FK-side join’s; Vars of the table’s own user columns only (no system column, no whole row); nothing volatile, no subplan. Whatever the analysis had appended for it (the first arms of an OR, a range’s bound) is undone from a snapshot of the build taken before it (lion_count_path_snapshot()): its own copy of every list the analysis appends to, the per-column lists of range bounds inside rangecls and rangepos included, which are appended to in place - a bound left in its column’s list would name the position the next clause takes, or one past the end of the lists. A restriction every partition’s bounds imply (§16) is taken back from the same snapshot. The query is then taken only when the other clauses SELECT rows - a positive clause, an OR of them or a range taken as a source (§32) - and the shape counts rows rather than entries: not a sum over every row’s entries or a range’s on the column it walks (§28, §35), §37’s aggregates over keys, the decoded walk, the parallel GROUP BY, or beside a multi-key query known only at run time (§17), whose candidates may be every row. And the rows the other clauses select must be at most half the table (LION_RESID_MAX_SHARE): past that, fetching them one by one loses to the sequential scan that reads them page by page whatever the prices say (5M rows, c2 IN (0, 1) with a residual: 2.2 s against 1.1).

How it is carried. The clauses travel in custom_private’s RESID member (21) at the path stage, and lion_plan_custom_path() appends them to custom_exprs after the aggregates' arguments (LION_FLAG_RESID, shape 25). LION_TL_RESID kinds lie above the filtered counts' LION_TL_FCOUNT range, so neither encoding can be read as the other. Every column they read is a column of custom_scan_tlist - one the target list prints already, or a resjunk LION_TL_RESID one no row prints (it is emitted NULL) - so setrefs.c rewrites their Vars into references to it, records their functions as plan dependencies, and SS_finalize_plan() sees their Params: a LATERAL value in a residual rescans the node.

How it is evaluated. lion_begin_residual() initialises the clauses with ExecInitQual() over a virtual slot of the node’s scan tuple, as the ordinary plan’s scan initialises its quals, which checks EXECUTE on every function they call and calls the object access hook - at executor start, plain EXPLAIN included, and as the role that runs the plan. The test of a row (lion_resid_test()) copies the columns they read from the heap tuple into their places in that slot and evaluates the qual: a NULL result rejects the row, as a qual’s does. The columns are read at the numbers the relation counted gives them, looked up when it is made so (lion_open_relation(), lion_heap_attno_in()): the plan’s are the parent’s (§16). It is the row filter’s rowtest (LionRowFilter, §17), set on the visibility cache every count of the execution is handed (lion_build_filter()), so every count the node makes - a plain count, each group of a walk or a batch, each test of a count(DISTINCT), the top k’s candidates, both counts of an OR with a complement arm (§35) - sends every candidate to the heap whatever the visibility map says, resolves its HOT chain under the snapshot there (lion_recheck_heap_filtered()), and counts the visible row only when it passes. A row the snapshot does not see is never tested, so a residual 100 / w > 20 raises nothing over a deleted row whose w is 0, as the ordinary plan raises nothing. From PostgreSQL 17 the candidates' pages are read through a read stream over their blocks in order, as a bitmap heap scan reads its pages.

Beside filtered counts (“Filtered counts” above) the two are taken together. A residual clause is a clause of the WHERE, and the row filter that tests it is built again for every run’s located WHERE (lion_locate_where()), so the WHERE’s run and each filter’s count only the rows that pass it: count(*) FILTER (WHERE b = 1), count(*) WHERE a = 2 AND p LIKE 'a%' counts a = 2 and a = 2 AND b = 1, each testing p LIKE 'a%' on its candidates. A filter’s own condition is never residual - a clause of one that no posting set answers leaves the query to the ordinary plan - and the residual’s price is paid once per run.

The price (lion_cost_residual()) is that bitmap heap scan’s: the rows the other clauses select (lion_residual_rows()), the distinct pages index_pages_fetched() expects them on, each at a cost between random_page_cost and seq_page_cost by the share of the table they are, and per row cpu_tuple_cost, the clauses' evaluation and cpu_index_tuple_cost for the candidate’s own handling. Each count fetches its own rows, and the rows of different groups share pages, so a GROUP BY pays the pages of each group’s rows: c2, count(*) WHERE c20 = 3 AND length(payload) = 64 GROUP BY c2 visits most pages twice (419 ms against the ordinary plan’s 371), and is left to it.

EXPLAIN prints the clauses as Residual Filter, and the ANALYZE of a count that is not grouped prints the rows they turned away as Rows Removed by Recheck. Measured on 5M rows (assert-enabled PostgreSQL 18.6, four shared cores, the table larger than shared_buffers): count(*) WHERE c20 = 3 AND payload LIKE 'ab%' 296 ms to 241, ... AND length(payload) = 64 334 to 278, ... AND id > 2.5e6 297 to 264, all against a bitmap heap scan; c200 = 17 AND payload LIKE 'ab%' 9.3 to 8.7 against a plain lion index scan. test/sql/countresid.sql checks the shapes against the sequential plan, with dead versions the residual must not see; node_contracts.sql revokes int4mod under a residual.

Filtered counts, residual clauses and complements beside the other shapes

Which of the WHERE’s newer forms each of the other shapes takes, every answer compared with the sequential plan in test/sql/countcombos.sql:

  • Grouping sets and the finer grain of a grouped count(DISTINCT) (“Grouping sets”) count the finer grouping with an ordinary count path, whose WHERE is its own: a complement OR (§35) and a residual clause are taken or declined as for a plain GROUP BY of those columns (a residual clause beside one grouping column only), and the rows are the WHERE’s either way. That count makes no filtered count: the aggregates below the grouping are counts alone, and a FILTER of the query’s is the grouping Agg’s, over the finer groups, when it reads grouping columns only - the finer grain of a count(DISTINCT) groups by the columns its FILTER reads as well. A FILTER over another column declines grouping sets.
  • The decoded walk (§34) takes no complement OR and no residual clause (the plan check refuses a residual clause beside it). Filtered counts are counted against the WHERE’s collected copy, a range of the WHERE’s collected into it for every filter’s run as for the WHERE’s own (“A range in the WHERE”); a range in a filter’s own condition keeps a filtered count off the decoded walk.
  • The FK-side join (§27) - an inner join of two tables, the dimension LEFT JOIN the fact, and a join with dimensions folded into fact filters alike - takes no complement OR among the fact filters, no residual clause (a fact clause no posting set answers declines the join) and no filtered count. A folded dimension’s own filters are its child’s, core’s; what the node adds for it is an IN list, a positive clause like any other.
  • The parallel GROUP BY (“A GROUP BY in parallel”) takes none of them: its WHERE is collected range by range, and a count against the copy has no second set to subtract or row to test. The parallel count (“A count in parallel”, 2026-10-19) takes all three: a filtered count’s run is cut into the same ranges as the WHERE’s, a residual clause is tested on every row the counts of a range are handed, and a complement OR’s two counts are made in each range, the identity holding of the rows at any keys.

Counts by cells (2026-10-20)

A report that buckets rows by a score - SELECT CASE WHEN score >= 70 THEN 'high' WHEN score >= 40 THEN 'medium' WHEN score IS NOT NULL THEN 'low' ELSE 'none' END AS tier, count(*) FROM t WHERE ... GROUP BY 1, the score a sum of CASEs over c = 3, c IN (4, 5), tags && '{1,2}', s IS NULL, f, rounded, the greater of two parts, and NULL outside a gate CASE WHEN r = 'r1' OR r = 'r3' THEN ... END - grouped by an expression no index holds, so it went to core, which fetched every row the WHERE selects to evaluate the CASE: a bitmap heap scan and a HashAggregate, seconds at a few million rows. But the expression reads its columns only through PREDICATES the posting sets decide for every row, so its value in a row is a function of the predicates' truths alone: the rows can be counted by CELL - one truth, true, false or NULL, of every distinct predicate - and the expression computed once a cell.

The predicates (lion_cells_analyze(), lion_plan_cells.c). The GROUP BY’s expressions, and the WHERE’s clauses no posting set answers, are walked down to their predicates:

  • a SCALAR TREE: an AND, OR or NOT of leaves of one column - c = v, c <> v, c IN (...), c NOT IN (...), c IS [NOT] DISTINCT FROM v, c IS [NOT] NULL, a boolean column by itself and IS [NOT] TRUE / FALSE / UNKNOWN of one - its truth in a row a function of the row’s value of the column, the values compared by the equality of a scalar lion index on the column. Every leaf of a column that names a value names it by one equality and collation, of one type on both sides, and the values are constants of that type - or, for an equality declared on a polymorphic type, of the column’s own: enum_ops' = is (anyenum, anyenum), and c = 'live' compares the column with a constant of its enum, a column of a domain over an enum being taken cast to that enum (dc::mood = 'live'). Such values are of the column’s BASE type, as the key column’s own are (cells_dim_valop(), the rule of lion_type_is_column()): the index match resolves them to the class’s type, and the labels' lookups probe them as the column’s;
  • a MULTI-KEY LEAF: c @> q, c && q or c @@ q with a literal q that a multi-key lion index answers exactly (§17): a tree of the query’s keys, AND for @>, OR for &&, the tsquery’s own for @@; a query that selects no row (c && '{}') is false, or NULL in a NULL row.

The predicates are told apart by equal(). Anything else in the expressions - a column read outside a predicate, a range (“Ranges”, below), a CASE c WHEN ... (whose WHEN tests a placeholder), a subquery, an aggregate, a mutable function, a set-returning one - declines the count; so does a tsquery with a NOT, a query a stored position would have to narrow (§17, “Stored positions”), and a GROUP BY that holds a bare column - that column’s walk is the plain GROUP BY’s, and a cell is no group of it.

Dimensions and outcomes. The predicates are grouped by column into DIMENSIONS. A scalar column’s rows are told apart by the values its predicates name: every value named, any other value and NULL, each with its SIGNATURE - the truth of each of the column’s predicates in it, by SQL’s three-valued logic (cells_eval(), the IN list’s NULL element and empty list as ExecEvalScalarArrayOp has them) - and the values of one signature are one OUTCOME, any other value’s being outcome 0. So fifteen equalities of one column in a CASE, which are mutually exclusive, are one dimension of sixteen outcomes and a NULL, not fifteen independent bits; c IN (1, 2, 3) beside c IN (3, 4) is four outcomes. A boolean both of whose values are named has no other value, and a column declared NOT NULL no NULL: either is outcome 0 too. A multi-key column’s leaves are BITS, its outcome the set of leaves a row satisfies, or NULL: 2^bits + 1 outcomes, at most LION_CELLS_MAX_BITS (10) leaves and LION_CELLS_MAX_KEYS (63) keys among them; a null test of the column is a predicate of its NULL outcome. A column only a multi-key index covers, tested by null tests alone, is such a dimension too (cells_dim_index()), although the WHERE’s null test on it is answered by that index’s NULL entry (§17, “Non-empty arrays”): its entries are keys, not the values a scalar dimension’s driver reads.

The bound. A cell is an outcome of every dimension, so the plan has at most the product of the dimensions' outcomes - the mixed radix the tally codes a cell in (§34) - and the planner declines more than pg_lion.count_cells_max, 227 = 134,217,728 by default. The product counts each column once whatever its predicates: the tier measured below - fifteen values of one column, eight overlaps of an array column and three more columns, two of them NOT NULL - is 2 x 17 x 257 x 3 x 2 = 52,428 cells at most, and 7,000 to 13,000 with rows. The walk’s time follows the cells with rows, not the product, and the price counts those; the default is where the measurements stop, just above the largest product measured, 125 million cells of five columns and ten overlaps over 3 million rows, of which 700,000 had rows: the cells were still 5.8 to 8.6 times faster than core there, and their estimate of the cells with rows 6 % high. A product past it is a grouping finer than any measured, with ever fewer rows to a cell for the cells to share.

The walk (lion_count_cells(), lion_count_decode.c) is the decoded walk of §34 with one column. One scalar dimension DRIVES (lion_cells_choose_driver()): its entries are walked a pass of LION_GROUP_BATCH_MAX at a time, located under their pins, and each entry’s key is compared with the dimension’s values by its equality - entry offsets are no names (§21) - to give the outcome every row of the entry has. The other dimensions are LABELS: a scalar column’s sets of its named values with an outcome other than 0, and its NULL entry; a multi-key column’s set of each key of its leaves, and its NULL entry. At each container key the WHERE’s collected copy keeps rows at, the driver’s containers are decoded into each row’s code - its outcome times the dimension’s weight in the radix - and each label’s container there adds its weight to the code of each row it holds: a scalar dimension’s labels are disjoint entries, so a row is in one at most; a multi-key one’s keys are gathered into a mask a row first (a generation stamping each row’s), and its leaves' trees evaluated over the mask - a table of the masks for sixteen keys or fewer - give the outcome. A row of no label has outcome 0. Then the map is asked about the key’s blocks, and each row is counted under its code, or queued for the heap with it, as a decoded walk’s under its combination; the tally is the decoded walk’s, dense or a hash that spills. The driver is the scalar dimension that leaves the fewest label rows to read times its passes - a dimension of one entry a value reads nothing for itself - with no more than 16 passes and 65,536 comparisons of keys with values; where none will do (every dimension a multi-key one), any lion-indexed column of the table no dimension is drives (lion_count_all_driver()), all of its rows outcome 0.

The cells out. The node’s tuple holds the truth of each predicate - LION_TL_CELLPRED(i) columns of custom_scan_tlist, a boolean or NULL - and the cell’s partial count. setrefs.c matches the plan’s target list and qual against custom_scan_tlist top down by equal(), so the predicates inside the grouping expressions become references to those columns and the node’s own projection computes the expressions from them, once a cell (lion_next_cells()); core’s Finalize HashAggregate adds a group’s cells up, and applies the HAVING. The expression’s value is whatever the expression makes of the truths, of its own type - a numeric from round(), the tier’s text - and a group no cell has rows of has no row. A WHERE clause over the predicates - count(*) WHERE n < 50 AND <the tier> = 'high' - is answered the same way: it is the node’s qual, tested once a cell, and core’s Finalize Aggregate adds up the cells it keeps. The predicates' functions are the node’s to check EXECUTE on (lion_expr_functions(), §9 “Privileges”), since the projection reads their truths and never calls them.

Why it is exact (§9, §11). It is the decoded walk’s argument with labels for columns (lion_count_groups_decode(), and “The groups of a walk, counted together” above for the WHERE’s copy). Every row counted is in exactly one pinned set of the driver, its entry of the driver’s column located for the pass under the entry’s pin, and the map is asked at a key once every driver cursor standing there has copied its container. The labels and the WHERE copy are read with no pin, after the snapshot was taken: a row the snapshot sees was in every index before its transaction committed, and is not removed from any while the snapshot sees it, so its labels are exactly its values' and keys' - its cell is its own. What a label lists that the snapshot does not see is a dead row whose TID the driver’s container still holds - so VACUUM has not got past the driver’s page, has not set the block all-visible, and the row goes to the heap, which does not show it - or one the driver no longer holds, which is in no pass. A row of a NOPIN set of the driver, of a set read from a copy, and every row in recovery go to the heap; so does a row whose code is past the tally’s - only a dead TID in two entries of one column can make one - which the recheck raises an ERROR for if it finds it visible. Under SERIALIZABLE the blocks counted are predicate-locked page by page, as the decoded walk’s, and the index pages as every reader locks them (§9, “SERIALIZABLE”): the driver’s leaves and its CHAIN roots as a walk’s, each label’s entry or the leaf its absent key would be on, and the WHERE’s sets as a count’s.

In parallel (“A GROUP BY in parallel” above): the participants claim ranges of container keys from the shared counter, collect the WHERE of each range alone - or read every row of it without a WHERE - and count every pass of the driver there into one tally of their own, whose cells go up as partial rows at the end, under the Gather and the Finalize Agg. Each range is the serial walk over a copy of its keys, the driver’s sets located afresh by the participant under its own pins and the labels read after the leader’s snapshot, so the argument above holds in each; every row is in one range, counted by one participant. The WHERE a range collects is every clause of it: an IN list on the driver’s own column too, which a count by cells never takes for §15’s driver of the groups (lion_ingroup_item()) - it walks every entry of the driver, serial or in parallel - and locates as any other list, one set a value, or one collected set past a batch (lion_locate_list()). The planner makes the parallel form for a WHERE with a clause positive whatever its value, or none, as a parallel GROUP BY’s, and no range taken as a source: a range can collect nothing else.

The planner (lion_try_count_cells(), lion_plan_count.c) runs first in each scoped build of a count (lion_try_count_path_scoped()), with phases of its own for the GROUP BY (every expression immutable, none a bare column, hashable; or none, with a WHERE clause over predicates), the outputs (the expressions and counts the node answers - count(*) and the count(col) forms already taken - and expressions over them; nothing else) and the strategy (the decoded walk’s WHERE: one collected copy, every range a source, a positive clause to begin from, no complement OR, no residual clause, no filtered count). The WHERE phase is the count’s own; with cells being built, a clause no posting set answers that is a function of predicates (lion_cells_shape_ok()) is taken as one over cells instead of declining. A plain table only, or the table an anti join over a small inner side counts (§27, “Counts by cells beside the anti join”); not a partition, an FK-side join, a DISTINCT, a partial index’s build (§45), a hot standby, or beside an expression column (§41). The GROUP BY of a column is never one: its plan is the plain walk’s, which the cells build leaves alone.

The price (lion_cost_cells_path()): the driver’s grouped count against the WHERE as lion_cost_count_rel() prices it, its containers decoded at the keys the WHERE’s rows lie at; at those keys each label’s containers, a step or a seek to the key each, a store per member - two for a multi-key column’s, gathered first - and the posting pages they span, again in every pass; each of the WHERE’s rows counted under its cell, a column read per labelled dimension, at the decoded walk’s row costs; the qual over predicates once a cell; and a partial row per cell with rows, each projected. The cells with rows (lion_cells_nonempty()) are estimated from each outcome’s share of the table - a value’s selectivity, NULL’s, a multi-key column’s bit patterns from its leaves' selectivities taken as independent - a cell of share p holding one of n rows with probability 1 - exp(-n * p); the shares are multiplied out as histograms over their log2, half a power of two a bin, rather than cell by cell. In the units of the cheapest core path (§39) like every count. The parallel form is priced as a parallel GROUP BY’s: each participant’s share of the serial count, what each of its ranges pays again (the driver’s walk begun anew and every label sought into the range in each pass), and its partial rows - the cells with rows of its share of the rows, one set a participant.

The plan. custom_private member 23, LION_PRIV_CELLS (shape 28): each dimension’s kind, column, index, outcomes, NULL’s outcome, the values with their outcomes or the leaves' operators and queries; each predicate’s dimension and its truth in each outcome - a table for a scalar dimension’s, a bit for a leaf, the two truths of a null test of a multi-key column; and the driver’s dimension, or -1. The anti join’s ANTI member (24, §27) and TLKINDS (25) follow it: 26 members, the magic 0x5242491d. At the path stage the member also carries the predicates and the clauses over them, which lion_plan_custom_path() makes custom_scan_tlist’s columns and the plan’s qual. LION_FLAG_DECODE with the driver as the one GROUPN column.

EXPLAIN prints the driver among Lion Indexes as a grouping column’s, Group Strategy: Cells, Cell Columns - each dimension’s index and column - Cell Predicates in their order, and Cells, the bound; the clause over predicates is the node’s Filter. Under ANALYZE: the decoded walk’s Decoded Passes, Decoded Keys, Decoded Rows, Decoded Rows Rechecked, Partial Rows - the cells with rows - and, when it spilled, Tally Spills.

Finalize HashAggregate
  Group Key: CASE WHEN (CASE WHEN ((r = 'r1'::text) OR ...
  ->  Custom Scan (LionCount)
        Lion Indexes: lcc_r (r), lcc_n (n < 50)
        Group Strategy: Cells
        Cell Columns: lcc_r (r), lcc_ks.k (k), lcc_tags (tags), lcc_ks.s (s), lcc_fg.f (f), ...
        Cell Predicates: ((r = 'r1'::text) OR (r = 'r3'::text) OR (r = 'r4'::text)), ...
        Cells: 4050

Measured (2026-10-20, PostgreSQL 18 release build, 4 CPUs shared with other work, warm, the heap all-visible, work_mem 64 MB; the median of 7 runs alternated with core’s, back to back). 3,000,000 synthetic rows of 908 MB, a lion index on each of six columns, 54 MB together: c, 20 integers and 2 % NULL; r, 10 texts, NOT NULL; g, 20 integers, NOT NULL, the WHERE’s; f, a boolean 5 % NULL; s, 6 texts and 10 % NULL; tags, two to four of 64 integers. The tier is CASE WHEN score >= 40 THEN 'high' WHEN score >= 25 THEN 'medium' WHEN score IS NOT NULL THEN 'low' ELSE 'none' END over CASE WHEN r = 'r1' OR r = 'r2' OR r = 'r5' THEN round(CASE WHEN c = 0 THEN 40 ... END * 0.6 + <eight CASE WHEN tags && '{...}' THEN n ELSE 0 END> + GREATEST(CASE WHEN f THEN 6 ELSE 0 END, CASE WHEN s IS NULL THEN 2 ELSE 4 END)) END, fifteen values of c named; the WHERE form is count(*) of the rows whose tier is 'high'. Core’s plan is the plan with pg_lion.enable_count_cells off, reading the heap; its counts are the same.

3M rows, ms g IN (0, ..., 9), 50 %: cells; core g = 3, 5 %: cells; core
tier, serial 113; 2,209 54; 271
WHERE form, serial 112; 2,040 54; 264
tier, 3 workers 72; 970 39; 267
WHERE form, 3 workers 70; 1,159 38; 246

13,352 cells had rows at 50 %, 6,967 at 5 %, of the 52,428. The cells grow with the predicates, the time with the cells that have rows. Without a WHERE, the same table grouped by the predicates themselves - a CASE over each of the first columns' values as its own grouping expression, then up to ten overlaps of tags - and the same predicates summed into a tier of three groups:

product of outcomes cells with rows (estimated) GROUP BY of the predicates: cells; core a tier of their sum: cells; core
462 420 (420) 33; 654 34; 1,040
5,082 4,200 (4,200) 40; 920 41; 1,644
40,656 29,400 (29,400) 61; 1,211 60; 1,989
121,968 71,397 (87,847) 94; 1,444 106; 2,264
1,097,712 251,526 (275,216) 252; 2,564 226; 3,738
7,927,920 460,961 (491,383) 601; 3,721 583; 5,425
31,345,776 616,597 (649,675) 763; 4,353 871; 6,254
125,017,200 695,944 (739,345) 868; 4,995 1,027; 8,852

No tally spilled. A tier of four columns and up to ten overlaps under g = 3 and under the ten values of g, products of 242 to 5,953,200, was planned by cells every time against core’s plan: 38 to 158 ms against 224 to 656 at 5 %, 68 to 219 against 1,212 to 4,119 at 50 %; the cells priced at 180 to 510 units a millisecond and core’s plans at 170 to 670, the cells' price a seventh to a twelfth of core’s. A GROUP BY of a column keeps the plain walk’s plan: the cells build declines it before pricing anything.

Tests. test/sql/countcells.sql compares every shape with the plan of the pushdown off on 40,000 synthetic rows, as multisets: the tier over a score of every predicate kind, its score and both together, the WHERE form, a HAVING; exclusive and overlapping equalities, NOT IN and IS DISTINCT FROM with NULLs, booleans with and without NULLs, a value no row has, NULL and empty arrays and arrays with a NULL element, an overlap with the empty array, text searches, round, GREATEST, LEAST and COALESCE over NULLs; the declines - a column read outside a predicate, a volatile function, the bound exceeded; generic plans with Params in the WHERE, an IN list of one on the driver’s column longer than a batch among them, in a decoded walk too; a rescan per outer row and SERIALIZABLE; deleted and updated rows before VACUUM, a transaction’s own changes, and after VACUUM; enum columns and a domain over one; and parallel plans, with rows the heap decides, an IN list on the driver’s column - short, of no value with rows, a Param’s past a batch, beside the anti join - and the WHEREs that stay serial: a range source, a Param’s multi-key query, a literal prefix too common to expand, a tsquery’s or a jsonpath’s.

Ranges (not done). The analysis takes a range of a scalar column for LionProbeJoin (§46, “Ranges”: the column’s values in the index’s order, the gaps between them and NULL its pieces, outcome 0 its largest), and declines it here (lion_cells_analyze(..., false); lion_cells_choose_driver() declines a ranged dimension too): the walk reads a scalar column by its values. The count would take it in four parts, a change of its own. The driver: a ranged dimension can drive, its entries walked in the column’s order; each entry’s outcome is its key’s piece, found by the family’s comparison (LION_CMP_PROC) - a merge of the pass’s keys with the m sorted values, a comparison an entry and a value, where an equality dimension compares each entry with its values - and the bound on comparisons counts those. The labels: a ranged dimension that does not drive is labelled by its outcomes' runs, as §46’s are; an interval is many entries, which read at each container key would be a seek each, so each run is collected first, as a range taken as a source is (lion_range_collect(), §28; lion_sources_collect_range() for a participant’s range of keys in parallel), within the hash table’s memory that the WHERE’s collections share, and read at each key as a copy, as the WHERE’s is; a run that would not fit declines the plan. The price: a run’s collection as lion_cost_range_source() prices it, its reads at the keys as a copy’s, and the cells with rows from each piece’s share, a gap’s its bounds'. The plan: LION_PRIV_CELLS carries, for a ranged dimension, its sorted values, each piece’s outcome and each outcome’s runs; EXPLAIN is unchanged, the ranges among Cell Predicates. The argument for exactness is the walk’s: the collected runs are read after the snapshot, as the labels are.

11. Additional VACUUM rule for phase 2 (binding on wave 2 lion_vacuum.c)

ambulkdelete acquires a cleanup lock (an exclusive content lock that also waits for all other pins to drop) on every page that can hold a TID — every directory LEAF and every LEAF of every key’s posting tree, whether or not it has anything to remove there — in chain order (the directory leaves left to right from the leftmost one, then each posting set’s leaves left to right from ITS leftmost one).

Chain order is leaf right-link order (§22). A posting set is a B-tree over container keys now, but its leaves are the container pages that were the chain, they are still rightlinked in ckey order, and VACUUM still walks exactly that list — it only has to DESCEND to the leftmost leaf instead of starting at the entry’s head, which is the tree’s root.

Internal DIRECTORY pages are not cleanup-locked, and need not be (§21): they hold downlinks and separator keys and no TIDs at all, and a directory page never changes level while a scan can hold a link to it - a root split keeps the old root as the left half at its own level, and a deleted page is reused only once no scan can (§21, “Page deletion”) - so no reader can be holding a container copy that came from one. The directory leaves are exactly the pages that used to be bucket pages, and VACUUM treats them the same way. A posting tree’s descent IS cleanup-locked, root first (§22, and a deviation from this section’s first version, which said internal posting pages “need not be” either): a posting set’s root is a LEAF for as long as the set fits one page, a reader that copied containers out of it pins it, and an insert that overflows it pushes it down - moving those very containers to a brand new child - under an exclusive lock the pin does not stop. A VACUUM that walked from the leftmost leaf would never ask for the old root at all, clean the child, and let the heap phase mark the rows all-visible under the reader (test/isolation/count_root_pushdown_race.spec answered 1000 for 500). So pass 2 descends from the root to the leftmost leaf with a cleanup lock on every page of the way (lion_vacuum_descend()), and the reader’s pin on the old root stops it before it reaches the page the containers moved to. The root is the only page that ever changes level; the internal pages below it are locked for uniformity, and there is one of them per several hundred leaves. A leaf SPLIT obeys the same rule the chain split does - the upper half goes to a brand new page immediately to the right - so an entry can only ever move onto a leaf the walk has not passed, and an entry that moves off a leaf the walk has already finished has by then been fully processed, posting sets and all. This is the same rule nbtree’s btvacuumscan follows, and it is what closes the page-split hole in the §9 argument: a reader pins page P, copies container C out, and drops the content lock; a concurrent insert may then split P and move C to a brand-new page N linked immediately right of P. VACUUM must reach P before N, and P’s cleanup lock is blocked by the reader’s pin, so it cannot clean C on N until the reader has finished its visibility-map checks. If VACUUM had already passed P when the reader copied C, C had already been cleaned of this cycle’s dead TIDs. Two properties make this sufficient: a page is only ever linked into the leaf list immediately right of the page whose split or spill allocated it, so chain order and VACUUM’s visit order agree, and each VACUUM cycle’s set of dead TIDs is fixed before index cleanup starts. If VACUUM only cleanup-locked pages it modified, it could walk past an emptied P and clean C on N while the reader still counts from its stale copy.

A reader that SEEKS rather than walks does not weaken any of that (§22). The argument was never about which pages the reader has visited; it is about the page the container it is holding came from. A cursor that descends straight to leaf Z and takes container C from it pins Z, and VACUUM cannot have cleaned Z - it has to hold Z’s cleanup lock to remove a TID from it, and that lock waits for the pin. If a split then moves C to a page right of Z, VACUUM reaches Z first, and is blocked there. The internal pages the descent reads on the way hold no TIDs.

A ROOT PUSH-DOWN is the one page transition VACUUM can meet (§22): an insert that needs a 17th leaf in a set with no internal level turns the set’s head into an internal page and moves its containers to a brand new child (the promotion). VACUUM, holding the cleanup lock on that page, finds it is no longer a leaf and descends again from the root, cleanup-locking the way down. That is safe for the reason above: the cleanup lock it holds waited for every pin on the page those containers came from, so no reader holds a stale copy of them, and the page they moved to did not exist a moment ago. The push-down VACUUM itself causes - its REGROW step (§18) re-placing a filtered container that no longer fits the head of a set of 16 leaves - keeps the new child EXCLUSIVE from its allocation until the filtered container is on it (lion_posting_root_pushdown() returns it locked): the push-down has just copied the UNFILTERED container there, and a reader that could lock the child in between would copy the dead TIDs, keep its pin, and have the write that follows - which takes no cleanup lock - remove them under it. (test/isolation/vacuum_regrow_pushdown.spec parks the regrow between the push-down and the write and sends a count’s descent at the child: with the child unlocked there it would answer 72,160 for 64,928; with it held the descent waits for the child and answers 64,928.)

Page recycling (§18) does not weaken either property. A recycled block is still linked immediately right of its split origin, so it is still visited after it. A page only ever LEAVES the leaf list as part of a whole-set free, which happens under a cleanup lock on that page after VACUUM has found it empty - so a reader pinning it blocks the free outright, and a reader that got past it is holding containers that VACUUM has already cleaned. And a page that a split moves items onto has had those items cleaned already, because they come from a page this cycle visited first.

The guarantee readers may rely on is therefore: a TID is never removed from any page of an index before VACUUM has held a cleanup lock on every page that precedes it in chain order - the pages of each posting tree’s descent included - and the removal itself happens under a cleanup lock on that page. A reader that checks the visibility map before dropping the pin on the page a container came from cannot be overtaken by VACUUM (§9).

On a standby the same sentence holds for REPLAY of an rmgr-mode index (§25). The startup process is the only writer there, it applies the primary’s records in the order the primary wrote them - so it meets the pages in chain order for the same reason VACUUM did, and a page is still only ever linked in immediately right of the page whose split created it - and it takes a CLEANUP lock on every block a record removes TIDs or items from (VACUUM_PAGE, ITEM_DELETE, PAGE_DELETED, the ENTRY record VACUUM deletes entries with, and every record VACUUM writes through shared placement code: the regrow of §18 and an INLINE spill its filtering causes). And on every page VACUUM cleanup-locked without writing to it: those write no record of their own, so VACUUM collects them and the next record whose replay takes a cleanup lock carries them as a barrier - ranges of blocks redo cleanup-locks one at a time before it touches the record’s own blocks (§25, xl_lion_visit). Without it the “every page that precedes it” half of the sentence above has no standby counterpart, and a split, a root push-down or a spill that moved a standby reader’s TIDs off its pinned page lets replay remove them elsewhere. Pages visited after the last removal of an ambulkdelete need no barrier - every dead TID of the cycle is gone by then - and are dropped. §11’s WAITING RULE applies to the startup process as well and is met the same way: the barrier comes first, then the blocks that need a cleanup lock (the shim puts them FIRST in the record whatever order the writer registered them in), so every wait happens with no other buffer lock held. Replay of a GENERIC record takes an exclusive lock instead and offers none of this, which is why §9’s last rule is per resource manager.

VACUUM waiting rule (binding): VACUUM must never wait for a cleanup lock while holding any other LWLock, because holding an LWLock implies HOLD_INTERRUPTS and the wait becomes uncancellable (a cursor pinning one container page could then freeze VACUUM until the session ends). The leaf lock is needed only while a page is actually modified (the entry counters are written in the same record), so ambulkdelete processes each leaf in two passes:

Pass 1 (directory leaves): pin the leaf; LockBufferForCleanup(leaf) with nothing else held; process every INLINE entry on it (filter, repack, spill if grown); note the offset, the head block and the KEY of every CHAIN entry; release the content lock but keep the leaf pinned; move to the right link.

Pass 2 (each CHAIN entry’s chain, left to right, holding no page lock between steps): for EVERY page X of the chain, including pages with nothing to remove: 1. LockBuffer(leaf, EXCLUSIVE) — blocking is fine, nothing else is held. 2. if ConditionalLockBufferForCleanup(X): filter X; if anything changed, write the fresh entry copy (re-read from the page; inserts may have changed it) with the deltas in the same record; release X, release the leaf. 3. else: release the leaf; LockBufferForCleanup(X) (blocking, cancellable, nothing else held); if ConditionalLockBuffer(leaf, EXCLUSIVE): as in step 2; else release X and go to 1. A page with nothing to remove still needs the cleanup lock (step 2/3) but no WAL record; the leaf lock may be skipped for such pages only if the page is inspected under the cleanup lock first and found clean (then release and move on). Inserts may interleave between steps; that is safe because splits only move items right to new pages (VACUUM revisits or re-filters them). Offsets of entries are NOT stable any more - §21’s sorted directory inserts in the middle of a leaf and splits it - so every use of the offset pass 1 recorded is checked against the entry’s kind and stored key first, and an entry that has moved is found again by a descent made with NOTHING else held (a directory lock taken while holding a container page would close exactly the cycle the waiting rule above avoids). Counters are always applied as deltas to the entry as it is at the moment of the write, never as absolutes.

This also removes the latent deadlock between a reader that pins a directory leaf (INLINE entry) while taking a container-page SHARE lock and a VACUUM holding that container page while waiting for the leaf: VACUUM never blocks on the leaf while holding a container page.

Parallel VACUUM does not change any of this (§18, “Measured, 2026-09-23”). Since then lion declares VACUUM_OPTION_PARALLEL_BULKDEL, so a VACUUM of a table with several indexes hands each index’s ambulkdelete to whichever participant - the leader or a parallel worker - claims it first. Everything above is a statement about ONE index, and one index is still vacuumed start to finish by one process: the same chain order, the same cleanup lock on every page, the same waiting rule, the same per-process standby barrier (lion_wal_visit() keeps one list per backend and one index at a time). Two things could have made the process matter, and neither does. The dead-TID set is still fixed before any index is cleaned (the workers read the leader’s store and nobody adds to it until every index is done), and the heap is still only marked all-visible after lazy_vacuum_all_indexes() has waited for every participant, so a reader’s pin still holds the whole cycle off from making its TIDs' heap pages all-visible - whichever process it is blocking. A reader that holds pins in two indexes (an AND across two columns) can now block two participants at once instead of one after the other, which is the same wait. amvacuumcleanup stays with the leader: it only vacuums the free space map. test/isolation/vacuum_parallel.spec makes a worker run a lion ambulkdelete and checks what it leaves.

Deadlock rule for readers: never acquire a directory lock while holding a pin on a container page. VACUUM holds the leaf and then waits for cleanup locks on that entry’s container pages; a reader holding a container pin and then asking for the leaf closes the cycle, and buffer LWLocks have no deadlock detection. Finish with the leaf (copy the entry out, drop its lock) before pinning container pages, and never go back. lion_chain_find_page() takes SHARE locks internally, so do not call it while holding a lock on any page of that chain.

On a standby the rule has a second half (§25): replay locks a record’s posting pages BEFORE the entry’s directory leaf and holds both to the end of the record, so a backend in recovery must also never wait for a posting page’s lock while it holds a directory page. Every reader already finishes with the leaf first; lion_index_verify() did not - its walk of a set and its heapallindexed lookups held the leaf - and replay deadlocked against it for good (2026-09-29 review, §7).

On-access pruning sets the visibility map too (PostgreSQL 19+)

§9 was written when VACUUM was the only thing that set a VM bit. PostgreSQL 19’s heap_page_prune_opt(rel, buf, &vmbuf, rel_read_only) (pruneheap.c) passes HEAP_PAGE_PRUNE_SET_VM when the scan’s relation is read-only for the query, and heap_page_prune_and_freeze() then marks the page all-visible if what is left after pruning is. Seq, bitmap-heap, index and index-only scans all call it (heapam.c heap_prepare_pagescan(), heapam_handler.c BitmapHeapScanNextBlock(), heapam_indexscan.c), so on 19 pages our counts read are made all-visible by core scans whatever pg_lion does, and since this section the count’s own heap recheck does it too. The claim §9 needs, and that this section proves, is:

At any moment at which a heap page is all-visible, every TID that a lion index holds for that page and that the counting reader holds a pin for (§9) is exactly one row visible to every snapshot, the reader’s included.

On 16-18 on-access pruning never sets the VM (the function has no vmbuffer argument there), so this section is about 19 and later; checked against 19’s and 20devel’s pruneheap.c, which differ only in one visibilitymap_clear() call.

What the index holds. A lion TID is always a ROOT line pointer: the index is not summarizing, so a HOT update inserts no index entry and a heap-only tuple is never pointed at. (A heap-only tuple’s slot may later be freed and reused, but reuse means a new tuple inserted with its own index entries, not an existing entry now naming it; see “LP_UNUSED” below.)

What an all-visible page may contain after on-access pruning. heap_page_prune_and_freeze() plans every line pointer (prune_freeze_plan()) and sets the page all-visible only if set_all_visible survives all of the following, each of which clears it: - an LP_NORMAL tuple left on the page that is not HEAPTUPLE_LIVE (heap_prune_record_unchanged_lp_normal(): RECENTLY_DEAD, INSERT_IN_PROGRESS and DELETE_IN_PROGRESS all clear it), or is LIVE but not yet hinted xmin-committed; - the newest xmin of the live tuples still being considered running by any snapshot (GlobalVisTestXidConsideredRunning(vistest, newest_live_xid), after the plan) - the reader’s registered snapshot advertises its xmin in its PGPROC, so a row committed after the reader’s snapshot keeps the page off the map; - any LP_DEAD item at all, whether pruning made it just now or it was there already: heap_prune_record_dead() and heap_prune_record_unchanged_lp_dead() both append to deadoffsets/lpdead_items, and if (prstate.lpdead_items > 0) set_all_visible = false runs before heap_page_will_set_vm(), with Assert(!set_all_visible || lpdead_items == 0) after it and a heap_page_is_all_visible() cross-check on assert builds. DEAD tuples themselves never survive into the all-visible page: a DEAD root (an aborted insert, a deleted or non-HOT-updated row whose deleter is older than every snapshot) becomes LP_DEAD, which by the last bullet blocks the bit; a chain whose root is followed by dead versions is redirected to its first live member, and the dead HEAP-ONLY members become LP_UNUSED. So on an all-visible page every root line pointer is one of: LP_UNUSED; LP_NORMAL holding a tuple visible to all; LP_REDIRECT to a chain whose surviving members are all visible to all.

Each of those, as a TID in the index. - LP_NORMAL, visible to all: one row, visible to the reader. It cannot also be HOT-updated to another visible version: a committed updater makes it at least RECENTLY_DEAD, and an aborted updater leaves a DEAD heap-only successor that prune_freeze_plan() frees separately. - LP_REDIRECT: heap_prune_chain() redirects the root to the first non-DEAD member and frees the dead ones before it; for the page to be all-visible that member is LIVE, and it is the only live one for the reason in the previous point. One row. (A redirect whose target is gone becomes LP_DEAD, heap_prune_chain()’s nchain < 2 case, and blocks the bit.) - LP_UNUSED: the one case that would count a row that does not exist, so it must not happen while the reader’s pin is held. On-access pruning never makes a root LP_UNUSED: it frees only heap-only tuples - the members after the root in heap_prune_chain() and the unchained DEAD heap-only tuples in prune_freeze_plan()’s last loop, and heap_page_prune_execute() asserts exactly that - while a dead ROOT goes through heap_prune_record_dead_or_unused(), which picks LP_DEAD unless HEAP_PAGE_PRUNE_MARK_UNUSED_NOW was passed. heap_page_prune_opt() never passes it (“cannot safely determine that during on-access pruning”), and VACUUM passes it only when the relation has no indexes (vacrel->nindexes == 0), which a relation with a lion index is not. LP_DEAD to LP_UNUSED is VACUUM’s second heap pass (lazy_vacuum_heap_page(), the lp_truncate_only path), which runs only after every index’s ambulkdelete for that cycle - and ambulkdelete cannot pass the page our container came from while we pin it (§11 above). So a pinned container’s TID is never LP_UNUSED, and nothing about that changed in 19. - LP_DEAD: excluded by the page being all-visible. - A heap-only tuple: excluded by “what the index holds”.

Every other path to the bit. Besides on-access pruning, 19 sets the VM in VACUUM’s first pass (the same heap_page_prune_and_freeze(), with the same LP_DEAD rule), in its second pass (after ambulkdelete), for empty pages, and in heap_multi_insert() for COPY FREEZE into a table created in the same transaction - none of which depend on who pruned. Nothing else calls visibilitymap_set(). heap_page_fix_vm_corruption() and the fast path for already all-visible pages only ever clear. An insert, update or delete still clears the bit under the heap page’s exclusive lock, so the “stale true” case of §9 is unchanged.

The cases asked about, one by one. Aborted insert: the root is DEAD, becomes LP_DEAD, no bit until VACUUM has removed the TID. Aborted HOT update: the new version is a DEAD heap-only tuple, freed; the old one is LIVE (its xmax aborted) and is the one row. HOT chain with committed updates: redirect to the live member, one row; while the replaced version is still RECENTLY_DEAD to some snapshot - the reader’s included - the page is not all-visible. In-progress insert, update or delete: INSERT_/DELETE_IN_PROGRESS, no bit (and the writer cleared it anyway). The reader’s own earlier writes: its XID is running, so the same. Frozen tuples: visible to all, one row each.

The standby. heap_page_prune_opt() returns at once in recovery, so a standby never prunes on access; it replays the primary’s XLOG_HEAP2_PRUNE_ON_ACCESS records, whose VM bit is set with the page state the primary had - no LP_DEAD - and whose snapshot_conflict_horizon (newest_live_xid) makes ResolveRecoveryConflictWithSnapshot() cancel any standby query that could still see the page as not visible to all, exactly as for VACUUM’s VM records. The LP_DEAD to LP_UNUSED step is replayed from VACUUM’s own records, which follow its index records in WAL order, and in rmgr mode those take the cleanup locks of §9’s standby rule. The argument therefore reads on the standby word for word; in generic-WAL mode the standby rechecks every TID anyway.

Conclusion. The claim holds on 19 and 20devel: on-access VM setting never exposes a TID of a pinned container that is not exactly one visible row. That covers the pages core cleans on access today with no change of ours, and the count’s own pruning below. Index-only scans depend on the same property (they too count a TID on an all-visible page without a heap visit), which is some reassurance, but it is not the argument: the bitmap-heap-scan skip-fetch that core removed in 18 failed on the TIDs a scan holds WITHOUT a pin, which is what §9’s pin rule is for.

The count prunes on access too (lion_recheck_heap_heap(), 19+ only). Before share-locking a heap block it rechecks, the count calls heap_page_prune_opt(heap, buf, &vmbuf, rel_read_only) once, exactly as BitmapHeapScanNextBlock() does: buffer pinned and NOT locked (the function takes the cleanup lock itself, conditionally, and pins the VM page before trying), the count’s own vmbuf pin reused and released where it always was. It takes no lock it could wait on while holding another: the index pages the merge still holds are pinned, not locked, and the cleanup lock is conditional. rel_read_only is decided when the LionCount node starts, as ScanRelIsReadOnly() decides it for core’s scans, but by relation and not by range-table index: the node’s own RTE is never a result relation (it is only planted in a SELECT with no row marks), so the question is whether the statement modifies or row-locks the counted relation, a partition of it, or an ancestor of that partition anywhere else - a scalar subquery counting t in an UPDATE t must not set bits the UPDATE is about to clear. The SQL-callable counts (lion_index_count() and friends) cannot see their statement and pass false: they prune, as any core scan of a modified relation does, and never set the VM. The flag is a heuristic about wasted work only; the argument above does not depend on it. - The per-query visibility cache (§9 step 6) stays valid: pruning only removes versions dead to every snapshot, keeps survivors reachable from the root, and never moves a root line pointer, which is the argument the cache already rests on (its point 2); a block answered from the cache is not read at all and is not pruned. - A grouped count may now meet, in a later group, a page that an earlier group’s recheck just made all-visible, and count it from the map: that is the claim above, applied to a container whose page is pinned at the time of the VM check, like any other. - SERIALIZABLE is unchanged: pruning takes no predicate locks and removes only versions no snapshot can see; the tuple locks of the recheck and the page locks of the VM path are what they were. - The trade: a count may dirty heap pages and write WAL (a prune record), as core’s scans on 19 already do; it never does so in recovery.

Every bulk delete counts itself (§37)

The aggregates over keys (§37) read entry headers with no pin held, and learn that a TID they counted may have left the index - an aborted row’s, which no snapshot holds VACUUM back from - from the meta page’s bulkdeletes. So, binding on lionbulkdelete(): every call ends by adding one to it (lion_meta_count_bulkdelete(), in the NDISTINCT record that carries §33’s counts), after the last TID it takes out and before it returns - which is before VACUUM’s second heap pass can mark any page whose dead items it was handed. A call that found nothing to take out counts too: the TIDs it was handed may have been taken out by an earlier call that failed before its end. The one call that cannot write the record, on an rmgr-mode index whose server has not registered the resource manager, cannot take a TID out either (§25: its first removal stops the VACUUM). Nothing else removes a TID from an entry; a path that ever does - an opportunistic cleanup on insert, say - has to count itself the same way, or the aggregates over keys will count the rows it removed.

12. Measured on 20M rows (2026-09-20) and v1 priorities

Measured (optimized build, see README; final run with sparse segments, inline_limit 4096 and byte-sized buckets): count pushdown 2.6 ms for a 10M-row key, 56 ms for a 200-group GROUP BY, 4.9 ms for a 3-index AND; bitmap-scan paths heap-bound at parity with btree/GIN; index build 11-19 s per column; inserts ~4x btree cost. Sizes on 20M rows: clustered keys 9x smaller than GIN, dense random keys 1.3-2.7x GIN, sparse high-cardinality keys 1.2-1.3x GIN (c20k 209 MB vs 157, c1m 240 MB vs 199).

v1 priorities, in order of measured impact: 1. DONE (§13): sparse posting representation for keys with few members per container. 2. DONE (§13, “Build policy, measured”): the space policy around it - inline_limit defaulting to 4096 and bucket counts sized by bytes - which is what turned the saved bytes into a smaller index (c20k 164 → 37 MB, c1m 256 → 116 MB). 3. Bitset page packing: a 4104-byte 15-bit bitset fits once per 8152-byte page (c2: 56 MB vs GIN 21 MB). 14-bit containers pack three per page and shrink c2 to 38 MB but cost mid-cardinality keys ~18%; a dedicated two-bitset page format, or splitting only bitset containers, would get both. Sharing a container page between keys is the other half of this. PARTLY DONE (§38): a dense container is a NARROW of the width its heap pages' offsets need, 520 bytes for 63 rows a page or fewer and 2568 at most, fifteen to three to a page. 4. Insert cost: a container is copied out and back per insert (bitset fast path exists). Consider an in-place add for ARRAY containers with slack and a GIN-style pending list for bulk loads. 5. IN predicates in the AM and the count pushdown are DONE (§15, amsearcharray), and so are NULL keys (§14, amsearchnulls) and the multi-key opclasses of §17. Range predicates are §28: strategies 6..9 on every ordered scalar class, a bounded walk of the sorted directory (§21) in the bitmap scan, and in the count pushdown a range that bounds the entry walk driving the count rather than a source of its own. 6. DONE (§18): page recycling and entry deletion. 7. Params in the count pushdown; multi-column GROUP BY. Partitions are DONE (§16). The per-container visibility-map read (lion_vm_allvisible_mask) is already in: it turned the GROUP BY from O(heap blocks × keys) (1034 ms) into O(containers) (50 ms).

13. Sparse segments (v1 format addition)

Problem (measured, §12): a container costs 12 bytes plus alignment even for one member, so keys with fewer than ~4 members per 64-page window (high-cardinality columns, tsvector lexemes, c20k/c1m) cost 24 bytes per row where GIN costs 3.

Item kind. Container pages and INLINE payloads may now hold a fourth item type, the sparse segment, sharing the 8-byte LionContainer header. Its pairs are heap TID codes (code = ckey << 15 | lo, 41 bits at 8K), sorted and unique, stored as a stream of delta varints (format version 9):

type        = LION_CT_SPARSE (4)
ckey        = container key of the first pair; the stream's base is ckey << 15
cardinality = number of pairs n (1 .. LION_SPARSE_MAX_PAIRS = 255)
flags       = the stream's length in bytes (n .. LION_SPARSE_MAX_BYTES = 255)
stream      = n LEB128 varints: the first pair's lo (its code minus the base, so below 32768),
              then each code minus the one before it (at least 1)
total size  = 8 + flags <= LION_SPARSE_MAX_SIZE = 263

Every varint is in its shortest form, so a set of pairs has exactly one encoding: the redo of an in-place insert (LION_OP_SPARSE_INS, §25) calls the same lion_sparse_insert() on the page item and has to land on the same bytes, which wal_consistency_checking = 'pg_lion' checks. A delta is at most LION_SPARSE_VARINT_MAX bytes (6 at 8K: a 41-bit code), a zero delta is a duplicate pair and is rejected by lion_sparse_check() with any varint that is overlong, truncated or carries a code past the largest heap TID, and with bytes after the n-th pair.

Why a stream. The pairs of a key too sparse for a container are, by definition, far apart in the heap: a few to a few hundred heap blocks, a code delta of a few hundred to a few hundred thousand, which is two or three bytes as a varint against the six of a (uint32 ckey, uint16 lo) pair. And a pair above the last one - every insert of a row the heap appended - changes the header’s count and length and adds bytes at the END, where with a fixed-width ckeys[] array followed by los[] it shifted the whole los[] array: the DELTA record of an INLINE entry (§25) went from covering the tail of both arrays to covering the header and the new varint. Measured on the 5M-row table of §12 (50,000 rows appended, pg_walinspect; lion WAL per inserted row, index after the build):

column   lion WAL per row   index size
c20k     576 B -> 78 B      41.2 MB -> 19.1 MB
c1m       88 B -> 79 B      96.6 MB -> 83.2 MB
skew     158 B -> 80 B      4.98 MB -> 2.91 MB
c200      87 B -> 86 B      16.5 MB (containers only: unchanged)

A middle insert (a row placed in space VACUUM freed) rewrites the tail from the insertion point, which is at most the segment’s 255 bytes.

The cap. 255 bytes of stream, which is what the one-byte flags field holds, and so at most 255 pairs (about 85 at three bytes a pair). It keeps every look inside a segment a sequential decode of at most 255 bytes - a lookup by ckey, lion_item_last_ckey() (which the page ordering asks of every item), the cursor’s seek past a segment’s ckeys below a target - so there is no skip table: the GROUP BY pushdown over c20k’s 20,000 keys, a 2,000-key count and a bitmap scan of ten keys take the same time as over the fixed-width segments, within run-to-run noise. A key with several hundred sparse pairs takes several segments, eight bytes of header each.

A segment covers the ckey range [ckey, ckey of its last pair]. Invariants: every ckey of a posting set appears in at most one item (a regular container or inside one segment); items on a page are ordered by first ckey and their ranges do not overlap or interleave; page minckey/maxckey and chain ordering use each item’s first and last ckey; lion_item_size() dispatches on type for containers and segments. The entry tuple’s ncontainers counts ITEMS, containers and segments alike (it is the number the chain machinery maintains and verify() checks against the items it finds); the containers, array_containers, bitset_containers, run_containers and narrow_containers (§38) columns of lion_index_stats() count only real containers, and container_bytes is the bytes of every item, segments included.

Policy. LION_SPARSE_THRESHOLD = 4: a ckey with ≥ 4 members is a regular container; with ≤ 3 members its pairs live in a segment. (The threshold was chosen when a pair took six bytes and an ARRAY’s 12 + 2n beat 6n from n = 4. At two or three bytes a pair a segment stays the smaller form for more members than that, but the threshold also decides what the count cursor and the set algebra get as real containers, and it is unchanged by the stream format.) - Build: per key, group sorted codes by ckey; dense ckeys emit containers, sparse ckeys append to the open segment; a container closes the open segment (ranges must not interleave); a segment also closes when a ckey’s pairs would take its stream past 255 bytes (lion_sparse_append(), which writes the same bytes as an insert of the same pair). - Insert: locate the item for the ckey (binary search by first ckey; a segment matches when its range covers the ckey or when the ckey falls between two items — then prefer appending to the preceding segment if it has room and no container lies between, else the following one, else a new 1-member segment). Insert the pair in sorted position. If the ckey’s members within the segment reach the threshold, extract them into a new regular container item and split the segment around it (left part, container, right part; empty parts vanish). If the pair does not fit the segment’s 255 bytes, split it in half. All through lion_chain_put_container-style placement so page splits work unchanged. Order matters: the threshold test comes FIRST, because promoting a ckey shrinks the segment, so a full segment only has to split in half when the pair really stays in it and one insert can never produce more than three items. The half-split therefore only happens for a pair that lands inside a full segment’s range (out-of-order inserts, e.g. heap pages freed by VACUUM and reused); a pair beyond the last item simply starts a new segment when the preceding one is full. A full segment all of whose pairs share one ckey cannot be split in half; that state is unreachable under the policy, and the insert promotes the ckey to a container instead of failing. A key’s first TID is a one-pair segment (9 to 11 bytes: the header and its lo), not a one-member ARRAY container (10), and every pair after it two or three bytes instead of ten for a container. A container built by promoting a ckey out of a segment gets one lion_container_optimize() call, exactly as the build path does when it closes a container — that is what makes 200 consecutive inserts of one key a 14-byte single-run container instead of a 408-byte array. - Placement: lion_chain_put_items_locked() replaces the item at an offset (or inserts at it) with 1..3 items in ONE GenericXLog record, splitting the page exactly as before when they do not fit; lion_chain_put_container_locked() is a one-item wrapper over it. Atomicity matters here: a crash between writing a promoted container and the remains of its segment would leave a ckey in two items, which every reader would then see twice. - VACUUM: filter pairs; delete empty segments; a regular container that falls below the threshold may stay a container (conversion back is optional). - Scan: emit pairs grouped by heap block via tbm_add_tuples. - Count (§9): the set cursor presents each ckey of a segment as a temporary ARRAY container built on the fly (≤ 3 members), so the merge/AND/VM-mask code is unchanged; one VM mask read per ckey. - verify: segments sorted, non-overlapping with neighbours, sizes in range, cardinality == n, each ckey inside a segment has < threshold members (after build; inserts may transiently violate only until extraction). stats: add sparse_segments and sparse_members columns. “Sizes in range” means the item’s allocated length is at least what its header needs and at most that plus one slack allowance (§4): a segment an insert wrote has room for a few more pairs inside it. - Container library: lion_container_check() rejects type 4 (“item is a sparse segment, not a container”); segment helpers live in src/lion_sparse.[ch] with their own standalone unit test (test/unit/sparse_test.c, make unit). Segments never grow while being filtered, so VACUUM’s regrow path stays container-only. - Damaged segments are handled as §3 handles damaged containers: every function decodes at most the header’s flags byte of stream - 255 bytes - and at most lion_sparse_npairs() pairs, the header’s count clamped to 255, stopping at a truncated or overlong varint, so a header claiming more never walks a reader past 263 bytes, and a mutator rewrites the claim first. VACUUM copies a page item into its work buffer by the line pointer’s length, not by the header, which is why the bound is the stream and not the count. lion_sparse_extract() builds its container with lion_container_add() rather than the bulk builder, which is promised ascending, unique members that a damaged segment need not have (the same bytes for a well-formed one, and at most three members). - And (2026-09-27 review): lion_sparse_iterate() and lion_sparse_remove_if() hand their callbacks lo values masked into range - the window scan indexed a 4 KiB bitset image with them - and the count cursor, which expands a segment’s pairs through the bulk builder, masks them too, while the builder itself now checks its promise instead of Assert()ing it (a value out of order or repeated is added the slow way). An EMPTY segment is never stored, but a damaged page can hold one, and lion_item_last_ckey() read its pair n - 1 far past it: it now answers the header ckey, and lion_sparse_check(), lion_inline_fetch() and lion_page_item_fetch() refuse one.

Expected effect: c20k 475 MB → ~125 MB, c1m 337 MB → ~170 MB (GIN: 157 / 199 MB).

Measured (2026-09-20, implemented; 5M rows, 27072 heap pages, same table before and after):

column   posting-set bytes   free bytes in those pages   items
c20k     40.2 MB → 30.1 MB   94.9 MB → 139.8 MB          3.77M containers → 26.9k containers + 46.4k segments
c1m      49.8 MB → 37.9 MB   170.5 MB → 182.3 MB         4.97M containers → 0 + 993k segments
c200     unchanged (25 members per ckey: no segment is ever opened)
c2       unchanged

pg_relation_size is the same before and after (c20k 164 MB, c1m 256 MB, c200 12 MB, c2 7272 kB) because at this scale both keys are already at a page-granularity floor that the item format cannot move: a CHAIN posting set owns whole container pages (c20k: 20000 keys × 1 page, 156 MB, now 85% empty), and c1m’s entries are INLINE in 32768 bucket pages (256 MB, now 71% empty). The bytes the format saves therefore only show up as free space until the next two v1 items are done — sharing a container page between keys, and sizing buckets by bytes rather than by distinct keys. What it already buys where the posting set can stay inline: c20k with inline_limit = 4096 is 35 MB (GIN 26 MB, btree 34 MB) against 164 MB with the default limit. The write path gains as well: 100k inserts into a 1M-row table indexed on c20k take 1.64 s → 1.51 s and grow the index by 9.3 MB → 5.1 MB. Counting costs a little: the GROUP BY pushdown over 20000 c20k keys goes 395 ms → 415 ms (same 3.77M container keys, now built on the fly out of segments); a single-key count is unchanged at 0.03 ms.

Build policy, measured (2026-09-20, same 5M-row table, 31848 heap pages)

The two changes of this wave - inline_limit defaulting to 4096 instead of 1024, and bucket counts sized by the bytes the entries need instead of by the number of distinct keys - are what turn the free space §13 found into a smaller index:

column   roaring before   roaring after   GIN      btree
c2            8488 kB         8488 kB     5280 kB  33 MB
c200            13 MB           14 MB       13 MB  33 MB
c20k           164 MB           37 MB       26 MB  34 MB
c1m            256 MB          116 MB       87 MB  56 MB

c20k: 20000 keys × ~1.5 KB of sparse segments each. They used to spill to a container page apiece (1024-byte limit) - 20000 pages, 85% empty - and now stay inline in 4732 bucket pages across 1807 buckets. c1m: 993308 keys × ~84 bytes, which used to own 32768 bucket head pages because the count came from ndistinct/32 rounded up to a power of two, and now has 10660 buckets in 14818 pages (the byte estimate is about 30% low for keys whose members are spread thinly over container keys, so the buckets overflow by that much; the overflow pages cost nothing but a longer chain). c200 grows by 71 pages: 135 buckets instead of 64. Nothing else moved - c2’s two keys own bitset chains either way, and the posting-set bytes are exactly the ones §13 measured.

14. NULL keys (v1, implemented)

One reserved entry per index holds the posting set of rows whose key is NULL: flag LION_ENTRY_NULLKEY, keylen 0, hash 0, living in bucket 0 and matched by the flag rather than by key. Everything that searches a bucket for a key skips it (lion_find_entry_ext), and everything that reads a stored key asks first; lion_find_null_entry() is the only way to it. Once located it is an ordinary entry: it spills to a chain, is filtered by VACUUM and is counted exactly like any other, and the spill keeps the flag - every path that rewrites an entry’s flags (insert spill, VACUUM spill, lion_entry_rebuild, build) preserves all of LION_ENTRY_RESERVED, and verify() reports a key-less entry without one as corruption (keylen 0 is not a legal key length).

Build: the tuplesort’s key column may be NULL (hash 0 for NULLs; both passes group by the isnull flag first, then by key). (Since §24 “Build” there is no sort: a NULL row’s code is appended to the column’s reserved NULL entry directly.) Insert: NULL → the null entry (created on first use). Scan: amsearchnulls = true; SK_SEARCHNULL returns the null entry’s set; SK_SEARCHNOTNULL alone returns every other entry (a full walk of the index, correct but expensive; the cost model prices it as the whole index), and beside other quals takes the null entry away from their answer (“IS NOT NULL in the scans”, below). Stats gain null_tids. verify checks there is at most one null entry, that it is in bucket 0 with keylen 0, and that no entry carries unknown flag bits.

Count pushdown as built. The second and third bullets deviate from the sketch this section started as, both to keep the buffer-pin budget of §9 bounded:

  • col IS NULL is a positive source: the null entry’s posting set, intersected with the rest.
  • col IS NOT NULL is a NEGATED source, not a union of every other entry. The rows whose key is NULL are exactly the members of the null entry, so the answer is the intersection of the other clauses MINUS that set, which the merge computes per container key with lion_container_andnot(). A union of every entry would need one cursor - and up to one buffer pin - per distinct key. Any number of IS NOT NULL clauses cost one extra cursor each, and no inclusion-exclusion.
  • col IS NOT NULL with no other clause and no GROUP BY has nothing to drive the merge, so the node instead iterates every entry of that column’s index and sums the counts (the entries of one index are disjoint - a row has one value per column - so the sum is the count of their union; §15 states the argument in full). The same disjointness means the clause’s own negated source has nothing to subtract from any OTHER entry of that index, and nothing to leave of the null entry itself, so the driver skips the null entry and drops the source rather than subtracting it at every container key of every entry (§15, st->sumallitem: 36.9 ms to 13.0 on a 1M-row nullable column). A IS NOT NULL on a different column keeps its source. EXPLAIN shows this driver as idx (all keys), and EXPLAIN ANALYZE counts the entries in Posting Sets Summed.
  • GROUP BY on a nullable column no longer bails: the NULL group is the null entry’s count, emitted with a NULL datum.
  • count(col) is answered when col is declared NOT NULL, or is the group column (count() for a real group, 0 for the NULL group), or a clause pins it with a strict operator or IS NOT NULL (count()), or a clause says col IS NULL (0). The attnotnull requirement of §10 is gone.

IS NOT NULL in the scans (2026-10-19)

Every lion scan builds its rows from its quals the same way (§5 SCAN step 5, §29.2, lion_scan_choose()): the bitmap scan (lion_getbitmap_choice(): lion_emit_columns(), or lion_emit_intersect() beside a range), the plain scan’s source (lion_source_build(), §29.3), and LionOrdered’s and LionBtreeScan’s sets, which are built from that source (lion_source_open(), §30.4) or probed lazily (lo_probe_node()). Until this date col IS NOT NULL was ANSWERED by them only when nothing else answered - a walk of every entry of its column but the NULL one - and beside anything that did it was DROPPED (lion_scan_col_tree() had no tree for it; lion_source_build() set the recheck “next to anything that answers”; lo_lazy_ok() refused it): the bitmap heap scan rechecked every row of the others' answer, the plain scan set xs_recheck, and the sets of §30 and §40 were reported inexact, so that the index-only walk fetched every candidate’s heap tuple to recheck it. Beside a strict qual of its own column it was, and is, IMPLIED, neither answered nor rechecked: the strict qual never selects a NULL. Where such a set fed LionOrdered’s index-only walk of a covering btree, every candidate was fetched from the heap and about half of them thrown away by the recheck. Now the set is exact: no heap fetch, a fraction of the buffers, and many times faster cold and warm.

The count pushdown answered it exactly all along, as a NEGATED source (above), and the scans now do the same. Every row the index holds is under exactly one of a column’s NULL entry and its other entries, so a row of the other quals' answer has a value in the column exactly when the NULL entry does not hold it: IS NOT NULL beside them is their answer LESS the NULL entry. A node of the shared evaluator, LION_KN_ANDNOT (lion_expr.c), is its first child’s members less every other child’s. Over the AND of the other quals' trees and the NULL entries of every IS NOT NULL column (lion_scan_and()), it seeks each NULL entry forward to the container key the AND stands at and takes it out there (lion_container_andnot_raw()), so a NULL entry is read only where the answer has rows, as a leapfrog’s other sources are (§22), and a key left empty is passed over. Only the first child stands at the keys the node hands out, so its pin is the node’s (§9) and the others, subtracted raw, keep theirs only within the budget. The node is the scans' (the count keeps its negated sources) and sits at the top of a tree: over the SETS' AND and the entry-by-entry WALK’s, over a WINDOW’s rest and each LIST batch’s (§29.3), over the bitmap scan’s intersection, batched or not, and over a lone range’s collected union. A UNION (§29.3) takes the OR of the NULL entries as one stream sought forward to each container key it hands out (lion_source_union_next()). LionOrdered’s lazy set (§30.4) takes IS NOT NULL as a NEGATED probe key: the leaf’s other keys are ANDed at the probed key and the NULL entries taken out of that; a leaf of nothing but IS NOT NULL keys is built, not lazy, having nothing to take them out of.

Alone, the first IS NOT NULL is walked - every entry of its column but the NULL one, a multi-key column’s as a UNION - and the NULL entries of any others are taken away from that walk. The bitmap scan walks a lone one itself (lion_emit_all_keys()) and runs the plain scan’s source into the bitmap for more than one, or for one beside quals none of which the sets express (lion_emit_source()).

Multi-key columns. A multi-key column keeps a NULL value in its NULL entry, a value with no key - an empty array, an empty tsvector, '{}'::jsonb - in its EMPTY entry, or for an empty array with dimensions its EMPTYDIM or EMPTYINNER one, and an array whose elements are all NULL in its NULLELEMS entry (§17); they are mutually exclusive, and every other row is under its keys. IS NOT NULL is true of an empty value, so it is every row the index holds but the NULL entry’s, as for a scalar column: the same subtraction, and the key-less entries are no part of it. Alone, it is the UNION of every entry but the NULL one, the key-less ones included, which is now exact: the UNION had been rechecked for the mode-ALL queries it also reads (§17), and now is only for those.

Walking instead (lion_scan_notnull_walks()). Where a column is mostly NULL with a few values, walking its other entries beside the other quals' sets - as a range is walked, an unbounded one - reads less of the index than its NULL entry, and the scan walks: on a scalar column (a multi-key column’s walk would repeat rows), beside sets, with no range or long list already taking the walk’s place, when the walk - a descent, the column’s directory leaves, an INLINE entry being its share of one and a CHAIN entry its posting pages, once per window (§29.3) - reads LION_NOTNULL_WALK_MARGIN (2) times less than the NULL entry. The NULL entry’s pages are those of the form it takes (lion_null_entry_pages()): the model the scans size sets by, two bytes a member and a bitset at most, prices a NULL entry that holds nearly every row at its keys - the one worth walking past - as the bitsets it is not; it is RUNs, a run a heap block and one more for each other row there, or a NARROW as wide as a block’s rows (§38). A column NULL on 29 rows in 30 at 44 rows a block takes 520 bytes a container, not 4104, and the first cut, which priced it as bitsets, walked its three values beside b = 5 over 60,000 rows and read 11 index buffers where taking the NULL entry away reads 7 (9 with the choice’s own lookup). The other rows are the index’s reltuples less the NULL entry’s; an index without reltuples is held to the bitset bound. The choice reads the directory alone - the NULL entry’s header, and the column’s leaves only when the NULL entry is large enough for a walk to win at all - and says what it chose at DEBUG1. A walk pays on a large heap: a heap of 72 container keys (36 MB at fillfactor 10), its column NULL on 399 rows in 400 and its two values INLINE, walks.

Concurrency. Under an MVCC snapshot a NULL entry read at another moment than the sets it is taken from changes nothing (§9): a row whose insert the snapshot sees is in every entry it belongs to, and a row it does not see is not returned whichever entry holds it. A non-MVCC scan (§29.5) could meet a row being inserted in the other quals' entries before its insert reaches the NULL entry, so such a scan with a NULL entry to take away sets xs_recheck.

Pricing. lioncostestimate() charges the subtraction, or the walk, as the scan chooses it (lion_cost_notnull(), §29.11), and no longer marks the plain scan rechecked for it. §40.3’s index-only walk prices the quals its set still leaves to the recheck as the heap fetches they cause.

Tests: test/sql/scan_notnull.sql (§29.12).

15. IN lists and ScalarArrayOp (v1, implemented)

AM: amsearcharray = true. liongetbitmap receives an SK_SEARCHARRAY key whose argument is an array datum: deconstruct it, skip NULL elements, look up each value and emit its set. Duplicates are not filtered there - set semantics make them harmless, they only cost a second lookup.

Count pushdown: accept ScalarArrayOpExpr with useOr = true whose operator is strategy 1 of the index opfamily and whose array is a non-NULL one-dimensional Const of at most LION_MAX_LIST_ELEMS = 100,000 elements; the clause’s posting set is the UNION of the elements' sets. Values are deduped, and since the disjoint-sum short-circuit below they are deduped by entry and not merely bytewise (see there). An empty array, an all-NULL one, or a list none of whose values has an entry means zero rows.

Long lists. The cap was a thousand while every listed value’s set was located at once, each holding a pin for as long as the node ran. Neither holds now: the pins are bounded by the list pin budget (“The pin budget” below), and a list longer than lion_array_batch_size() is located a batch at a time or collected into one union (“A list too long to locate at once” below), as a parameter’s array of any length always was. What a literal’s length still costs is a Datum and a hash per value in the node, and the planner’s work over the list (core’s selectivity of an IN list walks it once per value, and the parser has built it already); the cap bounds that. Measured on a release build of PostgreSQL 18 at 1M rows, in ten count shapes (plain, AND, OR, GROUP BY on the list’s column and another, count(DISTINCT), an FK-side join, a partitioned table, a partitioned GROUP BY), lists of 1,000 to 100,000 values: every answer the same with and without the pushdown, and the backend’s peak memory at 100,000 values 114 to 210 MB against 83 to 161 MB with the pushdown off (the list’s text and core’s planning of it are most of both; the pushdown’s planning adds 30 to 50 MB, some 400 bytes a value, whether or not it is chosen). The cost model chooses: a 10,000-value list on a 5,000-key column without a B-tree counted in 3.5 ms against a sequential scan’s 60, and GROUP BY k over a 30,000-value list in 7.3 ms against 68; with a B-tree on the column the index-only scan is chosen from about 3,000 values, which is the faster plan there. LION_MAX_ARRAY_ELEMS = 1000 stays the bound where sets really are held at once: an OR’s leaves (§19), a rewritten NOT IN (§10) and the frequencies the planner looks up (§33).

A list longer than those thousand frequencies is estimated by core as one equal share of the column’s rows per listed value, which for ids from elsewhere, few of them in this table, is many times its rows; the count’s sets were priced for them (lion_cost_set_clause()), and from about 12,000 such values on that 1M-row table the sequential scan was chosen where the count was ten times faster. The count’s price now takes the list’s rows from the index instead (lion_list_sample_sel()): LION_LIST_SAMPLE_KEYS (1,000) values evenly spaced over the list, each counted by a descent of the directory as §33 counts a listed value (lion_key_ntids()), and their rows scaled to the list’s length. It is made where §33 would serve the values' frequencies - pg_lion.enable_index_mcv on, a literal list of the key’s type compared by its equality, not a multi-key column, an index neither partial nor hypothetical, a user who may read every row of the column - and once a list a planner run (LionProbeCache.listsels). Core’s estimate of the relation’s rows is left as it is: this prices lion’s sets, which is what chose against the count. A value listed more than once is counted once, as the executor looks it up once. Measured as above (lists of every 17th integer over a 5,000-key column, 294 of the values in it): 30,000 values counted in 7.7 ms against the sequential scan’s 53 and 70,000 in 16.5 ms against 59, both now chosen; at 100,000 the sequential scan still is, where the count takes 22 ms. What is left is the price of each value’s search (LION_DESCENT_COST a value), the same for a value without an entry, and a list’s values walked in order find their leaves cheaper than that; it is a constant every list’s lookups share, and is left as it is. test/sql/isectlist.sql pins a 5,000-value list of 294 present values counted, and the sequential scan chosen without the sample.

The disjoint-sum short-circuit

The count of a union is not in general the sum of the counts, but for the posting sets of DIFFERENT entries of one SCALAR index it is. A scalar opclass extracts exactly one key from a row’s column value, so the build and the insert path put that row’s TID under exactly one entry - the entry of its value, or the reserved NULL entry of §14 - and no TID is therefore in two entries. §14 already rests on precisely this: col IS NOT NULL with nothing else to drive the merge is answered by summing the counts of every entry of that column’s index, and that is only the count of their union because the entries are disjoint. The short-circuit is the same argument applied to the entries an IN list names instead of to all of them.

lion_count_sources_cached() therefore counts each entry’s posting set on its own - lion_count_one_set(), one pass of the ordinary single-key machinery per entry, sharing the visibility-map pin, the recheck batch (spilled to a sort rather than flushed while an entry is still to come, so that a dirty block is read once, §9 step 5) and the visibility cache - and adds the results up, whenever

  • the IN list is the ONLY positive source (count(*) WHERE k IN (...), the common shape), and
  • there is no GROUP BY driver, and
  • every set belongs to one index and that index is not multi-key, and
  • the sum is the CHEAPER of the two, which is a question about the data rather than about correctness and is answered separately below.

§9 is unchanged and needs no new argument: the sets are counted one at a time, so at every container key exactly one leaf cursor is live, it pins the page its container came from, the visibility map is consulted (lion_count_container_vm()) and only then is the cursor advanced. Nothing is materialised - each set is read once - so the interlock is carried by construction.

Two conditions on the input, both enforced rather than assumed:

  • Multi-key opclasses never take it (§17). One row yields many keys there, so it is under several entries and the sum over them is not a row count - the same reason §17 refuses such an index as a GROUP BY or sum-over-all driver. lion_sources_disjoint_sum() asks the index (LionState.multikey) rather than trusting the caller.
  • No two sets may be the SAME entry. The bytewise dedupe in lion_posting_set_lookup_many() is not enough: an opclass whose equality is not byte equality - citext - has distinct values that hash alike and reach one entry, and summing it twice would count its rows twice. Every located entry is therefore also compared, by the (page, offset) its entry tuple lives at, with the entries the same hash run has already found; LionPostingSet carries that pair. The caller states the result in LionCountSource.disjoint, which only a caller that used that function may set - an IN list under an OR (§19) shares one set array with the other leaves and does not.

… and when it is FASTER

Whether the sum is the right answer and whether it is the cheap one are different questions, and lion_sum_is_cheaper() answers the second. Both paths read every container of every set exactly once and count the same members; what they do not share is where the per-container overheads land. The SUM asks the visibility map once per (set, container key), because each set is counted on its own. The MERGE asks it once per container key, for the union, and pays instead a heap sift and a union step per container. So the sum wins exactly when there is nothing to amortize, and two things say there is - either one sends the count back to the merge:

  • DENSE sets. If an entry has many rows at the SAME container key, the merge folds all the sets' members at that key into one container and asks the visibility map about it once. Measured at 1M rows (15385 heap pages, so 241 container keys), k IN (1000 values) with the sum against the same binary forced to merge: 20000 distinct keys (1.0 rows per key per container key) 3.6 ms against 9.9, 5000 keys (1.0) 11.5 against 21.5, 2000 keys (2.1) 25.1 against 32.2, 500 keys (8.3) 21.2 against 9.0, 200 keys (20.7) 14.0 against 6.5. The crossover is between two and eight rows per container key, so LION_SUM_MAX_DENSITY is four.
  • MANY of them, because the merge only becomes good at dense sets once it reaches its bitset image (LION_OR_BITSET_MIN containers at one key); below that it folds them pairwise, which is quadratic in the sets. On the 200-key column above, sum against merge at 5 values is 0.49 ms against 0.64, at 15 values 1.14 against 2.37, at 30 values 2.15 against 7.6 - and at 50, where the image takes over, 3.5 against 2.5. (A range collected as a source had the same quadratic fold, one container at a time into its union, and now keeps the union as an image once it is one - §32, “Summed ranges: dense and probed”.)

The density is read off the entries' own row counts, which the located posting sets carry already (ntids), against the number of container keys the heap has; no extra page is read to decide it. lion_cost_count_rel() makes the same test from the planner’s estimates, which is why both thresholds live in lion_count.h rather than in lion_count.c.

The GROUP BY driver

A GROUP BY on the very column the list constrains takes the short-circuit too, in a different shape: the listed values ARE the groups, so the node walks the clause’s located sets instead of the index’s entries (lion_next_group_inlist()) - a thousand sets instead of twenty thousand entries on c20k - and the clause stops being a source of the merge, because a group intersected with the union of a disjoint list is the group. Any other clause is still intersected with each group, so this is the group DRIVER half and applies whether or not the sum half does; it is not subject to the density test above, because a GROUP BY counts each group on its own whatever happens. It is the difference between a plan and no plan: SELECT c20k, count(*) ... WHERE c20k IN (1000 values) GROUP BY c20k took about three minutes when forced before this (182 and 189 s over two runs) - intersecting all twenty thousand entries with the thousand-way union, which is why the cost model rightly refused it - and takes 4.6 ms now against the B-tree’s 5.1, which it is chosen over.

The list drives the walk of one column’s entries: lion_next_group_inlist(), and §26’s count(DISTINCT k) walk. A decoded walk (§34) and a count by cells (§10, “Counts by cells”) walk every entry of their column 0 against the WHERE collected whole, so a list on that column is a clause of the copy like any other, located as one (lion_ingroup_item() names no driver for them): a list past a batch is collected into one set (“A list too long to locate at once”, below), where the driver’s batches would be left for a walk of them that never comes.

EXPLAIN ANALYZE reports Posting Sets Summed: how many posting sets had their counts added into a total instead of being merged - the entries of a summed IN list, or the entries of a sum-over-all (§14). Zero means the k-way union below ran, which is what an IN list ANDed with another clause, an IN list under an OR, a list the density test declined, and every multi-key clause still do. A GROUP BY reports zero as well: its groups are emitted, not summed.

Groups a list drives (2026-11-01)

Two things in the price of the driver above were wrong, and together they handed the GROUP BY to a B-tree on the column walked in order with the WHERE tested on each entry (LionOrdered, §30) - over uuid keys of up to a few hundred rows each, at lists of a thousand ids and more, a plan three to four times slower.

  • The groups. Core’s estimate of a column’s groups scales its distinct values to the rows the restrictions keep (estimate_num_groups()), and knows nothing of a list on the column: on a 4M-row table, 1,200 ids were estimated at 12,826 groups. A GROUP BY of a column can have no more groups than a col = ANY (const array) of it lists, nor more than one under a col = const, so the node’s groups are held to that (lion_group_list_bound()), the non-NULL elements counted, repeats included, which is never below the groups there can be. The column is matched under binary-compatible relabels alone - a value-changing cast (a numeric column compared as float8) may map many values to one constant - and the equality under the grouping’s collation alone, since a case-insensitive one bounds nothing of a GROUP BY under “C”. Of two grouping columns, one of them bounded, the node’s groups are held to the product. The bound is of the output only: each column’s estimate of its entries, which prices the walks (the nested loop of §20 visits every pair whatever the list), the top k and the decoded walk, is core’s. Nor is it anyone else’s: an Agg core puts above other paths keeps core’s estimate.
  • Each group’s work. It was priced as a merge of the WHERE’s sources at every group, each probed in memory (LION_MEMORY_PROBE_COST). The executor does not do that: once a group’s count would read as much as the WHERE, it collects the WHERE once (WHERE Sets Collected: 1) and ANDs each listed value’s set with the copy, sought from its key index (lion_mat_seek()). The price says so now (lion_count_rel_list_once()): the collection, and each group’s containers visited with a seek step (LION_PROBE_STEP_COST) and an AND of its members against the copy, where the WHERE could be collected (lion_cost_where_collected()) and two groups or more are walked - not on a hot standby, which collects nothing and merges each group with the WHERE’s sources.

On the 4M-row table (a tsvector WHERE of 5% of the rows, 400,000 keys of a skewed size): lists of 1,200 ids, and of 2,600 of which half have no entry, took 94 and 97 ms through LionOrdered and a GroupAggregate, and take 26 and 28 ms as the count, which was priced 60,131 and 66,135 and is now 30,283 and 33,772; an EXISTS over a table of the ids, the same groups, takes 27 and 30 ms. grouplist.sql checks the choice and the bound at 200,000 rows (8.2 ms through LionOrdered, 2.6 ms counted).

§14’s sum-over-all is the same argument

col IS NOT NULL with nothing else (§14) sums the counts of every entry of that column’s index, and the reason it may is the one above. The same disjointness also says that the clause’s own NEGATED source - the column’s NULL entry, subtracted - has nothing to do: subtracting it from another entry of the same index removes nothing, and subtracting it from itself leaves nothing, so the driver skips the NULL entry and drops the source (st->sumallitem). That is not bookkeeping: the NULL entry of a column with many NULLs has a container at every container key, so the merge was running a lion_container_andnot() against a dense bitset at every container key of every entry. A IS NOT NULL on a DIFFERENT column is a different index’s set and keeps its source. Measured, 1M rows, nullable (200 values plus 10% NULL): 36.9 ms before, 13.0 ms now, against the sequential aggregate’s 69.1.

The array may also be a Param, or an ArrayExpr over literals and Params, which is what k IN ($1, $2) and k = ANY ($1) keep in a generic plan (§10) - or, since 2026-09-27, any stable expression, or an ArrayExpr over stable expressions (k IN (current_date, current_date - 1)). The length cap then applies only when the length is known at plan time - a literal array’s, an ArrayExpr’s, and a stable expression’s that estimate_expression_value() reduces to a literal. A Param that IS an array has no length until the executor has it, and by then there is no plan left to decline in favour of, so it is answered whatever its length, and the pins are bounded by the lookup instead (below) - and, in a plain count, the memory by locating it a batch at a time (“A list too long to locate at once”).

The pin budget (2026-09-23 review)

This section used to say that an unbounded list’s pins were “bounded by the index’s bucket pages, since every element’s entry lives on one of those”. Since §21 that is the number of directory LEAVES, which grows with the index, and the bound was no bound: with 1.9 kB keys, a few per leaf, lion_index_count_any(idx, (SELECT array_agg(k) FROM t)), the pushdown’s k = ANY ((SELECT array_agg(k) ...)) and the multicolumn bitmap scan of the same clause all failed with “no unpinned buffers available” at shared_buffers = 16MB and 9000 keys - and a list one short of that would have starved every other backend of buffers instead.

So lion_posting_set_lookup_many() keeps pins on at most a BUDGET of distinct leaves, and the budget is the BACKEND’s: every list it has located and not yet released draws on the same one, so two unbounded lists in a query share it rather than taking one each. It is the smaller of LION_LOOKUP_MAX_PINS = 1000, so that a list of up to a thousand values keeps every leaf it located pinned, and an eighth of the pool the index is read into: shared_buffers, or for a TEMPORARY index the backend’s own temp_buffers (as few as 100), which the first version of this budget did not tell apart - a list over a temporary table pinned up to a thousand local buffers of 1024 and failed with “no empty local buffer available” (2026-09-25 review). The eighth is what keeps one backend from exhausting the pool - the failure above was 9000 leaves against 2048 buffers, where an eighth is 256 and leaves seven eighths to the query’s own heap, visibility-map and chain pages and to every other backend - and it only binds below 64MB (on a shared pool the fair-share bound below binds before it, since 2026-09-29 below about 340MB of shared_buffers). The CURSORS that read the located sets draw on what the lists leave of the same limit (“Bounded cursors”, below). Pins on the leaf the previous set already pins cost no buffer and are not counted. Every set that took a new leaf is marked budgeted and returns it when released. A set abandoned by an error never is, but its PIN is released - by the resource owner that was current when it was taken, with the transaction, subtransaction or portal the error ends - so each charge is recorded against that owner too, and an owner being released gives back what its sets still had charged (lion_list_pins_resowner(), a resource-release callback; 2026-09-28 review). Before that the count was only zeroed at the end of the top-level transaction, and a subtransaction failing in a loop - a PL/pgSQL EXCEPTION block around a count - left its charges behind until then, so every list after it came out NOPIN: never wrong, but a cliff. Restoring the count saved at the start of the subtransaction would not have been right either: a cursor opened outside the subtransaction and FETCHed inside it keeps its sets, and their pins, when the subtransaction is rolled back, and the owner is what tells those apart. The top-level zeroing stays as a backstop. An INLINE set found past the budget comes out NOPIN: its payload copied, its leaf let go, exactly like a materialized set (§9).

The limit is per backend, and two things it did not account for were other backends and a query’s own parallel workers (2026-09-28 review). An eighth of the pool keeps one backend from exhausting it, but eight at once still could. So for the shared pool the limit is also at most LION_LOOKUP_SHARES = 4 times the backend’s fair share of it: the pool over every process that may pin a buffer of it, core’s GetPinLimit() - NBuffers / (MaxBackends + NUM_AUXILIARY_PROCS), which is what core holds a read stream or a relation extension to, and which 16 and 17 compute the same way without exporting it (lion_pin_fair_share(), lion_compat.h). Exhausting the pool that way takes a quarter of all the processes the server is configured for, running lists over the budget at the same moment. On a stock 18 server (128MB, 100 connections: 94 buffers a backend) that is 376, and the thousand binds from about 340MB of shared_buffers up - the regression suite’s servers pin exactly what they did. A plain fair share, the budget’s first version below, sent ordinary lists to the heap; four of them keep a pinned leaf for most of a stock server’s thousand-value lists.

The first cut of this bound (2026-09-28 review) was sixteen times NBuffers / MaxBackends and never less than 64 pins, and the second review of it (2026-09-29) found both out of proportion to the pool: a thousand leaves pinned by one list count on a stock server, where seventeen such counts at once pin all 16384 buffers; and floors of their own, 64 list pins on any pool of 512 buffers or more however many connections share it, and 16 cursor pins (LION_OPEN_MIN_PINS) and 16 groups of a GROUP BY batch on any pool at all - the whole of a 128kB one. At shared_buffers = 128kB - sixteen buffers, one session - or 512kB with twelve pgbench clients, k IN (30 CHAIN values) AND x = 1, a GROUP BY over such a list and lion_index_count_any() of fifty CHAIN values failed with “no unpinned buffers available” where the plan without the pushdown answers. Nothing floors the share now: a pool too small for its connections gives a limit of NO pins, every list is located NOPIN, and the pins of the count’s cursors follow the same limit (“Bounded cursors”, below: their floor of 16 gives way to it, and past it an OR is read without its pins). What is left is what a count cannot do without: a heap page, a visibility-map page, and the page each source’s CHAIN set is read from - past the budget a list is taken one set at a time, an OR keeps no pin, an AND one child’s, and a GROUP BY counts one group at a time. test/hook-check.sh restarts the dev cluster with shared_buffers = 128kB and runs those counts (test/modules/lion_hooktest/sql/pinpool.sql), because installcheck’s pool of gigabytes never reaches any of this.

And each participant of a parallel FK-side join locates the fact filters for itself (§27), so a leader and seven workers took eight budgets: the node now tells each participant how many the plan was started with (lion_list_pin_participants(), the workers planned plus the leader, whether it takes part or not), and each takes that share of the limit. No bound kept per backend promises the pool to every backend at once - that needs a count in shared memory, which an extension that need not be preloaded does not have - and what the bound costs past it is only time: sets come out NOPIN.

The first version of this budget (2fb790e) was this backend’s “fair share” of the pool, GetAdditionalPinLimit(), which is NBuffers / MaxBackends: 86 buffers on a stock 128MB, 100-connection server. That sent ordinary queries off the visibility map - a thousand-value literal k IN (...) OR x = 1 over a 963-leaf index rechecked 20783 TIDs on 3540 heap blocks (9.9 ms) where the map answered before - and 18’s function returns 0 outright once the share is at most eight, which made the same version drop the pin of EVERY single lookup (lion_posting_set_take()) on a small pool with many connections, so a plain k = 5 AND x = 5 over two INLINE sets lost the map. Single lookups keep their pin unconditionally now. A caller’s LOOP of them does not: a multi-key clause extracts up to LION_MAX_QUERY_KEYS = 1000 keys (lion_multikey.c; beyond that the query is answered as ALL and rechecked), and since a clause’s query may be a parameter (§17, “A query known only at run time”) nothing in the query text shows how many - one @> of a thousand-element array parameter pinned a thousand leaves, and a query of several such clauses a thousand each, outside the budget (2026-09-28 review). The pushdown locates a multi-key clause’s keys with lion_posting_set_lookup_budgeted_col(): the single lookup, but a pin it keeps is charged to the list budget like a list’s (another pin on the leaf the clause’s previous key took is free), and a key found past the budget comes out NOPIN. The keys are not sorted into the directory’s order first - their positions are the tree’s leaves - so a leaf two keys share at a distance is charged twice: the budget runs out early, never late. A multi-key source copes with NOPIN leaves as any source does (below): its plan is not pinned, so another source carries the interlock or the count trusts no map. The bitmap scan holds none of them past the lookup (below).

A NOPIN set carries no §9 interlock of its own, and the count restores one in each of its shapes:

  • The disjoint sum (the common case, the list the only positive source). Counting the sets one at a time needs only one of them pinned at a time, so lion_count_one_set() locates a NOPIN set’s entry AGAIN, by its stored key, keeps that pin, and counts the fresh copy under it - the ordinary §9 order, one set per pass. A list with NOPIN sets takes the sum whatever lion_sum_is_cheaper() says, because the union would have nothing to carry the interlock. Locating again is sound for the reason a chain may be read page by page after its entry was found: the entry is the same one (a column has one entry per key), read later under the same snapshot, and an entry VACUUM deleted in between held nothing visible to anyone. One count of a single set - a group of the §15 GROUP BY driver - does the same.
  • An intersection (k = ANY ($1) AND x = 1): the rule that already lets a set be materialized applies unchanged - the count of an intersection may be taken from the visibility map while ONE positive source holds a pin at every container key (lion_source_pinned(), which now counts a NOPIN leaf as unpinned), and the argument on lion_posting_set_materialize() is exactly the argument for a stale copy. x’s set carries it.
  • Nothing carries it (an OR across columns, §19, whose leaf is a list past the budget; or an intersection of sources that are all NOPIN or materialized): the count trusts no visibility map (cx.novm) and rechecks every candidate in the heap, as a standby with a generic-WAL index does. Correct, and slower only for lists over the budget. A union read as a WINDOWED union (“Bounded cursors”, below) is in the same position: it holds no pin at all.

The bitmap scan needs no interlock at all - every TID it emits is visited by the executor - so lion_scan.c drops the pins of its per-key lookups at once (lion_posting_set_unpin()), and its IN lists are budgeted by the lookup like everyone’s; since “Bounded cursors” the multicolumn walk drops its lists' leaf pins as soon as every set is located, and its cursors keep none either. test/sql/pinbudget.sql parks a GROUP BY count with a cursor, whose WHERE sets stay located from the first group to the last, and counts the index’s pinned buffers in pg_buffercache: 1500 before the budget, at most 1000 with it; it proves every shape above still exact and which of them still answer from the map, and it pins down that a list within the budget under an OR and a plain AND of two INLINE sets DO answer from the map.

Measured on a stock PostgreSQL 20 server (shared_buffers = 128MB, max_connections = 100), 800000 rows, a 963-page index on k (40000 keys, 20 rows each, INLINE), a thousand-value literal list:

query 2fb790e (fair-share budget, 86) now (budget 1000)
k IN (1000) OR x = 1 9.9 ms, 20783 TIDs rechecked on 3540 blocks 8.1 ms, all 3540 blocks from the map
k IN (1000) (disjoint sum) 4.6 ms 3.7 ms

The merge’s sources are therefore unions: k sub-cursors merged by container key, yielding the union of the containers that share the smallest key any of them still has. The §9 pin rule is per sub-cursor and unchanged - a sub-cursor that contributed to the merged container still pins the page that container came from, because only the merge advances it, from lion_count_container(), after the visibility-map checks.

Neither half of that may be done by walking all k sub-cursors, because k is up to 1000: - Which sub-cursors are at the smallest key: a binary MIN-HEAP of the sub-cursors, keyed by their current container key. The ones standing at the root’s key come off the heap into a “hot” list and go back on as they are advanced, so the per-container-key cost is proportional to the sub-cursors that take part rather than to k. The heap entry carries a COPY of the child’s container key and of its container: reading them out of the child while sifting chases a thousand ~300-byte cursors in a random access pattern, which measured slower than the linear scan it replaced (+4 ms on a 1000-value list at 1M rows). Both fields change only when the child is advanced, which is also when it is pushed back on, so the copy cannot be stale. - The union itself: folding m containers pairwise builds and re-optimizes m-1 intermediate containers, which is quadratic in them. Above LION_OR_BITSET_MIN (32) of them at one container key, they are ORed into one bitset image of the key’s range instead and the result container is built from it once. Below that the pairwise fold is cheaper, because the image costs a fixed pass over the whole 32768-value range however few members arrive. The threshold is worth a lot in both directions: raising it past any list length, so that the fold runs everywhere, took c20k IN (1000 values) AND c2 = 1 from 11.1 ms to 21.0 and c200 IN (1000 values) AND c2 = 1 from 7.6 to 263. (Since 2026-09-28 a count folds only ARRAYs of 128 members together or fewer, and takes the image for any other union, which it no longer optimizes: “Unions and intersections unoptimized”, below. A scan’s stream keeps this rule.) A third construction was tried and rejected, and it is recorded because the shape that suggests it is the common one: a union over sparse segments (§13) arrives at every container key as a hundred or two throw-away one-member ARRAYs, for which the image’s 4 KiB memset and 512-word pass look absurd. Gathering those members into one buffer, sorting and writing them out as an ARRAY cannot win more than the image’s FIXED cost, and that cost is per CONTAINER KEY - the heap has only heap_pages / LION_BLOCKS_PER_CONTAINER of them, 241 at a million rows, and that count grows with the heap exactly as the member count does, so the ratio never improves. It is under a millisecond of that eleven-millisecond query. What the query spends its time on is the per-CHILD work: 45006 sub-cursor advances and their heap sifts. - Locating the entries: lion_posting_set_lookup_many() hashes every value first and then looks the entries up in (bucket, hash) order, so the bucket pages are read in ascending block order and duplicates - which hash equally and are therefore adjacent in that order - are dropped in one pass instead of by comparing every value with every earlier one (half a million datumIsEqual() calls at 1000 values). lion_index_count_any(idx, keys) is the SQL form of the whole path. A list whose length is a parameter’s has no cap, so all three steps answer a cancel (2026-09-28 review): the hashing every thousand values, the sort every 65536 comparisons, and the walk every 64 values - where it lets go of the leaf it has been keeping LOCKED from one value to the next and lets the next value descend again, because a content lock holds interrupts off and the check the walk used to make under it did nothing. (A lookup that steps right with lock COUPLING - the insert path’s run of one prefix - holds a lock at every moment and cannot check; its run is one prefix’s collisions.) So do the scan’s own passes over such an array (2026-10-18 review): the ordered walk’s sort of a list’s values (lion_order_points(), every 65536 comparisons, as the lookup’s sort), and the searches for the widest element of k < ANY (array) and for two distinct elements of k <> ANY (array), every thousand elements.

Cost. A list is priced per element and not per clause (§10’s lion_cost_count_rel()): one bucket page each - but read in ascending block order, so at lion_heap_page_cost()’s interpolated page cost rather than at random_page_cost, and shared once the list is longer than the index has buckets (Min(nelems, nbuckets)); one chain page each at seq_page_cost, capped at the container pages the index actually has; one container step per element per container key; and, when the merge runs at all, lion_merge_ops(). Since §22 the chain pages are what the list really reads: a list that is one source of an AND and does not drive it is SOUGHT - the merge seeks every sub-cursor of the union - so each of its k sets is priced by §22’s probed bound, and only a list that drives the merge (or is the only source) pays for all k chains in full. Which one drives is decided from the MEMBERS, as the executor decides it: a thousand-element list of a high-cardinality column has fewer members than a dense equality and therefore drives, even though its k sets lie in many more containers between them. Three things were wrong here and all three refused a query the node answers several times faster:

  • charging a list as a SINGLE bucket page made a thousand-element list look four times cheaper than it is, and the planner chose it anyway (the 2026-09-21 follow-up review);
  • charging every element a RANDOM read of a one-megabyte index then refused it;
  • and charging the merge members × log2(nelems), as if every row were compared its way through the heap, asked 24750 of the 26481 cost units for c200 IN (1000 values) at 1M rows - a query the node answers in 6.3 ms against the B-tree index-only scan’s 66 - and refused that. lion_merge_ops() charges what lion_ecursor_build() does instead: a heap sift per CONTAINER, log2(k) deep, plus the union of the containers at one key, which is a bitset image per container key above LION_OR_BITSET_MIN and the members once per fold below it. And when the disjoint sum applies there is no merge term at all.

Measured at 1M rows, all-visible heap, warm cache, assert build, pgbench -M prepared: the node with the other scan types disabled, against the same query with pg_lion.enable_count_pushdown = off, which picks a B-tree index-only scan. Unless a row says otherwise, the planner picks the node on cost with nothing disabled.

query lion B-tree before (§15 as of 081e067)
c20k IN (3) 0.15 ms 0.16 0.17
c20k IN (10) 0.18 0.19 0.24
c20k IN (100) 0.50 0.50 1.49
c20k IN (1000) 3.76 3.52 10.99
c20k IN (1000) AND c2 = 1 11.14 18.77 11.50
c20k IN (1000) GROUP BY c20k 4.63 5.07 ~185000 (refused)
t20k IN (1000 texts) 3.70 18.16 10.47
c200 IN (3) 0.34 1.13 0.43
c200 IN (10) 0.83 3.45 1.40
c200 IN (100) 3.07 34.26 3.11
c200 IN (1000) 6.27 65.98 6.64 (refused)
c2 IN (0, 1) 0.89 64.73 2.57
nullable IS NOT NULL 13.04 69.06 36.69

The two remaining mispredictions are ties the planner resolves the wrong way by a hair: c20k IN (1000) at 1.07x of the B-tree and c20k IN (3) AND c2 = 1 at 1.11x. test/sql/pushdown.sql pins the choice at 3, 10 and 1000 values on a table of its own, and test/sql/inlist.sql at 3, 10, 100 and 1000 on a high- and a low-cardinality column.

Absolute numbers from this build are worth less than the ratios: it is an -O1 assert build, and one intermediate binary of identical source measured c20k IN (1000) at 2.75 ms where every clean rebuild of it measures 3.8-4.0. Every comparison above is between two runs of the SAME binary.

A GROUP BY over the same column restricts the groups to the listed values (a group outside the list counts 0 and is not emitted). col = ANY (...) with useOr = false (= ALL) is not pushed down. EXPLAIN prints the list as idx (col = ANY ('{1,2,3}'::integer[])) (the clause’s own operator and a deparsed value, §10).

Bounded cursors (2026-09-25 review)

The pin budget above bounds what a list’s LOOKUP keeps. Nothing bounded the CURSORS that then read the located sets, and those were all built at once: the evaluator (§17) opens a cursor for every leaf of a source’s tree when the merge starts, and an IN list is a leaf per value. Each held ~17 kB - an 8 kB staging buffer for an INLINE entry (LION_CONTAINER_MAX_SIZE, 4104 bytes, which aset rounds up), a page image for a CHAIN one, a second 8 kB the first time it expanded a sparse segment - and each CHAIN cursor a buffer pin, from the moment it was built until it moved past its page. Only the disjoint sum, one set at a time, escaped it; an IN list ANDed with another clause, a dense list (lion_sum_is_cheaper() sends it to the merge), a list under an OR, a GROUP BY with a list on another column (its copies too, below) and the bitmap walk of a multicolumn index all paid it, and work_mem never came into it. Measured on 18.6 (assert build), 300k rows, work_mem 4MB: k = ANY (50000 values) AND x = 1 peaked at 857 MB of VmHWM in 4.2 s (300k values: 5 GB, 39 s), the multicolumn bitmap scan of 100k values at 1.7 GB, and over a TEMPORARY table a list of 1100 CHAIN entries (inline_limit = 64) failed with “no empty local buffer available” - a shared table has no such error, its backends just take the pool from everyone else.

What changed, each where the cost was:

  • Staging buffers sized for their items. lion_inline_fetch() never copies more than is left of the payload, so an INLINE cursor’s buffer is the payload’s length (at most the largest item); the segment buffer holds one container key’s pairs, fewer than LION_SPARSE_THRESHOLD in a well-formed segment and grown if one ever holds more. Most of the 17 kB was allocator rounding and most of the 4.2 s was touching it: that change alone took the query to 59 MB and 0.25 s.
  • An open budget, and a plan against it. Before a merge builds its cursors, each source’s tree is PLANNED (lion_plan_node()) against an open budget: work_mem of cursor memory (at least 256 kB) and, in pins, what this backend’s lists have left of the list pin limit above (at least 16, or the limit itself where that is smaller - 2026-09-29 review), so that a list’s leaf pins and its cursors' page pins come out of ONE limit per backend. The plan estimates bottom-up what each node’s cursors would hold - the LionExprCursor, the staging buffer or page image, an OR’s heap entries and accumulators, a LION_KN_POSFILTER’s filter (§17, “Stored positions”: lion_posfilter_mem(), which grows with its query) - and decides the three shapes below. An ordinary query is under the budget everywhere and is built exactly as before; the plan also carries each node’s pinned, the §9 property the count needs of one positive source, so the shape and the interlock are one decision.
  • A disjoint list is counted in BATCHES (lion_run_batches()). A source that is a disjoint list (the §15 short-circuit’s test, minus “the only source”) and would not fit the budget opened whole is cut into batches of consecutive entries that do fit (at least LION_BATCH_MIN_SETS = 16 found sets, or on a pin budget smaller than that as many as it allows and one at the least: one set is never read as a windowed union), and the merge runs once per batch with the list replaced by that batch’s union. With the list’s sets pairwise disjoint and U_1 .. U_m the batches' unions, R the intersection of the other positive sources and N the union of the negated ones, |((U_1 ∪ ... ∪ U_m) ∩ R) \ N| = Σ_j |(U_j ∩ R) \ N|, the terms being disjoint - the sum’s own argument, one batch at a time instead of one set. Every pass accumulates into the same recheck queue, visibility-map pin and visibility cache, as the sum’s passes do, and the queue is spilled to a sort rather than flushed while a batch is still to come (§9 step 5). A list that fits is one pass, and nothing changes for it.
  • Any other union too wide is WINDOWED (lion_wide_fill()). A union that is not disjoint - an OR across columns (§19), a multi-key OR (§17), a list under an OR - cannot be cut into batches that add up, so an OR node too big for the budget opens its children ONE AT A TIME: for a window of container keys each child is read from the window’s first key to its end and ORed into one bitset image per key, and closed before the next is opened; the images come out in ascending key order, as the merge’s would. The window holds at most half the budget in images and its end moves DOWN to where they run out (an image full: the largest key is evicted and becomes the end, and the next window starts there), so a sparse union over any heap is one window and a dense one as many as it must be. The children are walked pinless, so a windowed union is never pinned.
  • An AND over the pin budget keeps one child’s pins. A multi-key @> of many CHAIN keys pins a page per key; the intersection needs only the pages of ONE input, so the first child that has a pin at every key keeps its pins and the others let go of each posting leaf as they copy it.
  • The bitmap walk keeps no pin. lion_sets_iterate() used to keep the §9 pins for a bitmap scan, which needs none; it now walks pinless, planned against work_mem, and the multicolumn scan drops its lists' leaf pins once every set is located (lion_emit_columns()).
  • The copies of a GROUP BY’s WHERE sets are budgeted. lion_posting_set_materialize() copies a CHAIN set counted more than once (§9), up to 256 kB each, and a list on another column made every one of its sets a copy, kept for the relation’s whole turn with no total. All the copies one count’s sources hold, those of earlier groups included, now stay within work_mem; a copy is kept at its exact size (it grew by doubling from a page), and a set that does not fit is walked page by page as a set too big to copy always was, without being tried again at every group.

§9 is untouched, shape by shape. A batch pass is an ordinary merge: its cursors pin the page each current container came from, the map is asked under those pins, and they are closed before the next pass builds any; a batch is sized to fit, so it is never windowed, and whether a pass may trust the map is decided per pass from that pass’s plans (a batch holding a NOPIN set is not pinned). No TID is in two batches - a scalar index holds a row under one entry - and a line pointer VACUUM frees between two passes and an insert reuses under another entry is a row inserted after our snapshot: invisible to the recheck, and its page cannot be all-visible while our snapshot is registered, the case the sum’s passes already live with. A trimmed AND is the materialization argument of §9: every member of the intersection is in the kept child’s container, whose page is pinned, so VACUUM cannot have finished ambulkdelete on that index. A windowed union carries nothing, and says so: its plan is never pinned, so another positive source carries the interlock or the count trusts no map at all (cx.novm) and rechecks every candidate, as for a NOPIN list under an OR. The pins dropped are pins that carried no interlock; a pin too many never made a count right.

What is still per value. The located sets themselves: a LionPostingSet, its INLINE payload copy and a tree node, about 200 bytes a value, which is what the executor spends on the array anyway (a 30000-value list costs 7.8 MB of VmHWM with the pushdown off, 15.6 MB with it on at work_mem 4MB, 499 MB before) - except in a plain count and a bitmap scan, which locate a long list a batch at a time (below). And the query’s own text: a multi-key AND of 1000 CHAIN keys (the extraction’s cap) still holds 1000 page images, 8 MB, though past the pin budget only one child’s pins.

A list too long to locate at once (2026-09-28 review)

A parameter’s array has no length cap, and every value’s located set was held at once whatever work_mem said: some 200 bytes a value, and past about nine million values an array of sets larger than the 1GB a plain allocation may be, so k = ANY ($1) failed with “invalid memory alloc request size” where the ordinary plan answers. Two things now:

  • A plain count locates it a batch at a time (lion_array_batch_prepare(), lion_count_batched()). When the node counts the relation as one row - no GROUP BY, no count(DISTINCT) walk, no FK-side join - and an IN list that is a source of its own has more values than lion_array_batch_size() (a work_mem of 256-byte sets, never fewer than a thousand), the list is not located with the other clauses. Its values are sorted as a lookup sorts them (lion_probe_sort()) and kept with their hashes, and the count locates a batch, counts it with the other sources, releases it - pins and all - and goes on to the next. The count of the list is the SUM of the batches' counts for the reason lion_run_batches() gives one level down: with the batches B_j of whole entries of one scalar index, R the other positive sources and N the negated ones, |((B_1 ∪ ... ∪ B_m) ∩ R) \ N| = Σ_j |(B_j ∩ R) \ N|, the terms being disjoint. A batch ends only where the hash changes, so two values of one equality class - which hash alike and sort together - are never in two batches, and neither is any entry: the argument of the plain scan’s pieces (§29.4). A batch none of whose values has an entry adds nothing. EXPLAIN ANALYZE prints List Batches when there were any.
  • So does lion_index_count_any() (lion_count_any_batched(), 2026-09-29 review), which is that count’s SQL form and had been left out of it: it still located every value’s set at once, so its array of sets failed the same way at nine million values (“invalid memory alloc request size 1080000000”, where k = ANY (...) answered) and before that held a gigabyte at five million (1.09 GB of VmHWM against the node’s 260 MB). A list longer than lion_array_batch_size() is now sorted, byte-for-byte repeats dropped (lion_probe_sort_unique(), the node’s own), and located, counted and released a batch at a time, the batches sharing one visibility cache as the node’s do: 243 MB of VmHWM at five million values, against the node’s 263 MB on the same server. A list within the batch size is located whole, exactly as before; the element types, collations and NULLs a call accepts are the same either way, because everything the batches do is what one lookup of the whole list did, in pieces. test/sql/countany_long.sql checks it against the query’s own answer across the batch boundaries, with repeats, NULLs, other element types and citext’s equal-but-not-identical keys, and at nine million elements.
  • So does a bitmap scan of several quals (lion_emit_columns(), and lion_emit_intersect() beside a range; 2026-10-18 review), which had been left out the same way. The multicolumn scan of k = ANY ($1) AND x = 1 reserved room for every value’s set in one plain allocation, 120 bytes a set: “invalid memory alloc request size 1080000360” at nine million values, where the scan of the list on its own (lion_emit_array(), a value at a time) and the ordinary plan answer. Below that it held some 250 bytes a value outside work_mem - on the regression suite’s PostgreSQL 16 server, at work_mem 4MB, 84 MB of VmHWM for 300,000 values and 246 MB for a million, now 10 MB and 21.5 MB, in 0.71 s for the million where it took 1.0 to 1.1. The longest of the scan’s lists longer than lion_array_batch_size() is sorted, its byte-for-byte repeats dropped and cut where the hash changes, as the count does it (lion_scan_list_begin(), lion_scan_list_locate(), which the plain scan’s LIST takes its batches from too, §29.4), and each batch is located, its union ANDed with the other quals' trees and walked into the caller’s bitmap - beside a range, as a stream of its own, window by window through the range walks - and released before the next is located (lion_bitmap_list_batch()). A bitmap is a set, so the batches' answers make the list’s: (B_1 ∪ ... ∪ B_m) ∩ R = ∪_j (B_j ∩ R), disjoint or not (they are). Each batch reads the other trees again, as each of the count’s batches does; a list within the batch is located whole, as before. A second long list is held whole there, and so is a multi-key op ANY (array) - in a huge allocation now (lion_sets_reserve()), as are the sets the planner’s intersection probe and the facets' filter locate for a scan key through the same code (lion_scankey_sets()). test/sql/scanlist.sql checks the scans against the heap’s answer across the batch boundaries, with repeats, NULLs, values without an entry, two lists, a multi-key column, a range and two beside the list, on a dirty heap and past nine million elements, and the VmHWM of the 300,000 values.
  • Every other count collects it (lion_locate_list()). A GROUP BY, a count(DISTINCT) and an FK-side join count the list many times over - batching it would locate it again for every group - and a plain count’s second long list is no batch of the first’s. Such a list, a source of its own and longer than lion_array_batch_size(), is located a batch of values at a time, and each batch’s sets are ORed into ONE union and released before the next batch is located: the list is collected as a range taken as a source is (§32, “A range as a source”; lion_union_collect()), in what is left of the hash table’s memory the relation’s ranges and lists share, and past it a window of container keys at a time into a temporary file: the values are located once, into the first window, and what it does not keep goes to an overflow the windows after it read in their place (§32, “Too large to collect”). One that does not fit what the sources collected before it left sends them to files, as a range does (lion_collect_within()). A value in two batches is ORed in twice, which a union takes once. The union is one pinless set, counted beside a source that carries the §9 interlock or rechecked in the heap, as a list too wide for the open budget always was (“Bounded cursors”). So a filtered count’s run whose one source that selects rows is the list - count(*) FILTER (WHERE k = ANY ($1)) in a node that sums every entry for its run 0 (§10, “Filtered counts”), which takes no batches - is counted as a run whose source is a range taken as a source is: the sum over the driving walk’s entries, each ANDed with the union, the walk carrying the interlock (lion_run_has_plain_source()). On its own the union would have nothing pinned beside it, and every row of it would go to the heap (lion_count_one_set()): on an all-visible heap of 3 million rows, 50,000 values spread over all of it took 72 to 79 ms that way, reading every heap page, against 49 ms for the sum, which reads none. EXPLAIN ANALYZE prints Lists Collected and Lists Spilled, the lists collected and those of them that went to a file: each list once, however many partitions, runs, rescans and participants collected it again, as the range sources are counted (§32, “Walked at every count”). A parallel plan printed the most any one participant had collected before the 2026-10-23 review, which counted two lists that two workers spilled one each as one.
  • So does a list under an OR (lion_locate_or(), 2026-10-22 review). Its union with the other leaves is no disjoint one, so it cannot be taken a batch at a time; a leaf longer than lion_array_batch_size() is collected as above, in the same memory, and is one set of the OR’s like a range leaf. Like a range leaf it holds no pin, so the OR carries no interlock: the run is counted beside another source that does, or as the sum over the driving walk when it has none (lion_run_has_plain_source(), LionOrState.listcollected), and otherwise rechecked. A leaf within the batch size is located as it was.
  • The list that drives the groups is located a batch at a time as they are walked (lion_group_list_prepare(), lion_ingroup_next(), 2026-10-22 review). GROUP BY k over k = ANY ($1) counts each value’s set as a group of its own, so a union would not do; past lion_array_batch_size() the values are sorted as a lookup sorts them, repeats dropped, and cut where the hash changes, as a plain count’s are, and a batch’s sets are located when the groups of the one before it are done, its predecessor’s released - pins and all - first. The groups come in the order the lookup takes the values in, as they did when the list was located whole; a batch ends only where the hash changes, so no entry, and no group, is in two. Its sets keep the pins they are located with until the next pause, as the WHERE’s do (lion_unpin_where()). List Batches counts these batches too.

Measured on a PostgreSQL 18.6 build without assertions (2026-10-22): 3 million rows, k 0 to 499,999 over 30,928 heap pages, at work_mem 64 kB, the backend’s peak anonymous memory with the list held whole and now. GROUP BY k over k = ANY (500,000 values): 112 MB, now 20 MB, in 782 ms where it took 813. count(DISTINCT k) over the same list: 112 MB, now 20 MB, 361 ms against 640. k = ANY (250,000 values) OR x = 1: 86 MB, now 4.7 MB, 392 ms against 727. What is left is the values and the batch.

What a batched or collected list still holds per value is the values: the array, and for a batched one a Datum and a hash each - what the executor’s own = ANY holds, and core’s B-tree scan of the same array deconstructs and sorts it just the same. A list that the node locates whole is one no longer than a batch, a work_mem of sets; the planner cannot see a parameter’s length to decline a longer one (the literal cap), and at run time there is no plan left to fall back to, so none is.

Measured on the same 18.6 assert build, work_mem 4MB. What the review measured:

query before after
k = ANY (50000 values) AND x = 1, 300k rows 857 MB VmHWM, 4.2 s 48 MB, 0.22 s
the same, 300k values 5 GB, 39 s 102 MB, 0.54 s
the multicolumn bitmap scan of it, 100k values 1.7 GB 59 MB, 0.34 s
temporary table, k = ANY (1100 CHAIN entries) AND x = 1 “no empty local buffer available” 29 ms
GROUP BY x, k = ANY (3000 CHAIN entries): copies held after three groups (work_mem 1MB) 24 MB 1.6 MB
a batched count parked at its first visibility-map question: posting pages pinned 400 28 (one batch)

(the VmHWM figures include the shared buffers the query touches, about 10 MB here). And what an ordinary list costs, 1M rows, 8334 heap pages, c20k INLINE (50 rows a key), c200 CHAIN (5000): backend CPU time per execution (utime + stime over thousands of executions in one session, libraries alternated, median of four to six sessions each - wall-clock timings on the shared machine this ran on swung by half between two runs of ONE binary, so CPU time is the figure):

query before after
c20k = 77 AND c2 = 1 0.19 ms 0.20
c20k IN (3) (the sum) 0.08 0.08
c20k IN (10) AND c2 = 1 0.48 0.35
c20k IN (100) 1.23 1.17
c20k IN (1000) 9.9 8.8
c20k IN (1000) AND c2 = 1 28.1 13.1
c200 IN (3) 0.35 0.34
c200 IN (3) AND c2 = 0 0.43 0.43
c200 IN (100) (the merge) 3.0 3.1
c200 IN (100) AND c2 = 1 4.3 4.3
c20k IN (100) OR c200 = 5 1.9 1.9
c20k = ANY (5000 values) AND c2 = 1 (wall, pgbench median) 337 122

Nothing an ordinary list does got slower; what got faster did so because it no longer allocates and touches 8 kB per INLINE value. A list over the budget pays its batches - the other sources read once per pass - where it used to pay its memory.

test/sql/countbudget.sql proves the lists exact at every length across several batch and window boundaries (CHAIN and INLINE entries, NULLs and repeats, a negated source, two lists, GROUP BY on the list’s own column and another, count(DISTINCT), OR across columns, multi-key && and @>, the multicolumn bitmap scan) on an all-visible and a dirty heap; that VmHWM grows by less than 16 MB for 5000 CHAIN sets ANDed with another clause and for the bitmap walk of them, and by less than 48 MB for 30000 INLINE ones (Linux only: elsewhere /proc is missing and the checks pass vacuously); that a GROUP BY’s copies of its WHERE sets stay within work_mem; and that a temporary table in 100 local buffers answers what failed before. On the old code every one of those checks fails. test/isolation/count_batch_race.spec parks a batched count at its first visibility-map question: one batch’s pages are pinned where the whole list’s were, and a VACUUM waits for them.

Paused and finished counts (2026-09-28)

The cursors above are those of ONE count. Between two counts the node held pins as well (2026-09-27 review): the WHERE sets are located once per relation (§10) and an INLINE one keeps its leaf pinned (§9), so a GROUP BY paused in a cursor held those pins for as long as the client left it open, and a node that had returned its last row - the one row of a plain count included - held them until the transaction ended; the outer group of §20 kept its set’s pin across the rows of its inner loop. VACUUM’s cleanup lock on those leaves (§11) waited for all of it. §9 wants the pin only while a container of the set is checked against the visibility map, which no count does between two rows, so now:

  • A node that is done lets go of everything at once (lion_finish_run()): its walks, its group sets and its WHERE sets. A rescan locates them afresh, as it always did. (A partitioned scan already let go of each partition’s at the end of that partition’s turn.)
  • A row with more to come keeps the sets but not their pins (lion_pause_run()). Besides the walk’s own pause (§28, lion_entry_scan_pause()), every located WHERE set that holds a pin gives it up and becomes NOPIN (lion_posting_set_unpin()): the private copy a set located past the pin budget above has always been, which every count already knows how to take. The outer set of §20 does the same.

Why the counts after a pause stay exact and still trust the map. A NOPIN copy carries no interlock of its own; a count needs ONE positive source that holds a pin at every container key (lion_source_pinned()) and otherwise trusts no map (cx.novm). Every shape that returns more than one row locates the set that drives each count afresh, under a pin of its own: a group’s entry from the walk (§10), the inner set of a (g, k) pair (§20, §26), the fk set of a dimension row (§27). That is §9’s argument for a materialized WHERE set: every member of the intersection is in the pinned source’s container, so VACUUM cannot have finished ambulkdelete on the page it came from, nor set any of those heap pages all-visible. The one driver that was not located per count is the IN list that drives its own groups, whose groups ARE the clause’s located sets; a set of it that holds no pin is now located again for its group (lion_next_group_inlist()) - which also gives the interlock back to the sets past the pin budget, whose groups went to the heap when every other source was NOPIN too. What a copy can disagree with a fresh locate about does not matter: a TID removed since is a dead row’s, and one added since is a row the snapshot cannot see (§29.5 cases 1 to 3).

The cost is a loop over the located sets per row, with nothing left to unpin after the first, and one lookup per group of a list-driven GROUP BY. Between two rows the node holds private memory only: the located sets' copies, the walk’s batch of CHAIN copies, §20’s inner keys - and, once a GROUP BY has collected its WHERE (§10, “The WHERE sets, collected once”), that one set, which never held a pin and is counted beside each group’s set under the group’s own pin, as a NOPIN copy is. test/sql/countpause.sql checks with pg_buffercache that each shape, paused in a cursor, pins no page of any lion index and that a finished one pins none either, and that the rows after the pause are the ordinary plan’s; test/sql/pinbudget.sql’s GROUP BY under a long list, which held a pin per leaf of the list while paused, holds none. test/isolation/count_pause_vacuum.spec has VACUUM complete under a GROUP BY paused after its first group and under a plain count done after its row, and the paused cursor’s other groups come out as its snapshot saw them after VACUUM has removed TIDs its copy of the WHERE set still lists and an insert has taken their slots back.

The counts of one node share two things (2026-09-28, §27 “The per-key path, end to end”): the scratch memory they work in, reset after each, and the visibility-map page the last one left pinned, which the node keeps between rows as core’s index-only scan keeps its map page. Neither is an index page or part of the §9 interlock, and no VACUUM waits for a map page’s pin; it goes when the node is done, rescanned or ended, or moves to another partition (lion_vis_cache_release_vm()). test/sql/countpause.sql’s check - no page of a lion index pinned by a paused cursor - holds as before.

Unions and intersections unoptimized (2026-09-28)

A benchmark’s count over a wide filter - a multi-key union of dense elements beside several dense clauses - took several times what bitset unions and ANDs of its data should take. The synthetic repro of §10 (“The groups of a walk, counted together”, which describes it) has such a filter: tags && '{1,2,3,4}' AND tags2 && '{1}' AND c20 IN (3, 7, 11) AND f1 AND NOT f2 AND lvl = 1, 224 ms, over 3,896 container keys - 57 us a key for eleven containers of a few hundred to two thousand members. Its profile (perf, the backend alone):

function share
lion_container_or() 30.5%
lion_container_optimize() 24.4%
lion_container_and() 17.4%
container_make_run() 12.0%
bits_extract_runs() 5.8%

Nine tenths of it was the set algebra converting its results. Every lion_container_and(), _or() and _andnot() optimizes its result - counts its runs, picks the smallest representation, converts, copies it out - which is what a container kept on a page needs, and what an intermediate result of the merge, ANDed with the next source at once, does not. With 34 rows a heap page a dense column’s containers are runs of a block’s rows: the union of the four elements' ARRAYs came out a RUN at each fold, which the next fold filled back into a bitset image, and the AND chain’s ARRAY of an IN list was checked for runs after each of its five ANDs. And the merges that remained were three-way branches on members that interleave at random, mispredicted at most steps.

So, for a count (and a collection, §10 and §27) and nothing else:

  • The merge’s results are left unoptimized (LionCountCtx.raw; lion_container_and_raw(), _andnot_raw(), _or_raw()). An AND is an ARRAY when either side is one, a RUN when both are RUNs whose runs fit, else a BITSET; an ANDNOT an ARRAY when its left side is one; an OR an ARRAY when two ARRAYs fit one, else a BITSET - written straight into the destination, never optimized. A result may be larger than its smallest form (a BITSET of a few members), and every reader takes it: the next AND, the visibility map’s block mask, the count, the recheck’s iteration. What keeps a container optimizes it first: a collection’s copy (lion_collect_container()), so a copy takes what it always took. The streams a plain, bitmap or ordered scan reads (lion_stream_*) keep the optimizing forms, since their callers copy the containers and budget them by size.
  • A union of dense containers is a bitset image (lion_or_hot_raw()). The containers standing at a key are folded pairwise only while they are ARRAYs of LION_OR_FOLD_MEMBERS (128) members together or fewer, and fewer than LION_OR_BITSET_MIN of them; anything else is ORed into the payload of a BITSET container in place and counted once, where it used to take 32 containers before the image did. Measured: three ARRAYs of 110 members, folded in two branch-free merges, took a sixth longer than their image.
  • An AND of an ARRAY meets the other side one of three ways (lion_container.c, “how two sides of an AND meet”). A merge of two sorted sides is a chain of dependent steps, each comparison deciding where the next loads are, and costs 2.5 to 4 ns a step however it is branched; setting the larger side’s members in a stack image and testing the smaller’s against it are independent steps of about a nanosecond, after a 4 kB clear. So two ARRAYs of more than 48 members together are intersected through an image of the larger - unless the smaller has fewer than an eighth of the larger’s (plus the clear’s worth), when it is galloped through as before, the gallop’s last binary search now branch-free; an ARRAY against a RUN through an image of the runs - or, for a few members against many runs, a branch-free binary search of the runs per member; two small sides by a merge without a branch on the comparison, whose steps are the three-way branch’s. The union merge and lion_container_and_cardinality()’s merges lost their branch the same way. Every choice is made per call from the two sides' counts, and every path is memory-safe for a damaged container as the rest of the library is (“untrusted containers”).

Measured by a harness beside the container library (-O2, containers of a heap of 34 rows a block, members only at offsets 1 to 34 of each of the 64 blocks; a key’s AND timed 20,000 times):

AND merged as an image searched
ARRAY 1,035 x ARRAY 1,022 4.2 us 1.8 us
ARRAY 356 x ARRAY 353 1.8 us 0.64 us
ARRAY 15 x ARRAY 340 (galloped) 0.19 us 0.33 us
ARRAY 591 x RUN 1,740 (about 400 runs) 4.3 us 1.1 us 4.7 us
ARRAY 53 x RUN 1,724 2.1 us 0.69 us 0.53 us
ARRAY 8 x RUN 1,716 1.3 us 0.74 us 0.09 us

Measured on the repro, the node against itself before and after (PostgreSQL 18.6, medians of 18 runs, three libraries interleaved, serial; §10’s subsection has the setup):

count before after cost units a millisecond, before and after
the wide filter above 223.9 ms 33.6 ms 99, 659
tags && '{1,2,3,4}' alone 152.7 10.7 28, 404
tags @> '{1,2}' 17.7 4.4 19, 76
f1 AND NOT f2 AND lvl = 1 AND tags && '{4}' 72.4 18.0 256, 1,031
c20 IN (1, 2, 3) AND f1 30.7 9.5 356, 1,147
tags && '{1}' AND tags2 && '{1}' 21.3 8.0 299, 800
grp IN (1, 2, 3) AND c20 IN (4, 5) 11.1 3.8 427, 1,264
NOT f1 AND f2 10.6 3.8 277, 771
c20 = 3 AND lvl = 1 8.4 5.0 327, 542
grp = 7 AND tags && '{2}' 3.0 2.7 502, 559

The wide filter’s profile after: container_or_bitset() (the images of the unions and of the runs an ARRAY is tested against) 35%, array_union() (the IN list’s fold) 22% before the fold’s threshold came down to 128, array_and_image() 11%, the page copies of lion_cursor_take_page() 6%, buffer lookups 5%. A GROUP BY’s WHERE is collected with the same merge, and the collection of §10’s grouped count took about what its ungrouped count does, 18 ms where it took 72.

The cost model is not refitted. Its AND terms (LION_PROBE_COST, LION_AND_MEMBER_COST, §10, “The units”) were fitted to the optimizing merge on §31’s data sets, which are outside the tree and were not at hand. What the repro says: the counts whose time went to converting unions were priced far below core’s scans - the union of four elements at 28 units a millisecond - and are in the band now; the ANDs of dense columns, which gained most from the images, are priced at about twice the reference (1,000 to 1,260 units a millisecond, against 500). Both errors are on the side of refusing the node, and no plan the regression suite pins moved. Refitting LION_AND_MEMBER_COST (4 to 5 ns a member when it was fitted; about a nanosecond now for the dense sides) and LION_PROBE_COST on §31’s counts is the open item. lion_merge_ops() still prices a union by LION_OR_BITSET_MIN alone. (Refitted 2026-09-28, §29.11, “Unions probed”, on 77 counts of three synthetic tables of the same build, §31’s sets still not at hand. What was out of the band was the unions: an AND’s IN lists, && and ORs were charged every container of their sets and lion_merge_ops()’s members x log2(k) comparisons at 5 ns each, a median of 1,785 units a millisecond and up to 7,349, where a raw union costs a nanosecond or so a member - and once they were probed, less. They are priced as the leapfrog meets them now (lion_merge_cpu_cost_sets()), at 688 units a millisecond (462 to 1,213 for nine in ten). The AND’s own terms held: the 22 counts of sets alone ran at 773 units a millisecond, the page reads priced as I/O by the convention of §10 (“The reference”) making up most of what is above 500, and their CPU as the constants say it - so LION_PROBE_COST and LION_AND_MEMBER_COST stand. What they misprice is the AND of two dense RUN containers of hundreds of runs each (run_and_run()’s merge, 7 ns a run), which LION_MEMBER_CAP charges as a thousand members: 335 to 375 units a millisecond, on the side of taking the node, whose competitor for such a count is a sequential scan thirty times its price.)

16. Partitioned tables (v1, implemented)

At UPPERREL_GROUP_AGG the input rel may be a partitioned parent: rte->inh, relkind p, IS_PARTITIONED_REL() (part_scheme, boundinfo, nparts > 0, part_rels, not dummy), reloptkind RELOPT_BASEREL. The parent has no storage and, because get_relation_info() skips indexes for an inheritance parent, no indexlist either, so everything is resolved per leaf. The fact side of §27’s FK-side join may be one too, resolved per leaf the same way and counted a batch of keys at a time in each (§27, “A partitioned fact table”).

Planning (lion_try_count_path, lion_collect_targets) - The query-level, GROUP BY, WHERE-clause and target-list checks of §10/§14/§15 are unchanged and are made against the PARENT: the Vars, attnums and constants the node carries are the parent’s. Clause analysis no longer looks an index up as it goes; it records (attnum, operator, compared type) and the indexes are matched afterwards, once per relation. - lion_collect_targets() then walks the partition tree and produces one target - heap Oid, the index whose entries drive the scan (GROUP BY or the sum-over-all of §14), one index per WHERE clause, and the leaf’s RelOptInfo for costing - per live leaf partition. It recurses through sub-partitioned children, using the planner’s already-pruned set (part_rels[i] non-NULL and i in live_parts) and additionally skipping children the planner has proved empty (IS_DUMMY_REL), which count nothing. - Column numbers are translated for every child through root->append_rel_array[childrelid]-> translated_vars, since partitions may number their columns differently or have dropped ones. A column missing in some partition is a bail-out. - Every leaf must be a plain heap table (relkind r, and a heap by table-AM routine, §2 - partitions may each have their own table AM; a materialized view is accepted by the same code path but cannot be a partition) with a usable lion index - the rules of lion_find_roaring_index plus, per clause, strategy 1 of THAT index’s opfamily for the clause’s operator and an opfamily member and hash function for the compared type, all checked per partition because nothing stops two partitions from using different opclasses. A foreign table, or a leaf without one of the indexes, bails out - unless the leaf’s bounds imply the clause, which it then leaves out (“Clauses the partition bounds imply”, below), or the clause can be tested on the leaf’s rows instead (“Residual clauses over a partitioned table”, below). - The two §10 rules about what an index’s entries MEAN are checked per partition for the same reason: the driving index’s strategy-1 operator must be the grouping equality (SortGroupClause.eqop), and an index whose stored key will be printed - the group key in the output, or a column a = clause pins - must have the type’s own equality and a representation-preserving one (BTEQUALIMAGE_PROC under that index’s collation). One partition with a coarser or a differently spelled opclass declines the whole query; there is no per- partition fallback, because the node produces one result set. - No live partitions (everything pruned) adds no path at all: the planner’s own dummy-rel handling gives the right answer. - Partial indexes (2026-10-19, §45): each leaf has its own candidates, proved with its partition constraint beside the restrictions, and reads its own in an attempt - a partial index made on the parent is one per leaf. A restriction is left out only where every leaf’s index implies it, and each leaf’s count is held to its own index’s rows; a leaf with no candidate reads whole indexes and leaves nothing out. EXPLAIN lists the leaves' partial indexes (Partial Indexes). - One CustomPath per query - added to the grouped rel directly, or, for a GROUP BY, as the subpath of the Finalize Agg that is added instead (below). custom_private gains one member: LION_PRIV_PARTS, a list of one OidList per partition (heap Oid, driving index Oid or InvalidOid, then one index Oid per WHERE clause in clause order, then - shape 30 - one Oid per residual clause, 1 where the partition’s bounds imply it), empty for a plain table. A partitioned plan leaves the index Oids in LION_PRIV_OIDS invalid - there is no single index - and keeps the parent’s heap Oid there, which is what EXPLAIN resolves column names against. - Cost: lion_cost_count_rel() is the per-relation estimate of §10, taking one relation’s pages, allvisfrac and rows and its own indexes; the path’s cost is the sum over the leaves. numgroups comes from estimate_num_groups on the PARENT (a partition may hold rows of every group), and is used unchanged for each of them. Each partition’s dirty working set is therefore charged once, for the same reason a single table’s is (§10): one visibility cache serves the whole node execution and empties itself as the partition under it changes, so a partition’s dirty pages are fetched once per query however many of its groups come back to them (§9). A partition’s own effective_cache_size share is prorated from its own page count, so a partitioned table is not quietly treated as one huge relation. Nothing is added for a merge, because there is none, and the Finalize Agg on top is costed by core’s own create_agg_path()/cost_agg() - including its spill. The node’s rows is what it really emits: for a partitioned GROUP BY the SUM over the partitions of that partition’s own estimate_num_groups (made against the child Var, so against the child’s statistics, and capped by its row count), which is also what cost_agg() is then handed as its input row count. - A partitioned GROUP BY emits PARTIAL aggregates; core’s Finalize HashAggregate combines them (2026-09-21 follow-up review). This is what bounds the memory, and it replaces the cross-partition merge the node used to do itself.

The node used to hold every distinct group of every partition in a TupleHashTable until the last partition had been counted, and that table had no spill path: unlike HashAggregate it could not fall back to batches on disk. A plan-time budget guarded it - decline when numgroups × hashentrysize exceeds get_hash_memory_limit() - but a plan-time bound is only as good as estimate_num_groups, and a partitioned parent is never auto-analyzed. Ten estimated groups against twenty thousand actual ones (reproduced: work_mem = 64kB, correct output, no spill) built a table with no bound at all. So the merge is gone.

Instead the pushdown builds the plan shape core builds for a partial aggregation:

  • the path’s target is a partially-grouped PathTarget made by lion_make_partial_target(), which is make_partial_grouping_target() (src/backend/optimizer/plan/planner.c) applied to the grouped rel’s own target: the grouping column carried through with its sortgroupref, every other column (one a WHERE clause pins, say) carried through as a plain column, and each Aggref flat-copied and put through mark_partial_aggref(AGGSPLIT_INITIAL_SERIAL). For count(*) and the count(col) cases of §14 the transition type is int8 and nothing is serialized, so a partial row is just (group key, int8 partial count). It has to be that function’s output, node for node: setrefs.c re-derives the partial Aggrefs from the Finalize Agg’s own ones (convert_combining_aggrefs()) and matches them against the subplan’s target list with equal(), so an Aggref differing in any field - the aggsplit above all - would not be found. custom_scan_tlist is copied from that target list, which is what set_customscan_references() then resolves the plan’s INDEX_VAR references against;
  • and what goes into the grouped rel is not the CustomPath but create_agg_path(root, output_rel, &cpath->path, output_rel->reltarget, AGG_HASHED, AGGSPLIT_FINAL_DESERIAL, root->processed_groupClause, NIL, &agg_final_costs, numgroups), with agg_final_costs from get_agg_clause_costs(root, AGGSPLIT_FINAL_DESERIAL, ...). EXPLAIN therefore reads Finalize HashAggregate -> Custom Scan (LionCount).

The Finalize node combines the partial counts with int8pl and spills to disk under work_mem × hash_mem_multiplier like any HashAggregate, so a grouping that does not fit is batched rather than refused, and a wrong estimate costs performance instead of memory. The node itself keeps nothing between rows: one partition open, one entry scan, one group’s iteration state. Neither a count (no groups) nor a single table (its groups already stream) has anything above it. test/sql/partition.sql pins the shape with EXPLAIN VERBOSE ((PARTIAL count(*)) in the node’s output), shows batches > 1 / disk usage > 0 on the Finalize node at work_mem = 64kB via EXPLAIN (FORMAT JSON), and replays the stale-statistics reproduction above: 20,010 exact groups out of a plan that estimated 10. - A partitioned GROUP BY needs a hashable grouping column (SortGroupClause.hashable) for the Finalize HashAggregate, and needs the planner to consider the aggregates splittable (extra->flags & GROUPING_CAN_PARTIAL_AGG); one table has nothing above it and cares about neither. - Partitionwise aggregation is still left alone: the patype != PARTITIONWISE_AGGREGATE_NONE bail-out means that with enable_partitionwise_aggregate = on a partitioned table gets the planner’s own per-partition Aggs and not this node. Two things about that guard are worth writing down now that the node produces partials itself: - it is wider than its name suggests. grouping_planner() sets extra.patype = PARTITIONWISE_AGGREGATE_FULL whenever the GUC is on and the query has no grouping sets, before anything has asked whether the input rel is partitioned at all, so the bail-out currently turns the count pushdown off for PLAIN tables too. The GUC defaults to off, so this only bites a session that turned it on; - and the shape reason for it is gone. The node’s output is now exactly what core’s own partial aggregation produces, so letting the path compete with the partitionwise Aggs on cost would be structurally sound; a child level doing PARTITIONWISE_AGGREGATE_PARTIAL never reaches the hook anyway (create_ordinary_grouping_paths() returns before it). Lifting the guard is therefore a costing decision, not a correctness one, and wants its own calibration and tests.

Executor - Every relation the node reads is open for the life of the node, from Begin to End, as the scans under core’s Append keep theirs from ExecInitNode() to ExecEndNode(): a plain table’s heap and indexes, and every leaf partition’s (lion_open_parts() / lion_close_parts(), into LionPartState). A heap is opened with NoLock - the executor holds a lock on every range table entry of the plan, partitions included (the planner locked them when it expanded the parent, and AcquireExecutorLocks() relocks the whole flat range table for a cached plan), and cassert builds check it with CheckRelationLockedByMe - and the indexes, which are not range table entries and which nothing relocks for a cached plan, with AccessShareLock, as ExecInitIndexScan() locks its index. A parallel worker takes AccessShareLock on the heaps too. The partition set is fixed at plan time (no run-time pruning, below), so nothing is opened later than Begin; EXPLAIN without ANALYZE opens nothing (EXEC_FLAG_EXPLAIN_ONLY), as ExecInitIndexScan() opens no index then: nothing runs between its Begin and its End that could drop one. - lion_open_relation() / lion_close_relation() then make ONE of them the relation counted: a plain table once for the life of the node, a partition for the length of its own turn, whose end releases what the turn located but leaves the relations open. - Why every leaf is open for the whole run (2026-09-30). Core refuses a TRUNCATE, DROP, REINDEX, CREATE INDEX, CLUSTER or ALTER of a table or index that a query of the same session still uses - a cursor between two FETCHes, say - and it knows which ones are in use only from the relcache references the query holds (CheckTableNotInUse()). The node used to open a partition only for its turn and so held none between turns: under a cursor over a correlated count a TRUNCATE of a leaf was accepted and the cursor’s later counts dropped (30000 to 20000 inside one snapshot), and a DROP TABLE of one was accepted and the next FETCH failed with “could not open relation with OID”, where core, with the pushdown off, refused both. Now each is refused with core’s own error; test/sql/partition_cursor.sql runs them under a cursor with the pushdown on and off and gets the same output. ALTER TABLE ... DETACH PARTITION alters the parent, which neither the node nor core’s Append has open, so both accept it, and both go on counting the detached table they hold. The cost, measured with 2000 leaves of one lion index each: a reference and a resource owner entry per relation (about 0.2 MB of backend memory in all); no file opened at Begin (smgr opens a relation’s files when a page is first read, through fd.c’s pooled descriptors, and keeps them past a relation’s close anyway: 993 descriptors either way); and the same locks for a fresh plan, whose planner locked every leaf’s indexes in get_relation_info() - a cached plan now holds its 2000 index locks for the run, as an Append of index scans does, instead of one turn at a time. - Per relation the node then runs one of lion_count_relation() (the intersection of the WHERE clauses), lion_sumall_relation() (§14’s sum over every entry) or lion_next_group() (§10’s streaming group loop); all three are the single-table code, unchanged. - Without GROUP BY the partition counts are summed and one row is emitted, as §10 says: nothing comes out until the last partition has been counted, because the one row is the total. - With GROUP BY the node streams. lion_next_partial_group() keeps two pieces of state, which partition has its turn (curpart) and whether the turn has begun (partopen), and on each call: begin the partition’s turn if it has not begun and locate its WHERE clauses; ask lion_next_group() for the next group of it and return that row; and when its entry scan runs out, release its posting sets, end its turn, and move to the next one. Each group is emitted as a partial aggregate the moment it is counted, so a group with rows in three partitions comes out three times and the Finalize Agg adds them up. The NULL group is emitted per partition like any other and hash aggregation groups NULLs together. Groups whose count is 0 are still not emitted - there is no such group in that partition - and a partition where a positive clause has no entry at all (wheremissing) is skipped without an entry scan. The per-partition work uses the same pergroup context as before, reset per group, and the located WHERE payloads are released and their context reset at the end of each partition. There is no cross-partition state beyond the open relations: no hash table, no per-node group memory beyond one partition’s iteration state. - The key a target list prints for a column a clause pins to one value is remembered on the clause (LionClauseState.storedkey, copied into a small context of its own) the first time a relation’s entry has it, because the posting set is gone by the time the row comes out. With partitions that is the first partition that holds the key; any other partition’s key compares equal to it by the index’s own equality - and, since the planner only builds a value-producing node when every partition’s index has the type’s own representation-preserving equality (§10), “compares equal” there means “is the same bytes”. - ReScan throws all of it away - entry scan, posting sets, the current partition’s turn and the remembered keys - and starts again from the first partition; the relations stay open. A Finalize HashAggregate that has not spilled re-reads its own table instead of rescanning the node (ExecReScanAgg()), so the case that really exercises this is a nested loop over a spilling one, which test/sql/partition.sql has. - EXPLAIN prints Partitions: p1, p2, ... in the planner’s order. The Lion Indexes line keeps its per-clause text but drops the index name for a partitioned scan ((a), (b = 2) rather than idx_a (a), idx_b (b = 2)), because there is one index per partition and no single name to give. Group Key and the ANALYZE counters are unchanged and are summed over the partitions.

Locking and the §9 pin discipline are per partition and unchanged. A partition’s posting sets are all released before its turn ends, so no partition’s index pin outlives its turn, and a VACUUM of one partition cannot affect the count of another: the interlock argument of §9 is about one heap and its indexes, and each partition is its own.

Not supported: run-time pruning (the node has no Append and no PartitionPruneInfo, so the partition set is fixed at plan time), parallel execution of a count (flags = 0, parallel_safe = false; the FK-side join over a partitioned fact table, §27’s “A partitioned fact table”, is parallel), and partitionwise aggregation as above, for the join as for a count. Planning costs one index_open per clause per partition (lion_index_bucket_pages reads the meta page), and EXPLAIN’s Partitions line names every one of them, so both are linear in the partition count. Also not supported, and refused rather than attempted: a partition whose opclass groups rows differently from the query, and a value-producing grouping over a type whose equality does not preserve the representation (§10). A grouping too large for hash_mem is no longer among them: the Finalize HashAggregate spills.

Clauses the partition bounds imply (2026-09-28)

A count over a partitioned table needed a lion index for every WHERE clause in every partition it counted. A clause on the partition key that no lion index covers - kind = 'a' over a table LIST-partitioned by kind - then declined the query, although pruning had left only partitions every row of which satisfies it: a count such as count(*) ... WHERE kind = 'a' AND tags && '{x}' AND ts >= now() - interval '30 days', over leaves that each have a lion index on (tags, ts), got a parallel bitmap heap scan and no LionCount. The key clause was the only blocker: the stable range on ts is a clause value like any other the node evaluates once per scan (§10), and the same query without kind = 'a' was pushed down.

A leaf partition’s constraint - RelationGetPartitionQual(), which adds every ancestor’s bound to the leaf’s own, so that a sub-partition of kind = 'a' by year on ts carries both - is TRUE for every row the leaf holds: partbounds.c builds it with explicit null tests so that it never evaluates to NULL, tuple routing puts a row only where its bound says, and neither ExecPartitionCheck() nor ATTACH PARTITION lets anything else in. So a WHERE clause the constraint STRONGLY implies - core’s own predicate_implied_by(clause, constraint, false), the proof partial indexes and constraint exclusion are decided by, with the clause mapped onto the leaf’s columns through every level of partitioning (adjust_appendrel_attrs_multilevel()) - is TRUE of every row of the partition, selects nothing there, and is left out of that partition’s count:

  • Per partition, per restriction. lion_collect_targets() asks it of each leaf for each restriction - each entry of the parent’s baserestrictinfo, so that an OR is left out whole or not at all (lion_leaf_drops()). A restriction the leaf has no index for has to be implied, or the query declines as before. One it has indexes for is left out too, which saves its lookups and an AND with a set that holds every row of the partition - but not where the count would then have nothing left to select rows by (below). The plan says which: the partition’s index Oid for the clause is InvalidOid (LION_PRIV_PARTS, shape 15), and the executor leaves the clause’s source out of that partition’s items (lion_relation_items()), opens no index for it and asks it nothing. The partition is priced for the clauses it keeps (lion_target_lists()).
  • Never left out: a range that bounds the walk driving the count (§28), which walks the same entries with it or without it and is on the driver’s own index; and a pinned column whose value the target list prints, which comes out of the index’s stored key (§10’s value gate).
  • Something must still select the rows. A count that nothing but the WHERE clauses drives - no GROUP BY, no sum over a column’s entries, no fk key (§27) - needs, in every partition, a positive clause or an OR of them that carries the §9 interlock (not a range collected into memory, §32), as the planner asks of the query as a whole (pinnedsrc). A partition that would have to leave out every such clause declines the query. Since “Residual clauses over a partitioned table” (below) a clause no partition indexes is left out before the analysis when every partition implies it, as a clause no posting set answers always was, so count(*) ... WHERE kind = 'a' and ... AND fk IS NOT NULL, over partitions that do not index kind, are counted whole, by the entries of one column of each (§35, “Every row”); a partition that indexes kind keeps the clause as one its index answers, and leaves it out as before.
  • Implied by every partition. A clause no posting set answers at all - an operator no lion opclass has, an OR with an expression for an arm - declines the query unless every partition counted implies it (lion_implied_everywhere()); then it is left out of the plan altogether, and EXPLAIN is handed its text (LION_PRIV_IMPLIED). And a range every partition implies does not drive a count that another clause, not implied everywhere, selects rows for: as the driver it would walk every key of each partition, as a source it is left out.
  • NULLs and default partitions. The proof is strong implication by a constraint that is never NULL, which is exactly “every row of the partition satisfies the clause”. A partition FOR VALUES IN (NULL, 'n') implies neither kind = 'n' nor kind IS NULL, only their OR; a LIST partition without NULL implies kind IS NOT NULL; a default partition’s constraint is the negation of its siblings', which implies kind NOT IN (...) of all their values and no equality. A clause with a volatile function or a subquery is never taken as implied - the query asked for it per row - and a Param proves nothing at plan time, so a generic plan’s kind = $1 still needs an index, where a custom plan’s literal does not.
  • What it does not change. The row estimates, which are the planner’s from the parent’s clauses; the partitions counted, the planner’s pruned set (a partition only run-time pruning would remove is counted, and matches nothing); and the EXECUTE checks, made on every clause of the parent’s quals as before (§9, “Privileges”), since the ordinary plan evaluates the clause as a Filter. A plan relies on a constraint only ATTACH and DETACH change, and both invalidate every plan over the parent; the executor’s lock on each partition it counts keeps a concurrent ATTACH from changing a default partition’s bound under a running query - which is what partition pruning rests on too.

EXPLAIN prints what the partitions leave out on a line of its own, Implied by Partition Bounds, each clause followed by in N of M partitions where not all of them do; a clause every partition leaves out reads no index and is not in Lion Indexes:

Custom Scan (LionCount)
  Partitions: ip_a_2025, ip_a_2026, ip_b, ip_cd
  Lion Indexes: (kind = ANY ('{a,b,c}'::text[])), (tags && '{x1}'::text[])
  Implied by Partition Bounds: (kind = ANY ('{a,b,c}'::text[])) in 3 of 4 partitions

test/sql/partition_implied.sql checks every answer against the pushdown off, forcing the node wherever it has a path, over a table LIST-partitioned by an unindexed key with one list sub-partitioned by year, a partition with an index on the key, one that takes NULL and a default: clauses implied by every partition counted (through a parent’s bound, an IN list, IS NOT NULL, a year’s range beside another clause and alone, an OR with an unanswerable arm, the default partition’s NOT IN), by some and not others (a list one partition answers from its own index, a range taken as a source, an OR across columns), and not at all (the NULL partition’s equality and null test, the default’s equality, a range on the key that keeps the default partition - residual there since 2026-10-26); counts, GROUP BYs of one and two columns; partitions counted whole where nothing is left to select rows by; the declines (a printed pinned column, a volatile arm, a generic plan’s parameter against a custom plan’s literal); and a dirty heap before and after VACUUM. Since “OR arms the partition bounds refute” it also checks a filter per kind ORed, counted and grouped by one and two columns, with a range beside it and without, before and after VACUUM; an arm a sub-partition’s bound refutes, whose range then no longer keeps the partition from carrying the pin; the two-value bound, which refutes one arm and answers the others from its own index; the default and the NULL partition declining the arm they do not decide; with plan-time pruning off, the partitions whose bounds refute every arm, which are not counted; and a generic plan of an OR whose arm is an IN list over a stable expression, run under values other than the one it was planned under, where no partition refutes or implies the arm.

OR arms the partition bounds refute (2026-09-28)

A filter per kind is often written as one OR, an arm per kind:

WHERE ((kind = 'a' AND tags && '{x}' AND tags && '{y}') OR (kind = 'b' AND tags && '{z}')
       OR kind = 'c' OR kind = 'd')
  AND ts >= now() - interval '...'

over a table LIST-partitioned by kind. No partition’s bounds imply the OR - kind IN ('c', 'd') does, but kind = 'a' does not - and the leaves kind = ... have no lion index, so every partition but that one had to decline the query, and nothing was pushed down. But in the partition of ‘a’ the arms on ‘b’, ‘c’ and ’d' are never true, and in the arm that is left kind = 'a' always is: the partition’s filter is tags && '{x}' AND tags && '{y}', which the posting sets answer.

So after the implied restrictions, lion_leaf_drops() takes each OR restriction a partition’s bounds do not imply as a whole, arm by arm:

  • An arm the bounds refute is left out. The partition constraint is TRUE for every row the partition holds, so an arm it refutes in core’s weak sense - predicate_refuted_by(arm, constraint, true): the constraint TRUE, the arm FALSE or NULL - is not TRUE for any of them, and adds no row to the union. That is left out whether or not the partition has indexes for it. A volatile function or a subquery keeps the arm, as it keeps a clause from being implied. The proof is made from each leaf as the query has it (LionClauseInfo.clause), never from the clause the cost model is handed: an IN list over a stable expression, k = ANY (string_to_array(current_setting('...'), ',')), is priced over the array the expression has while the plan is made, and a cached plan runs under every value it takes later: refuted or implied on the strength of the estimate, the arm would be left out of a generic plan’s partitions whatever the setting then is. Only an exact rewrite (a bare boolean column’s col = true, costexact) stands in for the clause.
  • In an arm it keeps, a leaf the bounds imply is left out of the arm’s AND - TRUE of every row
    • except a range, whose bounds are one source together (§32) and are kept. An arm all of whose leaves are implied is TRUE of every row, and so is the OR, which is then left out whole as an implied restriction.
  • What is left needs an index, leaf by leaf; an arm the bounds decide neither way whose leaf has none declines the query, as before: the default partition does not refute kind = 'e', a partition FOR VALUES IN (NULL, 'n') does not decide kind IS NULL.
  • An OR every arm of which is refuted is never TRUE in the partition, which then has nothing to count and is not counted at all. Plan-time pruning uses the same arms, so this is met when pruning is off (enable_partition_pruning).

Nothing about the pin changes: an OR carries the §9 interlock when no leaf of it is a range collected into memory, and a range in an arm the partition refutes is no leaf of its union there.

The plan says it as it says an implied clause: the partition’s index Oid for each leaf left out is InvalidOid (LION_PRIV_PARTS). The executor tells the three cases apart by what is left of each arm (lion_locate_or(), lion_relation_items()): an OR with no leaf left is left out whole, an arm with none is refuted and out of the union, and a leaf missing from an arm that keeps others is implied and out of its AND - the planner never leaves an arm all of whose leaves are implied. The partition’s cost is for the OR it keeps (lion_target_lists()), and the FK-side join’s fact filters (§27) are reduced the same way, since lion_collect_targets() serves it too. EXPLAIN lists the refuted arms on a line of their own and the implied leaves with the implied clauses:

Custom Scan (LionCount)
  Partitions: ip_a_2025, ip_a_2026, ip_b, ip_cd
  Lion Indexes: (ts >= ...), (((kind = 'a') AND (tags && '{x1}') AND (tags && '{y2}')) OR
                ((kind = 'b') AND (tags && '{x3}')) OR (kind = 'c') OR (kind = 'd'))
  Implied by Partition Bounds: ((...) OR (kind = 'c') OR (kind = 'd')) in 1 of 4 partitions,
                (kind = 'a') in 2 of 4 partitions, (kind = 'b') in 1 of 4 partitions
  Refuted by Partition Bounds: ((kind = 'a') AND ...) in 1 of 4 partitions,
                ((kind = 'b') AND ...) in 2 of 4 partitions, (kind = 'c') in 3 of 4
                partitions, (kind = 'd') in 3 of 4 partitions

GROUP BY the partition key. The FK-side join takes the value from the bound (§27, “Grouped by a fact column”): in a partition whose bounds give kind one value every row is one group, and its count the partition’s count, once §10’s value gate is asked of the grouping equality itself - it has to be the type’s and preserve the representation (numeric 1.0 and 1.00 are one partition value; a nondeterministic collation groups different strings), which an opclass says for an index but the partition key has to be asked for (lion_leaf_bound_value()). A count of one table, SELECT kind, count(*) ... GROUP BY kind, still needs a lion index on kind in every partition: it would need an executor mode that counts one partition as one group, and a partition count with nothing left to select its rows by is one the node has never made.

Residual clauses over a partitioned table (2026-10-26)

A count over a partitioned table declined every WHERE clause no index of its leaves answers, unless the leaves' bounds implied it: count(*) ... WHERE country = 'US' AND ts >= '2026-10-19' AND ts < '2026-10-26' over monthly partitions, or ... AND payload LIKE 'ab%', got the ordinary plan from the parent where the same count named one of the leaves and tested the clause on the heap row of each row the other clauses select (§10, “Residual clauses”). Now a partitioned table takes a residual clause as one table does, each leaf testing it on its own rows - but for the leaves whose bounds decide it.

  • Which restrictions. The decision is still the WHERE analysis’s, restriction by restriction, and lion_residual_ok() takes a partitioned table’s as it takes a table’s. A clause of no lion shape - payload LIKE 'ab%', n % 2 = 0 - is residual unless every leaf’s bounds imply it, which is asked first (lion_implied_everywhere(), as before), so that a clause no row can fail is left out of the plan rather than tested. A clause of the right shape on a column no leaf indexes - ts >= '2026-10-19', id > 10 - is found out as a table’s is, right after its analysis (lion_where_indexed()): the parent has no indexes of its own, and its live leaves' stand for them, any one’s (lion_rel_has_index(), through every level, the column translated for each child). Asked then rather than when the partitions are matched, the clause can be residual before a range on an unindexed column is made the walk’s driver, whose missing index would decline the query outright. An OR is left as answered: its arms may be what the bounds refute or imply, which narrows it to clauses the indexes answer (“OR arms the partition bounds refute”, above), and testing the whole OR on the rows would throw that away. What the leaves' indexes do not settle - a column some leaf indexes and another does not, which that leaf’s own index answers where the others' bounds imply the clause, or an OR no bounds narrow in some leaf - the partitions settle: where a leaf can neither answer a restriction from an index nor leave it out by its bounds, lion_leaf_drops() no longer declines the query but asks for the restriction to be residual (LionImply.wantresid, among the restrictions lion_residual_ok() took, residable), the leaf declines, the other leaves are asked the same (lion_collect_targets() passes such a leaf over, and stops at a leaf that declines for anything else, hard), and the build is made once more from the start with those restrictions residual (lion_try_count_path_scoped(), forceresid): once, because the second attempt’s partitions answer or leave out every clause left, as the first established. Every restriction’s number is the same in both attempts.
  • Never on a partitioning key with a Param or a stable function. ts >= $1, ts >= now() - interval '7 days': that is a clause run-time pruning decides, and the node has none - it counts the partitions the planner kept (“What it does not change”, above). Counted with the clause residual, every partition pruning would have skipped fetches the rows the other clauses select there, only to reject them. So lion_residual_ok() refuses it over a partitioned table, and the query keeps the ordinary plan, whose Append prunes at run time; the same clause on any other column (payload LIKE $1, created_by = current_user) is residual, tested everywhere. The keys are the columns of every partitioning expression at every level the planner keeps, in the parent’s numbering (lion_partition_key_attnos()), a Var of a sub-partition’s key mapped back up through each level’s translated_vars (lion_parent_attno()).
  • Per leaf, implied or refuted. Each target says which residual clauses its bounds imply (LionCountTarget.residdrop, by lion_leaf_implies(), the proof of “Clauses the partition bounds imply”): every row of the leaf passes them, and it tests none of its rows for them - fetches no row at all where it keeps none, so the month in the middle of ts >= ... AND ts < ... is counted from its indexes alone, and only the months at the ends read their heaps. A leaf whose bounds refute a residual clause (lion_leaf_refutes(), weakly: false or NULL of every row) holds no row that passes it and is left out, as a leaf whose bounds refute every arm of an OR is - which shows with enable_partition_pruning = off, or for a clause pruning does not read. The plan carries the flags per partition after its index Oids (LION_PRIV_PARTS, shape 30; LionCountPrivPart.residdrop), as many as the residual clauses, which the executor counts against custom_exprs (lion_begin_partitions()).
  • The price, and the share. The rows a leaf tests are its own estimate (lion_residual_rows(root, toprel, rel, ...)): the parent’s other restrictions mapped onto the leaf’s columns and estimated by its statistics, as a range source’s are (lion_rel_clauses_selectivity()), and its own pages at its own tablespace’s costs (lion_cost_residual()), summed over the leaves that test the clause. The half-the-table rule (LION_RESID_MAX_SHARE) is asked of those leaves together - the rows they test against the rows they hold - rather than of each: a sub-partition of the one kind the WHERE selects fetches every row of its own, and would decline a query whose other partitions test a fifth of theirs, where the ordinary plan is the whole table’s and the question is of the whole. A query whose only testing partition is such a sub-partition keeps the ordinary plan.
  • The executor. The residual qual is initialised at Begin over every clause, as before (lion_begin_residual(), the EXECUTE checks included), and kept as residall. When a partition is made the relation counted (lion_open_relation()), its own qual takes its place (lion_part_residual()): the node’s where it leaves none out, NULL where it leaves all out - its counts then read the visibility map as a count without a residual does - and otherwise a qual over the clauses it keeps, made at its first turn with ExecInitQual() in the query’s context and kept for the rescans (LionPartState.resid). The columns the clauses read are looked up in the partition by name then (lion_heap_attno_in(), residheapattno), since a partition may number them differently from the parent, whose numbering the plan’s columns are in.

  • EXPLAIN prints the clauses under Residual Filter, as for a table, and the ones some partitions leave out under Implied by Partition Bounds, each followed by in N of M partitions, as it prints the indexed clauses they leave out:

    Custom Scan (LionCount)
  Partitions: ev_m1, ev_m2, ev_m3
  Lion Indexes: (country = 'US'::text)
  Residual Filter: ((day >= '2026-01-10'::date) AND (day < '2026-03-20'::date))
  Implied by Partition Bounds: (day >= '2026-01-10'::date) in 2 of 3 partitions,
                               (day < '2026-03-20'::date) in 2 of 3 partitions

test/sql/partition_residual.sql checks every answer against the pushdown off, forcing the node wherever it has a path, over a table RANGE-partitioned by day with one month sub-partitioned by LIST on kind and the first month attached with two columns dropped, so that it numbers the columns the clauses read differently: clauses on columns no index covers and expressions, tested in every partition; a range on the key implied by the partitions in the middle and tested at the ends, through a sub-partition’s parent, implied by every partition kept (no residual), and beside a clause on the key no bound decides; with pruning off, the partition whose bounds refute the clause left out; a GROUP BY of one and of two columns, HAVING, a top k, count(col), a filtered count, an IN list, a <>, an OR with a residual arm, a division by a column that is 0 in dead versions only, and a SELECT DISTINCT; a generic plan’s parameter on the key against one off it, a stable expression on the key against one off it, a volatile function, a subplan, a residual clause alone, more than half the rows to fetch of every partition and of the one sub-partition left to test the clause, and the switch; a dirty heap before and after VACUUM; and a LATERAL value in the clause, which rescans the node partition by partition.

A parent pruned to one leaf (2026-10-26)

An application names the partitioned table, and plan-time pruning leaves it one leaf: a statement names one value of each partition key of a table partitioned into many leaves. The nodes that scan one relation got the leaf: set_rel_pathlist_hook runs for each partition (RELOPT_OTHER_MEMBER_REL), LionOrdered and LionBtreeScan plan on it (§30.11, §40.10), and core’s Append of that one child costs nothing (create_append_path() takes the child’s cost and pathkeys) and is elided (set_append_references()), so the plan is the leaf’s. The join nodes did not: set_join_pathlist_hook is called with the parent - reloptkind RELOPT_BASEREL, rte->inh, no indexlist of its own, its paths an Append - and oj_side_ok() refused it, so a query naming the parent got core’s plan where the same query naming the leaf got LionOrderedSplit over LionOrderedJoin, and ran many times slower.

The gate is one, shared, in lion_plan_partition.c, so that a node supports a partitioned table by asking it rather than by a rule of its own:

  • lion_single_leaf(root, rel): the one relation a path over rel reads. rel itself for a table, a materialized view or a leaf reached through its parent; for a partitioned parent the one live, non-dummy leaf through every level of sub-partitioning; NULL for a parent several leaves are left of or none, an old-style inheritance parent (whose children need not share its columns), or anything else. A node offered a relation asks this of it and plans its physical side
    • indexes, statistics, scan - on the leaf.
  • lion_leaf_expr(root, expr, leaf, top): an expression written over the parent’s columns, as the leaf numbers them (adjust_appendrel_attrs_multilevel()): the join’s key and the descent’s qual, which the leaf’s scan evaluates.
  • lion_top_var(root, var), lion_top_expr(root, expr): a leaf’s Var as the topmost parent numbers it (lion_top_rel(), lion_parent_attno()): what the node outputs. The query’s target and the nodes above refer to the parent’s Vars, and set_customscan_references() resolves a custom_scan_tlist by varno and varattno. A leaf attached with columns dropped numbers them differently from the parent, which is what the translation is for. A whole-row reference to the parent (SELECT f, row_to_json(f)) is a ConvertRowtypeExpr over the leaf’s whole row in the leaf’s target (adjust_appendrel_attrs()), which the leaf’s scan evaluates to a row of the parent’s type; lion_top_expr() writes it as the parent’s whole-row Var. Left as the conversion, setrefs.c finds no entry for the parent’s Var and fails the plan.
  • lion_single_leaf_path(path): the one-child Append seen through to the leaf’s path, for what inspects a plan’s side (the walk a loop is over, oj_loop_over_walk()).

LionOrderedJoin and LionOrderedSplit (§47, §48) take either side so. oj_side_ok() asks for the leaf and tests it (lion_ordered_rel_ok(), which accepts a member rel; the leaf’s restrictions); oj_key_join() plans on the leaves - OjJoin.fact and dim are the leaves, the unique B-tree the dimension leaf’s own, fkvar, pkvar and the descent’s qual in the leaves' numbering - and keeps the relations the query names as facttop and dimtop, for what the join outputs (oj_plan_path() writes the custom_scan_tlist through lion_top_var()), for the competitor’s walk (oj_competitor()) and for the split’s test that its first sort key is the dimension’s (os_split_of()). The executor is unchanged: it composes its slot from the child scans' by position, and the leaf’s scan outputs its reltarget, the parent’s translated, in the parent’s order. Either side may be such a parent, or both, or a key set’s table (§47.6: a key set is a scan of core’s path, the Append included). Declined: a parent several leaves are left of - core’s Append or MergeAppend over them, which the node would have to merge in order - or none, which is a dummy rel and core’s empty answer. A clause on the partition key with a Param or a stable function is a filter of the leaf’s scan as any other: run-time pruning could only remove the one leaf, and the clause tested on its rows gives the same answer.

What LionCount does with a partitioned table (above) is a different thing: it counts every leaf the planner keeps, as one node. Its aggregates of sliced columns (§52) are each leaf’s own, added up in the node (§52.13). The join and the walks need the planner to keep one leaf; a parent with several leaves left gets core’s Append for them.

test/sql/ordjoin_partition.sql runs the join and the split through both parents - the fact LIST by grp, each sub-partitioned by LIST on ver, the leaf the queries hit attached with two dropped columns; the dimension LIST by kind, each leaf with its own unique B-tree on the key - with a key set from another reference to the dimension’s parent, a page further in, the whole join, every column of the fact, either parent’s whole row; declines a parent several leaves are left of on either side and one pruned to nothing; plans the same node with the leaves named; and gets LionOrdered on the fact’s parent and LionBtreeScan as a nested loop’s probe of the dimension’s, every answer compared with the ordinary plan’s row by row.

17. Multi-key operator classes: arrays and tsvector (v1, implemented)

GIN-style extraction so one row can contribute many keys: array_ops (DEFAULT FOR TYPE anyarray, STORAGE anyelement) and tsvector_ops (DEFAULT FOR TYPE tsvector, STORAGE text). The entry key type is the element type / the lexeme; everything below the key - buckets, entries, containers, sparse segments, VACUUM, the visibility-map interlock - is unchanged.

Handler and opclass shape

amsupport = 3     amstrategies = 5     amstorage = true

proc 1   hash of the KEY type                 (absent when the key type is polymorphic)
proc 2   GIN extractValue(value, &nkeys, &nullFlags) -> Datum *keys
proc 3   GIN extractQuery(query, &nkeys, strategy, &pmatch, &extra_data, &nullFlags,
                          &searchMode) -> Datum *keys

strategies   1 =      2 @>      3 &&      4 <@      5 @@      17 array =      18 @@@

The extraction functions are GIN’s, named directly in the install script:

CREATE OPERATOR CLASS array_ops DEFAULT FOR TYPE anyarray USING lion AS
    OPERATOR 2 @> (anyarray, anyarray),
    OPERATOR 3 && (anyarray, anyarray),
    OPERATOR 4 <@ (anyarray, anyarray),
    OPERATOR 17 = (anyarray, anyarray),
    FUNCTION 2 ginarrayextract(anyarray, internal, internal),
    FUNCTION 3 ginqueryarrayextract(anyarray, internal, int2, internal, internal, internal, internal),
    STORAGE  anyelement;

CREATE OPERATOR CLASS tsvector_ops DEFAULT FOR TYPE tsvector USING lion AS
    OPERATOR 5 @@ (tsvector, tsquery),
    OPERATOR 18 @@@ (tsvector, tsquery),
    FUNCTION 1 hashtext(text),
    FUNCTION 2 lion_tsvector_extract_value(tsvector, internal, internal),
    FUNCTION 3 lion_tsvector_extract_query(tsvector, internal, int2, internal, internal, internal, internal),
    STORAGE  text;

lion_tsvector_extract_value() and lion_tsvector_extract_query() call gin_extract_tsvector() and gin_extract_tsquery() and hash the lexemes too long to be keys (“Long lexemes” below); in everything said here about GIN’s functions they stand for them.

GIN’s consistent function is deliberately NOT reused: it answers “given which keys matched, does this one row match”, and a roaring scan never asks that - it combines whole posting sets. The query side therefore builds a boolean tree of its own (below).

Two consequences of borrowing GIN’s functions, both binding on any future multi-key opclass:

  • gin_extract_tsquery is declared (tsvector, internal, int2, ...) in pg_proc even though its first argument is really a tsquery; that is how GIN declares it (amproclefttype = opcintype), and the opclass above copies it verbatim.
  • the strategy number handed to proc 3 is GIN’s, not ours. GIN numbers the array operators 1 &&, 2 @>, 3 <@, 4 =, and tsvector’s @@ is its strategy 1. lion_gin_strategy() in lion_multikey.c is the single place that translation lives. Getting it wrong is silent - the extraction answers for the wrong operator and returns INCLUDE_EMPTY, which only costs a rechecking full scan - so it is one function with one switch.

Array = (strategy 17) and @@@ (18) are there because GIN’s classes have them, so a query that uses them can use the index. a = q holds only where a has every element of q, so it is the AND of those elements' posting sets, as for @>, and the bitmap heap scan’s recheck decides the rest - how often each element occurs, their order, the dimensions and lower bounds; a NULL element of q is left out of the AND, and an empty q is every row (ginqueryarrayextract()’s INCLUDE_EMPTY). It is never exact, so lion_extract_query() gives it no tree and only the superset answers it. @@@ is @@ under its deprecated name: LION_STRAT_CANONICAL() turns 18 into 5 at rescan and wherever a strategy reaches the extraction. Neither is in LION_STRAT_IS_MULTI(), so the count pushdown, facets and the planner’s probe decline them and the AM’s bitmap and index scans answer them, as GIN’s do.

lionvalidate has two shapes. A scalar opclass is checked exactly as before (proc 1 with signature (T) -> int4, strategy 1 only, every operator’s types hashable by the family). An opclass is multi-key iff the family has proc 2 for (opcintype, opcintype); then procs 2 and 3 must have GIN’s signatures, every operator’s strategy must be a multi-key one (2 to 5, jsonb’s 11 to 16, 17 or 18), and proc 1 is required unless opckeytype is polymorphic. The “every type an operator mentions must be hashable” rule does not apply to a multi-key family, whose operators mention anyarray and tsquery and whose hashing is of keys. amconsistentequality stays true: a scalar roaring family holds nothing but equality operators and a multi-key one at most one, array_ops' =, so equality_ops_are_compatible() is never asked about two members of the same roaring family that disagree.

Key type resolution (lion_fill_state, lion_state.c)

The index’s own tuple descriptor already carries the resolved key type: ConstructTupleDescriptor() substitutes opckeytype for the column type and replaces ANYELEMENT under an ANYARRAY opcintype with get_base_element_type() of the column. So LionState.typid is still TupleDescAttr(...)->atttypid and a roaring array_ops index on text[] has a text key column. (Deviation from the sketch this section started as, which said “opckeytype when set, else opcintype”: taking it from the tuple descriptor is the same answer for every class, needs no polymorphism handling of its own, and keeps enum_ops - whose opcintype is the pseudo-type anyenum - working as it did.)

  • Collation: the index column’s (rd_indcollation[0]), falling back to C when that is invalid and the key type is collatable. A tsvector is not collatable, so a tsvector_ops index has no collation to offer, and hashtext() refuses to run without one; C is also the right answer, because lexemes are byte strings and GIN’s own gin_cmp_tslexeme() compares them bytewise.
  • Hash: support proc 1 when the opclass has one, else the key type’s default hash opclass through lookup_type_cache(TYPECACHE_HASH_PROC_FINFO) - which is how GIN’s initGinState() resolves its comparison function for a polymorphic key type. A key type with a default btree opclass and no hash one - money, bit, varbit, tsvector, tsquery, all of which GIN’s array_ops takes - is hashed by lion_image_hash() (LionState.imagehash): the hash of the stored image (hash_bytes() over what datum_image_eq() compares) where the btree class’s equalimage proc says equal values are stored alike (money, bit, varbit), and 0 for every key where it does not (tsvector, tsquery). A hash only has to agree with equality: no entry stores one, and such a column is ORDERED by the btree comparison (rule 2 of §21 holds: its equality is the btree class’s), which the hash only breaks ties of, so a constant costs nothing there; the dedup of a row’s keys and the facet table fall back on the comparison and the equality the same way. A column that should come out unordered all the same is an ERROR, and so is a key type with neither class, as GIN needs a btree one.
  • Equality: strategy 1 of the opfamily for a scalar opclass; for a multi-key one the key type’s default equality (TYPECACHE_EQ_OPR_FINFO), because the family’s strategies are about the indexed value and not about two keys. The two always agree: a type’s default hash opclass hashes what its default btree equality calls equal.

The reserved EMPTY entry

A second key-less entry joins the NULL entry of §14: flag LION_ENTRY_EMPTYKEY, hash 0, keylen 0, bucket 0, one per key column. It holds the rows a multi-key opclass extracted NO key from - an empty array, a tsvector with no lexemes - which are under no key at all and which an ALL-mode scan (tags @> '{}') still has to find. A NULL column value goes to the NULL entry as before; the two are mutually exclusive and verify() says so. lion_find_entry_ext() skips both, and lion_find_reserved_entry() (with lion_find_null_entry() as a wrapper) is the only way to either. Like the NULL entry it keeps its flag through a spill, whether the spill happens on insert or inside VACUUM (§4, §14); test/sql/array.sql and test/sql/tsvector.sql take both transitions.

An array with no element has no dimension as PostgreSQL makes one nearly everywhere - '{}', construct_empty_array(), and array_in(), array_recv(), construct_md_array() and array_map(), which all turn an array of no element into that one - but not an empty int2vector or oidvector: those have one dimension of length 0, [0:-1], which their casts to int2[] and oid[] (binary coercions) keep, as array_cat() of two of them does, and ARRAY[] or array_agg() of such arrays has more, [1:2][0:-1]. Neither is equal to '{}' - array_eq() compares the dimensions before any element - and array_length(), array_ndims(), array_lower() and array_upper() of either are not NULL: array_length(a, 1) is 0 for the first and 2 for the second, where it is NULL for '{}'. An array column’s values with no element are therefore three classes, and each has an entry of its own, key-less like EMPTY and one per key column: EMPTY the arrays with no dimension, EMPTYDIM (LION_ENTRY_EMPTYDIM) those whose first dimension has length 0, and EMPTYINNER (LION_ENTRY_EMPTYINNER) those whose first dimension has a length and an inner one none. lion_array_empty_flag() names the entry from the dimensions alone. Only a column whose extraction is an array’s elements (lion_extracts_array_elements(): lion’s array_ops, or a class of the user’s that extracts with ginarrayextract() itself) ever files a value in EMPTYDIM or EMPTYINNER, and verify() refuses either anywhere else; a tsvector with no lexemes or '{}'::jsonb is EMPTY’s. test/sql/nonempty_dims.sql builds all three.

A further key-less entry, NULLELEMS (flag LION_ENTRY_NULLELEMS, hash 0, keylen 0), holds the rows whose extraction returned keys that are ALL NULL - '{NULL}', '{NULL,NULL}', '{{NULL},{NULL}}' - which are under no key either, since NULL keys are dropped (“Build and insert” below). Keeping them out of the empty entries makes those exactly the values with no element at all, which is what lets the non-emptiness predicates of “Non-empty arrays” below be answered from the entries: a row is in exactly one of NULL (the value is NULL), EMPTY, EMPTYDIM or EMPTYINNER (it has no element, by its dimensions), NULLELEMS, or the value entries of its non-NULL elements. Only ginarrayextract() returns NULL keys among the opclasses lion ships; a tsvector, a jsonb value and a range never reach the entry. lion_extract_value() reports which reserved entry a key-less value belongs to, and the build, insert, VACUUM and verify(heapallindexed) all take it from there. The five sort NULL < EMPTY < EMPTYDIM < EMPTYINNER < NULLELEMS < value. They are part of format version 14 (LION_VERSION_V14); an index of another version is refused with the REINDEX hint, as after every format change.

lion_index_stats() reports their members in null_tids, empty_tids, emptydim_tids, emptyinner_tids and nullelem_tids.

Build and insert

Build (lion_build.c): the tuplesort tuple grows a fourth column, kind int2 (REAL / NULL / EMPTY), still sorted by (hash, code). A multi-key row is pushed once per distinct extracted key, all with the same code; both passes group by (kind, key) instead of (isnull, key). Nothing else changes, because the codes of each key still arrive ascending. (Since §24 “Build” the row’s code is appended to each extracted key’s entry, and to the reserved entry its value names when there is none; the sort, and kind with it, is gone.)

Insert (lion_insert.c): lioninsert() extracts and then performs one ordinary single-key insert per key, each taking and releasing its own bucket lock. They are not atomic with respect to a reader, which is exactly the visibility the heap already gives: the inserting transaction has not committed, so no snapshot that can see the row can run before the last of them is written.

Extraction (lion_extract_value(), lion_multikey.c) drops NULL keys and duplicates. A row whose array holds a NULL element is indexed under its other elements, and one with no other element in the NULLELEMS entry; nothing ever looks for a NULL key, because a query with one falls back to a rechecking scan. Duplicates must go, or lion_container_add() would report “already indexed” and leave ntids wrong. The dedupe hashes every key once and sorts the keys by (hash, the key type’s btree order when it has one, §21), so the only candidates for equality are adjacent, and each key is compared with the equality proc against the distinct keys of its own run of neighbours that tie on both - one of them. A row with n keys therefore costs n hashes, one sort and about 2n comparisons; the keys come out in that order, which nothing depends on. Sorting by the hash ALONE made a run every key of one hash, and n distinct keys chosen to share one (int8 (i << 32) | i, which hashint8() folds to one word) cost n²/2 equality calls: 80000 such elements took 17 s per INSERT. A key type with no ordering (xid, cid) still has only the hash, so there a run is quadratic in the keys that genuinely collide, which the built-in hash functions keep small.

ntids therefore counts (key, row) pairs, and so does IndexBuildResult.index_tuples. A row with no keys contributes one pair, in one of the empty entries or the NULLELEMS entry. VACUUM is unchanged: its callback is per TID and a TID is removed from every entry that holds it.

verify(heapallindexed) extracts each heap row’s keys with the same function the build uses and checks that the TID is present under every one of them, or in the reserved entry its value belongs to (EMPTY, EMPTYDIM, EMPTYINNER or NULLELEMS) when there are none.

Queries (lion_extract_query(), lion_multikey.c)

searchMode starts at GIN_SEARCH_MODE_DEFAULT and an out-of-range answer is treated as GIN_SEARCH_MODE_ALL, as ginNewScanKey() does. The result is one of three modes:

NONE   nothing matches            DEFAULT mode with zero keys: `tags && '{}'`, an empty tsquery
KEYS   a boolean tree over the keys, exact
ALL    every indexed row, with recheck

ALL is the answer for: any searchMode but DEFAULT (INCLUDE_EMPTY and ALL, which is what ginqueryarrayextract returns for @> '{}' and for <@); any partial-match key (a prefix lexeme - the keys are hashed, so a range of them cannot be walked); any NULL key; more than LION_MAX_QUERY_KEYS = 1000 keys; and a tsquery shape the tree builder rejects.

KEYS trees:

  • @> an AND over every key, && an OR over every key.
  • @@: the TSQuery’s QueryItem array is walked directly. It is in prefix order - an operator’s RIGHT operand is at item + 1 and its LEFT at item + qoperator.left (ts_type.h) - and gin_extract_tsquery() numbers the keys it returns by scanning that array from 0 and counting QI_VAL items, so the j'th key is the j'th QI_VAL in array order; the same item -> key map is recomputed rather than fished out of extra_data. QI_VAL with no prefix flag and weight mask 0 becomes a leaf; OP_AND and OP_OR become AND and OR nodes; OP_NOT, OP_PHRASE, a prefix and a weight mask make the whole query ALL. (!a has no complement over posting sets that is not “every row minus a’s”, and the ALL fallback is that set anyway; a phrase needs lexeme positions and a weight needs weights, neither of which the index stores.)
  • <@ is unreachable (extractQuery says INCLUDE_EMPTY first) and would be ALL.

Bitmap scans (lion_scan.c)

liongetbitmap dispatches on LionState.multikey. NONE emits nothing; ALL emits the union of every entry but the NULL one - the key-less ones included - with recheck = true (lion_emit_all_keys(), which is the function IS NOT NULL already used); KEYS locates one posting set per key and hands the tree to the shared evaluator, emitting the result containers with recheck = false.

col IS NOT NULL on a multi-key column is NOT a query of the opclass: an empty array or tsvector, whose row is in a key-less entry, is not NULL, so the rows it selects are every row the index holds but the NULL entry’s, as on a scalar column (§14, “IS NOT NULL in the scans”, 2026-10-19): the NULL entry taken away from the other quals' answer, or alone the union of every other entry, the key-less ones included - exact either way.

The scan reuses the count cursors. lion_sets_iterate() (lion_expr.c) walks (tree over located posting sets) in ascending container-key order and calls back per container; the scan’s callback is lion_container_to_tbm(). The alternative - one TIDBitmap per key combined with tbm_intersect/tbm_union - was not needed: the cursors already present a set as one ascending run of containers, the AND and OR nodes are twenty lines each, and using the same evaluator for the scan and the count means the two cannot drift apart.

All the posting sets are located BEFORE any of them is walked, so every bucket-page lock the scan takes is taken before the first container page is pinned: the reader side of the §11 deadlock rule. col op ANY (const array) reaches a multi-key index too (amsearcharray is on for §15’s sake); each element is a query of its own and the answers go into the same bitmap, which is their union.

Costing the ALL fallback (lioncostestimate, lion_amcost.c; the 2026-09-21 follow-up review). An ALL-mode scan reads the WHOLE index and hands the heap every indexed row, and the generic estimate priced it with the PREDICATE’s output selectivity instead - it was handed an empty GenericCosts and told nothing. A prefix query on 200,000 documents was estimated at 192.15 cost units against the sequential scan’s 9,156.14, for a plan that emitted 1,176,746 posting TIDs and rechecked 198,020 rows across 6,628 heap blocks, and it was chosen by default: 62.3 ms against 23.8 ms, and the phrase form 68.6 against 31.8. So lioncostestimate now asks, before calling genericcostestimate(), whether the scan will walk the whole index: - it classifies the quals as liongetbitmap() does (§5 SCAN step 5), because the cost of the scan is the cost of what it answers: a column is read whole only when NONE of the set quals it answers selects from its posting sets - one that does is enough, the others are dropped and rechecked - or when it is walked for IS NOT NULL, and the first column decides when no column selects (the ranking of ONE qual per column, a plain operator first, went with the scan that answered one); - col IS NOT NULL with nothing else to answer walks every entry (§14; beside quals that answer it is the NULL entry taken away from their answer), and so does a multi-key query the extraction above answers with LION_QMODE_ALL - which it is asked at plan time, with the clause’s own strategy, the same way the count pushdown asks; - a multi-key query whose value is not a plan-time Const is costed as ALL, because the MODE follows the query’s shape and an unknown shape has to be assumed to be the expensive one; - then numIndexTuples is the index’s whole tuple count as the planner has it - its ROWS, the table’s or a partial index’s reltuples, which VACUUM and ANALYZE both set to rows (§18, “Statistics”; VACUUM used to set the postings) and which is therefore a lower bound on the postings a multi-key column’s walk reads - which prorates into every index page, and any page the proration still leaves out is added at random_page_cost; - and for the multi-key fallback ONLY, indexSelectivity becomes 1.0, so the bitmap heap scan above is costed as a recheck of the whole table. IS NOT NULL keeps the clause’s own selectivity: the rows it emits are exactly the rows it selects, so its heap side is not a full recheck even though its index side is a full walk.

Phrase, prefix, weighted and NOT tsqueries, <@, @> '{}' and a query with a NULL element then lose the bitmap plan to the sequential scan (measured 61.5/51.9/45.5/65.8 ms for phrase / prefix / weight / <@ against the bitmap plan’s 101.3/98.0/88.7/89.1), and to GIN when a GIN index is present (18.8/7.1/27.3 ms), while exact @>, && and AND/OR tsqueries keep the count pushdown and the bitmap plan unchanged. test/sql/tsvector.sql and test/sql/array.sql pin the plan shapes.

The expression evaluator (lion_expr.c)

§15’s union cursor is generalised into LionExprCursor, a cursor over an LionKeyNode tree whose leaves are located posting sets. LionCountSource gains a tree member; NULL still means “the union of all the sets”, which is what every pre-§17 caller gets.

LEAF   one LionSetCursor, as before
OR     stands at the smallest container key any child has left; the container is the OR of the
       children standing there (this is exactly §15's union cursor)
AND    winds the children forward until they all stand at one container key; the container is
       the AND of theirs, and a container key whose intersection comes out empty is skipped
       here rather than handed up

Since 2026-09-28 an OR that an AND meets - a source of the count’s merge, a child of an AND node - is lazy: it stands at the key with its children’s containers and builds their union only when the leapfrog asks for it, which it does for the driver and where building it is the cheaper way to AND it; elsewhere the running intersection’s members are looked up in the children’s containers (§29.11, “Unions probed”).

The §9 pin rule survives both operators: a leaf is only advanced by lion_ecursor_next(), and the merge only calls that from lion_count_container(), after the visibility map has been consulted - so every leaf that contributed to a counted container still pins the page that container came from. The two places an AND lets a pin go without anything having been counted (winding a laggard forward, dropping an empty intersection) are safe for the same reason the merge’s own !alleq branch is: nothing of that container key reaches the visibility map.

Materialization (§9) needed a sharper rule, because with a tree “at least one participating set is read the pinned way” is no longer implied by counting pinned sets. lion_source_pinned() decides, per source, whether EVERY container it can yield comes with a live pin: a leaf has it unless its set is materialized, an AND has it if ANY child has it (all children stand at the key), an OR only if EVERY child has it (which children contributed is not known in advance). A positive source’s set is only materialized while some positive source still has it.

Since §15’s “Bounded cursors” a tree is PLANNED against an open budget before its cursors are built (lion_plan_node()), and the plan decides two shapes the rules above did not have. An OR whose children would together hold more memory or pins than the budget is read as a WINDOWED union (lion_wide_fill()): its children are opened one at a time, pinless, and it is never pinned. An AND whose children would hold more pins than the budget keeps them on its first child that has them at every key and has the others drop theirs, which leaves the AND rule - any child - exactly as it was. lion_source_pinned() asks the plan, so what the count trusts and how the cursors are built are one decision.

Count pushdown (lion_plan_*.c, lion_exec_*.c)

A new clause kind, LION_CLAUSE_MULTI: an OpExpr whose operator is strategy 2, 3 or 5 of some roaring opfamily, with the column on the LEFT (these operators do not commute: '{a}' @> tags is strategy 4) and a non-NULL Const on the right - or, since 2026-09-27, a value the node evaluates at run time (“A query known only at run time” below). Strategy 4 and everything that is not a roaring operator bail.

  • The strategy is read from the OPERATOR (lion_op_roaring_strategy(), a pg_amop lookup restricted to the roaring AM), not from an index, because the parent of a partitioned table has no index list (§16) and the clause kind has to be known before any index is matched. lion_match_index() then re-checks the strategy against the index that will really answer the clause, per partition.
  • A LITERAL query is extracted AT PLAN TIME and the clause is pushed down when the mode is KEYS, or, since 2026-10-07, when its superset is LOSSY (“Literal queries through the superset” below). An ALL-mode query would have every row rechecked in the heap, which is what the ordinary bitmap plan already does, better. lion_match_index() additionally insists that each relation’s index carries the very extractQuery function the plan-time extraction used, so the run-time extraction cannot come out differently.
  • A multi-key clause used to require a Const, even though §10 accepts a Param for equality and for an IN list. What decides whether this node can answer the clause at all is the query’s SHAPE
    • tags @> $1 with $1 = '{}' extracts to ALL mode, as does a phrase or a prefix tsquery - and a Param has no shape until the executor has it, at which point the plan is fixed and there is nothing to fall back to: the node could not recheck the operator against the heap, so it would have had to error on a query it was handed legitimately. It can now, and so a generic plan’s tags @> $1 and a stable expression’s query are taken (“A query known only at run time” below). A custom plan folds the parameter to a literal and is pushed down as before; test/sql/array.sql pins both.
  • At run time the clause’s source is the tree over its keys' posting sets, ANDed with the other clauses by the merge, exactly like an IN list’s union. Several multi-key clauses on ONE column are allowed (the one-positive-clause-per-column rule of §10 is for clauses that pin a value; tags @> '{a}' AND tags && '{b,c}' is just an AND of three key sets).
  • A multi-key clause pins no value, so the column it constrains cannot be printed by the target list; it does make the column non-null, so count(col) is still answerable.
  • A multi-key index can never DRIVE a count. Its entries are keys, not column values, so neither §10’s GROUP BY (the groups would be lexemes) nor §14’s sum-over-all (a row appears under each of its keys, so the sum of the entries is not the number of rows) is correct. lion_find_roaring_index() takes a multikey flag and the driving lookup passes false, so count(*) WHERE tags IS NOT NULL is driven by a scalar column’s entries (§35, “Every row”), with the multi-key column’s NULL entry subtracted (“Non-empty arrays” below), and falls back to the ordinary plan on a table with no scalar lion column. GROUP BY on a scalar roaring column next to a multi-key WHERE clause is the supported and tested combination.
  • EXPLAIN prints the clause with its operator: Lion Indexes: idx (tags @> {t5,t7}), idx (tsv @@ 'w1' & 'w2').

A query known only at run time (2026-09-27)

A multi-key clause whose query is not a literal - a prepared statement’s GENERIC plan (tags @> $1, tsv @@ to_tsquery($1)), a stable expression (tsv @@ to_tsquery(current_setting('app.q'))), an exec Param under a LATERAL nested loop - was declined, for the reason above: its shape decides whether the posting sets can answer it, and the node could not answer the shapes they cannot. It is accepted now, under §10’s rule for a value (lion_is_value_expr(), an ARRAY[$1, $2] of values included), and the node answers EVERY value the clause can take: exactly when the key sets answer it, and otherwise from a SUPERSET of its rows whose every candidate is tested in the heap.

The superset (lion_extract_query_superset(), lion_multikey.c). The run-time value is extracted with the index’s own extractQuery, as a literal is, and the tree over its keys is widened wherever lion_extract_query() would have given up - which only a caller that rechecks may do:

  • @> with a NULL element is the AND of the other elements (a NULL is under no key, and an AND of fewer keys selects more rows); every row when there is no other;
  • && with a NULL or partial key is every row: an OR cannot leave a key out and stay a superset;
  • in a tsquery a lexeme with a weight mask is the lexeme at any weight; a prefix lexeme is every row (the keys are hashed, and a range of them cannot be walked); !a is every row; a phrase a <N> b is a & b, each side widened the same way, because a phrase matches only where both of its operands match - at positions the index does not store - and an operand that is itself every row (!a <-> b) constrains nothing; an AND drops the operands that are every row, and an OR is every row when either operand is;
  • INCLUDE_EMPTY and ALL search modes (@> '{}'), more keys than LION_MAX_QUERY_KEYS and any other strategy are every row.

A query lion_extract_query() answers exactly comes back KEYS with the same keys; one that selects nothing (&& '{}', an empty tsquery) NONE; a widened one LOSSY, with only the keys its tree names; and one no key narrows ALL. <@ is still declined whatever its value: ginqueryarrayextract() answers it with INCLUDE_EMPTY, so it would never be exact and always be the recheck the ordinary bitmap plan makes anyway. (= on arrays is no operator of array_ops.)

Execution (lion_locate_multikey(), lion_exec_locate.c). The value is evaluated once per scan and after every ReScan with the other clause values (lion_eval_clause_values(), in every participant of a parallel plan), and extracted per relation counted:

  • KEYS is the literal case: the tree over the keys' sets is the clause’s source;
  • NONE selects nothing, like a NULL value;
  • LOSSY makes the superset’s tree the source, and the clause is added to the ROW FILTER below;
  • ALL locates nothing and is no source of the intersection at all - a negated source with nothing to subtract, which is exactly how IS NOT NULL over a column without NULLs reads - and the clause is added to the row filter, which then tests the rows the OTHER sources select;
  • and when there is no other source that selects rows - a plain count whose only positive clause was an ALL query - the relation is read by a sequential scan under the count’s snapshot, and the filter and the negated clauses are tested on each row (lion_count_heap_filtered()): the ordinary plan’s work, which is what the cost model charges a value it cannot estimate. The negated clauses are those of a heap column - IS NOT NULL, <>, IS DISTINCT FROM, which a NULL column passes and an equality’s true fails, NOT of a multi-key query, which a NULL column fails as the query’s NULL does, and §35’s reserved tests (lion_filter_with_negated(), LionRowFilterClause’s negate and nullpass). A filtered count’s run and a partition’s count come to it the same way, and a decoded walk (§34) reads every row with the same filter. The scan makes one count and reads no index expression, so beside such a query the count takes no negated clause on an expression column (§41) and no OR with a complement arm (§35, “Complements in an OR”), which is two counts, wherever it may come to the scan: with no clause positive whatever its value beside the query, and no walk that finds each count’s rows by itself - a GROUP BY’s but the decoded walk, a count(DISTINCT)’s (lion_where_scan_untested()).

The row filter (LionRowFilter, lion_count.c). The node builds one per relation it counts - each clause col op value with the clause’s operator and this scan’s value, the column numbered as the index that answers it says, which for a partition is the partition’s own numbering (§16) - and hangs it on the visibility cache every count of the execution is handed (lion_vis_cache_set_filter()), so that no caller of the count changes. The operator’s function is handed the call expression the parser built, relabels included, since a polymorphic function reads its argument types from it (get_fn_expr_argtype()): a column of a domain over an array reaches @>(anyarray, anyarray) as the base type, never as the domain (lion_arg_type_passed(); the 2026-09-27 review, test/sql/rowfilter_domain.sql). A count that finds one:

  • asks nothing of the visibility map: it vouches for visibility, not for the query. Every candidate TID goes to the heap recheck, as on a standby or with no pinned source (§9’s novm);
  • resolves each candidate’s HOT chain under the page’s share lock exactly as an unfiltered recheck does (heap_hot_search_buffer(), with its serializable checks and tuple predicate locks), notes the visible member, and tests it after the lock is released, under the pin alone - which is what core’s page-at-a-time heap scans do: nothing moves a tuple while another backend pins its page, pruning and defragmentation need the cleanup lock, and the operator may detoast or run for a while, which must not happen under a content lock. A row counts when the snapshot sees it and it passes;
  • keeps no visibility cache: the cache knows which offsets are visible, not what their tuples hold.

Why the answer is exact: the candidates are a superset of the rows the WHERE selects - the sources are exact for their own clauses and a superset for the filtered one - and each candidate is then decided by the snapshot and by the very operator and value the query applies, on the very version the snapshot sees. §9’s interlock has nothing to protect: nothing is counted from the map.

Every shape of the node counts through the same functions, so every shape takes the filter: a GROUP BY per group, a count(DISTINCT) per test (an existence test stops at its first row that passes), a range walk per entry, a partition per partition, and the FK-side join (§27) per dimension row - where the collected copy of the fact filters is made WITHOUT the filter (it copies the superset; a collection is handed no cache) and each count of it applies the filter. A disjoint sum or a one-set count is not taken beside an ALL query’s empty source, which takes the ordinary merge instead.

Under an OR (§19) the query still has to be a literal: the filter tests the clause alone, which is right for a clause the rest are ANDed with, while a row satisfies an OR when ANY arm holds, and the other arms' leaves are answered from posting sets and never evaluated.

Cost (lion_multikey_cost_mode(), lion_cost_recheck()). The clause analysis hands the cost model the clause over the value’s plan-time estimate when estimate_expression_value() gives one (a stable expression; a custom plan’s parameter is a literal already), and the model extracts it as the executor will extract the value itself: KEYS costs what a literal costs; LOSSY rechecks at least the rows the clause selects (the superset is wider, by how much the estimate does not say); ALL rechecks every row the other clauses select, or every row of the relation. A value with no estimate - a generic plan’s parameter - is priced as ALL: its shape is unknown, and is assumed to be the expensive one, as lioncostestimate() assumes it for a bitmap scan. Each candidate is fetched on the pages each count reads for itself (lion_heap_page_cost(), a filtered recheck having no cache) and tested at the clause’s own evaluation cost; §10’s recheck of the dirty pages is still charged beside it. An ALL query is no source of the AND, in the price as in the executor: the model leaves it out of the sources lion_cost_count_rel() merges, by the same lion_multikey_cost_mode() the recheck is priced by, as a scan’s price leaves it out of the scan’s AND (§29.11). That is the count’s extraction, a superset, and not the scan’s exact one (lion_cost_qual_is_full()), which would leave out a phrase or a NULL element too, whose superset the count does locate and AND. Until 2026-09-28 the price counted it a source all the same, at its default selectivity: a lookup, and a probe at each key of the driver - or, where that selectivity made it the smallest source, the driver of every count, which priced a grouped count’s merges as though a set of a few rows cut each of them short. So a lone tags @> $1 in a generic plan, priced as ALL, is priced as the sequential scan the node then makes, and is chosen over the ordinary plan’s sequential scan by the Aggregate above that one: at worst the node does the ordinary plan’s work without the Aggregate, and any other value it answers from the posting sets (below). Beside a selective clause the rows the node would recheck are the ones the ordinary plan fetches anyway, and the two come out close, the node ahead by what the posting sets save (18% in the table below, 4% on the smaller table of the tests, which is why those pin the choice of the lone clause only, and beside another clause the price); and under plan_cache_mode = auto the generic plan’s price keeps the plan cache choosing custom plans, whose literals are pushed down exactly.

EXPLAIN prints the clause with its value as core does (tags @> $1, tsv @@ to_tsquery('simple'::regconfig, current_setting('app.q'::text))); with ANALYZE, Heap TIDs Rechecked counts the candidates tested, or the rows scanned, and Rows Removed by Recheck - printed only when there were some, as core prints its own - the ones the filter turned away.

Measured (2026-09-27, PostgreSQL 18.6 assert build, so ratios and not absolute numbers; 400,000 rows over 9,757 heap pages, all-visible; k 50 values, tags three elements out of 20, 7 and 5,000, tsv four lexemes; generic plans, medians of five, warm):

query and value node ordinary plan
tsv @@ $1 AND k = $2, 'w1 & x2' (exact) 0.4 ms (8,116) 8.6 (9,929)
… 'w1 <-> x1' (the AND of the lexemes, rechecked) 2.9 7.1
… '!w1' (every row of k = 1 rechecked) 7.2 6.6
tags @> $1 alone, '{t1}' (exact) 0.2 (14,784) 126 (14,762)
… '{t1,NULL}' (the rows of t1, rechecked) 33.5 122
… '{}' (a sequential scan) 79.5 105

Bold is the plan the model picked then, which does not know the value: beside a selective clause the node, whose worst case - !w1 - is the ordinary plan’s work to a tenth; alone the sequential scan, by 22 units in 14,762, because priced as every row the node IS a sequential scan - and one that runs faster than the ordinary plan’s, since the query is extracted once where the ordinary plan evaluates it per row, so the worst case the model assumes is not worse than what it replaces. The 22 units were the lookup and the probes of a source the node never reads; without them (2026-09-28, above) the lone clause goes to the node, and gives up nothing: measured again on PostgreSQL 16.15 (a distribution’s build, no assertions; generic plans, medians of fifteen, warm, all-visible), on the two tables the tests pin it on and on a million rows with countmultikey.sql’s tags (20, 7 and 500 values) beside a few other columns, 16,210 heap pages:

count(*) ... WHERE tags @> $1 node ordinary plan
array.sql’s 20,000 rows: '{u500}' (exact) / '{t3,NULL}' (rechecked) / '{}' 0.003 / 0.16 / 2.1 ms (457.01) 3.1 / 3.0 / 3.0 (457.26)
countmultikey.sql’s 12,000: '{t1}' / '{t1,NULL}' / '{}' 0.005 / 0.39 / 1.4 (400.01) 2.2 / 2.0 / 1.9 (400.16)
1,000,000: '{u499}' / '{t3,u7}' / '{t1,NULL}' / '{}' 0.013 / 0.11 / 58 / 127 (28,710) 193 / 192 / 184 / 170 (28,723)

Under plan_cache_mode = auto the plan cache still keeps choosing custom plans for it, whose literal is exact.

test/sql/countmultikey.sql runs every shape against the pushdown off, with the node’s recheck counters beside the answer: @>, && over text[] and int[] and @@ in generic plans with values that are exact, select nothing, are NULL, widen (a NULL element, a phrase, a weight, a & !b, a prefix beside a lexeme) and are every row ({}, !a, a:*, a | !b), queries built by to_tsquery($1) and websearch_to_tsquery($1), ARRAY[$1, $2]; beside other clauses, two such clauses and an IS NOT NULL read by the sequential scan, count(col), GROUP BY (one column and two), HAVING, a folded GROUP BY and count(DISTINCT); the OR that is declined; custom plans; stable expressions, and one cached generic plan executed as its setting moves through exact, every row and widened; an exec Param under LATERAL; the cost model’s choices; a partitioned table whose partitions number the column differently; the FK-side join’s inner, semi, anti and count(DISTINCT) forms, and in parallel plans; a dirty heap before and after VACUUM; and literal prefixes too common to expand, one that expands and a jsonpath beside each negated clause of a heap column, of an expression column and a complement OR, alone and beside a clause positive whatever its value, grouped by one column, two (the nested loop, where a lexeme takes the decoded walk) and three, by cells, in count(DISTINCT), in the FK-side join, in a filtered count and beside a residual clause, serially and in parallel, and the same beside a parameter’s query.

Literal queries through the superset (2026-10-07)

Until now a LITERAL phrase, weight mask or a & !b was extracted exactly, came out ALL, and every consumer treated it as every row: the bitmap scan emitted the union of every entry with recheck, the cost model priced that walk and lost to a sequential scan, and the count declined the clause. The superset above was used only for values known at run time, yet nothing in it depends on when the value is known. Every consumer that can recheck now asks for the superset of a literal too:

  • the bitmap and plain scans (lion_emit_query(), lion_scan_col_tree() for a multicolumn index’s AND and for the plain-scan source) take a LOSSY superset as its KEYS tree and set the scan’s recheck flag, so core’s Recheck Cond decides each candidate - which is GIN’s own answer to a phrase. The paths that must be exact (lion_scankey_sets(), a scalar column) still use the exact extraction;
  • the cost model (lion_query_is_full_scan()) calls only an ALL superset a full scan;
  • the count pushdown (lion_analyze_leaf()) accepts a literal whose superset is LOSSY wherever it accepts a run-time value - that is, where the clause can take the row filter, so not under an OR - and lion_locate_multikey() locates it exactly as a run-time LOSSY value: the superset’s tree is the source and the clause goes to the row filter. A plan-time LOSSY literal that the executor extracts as anything other than KEYS or LOSSY is an error: both extractions are pure functions of the same Const and the same extractQuery, which lion_match_index() already insists on.

A prefix lexeme and a bare NOT are still ALL, and still go where they went. The answer stays exact everywhere: the scans by core’s recheck, the count by the row filter’s. What changes is the price, which is now the superset’s size rather than the index’s: a rare phrase (1,033 rows, 1,273 rows with both lexemes) counted in 10 ms where it took 914 ms, against GIN’s 9; a common one (401,520 rows) in 985 ms against GIN’s 949 (PostgreSQL 16.15, 500,000 synthetic documents of a median 120 words, warm). The count’s gain over GIN is the heap-skipping kind only where the superset is narrow; on a wide superset both do the same detoast per candidate. Exact phrase counts need stored positions, which is a separate design.

Stored positions: position trees (in progress, 2026-10-07)

Not reachable from the extension’s SQL yet: nothing reads positions, so no operator class in it writes them.

  • Chunks (lion_positions.[ch], item type LION_CT_POSITIONS): a key’s members - one per row, its heap TID code and its tsvector positions with weights - varint-encoded in ascending code order, at most 4,104 bytes a chunk. The header’s ckey field holds a heap block, a lower bound on the block of every member. test/unit/positions_test.c checks every operation against a brute-force reference and fuzzes damaged chunks.
  • Position trees (lion_postree.c): a posting tree (§22) whose pages carry LION_PAGE_POSITIONS and are stamped with the position tree’s own head, whose leaf items are chunks, and whose keys are heap blocks. Like a posting set it has no internal level while it has at most 16 leaves - its head is its first leaf and the leaves are one chain - and the split that would make a 17th promotes it (§22, “A small set has no internal level”). Descent, promotion, root push-down, leaf and internal splits are §22’s code, reached through a LionTreeRef that carries no entry.
  • Routing. One block’s members can fill several leaves, so separators repeat. Readers and writers of block X descend to the last separator strictly below X (lion_posting_search_before()) and walk right. A tree with no internal level has nothing to descend: a writer walks its chain from the head to the leaf the descent would have found - the last one before the first whose last block is at or past X - and a reader’s position cursor walks right from the leaf it stands on, past the leaves whose last block is below X, however far. The invariants that make this complete are in the header of lion_postree.c: members ascend; a chunk’s members reach at most the next chunk’s header block; and a leaf’s first chunk carries the leaf’s separator. VACUUM, when it removes positions, must keep a leaf’s first chunk even when it empties it. Blocks rather than container keys because a run of equal separators is walked, not searched: one heap page’s rows bound it.

  • Tested by test/sql/postree.sql, through a C test function the test creates from the library: puts into a tree (runs in hot blocks, scattered, appended, replacing), then a walk and a lookup of every member against a reference. It reaches three levels, so it also covers internal page splits, which posting_tree.sql cannot reach. Replayed on a standby under wal_consistency_checking = 'pg_lion' with no difference.

  • Entries of a column whose opclass has support function 5 (lion_tsvector_positions()) carry the flag LION_ENTRY_POSITIONS and an 8-byte extension after the key (LionEntryPosExt): an INLINE entry keeps one chunk after its payload, at a MAXALIGNed offset, pos_len bytes long; a CHAIN entry points at its position tree (pos_root). Reserved entries never store positions.

  • Insert writes a row’s positions before its TID (W1): in the INLINE entry’s one record, or into the position tree before the posting set. An INLINE entry that outgrows itself spills both, the new position head written in the spill’s last record with the entry. “Outgrows” is judged on the payload at its own length plus the chunk, against lion_inline_max() - which next to a long key is LION_MAX_ENTRY_SIZE less the key, not inline_limit - and the entry is then written with whatever growth slack (§4) still fits beside the chunk, none if none does: the area the last rewrite gave the payload is kept only while it does, since an entry built at that area beside a grown chunk can be over LION_MAX_ENTRY_SIZE although the payload itself fits.
  • VACUUM takes a TID’s positions out after the TID (V1), and asks the callback about every member (V2). INLINE: the chunk is filtered in the same rewrite as the payload. CHAIN: after the posting set’s leaves, every leaf of the position tree is filtered under an EXCLUSIVE lock (no reader’s correctness rests on a position page), and every page of the tree is marked visited. An entry that empties is deleted in one record with its position head; the rest of the tree - for a tree with no internal level, the chain to the right of the head - is freed afterwards whatever it holds, and the sweep (and verify) take an unreferenced position page whose head is gone for a leak.
  • Verify walks every position tree as it walks a posting set (§7, §22), and with the same code: level by level from the leaves up, every page a position page of that tree stamped as the entry’s, every internal page’s high key and separators, and every level’s downlinks matched with the pages of the level below. What differs is the keys: a leaf’s separator is the header of its first chunk (P3), no leaf’s last member is above the separator of its right neighbour (at it is fine), and an internal page’s last separators may equal its high key, which an internal split through a run of equal separators leaves. Every chunk, the members' order across the tree (P1, P2), and every leaf’s min and max are checked on the way; on a primary so is that every TID of the posting set has positions, the two trees' leaves read in step and nothing collected. The tree is walked with nothing held and kept when its entry reads the same before and after, and a walk that finds a TID without positions is made again with the entry’s leaf held before it is reported (§7, “Position trees”). On a standby a walk is held to the replay position it began at instead: what one that read every page at it found is reported, one that found no difference is kept, and a difference after replay changed a page is settled by passes of the comparison alone, or left uncompared with a WARNING (§7). With heapallindexed every heap row’s positions under each of its keys are compared with the ones its value gives (§7). verify.sql’s ninth case damages a downlink, a separator and a leaf’s min of a tree with an internal level, its seventeenth the members of a tree, its eighteenth a row’s positions (INLINE and in a tree), and verify() must report each.
  • Build: a row’s positions for a key travel through the build’s spool (§24) with the row’s code: in a key’s in-memory stream each code is followed by its positions (a varbyte count, then the WordEntryPos values), and a code with its positions - a member - stays one item through every spill, merge and parallel participant’s export, so it is under the spool’s one memory budget and comes out of the merge in code order with the key’s codes, a heap page’s members re-sorted where a heap-only tuple came under its chain’s root offset. lion_build.c hands each member to lion_posbuild.c as it adds the code to the posting set, and gets back the entry’s chunk, or a position tree written bottom up through the bulk writer as the members come - up to 16 leaves held back and written as a chain whose first leaf is at the head’s block, and a root over them only once a 17th is needed. Only then is it known what the chunk leaves of an INLINE entry, and a posting set that does not fit beside it goes to pages at the flush. A parallel build needs nothing more: the workers sort and merge positions as they do codes, and the leader merges the participants' tapes. The index is the same one, page for page, whatever the memory and the workers (test/sql/buildspill.sql compares the builds at the least maintenance_work_mem, serially and in parallel, with the one built in memory). On 1M synthetic rows with 3M distinct lexemes on PostgreSQL 18.6, the build takes 28 s at 256MB of maintenance_work_mem with 3 workers (every process within its share, 74-80MB resident), 63 s serially at 256MB and 74 s at 32MB; keeping the positions in a store of their own beside the spool, in half the memory and spilled to sorted runs of their own, took 52 s, 136 s and 148 s. Without positions the parallel build takes 24 s, and GIN 54 s.

  • Tested by test/sql/positions.sql, which compares what the index stores with unnest() of the rows after inserts, an UPDATE, deletes with VACUUM, a key emptied and refilled, a REINDEX, and a REINDEX at 1MB over heap-only tuples (spilled runs, page disorder).

  • Queries (lion_posfilter.c). The posting sets give the candidates, as lion_extract_query_superset() already did for a recheck, and PostgreSQL’s own TS_execute() decides each one from the stored positions: a cursor per lexeme the query names (those under a NOT too) walks the key’s members in TID order, skipping chunks by their headers - the rest of the chunk it stands in too, once the next chunk’s header is below the candidate’s block (P2) - and the callback is checkclass_str()’s over the candidate’s member, stripped members included. No member under a lexeme is a row without it (P ⊇ C). So phrases at any distance, weights, a & !b, a <-> !b and ORs of them come out exactly what the heap’s @@ gives. A query with a prefix lexeme is first expanded into the OR of its keys (see “Prefix lexemes” below).

    • A bitmap scan emits what the filter keeps with recheck = false, and a plain index scan puts its source’s containers through it (the scan’s keys are ANDed, so filtering the result by one of them is exact), under an MVCC snapshot.
    • The count pushdown puts every container through the filter before it asks the visibility map, while the container’s pages are still pinned (R1, R2), so an all-visible page is counted without the heap. EXPLAIN shows Position Checks and Rows Removed by Positions. This is for a clause ANDed with the rest of the WHERE, which the heap’s row filter would otherwise decide, and so for GROUP BY, count(DISTINCT) and the FK-side join too.
    • A copy is decided as it is made. A copy of the WHERE (§10, “The WHERE sets, collected once”) or of an FK-side join’s fact filters (§27) is put through the filters as its containers are copied (lion_sources_collect(), LionCollect.filter), a parallel plan’s shared copy chunk by chunk: every row of the superset the other sources leave is decided once, and the copy holds only the rows the query selects. The counts against it decide nothing again (LionMatSet.posfiltered, LionCountCtx.posdone) - a join’s thousands of keys, a GROUP BY’s groups. A count that reads the filters' sets themselves - one that probes them, or the WHERE of a walk that collects none - decides its own candidates, and its cursors start again at their lexemes' first members. Deciding positions at a moment of the scan’s own choosing is safe for the reason below (“Why reading positions at any moment of a scan is safe”): a row the snapshot sees keeps its TID and its positions, and a TID that changes hands in between is a row the snapshot does not see, which a copy may keep or drop as a stale copy may (§27, “Why a stale copy is safe”).
    • The price (lion_cost_posfilter()). A cursor decodes its chunk’s members up to the candidate’s, so a count whose candidates lie close together - a copy’s, a count of every row - decodes each member of its lexemes once, and one whose candidates lie further apart than a chunk holds decodes part of one at each: 1,606 to 1,703 members a seek, measured over trees of 400,000 to 7.5 million members of one position (LION_POS_SEEK_MEMBERS, 1,650). Each member decoded is pg_lion.position_member_cost (2.4 cpu_operator_cost), and each candidate pg_lion.position_check_cost (4.3) a lexeme beyond that, TS_execute() and its callbacks: 11.4 to 12.8 ns, the leaves a cursor steps and descends to included, and 21.6 ns on the release build (§10, “The units”), at the reference’s 5 ns a cpu_operator_cost. (Fitted on 2 and 10 million rows, foo and bar in three rows of four and new and york in a fifth, where the probing counts run at 453 to 487 units a millisecond and a count of every row at 476 to 511. A fit on 400,000 rows, whose trees are a few leaves, comes to 1.8 and 1,000, which over the larger trees prices the probing counts at 211 to 263 units a millisecond and takes the FK-side join of new <-> york at 3 and 20 keys of 2 million rows - 2.5 ms against the nested loop’s 0.37, 15 against 3.5 - and at 20 keys of 10 million, 21 against 2.7; at 2.4 and 1,650 each of those takes the nested loop. At 200 keys of 10 million the join still collects the filter, 144 ms against the nested loop’s 30: core prices that nested loop at its own rate, which §39 holds out.) The candidates are the SUPERSET’s rows, not the clause’s (lion_tsquery_pos_shape()): each operand is estimated as core estimates a query of it alone, the clause and its superset are combined from those as core combines a tsquery’s operands, and the candidates are the rows the caller’s filters select times the superset’s share over the clause’s - 1 for foo <-> bar, whose superset foo & bar core estimates as it estimates the phrase, and 8 for foo <-> !bar of foo in every row and bar in seven of eight - and never more than the superset holds: !foo <-> bar where every row has foo is estimated at next to no rows, and its candidates are the rows with bar. The operands' shares are combined over the rows whose column is not NULL (lion_tsvector_nonnull()), as core combines them: an operand’s share alone carries the NULLs once, and a product of n of them n times - with three rows of four NULL, that would price new <-> york at three times its candidates and foo <-> !bar at under half. The AND of the superset’s lexemes is as the intersection probe measures it (§29.11, “Correlated sets”): new <-> york of two lexemes in the same tenth of the rows is a hundredth of them by core’s product, and its candidates are the whole tenth. The probe’s measure is taken however near core’s product (lion_isect_measured_factor()): the three times §29.11 asks of a measure before it moves the rows estimate - which the plans above it are sized from, and which ANALYZE’s per-column estimates and the probe’s sample both wobble - is no reason to price a filter over rows it does not decide. alpha <-> beta of two lexemes in the same two rows of five, 2.5 times core’s product, would be priced over that product, and the FK-side join taken at 3 keys of 20,000 rows at 0.41 ms over a 0.04 ms nested loop (release build). A partition is probed in its own columns, as the planner hands it its restrictions (lion_clause_in_child()): a clause in the partitioned table’s columns matches no key of the partition’s index, and a count of a partitioned table would price its filters from core’s product: new <-> york’s, over 10,000 rows in two partitions, at 11 against 31 on the same rows unpartitioned. A cursor is a lexeme’s, one for each the query names, those under a NOT too, and its members are the rows that have the lexeme. Priced from the clause’s own rows and the superset’s one lexeme, foo <-> !bar would come to a sixteenth of what it takes: over 400,000 rows and 2,000 keys, the FK-side join would be taken at one to ten keys over a nested loop into a btree of the fact’s key that runs 8 to 40 times as fast (1.2 ms against 18 ms at two keys, on the release build); and the correlated phrase’s probing counts would be priced at a fifth of their time. The copy is priced once for its candidates, and every other count for its own, its lexemes decoded up to them - which an FK-side join’s probing counts, their candidates a few of every part of the heap, pay at almost every one. The heap recheck (lion_cost_recheck()) charges such a clause nothing. An AM scan, plain or bitmap, is priced the same way once a scan (lion_cost_scan_posfilter(), from lioncostestimate()): its cursors begin again at every scan, so a nested loop’s inner scan pays the seeks to its key’s candidates at every repetition: a nested loop over 20 keys of a dimension into the fact’s lion index, 3,000 candidates 2,000 rows apart, takes 68 ms, against 19 ms for the FK-side join that collects the filter once.
    • Under an OR no recheck sees a leaf alone: the row passes when any arm holds, and the other arms are answered from posting sets. A leaf the sets only bound is taken there when it is a literal an index that stores positions can follow (lion_query_posexact(); lion_match_index() then takes only such an index), and its tree is wrapped in a LION_KN_POSFILTER node: one child, the superset, whose every container the node’s cursor puts through a filter of its own as it gets there, skipping the containers it empties. The child is built eagerly and the node is never pending, so a leapfrog, a union or a probe above it sees an ordinary built container; the child stands at the key while its positions are read, so its pins carry §9 as any node’s do, and a wide union opens the node a window at a time like any child. Its cost is the filter’s, per candidate, as above.
    • What a filter holds grows with its query, and the node is charged that against the open budget (§15, “Bounded cursors”; lion_posfilter_mem()): a cursor per distinct lexeme, each a copy of its position tree’s leaf and a pointer per item that leaf may have (24 kB) or an INLINE entry’s chunk - which of them is not known before the entry is read, so every operand is charged a tree’s - beside the filter’s two member arrays (128 kB). So an OR of filters too big for work_mem is read as a windowed union, one filter open at a time. The charge used to be a flat 160 kB, and a phrase of a thousand lexemes allocated 24 MB under it (2026-10 review). The filter is in a memory context of its own, of sizes aset.c keeps no deleted context for (LION_POSFILTER_CXT_SIZES), so that what it holds is the same whatever context it is begun in and whatever the backend did before, and the charge is an upper bound of it: every buffer and array past 512 bytes is a block of its own, its size and the headers. A recycled small context keeps the chunk limit of the context it was first, and one first begun to grow to 8 MB put each 8 kB leaf in a block of 16 kB, so that the same filter held a third more in one session than in another. What only finding the cursors' keys takes - the opclass’s extracted keys, the entry lookups - goes with a context of its own when the filter begins, and what TS_execute() allocates for a candidate - a phrase’s positions - goes after each candidate, not after each container’s thousands of them.

    • Why reading positions at any moment of a scan is safe: a member’s TID can change hands only after VACUUM has removed the old row’s positions, and the new row is then inserted after the scan’s snapshot was taken, so it is invisible to the scan and its page is not all-visible to it either. A mix of two rows' positions can therefore only be applied to a TID the heap visit drops.

    • @@ ANY (array): a bitmap scan emits each element through its own filter (lion_emit_multikey()), and a plain index scan under an MVCC snapshot wraps each element’s superset in a LION_KN_POSFILTER node of its union (lion_scan_col_tree()), so neither rechecks unless an element has a prefix lexeme. The count takes a multi-key operator over a literal array (tsv @@ ANY) as the OR of one clause per element (lion_multikey_any_as_or(), §19), so an element the sets only bound is decided from the positions there as under any OR. An index stores them for every key column whose opclass has support function 5 (tsvector_ops) unless it is built WITH (store_positions = off): the reloption is an enum, auto (the default), on or off, with true and false kept as on and off from when it was a boolean defaulting to off. auto is GIN’s choice - a tsvector index keeps its positions without being asked, at about twice the size, so that phrases, weights, NOTs and BM25 are answered from it - and an index with no such column builds without them, where on is an ERROR (lion_settle_store_positions()). The build records what it did on the meta page (LION_META_POSITIONS in order_flags, set only on an index that has a column storing them; no format change), so ALTER INDEX changes nothing until a REINDEX (docs/REFERENCE.md, “Reloptions”).

Ranking: BM25 from the index (2026-10-07)

lion_bm25(index, query, k, k1, b, col) (lion_bm25.c) ranks the rows of an index by one key column that stores positions, by Okapi BM25, in Lucene’s form: idf = ln(1 + (N - df + 0.5) / (df + 0.5)) and tf * (k1 + 1) / (tf + k1 * (1 - b + b * |d| / avgdl)) per query lexeme, for any finite k1 of at least 0 and b between 0 and 1 (lion_bm25_params_ok(), which both functions and the LionBm25 planner ask). Each term is computed with its top and bottom divided by k1 + 1, as idf * tf / (tf / (k1 + 1) + k1 / (k1 + 1) * (1 - b + b * |d| / avgdl)), which no finite k1 overflows. Written as the first, a k1 near 1e308 makes both infinite and the term infinite or NaN, and an infinite k1 makes every term NaN; a NaN ranks below nothing, so once its heap is full a walk admits no other row, and the LionBm25 scan would return fewer rows than the sort of the same scores.

  • tf is the npos of the row’s member under the lexeme, a stripped lexeme counting once, read from the position tree with a counts-only cursor that steps over the positions. df is the lexeme’s entry’s ntids.
  • |d|, the row’s length, is its lexeme occurrences. The extraction files every row that has a lexeme under the EMPTY TEXT key as well, with one “position” holding the length in all 16 bits of a WordEntryPos (capped at 65535). The empty key is free: tsvector_in() and array_to_tsvector() refuse an empty lexeme and no tsquery can name one, and the extraction drops one that arrives some other way. It is an ordinary VALUE entry, so insert, build, parallel build, VACUUM and verify carry it with no change, and it costs one member per row (about 2 MB at 500k rows).
  • N is the empty key’s ntids. avgdl is the mean of its members' lengths; reading it walks the whole key, so a backend keeps it per index column until the key’s ntids moves by more than 1/64 or the index is rebuilt. VACUUM and ANALYZE walk the key too and record its members and the sum of their lengths on the meta page (§33’s area: the ndistinct and sumsq words of a multi-key column, which it has no other use for and whose valid_cols bit stays clear; lion_bm25_count_lengths()), and a backend without its own reading takes that on the same 1/64 terms. Without it every connection’s first score paid the walk, 8 ms at 500k rows and 31 ms at 2M, and with it 0.4 ms at either. A built index has none until its first VACUUM or ANALYZE, and walks. Each of them walks afresh, since rows replaced by longer ones leave the count where it was, and a key left with no rows records none, and drops a backend’s reading that sees it so, or rows inserted up to the old count would be read at the lengths of rows that are gone. The format is unchanged: an index whose words are zero reads as one with no reading.
  • The column. Keys are per key column (§24), so every column that stores positions has an empty key and lexeme entries of its own: its N, df, |d| and avgdl count that column alone, and a column of a multi-column index ranks exactly as an index on that column alone would, bit for bit. col names it as the index’s own pg_attribute does (the Column of \d index): a plain column by the table column’s name at CREATE INDEX, an expression by the name CREATE INDEX chose (to_tsvector, to_tsvector1 on PostgreSQL 19; the names differ between versions). NULL, the default, is the index’s one column that stores positions, and an error whose hint names the candidates when there are several (lion_bm25_column()); a name the index lacks, or a column without positions, is an error too. The functions are declared CALLED ON NULL INPUT for col’s NULL, and give no rows (a NULL score) for a NULL in any other argument, as STRICT would.

The query’s lexemes are its operands not under a NOT, once each; a prefix among them is refused, and one under a NOT scores nothing, as every operand there does ('w1 & !w7:*' ranks by w1). Candidates stream out of a merge of the lexemes' cursors in TID order, each scored as it comes, with the empty key’s cursor moving forward beside them; a min-heap keeps the best L = k + 16. Those are sorted and fetched best first under the caller’s snapshot, the TID following a HOT chain to the visible version, until k are visible; when dead rows leave fewer, the walk is made again for the next 4L: the rows ranked strictly below the last one already tried, by (score, TID). A row’s score depends only on its members and on the prepared query (N, df and avgdl are read once per call), so that is exactly where the last walk stopped. An offset into a bigger walk is not: a row inserted meanwhile, invisible to the snapshot but in the index, moves every row ranked below it down a place, and one row would be returned twice and another never. Dead rows count in N, df and avgdl until VACUUM removes them, as a search engine’s deleted documents do until a merge.

The walks stop only at one that returns fewer than it asked for, never on a count: rows inserted since the query was prepared are in the index too, and a walk that counted them towards the rows the lexemes had at the start stopped one short of the last visible row. Each walk asks for at most one row past that count, so in the usual case the walk that reaches the end comes up short and no further walk is made.

Ranking must answer what an ordinary scan of the index would, so lion_bm25() refuses, as the direct count functions do, a table with row-level security enabled (it cannot apply the policies), and so does lion_bm25_score(): N, df and avgdl count the hidden rows, so under a policy any score, even one of a made-up document, would tell whether hidden rows hold a lexeme and how many. The index lion_index_usable() rejects for the snapshot (an indcheckxmin index built over a broken HOT chain holds the newest version only, which no heap recheck can repair). Its lookups - the empty key’s, each lexeme’s, found or not - take the predicate locks of §9, “SERIALIZABLE”, as the LionBm25 scan’s do, so a SERIALIZABLE search that finds nothing still conflicts with a later insert of what it looked for.

MaxScore (Turtle and Flood) prunes the merge. A lexeme adds less than idf * (k1 + 1) to any row, since tf / (tf + norm) < 1 (infinite for a k1 near the largest double, which leaves every lexeme essential); with the lexemes in ascending order of that maximum, the first ones whose maxima sum to no more than the heap’s worst score, once the heap is full, are non-essential: a row with none of the others cannot get in. Only the essential lexemes' members are candidates. The non-essential ones are sought for a candidate only, from the largest maximum down, and only while what is left of them could still lift it past the heap’s worst. The heap’s worst only rises, so the essential set only shrinks. Equal scores rank by TID and candidates come in TID order, so a candidate needs a score strictly above the worst to get in; the maxima are nudged up by one part in 109 so a sum of them rounded down cannot pass for a score.

This pays most when a common lexeme sits beside rarer ones: on a synthetic 500k-row benchmark, top 10 for one very common lexeme and two rarer ones went from 41 ms to 16 ms. When every lexeme is common, each one’s maximum is close to the scores at the top, nothing becomes non-essential, and the walk still reads every member. Block-max bounds (a maximum per chunk, not per lexeme) would skip whole chunks there, but need a maximum tf and a minimum length stored per chunk, which the format does not have.

The planner path. lion_bm25_score(doc, query, index, k1, b, col) is the score of one row, computed from its tsvector with the N, df and avgdl of the index column col names, read once per call site. Its tf, length (every occurrence of every non-empty lexeme, capped at 65535) and order of addition are the walk’s: both add a row’s terms in descending order of idf, ties in query order, so a row gets the same float from either, bit for bit, as an incremental sort over the scan’s order needs.

Few matches. The walk is not offered when the query’s lexemes hold fewer rows than pg_lion.bm25_walk_min_share (0.75%) of the table’s, scaled down by how far the query’s tables and the index outgrow effective_cache_size. Before its first row the walk reads most of the empty key’s pages for candidates' lengths, a cost that levels off with the matches, while sorting grows with them, so warm the sort wins below a share of the rows that held across sizes: 1% at 100k rows, 0.72% at 500k, 0.8% at 2M (two rare words at 500k, 584 matches: 3.5 ms against 0.8). No price says so, because core’s estimate of a rare lexeme’s rows is flat - the plain plan cost 16,953 from 387 matches to 9,547 - so a declined path, decided on the walk’s exact ncand, is what works. Cold, the walk’s few heap pages beat the sort’s one per match, hence the scaling.

LionBm25 (lion_bm25_scan.c) is a CustomScan offered from set_rel_pathlist_hook for a base table whose query’s first pathkey is lion_bm25_score(doc, Const, Const, Const, Const, Const) DESC, the index a lion index on that table whose key column the function ranks indexes doc, a plain column. The planner resolves col with the function’s own lion_bm25_column(), and the walk ranks that key column, which the plan records. Anything else is left to the ordinary plan, which calls the function for each row, so that the rows and scores are the function’s in every plan: a doc that is another column, an expression, or a key column other than the one named is a tsvector the walk does not read, and a col that settles no column (NULL on an index with several, a name it lacks) is an error the function raises. Nor is the column ever taken from doc: the function cannot see where its tsvector came from, so the scan would rank where the function raises an error. The order is claimed only when the walk’s candidates hold every row the scan may return, so a restriction doc @@ q2 must be there that no document without the scored lexemes matches. lion_bm25_where_covered() evaluates q2 with TS_execute() in ternary logic: each scored lexeme is absent, and every other operand - another lexeme, a prefix - is unknown. q2 is covered when the answer is no whatever the unknowns are, NOTs and phrases included. Every row that passes it therefore has a scored lexeme and is a candidate. The usual query has the same q in both places, or a narrower q2: 'w1 & w2', 'w1 & !w9' or 'w9 <-> w1' under a score of 'w1'. Not 'w1 | w2', 'w9 & !w1' or a bare prefix, which a row without w1 can match.

The scan asks the walk for the best L rows: the planner’s limit_tuples plus 16, or 64 when the LIMIT is not known. It fetches them best first under the scan’s snapshot, following HOT chains, and applies the restrictions as its qual. When the node above pulls past them, it repeats the walk for the next 4L rows ranked below the last one it had, as lion_bm25() does, until the walk returns fewer than asked.

The scan also returns each row’s score: the ORDER BY expression is the last column of its scan tuple (custom_scan_tlist, after the table columns the plan uses), and setrefs.c replaces every copy of the expression above the scan with it. Evaluated on its own, lion_bm25_score() would read N, df and avgdl when the first row is fetched, the walk when the executor started; a cursor whose table gained rows in between would show scores out of the order it ranked by. With no call left above the scan for ExecInitFunc() to check, the scan checks EXECUTE on lion_bm25_score() for the current user itself, every time it is initialised and under plain EXPLAIN too (§9, “EXECUTE on what a count replaces”), so a cached plan answers a REVOKE or a SET ROLE as the ordinary plan does (test/sql/security_exec.sql). A relation locked FOR UPDATE or the like is left to the ordinary plan, since EvalPlanQual hands a scan the table’s row rather than that tuple, and so is one with LATERAL references (rel->lateral_relids: a PlaceHolderVar it computes for a LATERAL subquery pulled up from under an outer join), every path of which is parameterized by the relations they name - the scan’s is not, and its plan would compute that value from a column of a relation it does not scan. So are parameters the function refuses, whose error it raises. Its startup cost is two index tuples and two operators per member of the scored lexemes (their ntids, read at plan time from the lexemes' entries alone, lion_bm25_plan_ncodes(): a plan reads no row’s length for avgdl, which the price does not use, and takes none of a search’s predicate locks, which a plan that is never run would leave behind in a SERIALIZABLE transaction). Each row then costs a random page, a tuple and the restrictions. Against a bitmap scan that fetches, scores and sorts every match, on the synthetic 500k-row benchmark (top 10): 120 ms to 20 ms for two mid-frequency words, 157 ms to 51 ms for two common ones, and 160 ms to 21 ms for one very common word with two rarer ones. A rare AND keeps the bitmap scan, which is the cheaper plan.

Prefix lexemes (2026-10-08)

The tsvector opclass’s keys are text compared under the C collation, so the key directory is in byte order and every key starting with foo is one contiguous run of it. lion_tsquery_expand_prefixes() (lion_multikey.c) rewrites each foo:* in a tsquery into the OR of the index keys in that run, keeping the item’s weight mask on every key: a walk from >= foo that stops at the first key not starting with foo. The rewrite runs at scan time (bitmap scan, index scan, count pushdown), never in the planner, so new lexemes are always seen.

  • Exactness. A row visible to the scan’s snapshot was indexed under all its lexemes before it committed, so its foo... lexemes are all in the run and the OR selects exactly the rows foo:* does. Keys that only dead or uncommitted rows have add candidates and no matches.
  • Anywhere in the query. An OR of lexemes stands where the prefix stood, so under a phrase (foo:* <-> bar), a NOT or a weight it means what the prefix meant, and the position filter decides it like any other query.
  • Nothing matches. An empty run becomes the bare lexeme foo, which is not in the index either (an indexed foo would be in the run), so it selects no row.
  • The cap. At most LION_MAX_QUERY_KEYS (1000) lexemes over the whole query. Past it the original query is kept and the old path applies: the prefix is every row and the heap rechecks. So a literal query with a prefix may narrow nothing at run time, as a parameter’s may: the count pushdown does not take it for a clause positive whatever its value (plainpositive), which a parallel GROUP BY or count by cells collects by range and a filtered count’s decoded walk begins from (§10). Nor, with no such clause beside it and no walk driving each count, does it take a negated clause on an expression column or an OR with a complement arm beside it, which the heap scan of a WHERE that narrows nothing does not test (“A query known only at run time” above, prefixmulti).
  • Planner. Costing and the count pushdown’s mode choice see the query with its prefix flags cleared (lion_tsquery_strip_prefixes()), the closest stand-in for an expansion they cannot do. A prefix under an OR of clauses in the count pushdown is refused: that path has no recheck to fall back on if the expansion goes past the cap.

Long lexemes (2026-10-19)

A tsvector lexeme can be 2047 bytes long (MAXSTRLEN: text input stops at 2046, a dictionary or binary input does not), and a key with its 4-byte header at most LION_MAX_KEY_SIZE = 2047 (§4). So tsvector_ops puts every lexeme through lion_lexeme_key() (lion_multikey.c), in its extractValue and extractQuery wrappers and in the ranking:

  • A lexeme of up to LION_LEXEME_KEY_MAX = 2043 bytes is its own key.
  • A longer one is filed under a key of exactly 2043 bytes: its first bytes, as many whole characters as fit in LION_LEXEME_KEY_KEEP = 1978, then # up to byte 1979, then the 64 hex digits of the SHA-256 of the whole lexeme. Facets show the lexeme itself (§7, “Facet counts”).
  • Exactness. A 2043-byte key is ambiguous: it is a 2043-byte lexeme or the key of a longer one, and a 2043-byte lexeme can be spelled like another’s key. So the extractQuery wrapper turns a query that names one into LION_SEARCH_MODE_LOSSY: every candidate is rechecked in the heap (the bitmap scan’s recheck, the index scan’s xs_recheck, the count pushdown’s row filter), and the position filter stays out, since lion_tsquery_item_keys() needs the default search mode. The planner asks the same wrapper (lion_multikey_query_is_exact(), lion_query_posexact()), so such a literal under an OR of clauses stays with the ordinary plan. Lexemes of up to 2042 bytes keep every exact path.
  • Prefixes. A long lexeme’s key begins with its first 1978 bytes, or up to pg_database_encoding_max_length() - 1 fewer where a character straddles byte 1978, so its key is in the run of every prefix no longer than lion_lexeme_prefix_max(), and that key, being ambiguous, makes the expansion rechecked. A longer prefix is not expanded: the query keeps the path past the cap above, every candidate rechecked.
  • Ranking. BM25 scores keys: lion_bm25_lexemes() returns the query’s keys, deduplicated, and lion_bm25_score() finds a row’s term frequency by key (lion_bm25_doc_lexeme()), so the two agree with the index. A lexeme that shares its key with no other scores as itself. Two lexemes of one row with one key count as one term with the positions of the first in the tsvector’s order, which is the one the extraction keeps; the row’s length counts both.
  • Other keys: array_ops hashes long text, varchar and bytea elements the same way (“Long elements” below); a scalar key past LION_MAX_KEY_SIZE is stored compressed (§4, “Long keys”).

test/sql/tsvector_long.sql: lexemes of 1990 to 2047 bytes (the 2047-byte ones through binary COPY), long ones that share their first 2030 bytes, multibyte ones, and a 2043-byte lexeme spelled like another’s key; AND, OR, NOT, phrases, weights and prefixes at and past the longest a key keeps; bitmap, index, count and literal paths with and without positions against a sequential scan and GIN; BM25 against SQL and the LionBm25 scan against sorting; writes, VACUUM, REINDEX and verify; a scalar and an array key past the limit.

Long elements (2026-10-19)

GIN takes an array element of any length whose compressed index tuple fits in ~2712 bytes. array_ops names lion_array_extract_value() and lion_array_extract_query(), which call ginarrayextract() and ginqueryarrayextract() and treat an element as tsvector_ops treats a lexeme: one of more than 2043 bytes is filed under lion_long_key() of it - its first bytes, #, its SHA-256 in hex, 2043 bytes in all - and a query that names an element of 2043 bytes or more is LION_SEARCH_MODE_LOSSY, every candidate rechecked, with the same planner behaviour as a long lexeme. So a long element is never an error, where GIN’s limit applies to the compressed one.

Only for an element type whose equality is equality of its bytes, so that a hash of the bytes is the same for two equal elements: text and varchar under a deterministic collation (the column’s, as the extraction is called with it), and bytea, whose first bytes are cut anywhere rather than at a character. A hashed key is the type’s own value - the directory calls the type’s comparison and hash on it - and is equal only to a value spelled the same, which is 2043 bytes long and so rechecked as well. An element of any other type (numeric, citext, jsonb, …) is a key as it is or, past the limit, compressed (§4, “Long keys”), and an ERROR when that does not fit, as GIN’s is. The decision depends on the element’s raw length alone, never on how well it compresses, so an insert and a query always agree on whether an element is hashed.

lion_index_facets() shows a hashed key’s elements, each with its own rows (§7, “Facet counts”). test/sql/long_keys.sql: text, varchar and bytea elements of 2042 to 100000 bytes, multibyte and incompressible ones and one spelled like another’s key, under @>, && and <@, bitmap and count paths against a sequential scan and GIN, before and after writes, and the facets.

Non-empty arrays (2026-10-20)

A filter that asks whether an array has an element - array_length(tags, 1) IS NOT NULL, cardinality(tags) > 0, tags <> '{}' - was a heap filter beside the count: a function of the column is no clause of any kind the pushdown knows, and <> is no operator of array_ops. Every row of an array_ops column is in exactly one of five classes, which its entries tell apart (“The reserved EMPTY entry” above): the column is NULL (the NULL entry, §14); it has no element and no dimension (EMPTY), no element and a first dimension of length 0 (EMPTYDIM), or no element and a first dimension of length 1 or more (EMPTYINNER); or it has an element (NULLELEMS or the entries of its elements). A predicate whose truth is the same for every row of a class is then a union of reserved entries or the complement of one, and is answered with no heap visit.

The classes, as PostgreSQL’s arrays make them. An array with no element has no dimension nearly however it was made: '{}', '{{},{}}', array_fill(1, ARRAY[0]), an empty slice, trim_array() and array_remove() down to nothing and string_to_array('', ',') all come out as the one empty array of construct_empty_array(), and the binary input makes one of any array with no element. The exceptions are the empty int2vector and oidvector, [0:-1], their casts to int2[] and oid[], and what keeps their dimensions: array_cat() of them, and ARRAY[] and array_agg() of them, [1:2][0:-1]. Any other array has one to six dimensions of at least one element each, whatever its lower bounds and whether its elements are NULL. So over (NULL, EMPTY, EMPTYDIM, EMPTYINNER, has an element):

array_length(a, 1)                 NULL, NULL, 0,      >= 1,   >= 1
cardinality(a)                     NULL, 0,    0,      0,      >= 1
array_ndims(a)                     NULL, NULL, 1 .. 6, 2 .. 6, 1 .. 6
array_lower(a, 1), array_upper(a, 1)   NULL, NULL, any integer, any, any
a = '{}'                           NULL, true, false,  false,  false
                                   (array_eq() compares the dimensions first)

Which predicates (lion_nonempty_test(), lion_plan_nonempty.c). Each clause is evaluated over the five classes, a class’s answer being true, false, NULL for every row of it, or mixed; a clause with no mixed class is taken, and what it selects is the classes where it is true. The values are those of the table: the integer functions above with the dimension the constant 1, and coalesce() of one with an integer constant; compared with an integer constant of any width by an operator of the integer btree family, either way round, or with IS [NOT] DISTINCT FROM, or tested IS [NOT] NULL; the column compared with '{}' - an empty array constant with no dimension - by =, <> or IS [NOT] DISTINCT FROM; and any of these under NOT and IS [NOT] TRUE / FALSE / UNKNOWN. A value over a class is a range (>= 1, 1 to 6, any integer), so `array_length(a, 1)

= 0is true of every class with dimensions andarray_length(a, 1) > 1is mixed. Declined, and left to the heap: a mixed class (array_length(a, 1) = 1,array_ndims(a) = 1,array_ndims(a) > 1,array_lower(a, 1) = 1,tags <> ‘{1}’); an empty constant with dimensions (‘’::int2vector::int2[], which a row of EMPTYDIM equals or not by its lower bound); a dimension other than the constant 1 (array_length(a, 2)is NULL for a one-dimensional array with elements); a bound that is not a constant at plan time (the classes a parameter selects depend on its value); and a predicate true of every row or of none (coalesce(cardinality(a), 0) >= 0`), which has nothing to read.

The count (lion_nonempty_leaf(), first in lion_analyze_leaf()). The classes a clause selects become one of two clause kinds, whose value is a mask of the column’s reserved entries - one or more of EMPTY, EMPTYDIM and EMPTYINNER, and the NULL entry or not (LION_RESERVED_MASK_OK()):

  • LION_CLAUSE_RESERVED, the rows of the entries named - a = '{}' EMPTY’s, cardinality(a) = 0 the three empty entries', array_length(a, 1) = 0 EMPTYDIM’s, array_length(a, 1) IS NULL NULL’s and EMPTY’s - a positive source, located as the OR of those entries' sets (lion_locate_leaf()), and a plain one for the one-set counts as an equality’s set is;
  • LION_CLAUSE_NOTRESERVED, every row outside the entries named - cardinality(a) > 0 outside the NULL and the three empty entries, a <> '{}' and array_length(a, 1) IS NOT NULL outside NULL and EMPTY, array_length(a, 1) > 0 outside those and EMPTYDIM, a IS DISTINCT FROM '{}' outside EMPTY alone - a negated source subtracting their union, as <> subtracts its value’s entry and the NULL one (§35).
  • The NULL class alone, or every row but it, is a IS NULL or a IS NOT NULL (cardinality(a) IS NULL, cardinality(a) >= 0), the NULL and NOTNULL kinds of §14, now taken on a multi-key column too: from its NULL entry, when no index holds the column as a scalar one (a whole-array class, below) to answer them instead (LION_CLAUSE_ANY_COLUMN(), lion_match_index(), lion_index_col_for_kind() in the executor).

The rows with an element alone are never read as a union - NULLELEMS and every element’s entry, a row under each of its elements - only as what is left of the others. Both kinds are multi-key kinds, resolved on the column’s multi-key index, and only on one whose class extracts an array’s elements (lion_nonempty_index(), lion_extracts_array_elements()), which is what files the five classes in entries apart: a class of whole arrays, one key per row, has no key-less entry, and tsvector’s and jsonb’s classes extract other values. Both are taken wherever their models are, under the same rules: beside positive clauses, under a GROUP BY and a count(DISTINCT), in a filtered count, in parallel, on a partitioned table, as an arm of an OR (§19) and as a complement arm (§35), and beside nothing that selects rows, where the count of every row drives (§35). A RESERVED clause read from a partial index holds the count to the predicate’s rows, as an equality does (§45, lion_partial_anchor_kind()): its entries hold the predicate’s rows alone. count(col) reads the column as NOT NULL beside a RESERVED clause without the NULL entry and a NOTRESERVED one with it. Beside a multi-key query known only at run time (“A query known only at run time” above), which may narrow nothing and leave the rows to the row filter, the row filter tests a NOTRESERVED mask on each row the snapshot sees - the array in none of the empty entries named, which lion_array_empty_flag() reads off its dimensions, and with the NULL entry among them not NULL; without it a NULL array passes, as it passes IS DISTINCT FROM (lion_filter_with_negated(), LionRowFilterClause.reserved and nullpass). Two copies of one NOTRESERVED clause - array_length(a, 1) IS NOT NULL AND array_length(a, 1) >= 0 - are counted once. EXPLAIN prints the classes in one spelling whatever the query’s (lion_explain_reserved()): a = '{}', a <> '{}', a IS DISTINCT FROM '{}', cardinality(a) = 0, coalesce(cardinality(a), 0) = 0, cardinality(a) > 0, cardinality(a) IS DISTINCT FROM 0, array_length(a, 1) IS NULL, array_length(a, 1) = 0, array_length(a, 1) > 0, coalesce(array_length(a, 1), 0) = 0 and array_length(a, 1) IS DISTINCT FROM 0 - the twelve sets of classes one clause can select.

The estimates (lion_nonempty_rel_clauses(), §29.11 “The relation’s rows”). Core estimates each spelling as a function of the column, with no statistics of it: a null test at 0.995 or 0.005, an inequality at a third, multiplied with the others. array_length(a, 1) IS NOT NULL AND array_length(a, 1) >= 0 over a column with an element in 80% of its rows came out at a third of the rows, 2.4 times short, and array_length(a, 1) IS NULL at 0.5% where 19% were. Lion’s estimates take every such clause of one column of a lion array_ops index together, AND the classes each selects, and put in their place the one clause core estimates from the column’s own statistics for the same rows (lion_nonempty_clause()): a <> '{}', a = '{}', a IS DISTINCT FROM '{}', NOT (a <> '{}') (the empty array’s share and the NULLs'), a IS NULL, a IS NOT NULL, or false. Those clauses tell three classes apart, NULL, '{}' and the rest, and core’s statistics count an empty array with dimensions among the rest, as the clause does. So the EMPTYDIM and EMPTYINNER rows are taken from the column’s most common values (lion_ne_dims_clause()): the frequencies of those that are empty arrays with dimensions come off the clause’s estimate where the test selects the arrays with an element and not their class, and go on where it selects their class and not the arrays with an element - the clause then a RestrictInfo whose cached selectivity says so, as a run of <> is one (§29.11, “A run of <> is one exclusion”). A test of those rows alone (array_length(a, 1) = 0) is its own clause at their share. A value too rare to be a common one counts as none: such arrays come from a cast of an int2vector or oidvector, and a column with few of them loses little. A test alone that already is the clause it comes to (a = '{}') is left as it is, and changes nothing. The relation’s rows are corrected wherever this changes a clause, as wherever the intersection probe measures a correlation.

Planning cost. Every estimate of a relation lion indexes takes its clauses through lion_nonempty_rel_clauses(), at least once per relation and again for each index path’s quals, and the count pushdown takes every clause of its WHERE through lion_nonempty_leaf(), a table with no lion index included; nearly none is a test of whether an array is empty. So both look at a clause’s shape first, with no catalog lookup (lion_ne_shape(): one of the five functions above, coalesce() of one, array = or <>, or a null test), and only a clause of the shape is evaluated over the classes, its operator looked up last; a list with no clause of the shape and fewer than two null tests of a column is handed back as it is, without an allocation. Measured with callgrind on 18 (2026-10-21), the planning of a query with three or eight scalar clauses on a table with lion indexes spends 0.3-0.5% of its instructions there, where it spent 3-4.5%: 600 of 199,000 where it was 6,400 of 207,000 for three clauses, and 2,100 of 468,000 where it was 22,400 of 493,000 for eight under a count. A count on a table with no lion index spends 900 of 293,000 (0.3%) where it spent 8,400 of 301,000 (2.8%), and a query with no count none, as before.

The scans. There is no route. Core hands an index only the clauses whose operator is in the index’s operator family, and null tests of its columns (match_clause_to_indexcol()): a function of the column never reaches the access method, nor does <>, which is no operator of array_ops. a = '{}' does, as array = (strategy 17), and is answered as every row with a recheck, as above; a IS NULL and a IS NOT NULL are answered exactly (§14).

test/sql/nonempty.sql runs every spelling, the taken and the declined, against the pushdown off, as multisets both ways, over NULL arrays, empty ones, '{NULL}', '{NULL,NULL}', '{{NULL},{NULL}}', two-dimensional ones and lower bounds of 0 and -3; beside GROUP BY, filtered counts, ORs and complement arms, @> and && on the same and another column, a query known only at run time (with '{}', the row filter’s case), a text[] column under “C”, a domain over int[], a multicolumn index, a partitioned table (one partition without the index declines), a whole-array class and a class of the user’s over ginarrayextract(); after updates, deletes and inserts that move rows between the classes, before and after VACUUM; the plans; the rows estimated against core’s and the truth; and what reaches the scans. test/sql/nonempty_dims.sql does the same for the empty arrays with dimensions - ''::int2vector::int2[], array_cat() of two, ARRAY[] of them in two and three levels, ''::oidvector::oid[] - beside '{}', NULLs, '{NULL}' and arrays with elements and lower bounds of 0: every spelling, the empty constants with dimensions and the multi-key operators, alone and beside a key; the plans; GROUP BY, filtered counts, count(DISTINCT), ORs, the row filter beside a query known only at run time with each mask, counts by cells, the parallel node, a partial index the clause holds the count to, the facets, and inserts, updates, deletes, VACUUM and REINDEX moving rows between all five entries, with lion_index_verify() after.

Cardinality guard

Reloption max_entries (int, default 0 = unlimited, ShareUpdateExclusiveLock). Exceeding it is a WARNING, once per backend per index (a static HTAB keyed by relation Oid), and never rejects a row.

  • Build: exact. Pass 1 has just counted the distinct keys, so the test is ndistinct > max_entries.
  • Insert: a running estimate, and only on the one path that can add a key - creating a new entry. The heuristic is entries on this directory leaf + 1, times the leaves (§21), the leaves being the meta page’s count of directory pages less the directory’s height, and one at least (lion_insert_new_entry()). The index’s other blocks - posting and position trees, free pages - hold no entries: a column of five keys with 100,000 rows each is 62 blocks and one leaf. The leaf is already locked, and the meta page comes after it in a split’s lock order. It is deliberately crude: counting the whole index per new key would read every leaf.

Not supported

<@ from the posting sets (a row matches when it has no key OUTSIDE the query array, which the index cannot tell); prefix, phrase and weighted tsqueries from the posting sets, except as a superset rechecked in the heap (above); col op ANY (...) in the count pushdown (only in the bitmap scan); a query known only at run time under an OR (above); a multi-key index as the GROUP BY or sum-over-all driver; lion_index_count(idx, key) on a multi-key index (it needs a strategy-1 operator and errors out); a test of whether an array is empty in a plain or bitmap scan (“Non-empty arrays” above).

Measured (2026-09-20, 1M rows, 521 MB heap, all-visible, warm cache)

tags text[] of 5 tags out of 1000; tsv tsvector of 30 lexemes out of 20000 (4.99M and 29.98M (key, row) pairs).

column   roaring build   roaring size   GIN build   GIN size
tags          18.4 s         37 MB        2.9 s      23 MB
tsv          107.6 s        390 MB       19.3 s     157 MB

The posting-set BYTES are competitive - 20.4 MB for tags and 177 MB for tsv against GIN’s 23 and 157 - and the size difference is entirely the page-granularity floor of §12 item 3: every one of the 1000 / 20000 keys is too big to stay inline (5000 and 1500 members) and so owns whole container pages, 4 and 2 of them, leaving 10.8 MB and 214 MB of free space inside them. Sharing a container page between keys is what would close that gap, and it is the same item that was already the top open one before this section. The build time is the second half of the same story: 30M (key, row) pairs go through one tuplesort and then into 40000 pages that are each written once.

count(*) WHERE ...                     rows   pushdown   roaring bitmap+Agg   GIN bitmap+Agg
tags @> '{t17,t42}'                      16    0.14 ms        0.39 ms            0.72 ms
tags && '{t17,t42}'                   10031    0.38 ms        7.6 ms             8.0 ms
tsv  @@ to_tsquery('w17 & w42')            1    0.09 ms        0.11 ms            0.23 ms
tsv  @@ to_tsquery('w17 | w42')         2915    0.21 ms        2.3 ms             2.3 ms

The bitmap paths are at parity with GIN, heap-bound as §12 measured for scalar keys. What the multi-key classes are for is the first column: the pushdown answers the 10031-row && in 0.38 ms against 7.6 ms, because it never visits the heap - Heap Blocks Skipped via VM: 9358, Heap TIDs Rechecked: 0, Containers Visited: 2071.

18. VACUUM cost, entry deletion, and page reuse (format version 3)

Measured problems (bench/COMPARISON.md, bench/results/2026-09-21-stress): a VACUUM over the 1M-row, eight-index portfolio takes 1,141 ms and 316 MiB of WAL against btree’s 380 ms / 137 MiB; in the changing-key churn test the fifth-cycle VACUUM takes 1,017 ms against btree’s 41 ms, 600,000 historical entries remain for 100,000 live rows, and the two-index portfolio grows from 7.6 to 34 MiB until REINDEX.

Measured first (bench/vacuum_micro.sh, assert build, relative numbers only; the raw runs are in bench/results/vacuum-micro-before and bench/results/vacuum-micro-after). ambulkdelete reports its own breakdown at DEBUG1 - cleanup-lock waits, container filtering, GenericXLog apply, INLINE entry work, pages visited/changed, records, items rewritten/deleted - and pg_walinspect splits the WAL by resource manager. On the 1M-row eight-index portfolio with 1% of the rows deleted, the index side of the VACUUM was 696 ms of the 1,050 ms total, and it went:

INLINE entry handling                423 ms (61%)   ix_c1m alone 278 ms
container filtering (the callback)    96 ms (14%)
GenericXLog apply                     30 ms ( 4%)
bucket/chain page walking            147 ms (21%)
cleanup-lock waits                   1.0 ms

with 82 MiB of generic WAL (49 MiB of it full-page images) in 20,222 records against btree’s 58 MiB in 8,312. Two things stood out and shaped everything below.

  1. The INLINE work was almost all wasted. A VACUUM that removes 1% of the rows leaves 99% of the entries unchanged, and the old code found that out only after copying the payload out, filtering it into a StringInfo and building a new entry tuple. 632,130 entries × ~400 ns of allocation is the 278 ms. So the filter now runs a read-only PROBE first, straight out of the page (the cleanup lock makes it stable) into the shared work buffer, and only an entry that really loses a member pays for the second pass that builds its payload.
  2. One WAL record per changed ENTRY, each carrying the whole tail of its page. 20,222 records for 1,422 changed container pages: the rest were INLINE entries, one lion_replace_entry() record each, and because the entry shrank, PageIndexTupleOverwrite() moved every entry after it and the GenericXLog delta covered all of that - 33 MiB of deltas for a few hundred KiB of real change. Hence the two rules below: an entry keeps its bytes, and a bucket page’s entries are written in ONE record.

Shrink in place. Removing members from an ARRAY/RUN/segment item must not move the rest of the page: the item keeps the bytes the page has allotted it and the freed bytes become slack (§4 growth slack), so PageIndexTupleOverwrite() moves nothing and the GenericXLog delta is the item alone. An item that has lost more than LION_ITEM_SLACK_VACUUM = 256 bytes gives the space back and pays the page-tail delta once; LION_ITEM_SLACK_BOUND is the larger of that and the insert path’s bound, and is what verify() accepts and what lion_index_stats() reports as slack_bytes. A page whose filter removes nothing is never registered for WAL (already the rule). The same applies to an INLINE entry’s payload on a bucket page: a filtered payload is written back into the entry’s existing bytes with the remainder ZEROED, up to LION_ENTRY_SLACK_BOUND = 256 bytes. No length field was needed for that - no item kind has type 0, so lion_inline_fetch() stops at the first zero item header and the padding is self-describing; verify() bounds it and checks that every byte of it really is zero.

One record per bucket page. Pass 1 collects every INLINE entry of a bucket page that changed and writes them all in a single GenericXLog record. Entry offsets survive PageIndexTupleOverwrite(), so the writes do not disturb each other. A payload that outgrew its entry still spills onto container pages in records of its own (rare: it needs a RUN container to become a BITSET while losing members; “The spill” below says how many records).

Entry deletion. An entry with ntids = 0 and ncontainers = 0 is deleted by VACUUM: - The deletion happens in a final step for that bucket page, after every chain of that page’s entries has been walked, under a cleanup lock, with all of the page’s dead entries in ONE record. Emptiness is re-decided there, not from what pass 1 or pass 2 saw: an insert may have put a TID back, and then the entry stays. An INLINE entry that pass 1 emptied is still written back empty first - its dead TIDs have to leave the page under that cleanup lock - and deleted afterwards. Reserved NULL/EMPTY entries are deleted like any other; inserts recreate them on demand. - Entry offsets on a bucket page never change. This is the invariant readers rely on, and it is what lion_delete_entries() buys by going through PageIndexTupleDeleteNoCompact() instead of PageIndexMultiDelete(): the item’s bytes are freed but the line pointer array is left alone, the deleted entry shows up as an unused line pointer that every walk already skips, and nothing else moves. The freed line pointers are marked so that the next lion_add_entry() reuses them. (Deviation from the first draft of this section, which had readers copy the whole bucket page under a SHARE lock and iterate the copy. Offset stability is strictly cheaper - no BLCKSZ memcpy per page per reader - and it is the only thing the copy was for: lion_entry_scan_next() gives up its lock between two entries of one page and comes back to an offset. An INLINE posting set still keeps its bucket page PINNED until its containers have been through the visibility-map check, as §9 requires; that was always separate from the copy. It also needed no change to LionPostingSet/LionEntryScan, which live in a header this wave does not own.) - A group cannot come back twice by the other route either - the entry deleted and the key inserted again further along the page - because only an EMPTY posting set is deleted, and a posting set whose members are all dead to every snapshot has nothing the scan’s snapshot could have counted before. test/isolation/vacuum_entry_delete.spec parks a one-column and a two-column GROUP BY between two entries of a bucket page while a VACUUM deletes half of them. - Inserts are unaffected: they hold the bucket page EXCLUSIVE for the whole insert and look entries up by key.

Whole-chain free. When a CHAIN entry is deleted its chain is freed: (1) delete the entry (one record; from here the chain is unreachable for new readers and inserts), then (2) for each chain page, under ConditionalLockBufferForCleanup (skip the page on failure; it is merely leaked, and the walk stops because the rightlink cannot be read without the lock), check that the page still claims this chain and is empty, mark it LION_PAGE_DELETED, store safexid = ReadNextFullTransactionId() in the page body (as nbtree’s BTDeletedPageData), WAL-log, and keep it for the free space map, which gets it once it is reusable (“Reuse” below). Step 2 runs with the bucket page unlocked, because VACUUM never waits for a page while holding another LWLock (§11). A crash between (1) and (2) leaks pages, which is harmless: they are unreferenced and the sweep below finds them (test/recovery/run.sh phase 1b puts the crash exactly there). The leaves of a LIVE key’s set that VACUUM empties are taken out of the set one at a time and recycled the same way (§22, “Empty leaves”).

Reuse. (§22 adds one thing to the sweep below: a leaked posting set’s ROOT is an INTERNAL page with downlinks on it, not an empty one, so the sweep runs in ROUNDS - an unreferenced internal posting page is freed once every one of its downlinks names a DELETED page, which the round before made true of its leaves. A LIVE internal page can never pass that test: every leaf of a live posting set was visited by pass 2 and is therefore never swept, and a page a concurrent insert created since then is full of the items a split just put there. verify() reports such a page as the ordinary leak it is, with a WARNING.)

The free space map holds only pages that may be handed out, nbtree’s rule since PostgreSQL 14: a page out of the index’s trees - LION_PAGE_DELETED, or an UNLINKED posting leaf (§22, “Empty leaves”) - is recorded in it once GlobalVisCheckRemovableFullXid(heaprel, safexid) is true (no scan that could still hold a link to it is running), and not before. lion_alloc_page() then takes what the map offers, as _bt_allocbuf() does: GetFreeIndexPage() takes a block off the map, the page is locked with ConditionalLockBuffer and taken if it is all-zero or passes the same test, and otherwise the next block is asked for, until the map has none left and the relation is extended. A block refused is not put back. It is one the map is wrong about - the map is not WAL-logged, so a crash or a promotion can leave live pages listed, which must leave it - or a free page a reader holds locked, which the next VACUUM records again. Recording a page before it is reusable would fill the map with blocks every allocation reads and refuses: the map offers them by block, so young pages in front of old ones would have the inserts after a VACUUM extend the index while the old ones wait, and under a long snapshot every allocation would read one. heaprel comes from heapRel in aminsert and info->heaprel in VACUUM; a NULL heaprel means “never recycle”, which is what ambuildempty wants. ambuild never reuses. amvacuumcleanup calls IndexFreeSpaceMapVacuum().

So a page VACUUM frees is handed out by the first VACUUM that finds it reusable. That may be the same one: ambulkdelete keeps the pages it freed, and the ones its sweep found not reusable yet, with their safexids (bounded by work_mem, as nbtree’s _bt_pendingfsm_add(); one past the bound waits for the next VACUUM), computes the horizon afresh at its end and records those that are reusable by then (lion_vac_pending_record(), nbtree’s _bt_pendingfsm_finalize()) - a page freed early in a long VACUUM, beside short transactions, usually is. A page freed with no transaction started since is not, and the next VACUUM with rows to remove records it, as does one without them once such pages are more than a twentieth of the index (below). Leak recovery: ambulkdelete keeps a bitmap of the blocks it accounted for - the meta page, every bucket page it walked, every container page it reached from a live entry, and every page it freed - and sweeps the rest at the end of ambulkdelete: an all-zero page goes into the FSM, a DELETED page or an UNLINKED posting leaf goes there once it is reusable, and an unreferenced EMPTY container page becomes a DELETED one first, and so does an unreferenced FULL leaf whose root is not a live root of its key (the leftover of an interrupted spill, “The spill” below). A page already out of the trees is only read, under a SHARE lock, as lion_vacuum_recycle() and nbtree’s btvacuumpage() read one; only a page the sweep may free is cleanup-locked, conditionally. So a free page that the checkpointer or the background writer has pinned to write it out is recorded all the same, where a cleanup lock would leave it to the next VACUUM. The pages out of the index’s trees that are not reusable at the end of ambulkdelete - the ones it did not record - are counted on the meta page (the special area’s maxckey, nbtree’s btm_last_cleanup_num_delpages), and a VACUUM that calls no ambulkdelete reads every block of the index for them when they are more than a twentieth of it, nbtree’s _bt_vacuum_needs_cleanup() rule, recording the ones that are reusable now (lion_vacuum_recycle()). In a healthy index nothing but its free pages is unaccounted for, so the sweep reads those and nothing else. Every page ambulkdelete’s walk and its sweep read goes through VACUUM’s ring of buffers (info->strategy, lion_vac_read(); vacuum_buffer_usage_limit), as nbtree’s do, and so does every other page VACUUM reads to its own ends: the descent to the leftmost leaf that the walk, and the count of the keys in amvacuumcleanup, start from (lion_dir_leftmost_leaf_ext()); the descents and walks right that find an entry or a summary again after an insert moved it (lion_vac_ref_relocate()); the root of a set the sweep asks is live (lion_posting_root_live()); and what taking pages out of the trees reads - a directory page deletion’s leaf, siblings and parents (§21, “Page deletion”) and an empty posting leaf’s neighbour and parent (§22, “Empty leaves”), which a VACUUM that empties a run of leaves reads once per leaf. What is left reads without it, as the inserts it shares the code with do: the meta page, which every scan reads; and the pages of the writes a VACUUM makes beside its filtering through the inserts' code - a directory leaf split by an entry the filtering rewrote larger, with the parent its downlink goes to and any split left unfinished that the write finishes on the way (§21); a container that grew, put back by a split of its posting tree (lion_vacuum_regrow()), with the head a push-down copies, the parent and any unfinished split (§22); a page a payload spills to; and every candidate the free space map offers an allocation, up to the one it takes (lion_alloc_page()). So a VACUUM leaves no more of the index in shared buffers than the ring holds, besides the meta page and the pages those writes read, split or allocate - as many as the writes need, and none when its filtering rewrites nothing larger. It checks for interrupts at every block, with nothing locked, and each page it reads passes the cost-based delay first, as every page the walk reads does, so a sweep of a great many free pages answers a cancel and paces its reads and records. A page of a LIVE set is never mistaken for a leak, because pass 2 visits every page of every chain it found, and a chain created after pass 1 read its bucket page has no empty page in it (a spill and a split both fill every page they allocate inside the record that allocates it, under the lock they hold throughout) and names a root that is live. lion_index_stats() reports deleted_pages; VACUUM is told the counts below.

Page counts. IndexBulkDeleteResult has three: pages_newly_deleted, the pages this VACUUM freed; pages_deleted, the free pages the index holds; and pages_free, those the next allocation could take already. A page is counted once: a whole-set free counts the pages it marks DELETED as freed and as free and marks them visited, so the sweep passes them by, and the directory pages a deletion frees (§21) are counted as freed there and as free by the sweep, which meets them unvisited. pages_newly_deleted is ADDED UP over the calls of a VACUUM that calls ambulkdelete more than once (dead TIDs past maintenance_work_mem), as nbtree does it, and the other two describe the index as the call leaves it - the sweep counts them afresh every time, the earlier calls' pages among them - with a page reusable when it is all-zero, or DELETED with its safexid behind every snapshot, which is exactly lion_alloc_page()’s test. A page freed a moment ago is free and not yet reusable: of an index whose lion_index_stats() says 30 deleted pages, all freed by this VACUUM, VACUUM VERBOSE says “30 newly deleted, 30 currently deleted, 0 reusable”. (A VACUUM that needs no ambulkdelete at all reports them only when it reads the index for the pages an earlier one could not hand out yet, above; otherwise counting them would mean reading every page of the index.)

Statistics. num_index_tuples becomes the index’s pg_class.reltuples whenever it is not marked estimated, and the planner reads that as ROWS - a partial index’s size is estimate_rel_size()’s tuples-per-page density of it, clamped to the table’s rows - and ANALYZE writes rows there too. So ambulkdelete reports rows, not Σ ntids over every entry, which counts the NULL entry, every column of a multicolumn index, every element of a multi-key one and every SUMMARY entry (§32) all over again: every VACUUM would put a multiple of the rows there and every ANALYZE the rows back, and plans would flip between the two. It reports Σ ntids over the entries of the index’s FIRST SCALAR key column, SUMMARY entries left out, which is exact because a scalar column files every row the index holds under exactly one entry, the NULL entry for a NULL (§14, §24); for a partial index that is the count ANALYZE only estimates. An index whose key columns are all multi-key files a row under any number of entries, the key-less entries included (§17), and nothing short of their union counts its rows, so it reports the heap’s row count instead, exact when the heap’s is - ginvacuumcleanup()’s answer, and wrong for a partial index the same way. tuples_removed stays a count of postings, as GIN’s is. ambuild reports as index_tuples the rows it added, each participant’s summed (lion_spool_nrows()) - not, as GIN’s does, the postings it wrote - so that a partial index is sized by its rows from the start (§45.3).

The spill. An INLINE posting set that outgrows its entry moves onto container pages (§4, §5), and VACUUM’s filtering can make it outgrow the entry by an order of magnitude (§5, VACUUM step 2), so lion_entry_spill() writes as many leaves as the payload needs:

  1. the ROOT first, with reuse = false and held EXCLUSIVE to the end - every page of a set is stamped with its root’s block, and a root block is never handed out twice (above) - and LOGGED first, as an internal page of the set with no downlink yet, in a record of its own: an extension no record touches is one an OS crash can lose, and the next extension would then hand the same block out again as the root of another set, whose stamp the leaves the crashed spill had written could not be told from (every summary’s hash and every NULL entry’s is 0, §32). Replaying the record extends the relation past the root whatever blocks the leaves took, so the block is never a root again;
  2. then each LEAF, filled, linked to the next one and logged in a record of its own that registers that one buffer. The next leaf is allocated just before the record of the one in front of it, because that record carries the rightlink and a record may not allocate (§25), so at most two leaves are pinned at any time;
  3. then the root’s downlinks - ONE internal level: a leaf takes at least one item, a spilled payload has no more items than the INLINE payload it came from, which is at most LION_MAX_INLINE_LIMIT bytes of items of at least a header each, 512 against the 678 downlinks an 8 KB root holds (a static assertion keeps it so) - and the rewritten entry, together, in the LAST record. A set that fits one leaf is that leaf and its entry in one record, which is every spill an INSERT makes.

The entry changes in the last record and in no other, so it is INLINE until the whole set is on disk and CHAIN from the moment it is, never half of each. An ERROR or a crash before the last record loses nothing - the entry still holds the payload it held, dead TIDs included, which the next VACUUM removes - and leaves FULL leaves that nothing references, stamped with a root that has no downlink. The next VACUUM’s sweep collects them: of an unreferenced leaf that is not empty it asks the leaf’s ROOT (lion_posting_root_live()), and a leaf is an orphan when the page at its owner_head is not a live container page stamped with that same (owner_hash, owner_head), with a downlink on it when it is an internal page. Every page of a set that exists passes - its root is live until the entry is gone and the set freed, is internal only with a downlink (a push-down makes it one with one, in one record), and the whole-set free only frees EMPTY sets, whose leaves the sweep frees for being empty - and a spill that is still writing holds its root EXCLUSIVE, which the sweep’s conditional lock reads as live. A root seen unwritten, without a downlink, DELETED or stamped for another set under its lock stays that way for every leaf stamped with it, because the spill that stamped the leaf has ended and no block is a root twice. The root itself is freed by the same sweep: an internal page all of whose children are DELETED - here, none - is dead (§22). lion_index_verify() reports such a leaf as the leak it is, with a WARNING, like an empty one (which extends what §4 says verify() tolerates), and the root as an empty page. test/isolation/vacuum_spill_interrupted.spec stops a VACUUM’s spill between the leaves and the root with an ERROR (injection point lion-spill-leaves-written), and test/recovery/run.sh phase 1e with a crash.

Hot standby. Generic WAL cannot raise recovery conflicts - nbtree’s XLOG_BTREE_REUSE_PAGE has no equivalent - so on a standby a reader holding a stale chain link could land on a page that replay has already reused. Every container page therefore carries its owner in the special area, and readers validate it:

typedef struct LionPageOpaqueData {       /* version 3: 24 bytes */
    BlockNumber rightlink; uint32 minckey; uint32 maxckey;
    uint32      owner_hash;   /* owner stamp of the entry this chain belongs to (§21; 0 but for a value entry) */
    BlockNumber owner_head;   /* head block of that chain, fixed for the life of the chain */
    uint16 flags; uint16 page_id; }

Every reader that reaches a container page through an entry’s head or a rightlink (scan, count cursors, insert, VACUUM, verify) checks: CONTAINER flag set, DELETED clear, and the owner equal to the entry’s. A mismatch means the chain was freed after the reader copied the entry, which can only happen when the posting set held nothing visible to anyone: the reader treats it as the end of the chain (a count contributes nothing further). It is never an error outside verify(), except on the paths that hold the entry’s bucket page throughout - insert and VACUUM’s own walk - where the chain cannot go away and a mismatch is corruption.

owner_head alone identifies a chain, for the whole life of the index, and that is deliberate: a chain’s HEAD page is the one page lion_alloc_page() never takes from the free space map (lion_entry_spill() passes reuse = false), so a head block always comes from extending the relation, is a block number that has never been handed out before, and is therefore never reused as a head. Since §22 the head is the posting tree’s ROOT rather than its first leaf, and the rule is unchanged because a root split is a PUSH-DOWN that keeps the root at its block: head never moves for the life of the set, so owner_head is still an identity nothing else can collide with. A reused page always belongs to some other chain, whose head block differs. (Deviation from the first draft, which relied on the pair (owner_hash, owner_head) and accepted “a simultaneous 32-bit hash and head-block coincidence”. The reason for the change is not the coincidence but reachability: the count cursor’s LionPostingSet carries only the head block, not the entry’s hash, and LionPostingSet lives in a header this wave does not own - so making owner_head sufficient is what lets the count path validate at all. It is also strictly stronger than the pair check, since the freed head block is exactly the block the FSM is most likely to hand back. The cost is that a freed head block can only ever come back as a non-head page.) owner_hash stays as the second half of the check for the readers that do have the entry in hand, and is what lets verify() prove that a chain belongs to its key.

LION_VERSION becomes 3; older indexes are refused with the existing REINDEX hint. The bump is unavoidable and not merely advisory this time: the special area grows from 16 to 24 bytes, which moves every item on every page.

Measured after (same script, same cluster; bench/results/vacuum-micro-before and bench/results/vacuum-micro-after). The 78 MiB of XLOG/FPI_FOR_HINT in the WAL column is the heap’s own, identical on both sides, so the column that matters is the index’s:

portfolio, 1M rows, 8 indexes, DELETE id % 100 = 1, one VACUUM
                      VACUUM ms   total WAL   index WAL (of which FPI)   index records
  before (lion)          1050      162 MiB     82 MiB (49 MiB)              20,222
  after  (lion)           728      138 MiB     58 MiB (49 MiB)              10,198
  btree (control)         585      138 MiB     58 MiB (58 MiB)               8,311

... of which ambulkdelete itself (sum of the DEBUG1 breakdowns)
                      total    inline   filter   apply   walk   cleanup-lock wait
  before                696 ms   423      96       30     147     1.0
  after                 418 ms   186      89       28     115     0.6

churn, 100k rows, unique bigint key + two-valued key, UPDATE every key then VACUUM
                   cycle 1                          cycle 5
               ms      WAL     size   entries    ms      WAL     size   entries
  before      1343    428 MiB  14 MiB  200,000   1456   379 MiB  34 MiB  600,000
  after        135    4.6 MiB  14 MiB  100,000    140   6.2 MiB  15 MiB  100,000
  btree         58    3.3 MiB 5.8 MiB     -        60   3.3 MiB  10 MiB     -

The index WAL of the portfolio VACUUM is now exactly at btree’s, and all but 9 MiB of it is full-page images that the first touch of a page after a checkpoint costs whatever the AM is; the deltas went from 33 MiB to 9 MiB. The churn index no longer grows: entries stay at one per live row instead of accumulating a cycle of history each time, the VACUUM is ten times faster and writes sixty times less WAL, and the pages the dead keys owned are handed straight back through the free space map (deleted_pages is 0 at the end of every cycle because the next cycle’s inserts have already taken them). What is left between lion and btree on the portfolio is ix_c1m, whose 628,508 single-TID entries cost 196 ms of the 418: probing 990,000 dead-TID lookups in hash order, one bucket page at a time.

The §9/§11 proof is unchanged: TIDs are still only removed under a cleanup lock taken in chain order on every page; a page is freed only after it has been cleaned and found empty under that lock, so a reader’s pin on a page it took containers from still blocks everything that could make those TIDs' heap pages all-visible.

Measured, 2026-09-23: parallel VACUUM, and the callback asked once per TID

The focused benchmark of 2026-09-23 (PostgreSQL 18.6, release build, bench/results/2026-09-23-eb1579e-pg18-focused) had VACUUM (ANALYZE) of the 5M-row, seven-index scalar table after its maintenance sequence at B-tree 3,399 ms, GIN 3,823 ms and lion 4,374 ms. Most of that gap was not lion’s code at all: lion declared VACUUM_OPTION_NO_PARALLEL, and the benchmark runs VACUUM with the default max_parallel_maintenance_workers = 2, so B-tree’s seven indexes were vacuumed by three processes and lion’s by one. Measured separately (below), lion’s serial VACUUM was 14-20% slower than B-tree’s serial one, and its default (parallel for B-tree only) one 38-47% slower.

Set-up (release builds, -O2 and no assertions, PostgreSQL 19beta4; shared_buffers 512MB, maintenance_work_mem 512MB, max_wal_size 8GB, checkpoint_timeout 1h, fsync and synchronous_commit on, autovacuum off, max_parallel_maintenance_workers left at 2 - the benchmark’s settings). Two clusters, one without and one with shared_preload_libraries = 'pg_lion', so every index is generic-WAL in the first and rmgr-WAL in the second (§25). One run is: CREATE TABLE fact AS SELECT * FROM fact0 (fact0 is the benchmark generator’s table, bench/comprehensive/workloads.py scalar_data(), built once), VACUUM (FREEZE, ANALYZE) fact, the seven single-column indexes of ONE family (c2, c20, c200, c20k, c1m, skew, nullable; none for the heap alone), the benchmark’s maintenance sequence (INSERT of 1% new rows, UPDATE fact SET c200 = (c200 + 1) % 200 WHERE id <= rows/100, DELETE FROM fact WHERE id % 100 = 1), CHECKPOINT, and then the timed statement, VACUUM fact or VACUUM (PARALLEL 0) fact, with client_min_messages = debug1 to collect ambulkdelete’s own breakdown. The old and the new pg_lion.so were swapped (rename, restart) between every round, the configurations alternate inside a round, and the machine was shared with other work, so each cell is a median with its [min-max] and anything under 10% is noise. (The driver was a throwaway shell script around exactly these statements and is not in the tree; bench/vacuum_micro.sh measures the older eight-index portfolio above.)

Where the time goes (5M rows, rmgr, serial, old code; ms, sum over the seven indexes): 1,224 ms of ambulkdelete, of which chain-page filtering 400, INLINE entry work 618, WAL apply 94, cleanup-lock waits 1, and the walk itself the rest. Filtering is almost all the dead-TID callback: c2’s 5M TIDs filter in 82 ms, 16 ns a TID, and every index has 5M TIDs, so the callback alone is about half of ambulkdelete. nbtree asks the same callback about the same TIDs, and the API gives no cheaper question to ask (it is opaque; a range test against the dead-TID store would need the store, which is not the AM’s), so that half is the floor. The INLINE work had one real waste: an entry that changes was filtered TWICE, a probe pass to find out whether anything changes and then a second pass - asking the callback again - to build the new payload, and on c20k (twenty thousand entries of 250 TIDs) every entry changes. A cycle-counter split of pass 1 (temporary, on the new code) shows what is left: on c1m (a million entries of five TIDs) 40% is filtering the unchanged entries, 41% writing the changed leaves - one record per leaf, each with its full-page image after the checkpoint, the same as nbtree pays per leaf - and 9% the 95k entries that change; on c20k 62% is filtering the entries that change, now once, and 30% writing them.

What changed:

  1. amparallelvacuumoptions = VACUUM_OPTION_PARALLEL_BULKDEL (§6, lion_am.c). §11 has the argument for why the interlock holds whatever process runs an index’s ambulkdelete; test/isolation/vacuum_parallel.spec makes a worker run one (it reports f with the old setting) and checks the result. amvacuumcleanup stays with the leader: it only vacuums the FSM.
  2. One pass over an INLINE payload: the callback is asked about every member once (lion_vacuum_inline_filter()). The items in front of the first one that loses a member are copied from the page when it turns up; lion_container_optimize() is a function of the member set, so the payload built is byte for byte the old one (test/sql/vacuum.sql, “INLINE entries whose FIRST items lose nothing”, drives the prefix copy through ARRAY, RUN and sparse items).
  3. An INLINE entry that is not about to be deleted is not kept past pass 1 and its key is not copied: one allocation less per entry. Not measurable on its own.

Neither protocol nor WAL changed: the WAL of every run is identical before and after (the column).

VACUUM wall ms, median [min-max]       old lion              new lion              btree          WAL MiB (lion / btree)
1M rows (5 rounds)
  rmgr,    default (parallel)       561 [492-619]         471 [451-501]         370-392 [354-454]   139 / 136
  generic, default (parallel)       658 [640-694]         505 [497-552]            -                147
  rmgr,    PARALLEL 0               596 [568-651]         537 [520-596]         483-504 [452-544]
  generic, PARALLEL 0               654 [590-718]         662 [611-2027]           -
  heap alone (no index)             301-329 [275-375]
5M rows (3 rounds)
  rmgr,    default (parallel)     3,350 [3061-4012]     2,638 [2568-2794]     2,314-2,544 [2223-3628]  579 / 644
  generic, default (parallel)     3,347 [3283-4027]     2,648 [2551-3062]          -                580
  rmgr,    PARALLEL 0             3,391 [2991-3466]     3,775 [3422-4837]*    2,888-2,983 [2791-4800]
  heap alone (no index)           1,781 [1650-3370]     (1,509 min)

ambulkdelete, sum over the seven indexes (serial runs), median ms
                 old rmgr   new rmgr   old generic   new generic     of which c20k (rmgr)
  1M rows          293        262          372           350             44 -> 32
  5M rows        1,224      1,143        1,329         1,256            278 -> 200

(*) the new serial 5M cell and the second heap-alone cell were hit by other work on the machine - their minima are in line with the old ones; the ambulkdelete sums, which only time the index, went down. A B-tree cell gives two medians: its runs alternated with the old and with the new lion build, and it is the same B-tree both times.

So the VACUUM of the benchmark’s portfolio is now 16-23% faster in both WAL modes: 4-14% slower than B-tree’s at 5M rows (from 38%), 20-27% at 1M (from 47%). What is left is shape, not overhead: with three participants the wall time is bounded below by the longest index, and c1m - 632k INLINE entries at 1M rows - is that index there (133 ms of the parallel run’s 471). Its cost is one callback per TID, one pass over each entry and one full-page image per leaf, which is what nbtree pays for the same column. Not done: the REGROW path (§18, lion_vacuum_regrow()) filters a container that outgrew its slot a second time - on skew, 561 containers at 5M rows, about 30 ms of that index’s 128 - and could instead reuse the first result when the container’s bytes on the page are unchanged; below the noise here, and it touches the one path that re-places containers, so it waits for a workload where it matters.

19. OR across columns in the count pushdown (v1, implemented)

count(*) WHERE c200 = 17 OR c20 = 3 used to be declined and ran as BitmapOr + Bitmap Heap Scan (19 ms at 1M rows on the optimized build, the same as btree, because 5% selectivity touches nearly every heap page). The count evaluator already evaluates AND/OR trees over posting-set cursors (multi-key classes, IN lists), so the whole of this is a planner and a plumbing problem.

Planner. A top-level BoolExpr OR in baserestrictinfo is accepted when every arm is an accepted positive clause (equality, IN, IS NULL, exact multi-key) or an AND of such clauses, each on a plain column of the same relation; the collation, opfamily, cross-type, Param, IN-list-cap and per-partition rules are exactly the ones a plain clause goes through, because the analysis IS the same function (lion_analyze_leaf()). An arm of negated clauses (IS NOT NULL, <>, IS DISTINCT FROM, NOT of a multi-key query) is a COMPLEMENT arm, counted by §35’s De Morgan rewrite (“Complements in an OR”); an arm that ANDs a negated clause with a positive one is declined. So is an arm that is not indexable, one that would need a recheck (b > 3, a non-roaring operator, an ALL-mode multi-key query), and an AND arm one of whose leaves is any of those.

The OR is ONE count source whose expression tree has leaves in different indexes, and it is ANDed with the remaining clauses and with the GROUP BY driver exactly like any other source. Several OR restrictions in one query are several sources.

How it is carried. The leaves are appended to the flattened clause array alongside the plain clauses, contiguous and in arm order, and the structure over them travels in a new custom_private member, LION_PRIV_ORS: one IntList per OR of {first clause index, number of arms, leaves in each arm} (shape marker bumped to 4, LION_PRIV_NMEMBERS 9). Flattening is what makes everything else free: lion_collect_targets() matches an index for each leaf per partition, a Param in a leaf lands in custom_exprs where set_customscan_references() and SS_finalize_plan() can see it, and the cost model prices each leaf’s bucket page and chain. What the executor derives from the structure is inor[] (a clause is a leaf of an OR) and one item per source: a plain clause, or a whole OR.

Leaves constrain no column of the result. The other arms select rows this arm says nothing about, so an OR leaf enters none of the per-column bookkeeping of §10/§14: not posattnos (so a = 1 OR a = 2 is legal, and the one-positive-clause-per-column rule does not compare an OR leaf with a top-level clause), not eqattnos (so the pinned-column echo never prints a key from an OR leaf’s entry), and not nonnullattnos/nullattnos (so count(n) WHERE n = 2 OR a = 17 is NOT answerable and the query is declined - it would otherwise come out as count(*)).

Cost. Each leaf is charged its own bucket page and container chain, as if it had been a separate clause, plus the §15 union term for the merge they take part in: members × log2(nleaves) at cpu_operator_cost, where members is the sum of the leaves' own selectivities times the relation’s rows. lion_cost_count_rel() therefore takes the whole clause list and skips the negated ones itself, rather than being handed a filtered list, because the OR structure names its leaves by position.

Safety. For a union a dead TID may be contributed by a single leaf, so EVERY leaf under an OR must be read the pinned way until the merged container has been through the visibility-map check (lion_source_pinned(): OR → every child, which is what decides whether the source carries the §9 interlock at all). Leaves under an OR are therefore never materialized: LionCountSource gains nomaterialize, which lion_locate_or() sets and which is rule 3 of the materialization decision in lion_count_sources_cached(). The §9 argument would in fact still hold for a stale copy under a union - the intersection is only counted from the visibility map while some positive source is pinned, and a TID that survives into the counted container is in that source’s set too, so its index’s ambulkdelete is blocked - but keeping “every leaf of a union holds a pin” true by construction means the interlock can be checked by reading lion_source_pinned() alone. §15’s IN lists are unaffected and may still be materialized.

Executor. lion_locate_leaf() is one clause’s whole run-time meaning: it locates that clause’s posting sets and returns the tree over them, its leaves numbered from 0, or NULL when the clause selects nothing (a NULL Param, an empty IN list). lion_locate_or() calls it per leaf, concatenates the sets into one array for the source and renumbers each leaf’s tree onto its slice (lion_shift_keynos()), then builds AND nodes for the arms and one OR node over them. An arm with a leaf that selects nothing is dropped - which is how a NULL Param arm selects nothing while the OR of the rest remains - and an OR with no arm left sets wheremissing. A leaf whose key simply has no entry keeps its set and its leaf node; lion_sets_satisfiable() then answers for the whole tree. Pins: the OR’s sets are released by lion_release_where() with every other source’s, on ReScan, at End, and at the end of each partition’s turn.

EXPLAIN prints the arms after the indexes of all the leaves: Lion Indexes: ix_c200, ix_c20 ((c200 = 17) OR (c20 = 3)), an AND arm as ((a = 1) AND (b = 2)), and - on a partitioned scan, where there is one index per partition - the expression alone. A Param is deparsed as $1 like any other clause value.

An OR of equalities on one column is no union of leaves (2026-09-28). k = 1 OR k = 7, the arms one operator, one collation and one constant type, is the IN list it spells (lion_or_as_array()) and takes a list’s path through everything above and §15: one lookup walk, the disjoint sum when it is the only source, the batches past the budget, the list’s price. Only an OR the list does not answer - another operator, a Param, a second column, a range arm - is an OR of leaves as described here. §29.11, “An OR of equalities is its IN list”, has why.

Nested ORs, distributed (2026-09-27). The structure above is an OR of ANDs of leaves, one level deep, because that is what a source’s tree is built from. An AND arm with an OR inside it used to decline the whole query - and flag IS NOT TRUE is one (flag = false OR flag IS NULL, §10), so (status = 'val1' AND flag IS NOT TRUE) OR country = 'c1' was never counted while the same test at the top level was. lion_or_arms() now puts the restriction into disjunctive normal form before the leaves are analysed: an OR’s arms are its operands' arms, an AND’s are every combination of one arm of each operand, and flag IS NOT TRUE / IS NOT FALSE are the ORs they stand for wherever they appear, so that one is ((status = 'val1') AND (flag = false)) OR ((status = 'val1') AND (flag IS NULL)) OR (country = 'c1') - which is what EXPLAIN prints. The union of the arms is the same set of rows, and nothing downstream changes: the arms are ordinary arms, flattened and carried as before. Distributing repeats a term in every arm it is distributed into, and each repetition is a leaf of its own - its own lookup, priced as such, and its own set of the union - and it multiplies arms: an AND of k two-way ORs is 2k arms of k leaves. So the leaves of one OR are bounded by LION_MAX_ARRAY_ELEMS (§15): every set of the union may hold a buffer pin for as long as the node runs, and a union’s leaves are never materialized (above). Past it the query is declined; the bound applies to an OR as written as well, which no ordinary query comes near. The other option, an OR node inside an arm, would have kept a term to one lookup, at the price of a second level in LION_PRIV_ORS, lion_locate_or() and the model’s union terms; the distributed form is the one the rest of the machinery already knows, and the repeats cost what the model says they cost. test/sql/countclauses.sql covers both boolean tests in an arm, both in one arm (four arms, three of which select nothing), each in an arm of its own on both sides of the OR, an OR of other columns nested in an arm, a GROUP BY and a clause ANDed beside the OR, three nested ORs in one arm, and ten, which would be 1,024 arms of 11 leaves and is declined.

Measured (2026-09-21, 1M rows of (c200, c20), 18,182 heap pages all-visible, assert build): count(*) WHERE c200 = 17 OR c20 = 3 returns 55,000 rows in 0.55-0.62 ms against the BitmapOr + Bitmap Heap Scan’s 18.3-30.8 ms, reading 570 containers, skipping all 18,182 heap blocks via the visibility map and rechecking none. A three-arm form is 0.75 ms against 21.4 ms. test/sql/pushdown.sql covers two- and three-arm ORs, an AND arm, an OR beside a plain clause and under a GROUP BY, two ORs in one query, IN and IS NULL and multi-key arms, arms whose value has no entry, Params under force_generic_plan (including NULL ones), an exec Param under a LATERAL nested loop, a partitioned table, a dirty heap and an all-visible one, and the negative cases above - each compared against the same query with the pushdown off, as a multiset both ways round.

20. GROUP BY two indexed columns (v1, implemented)

SELECT a, b, count(*) FROM t [WHERE ...] GROUP BY a, b is a nested loop over the two indexes' entries. Everything §10 says about ONE grouping column is said again per column: its own scalar lion index, its own collation, its own grouping equality (SortGroupClause.eqop = strategy 1 of that index’s opfamily), its own value-representation gate (lion_index_can_emit_value() when the key is printed), and - for a partitioned parent - all of that per partition. Three or more columns are the decoded walk of §34; the same column twice is declined.

Which one drives. The column with FEWER estimated distinct values is the OUTER one: its entry scan drives the node exactly as a single group column’s does, and the other index is the INNER one. The choice is made at plan time from estimate_num_groups() per column, and the two are swapped before anything else looks at them, so groupattno/groupidxoid are always the outer pair and EXPLAIN names them first. The choice does not change the total intersection work (below); what it decides is how many entry scans there are and whose keys are held in memory.

The loop (lion_next_group2()). For each outer entry: hold its key and its posting set in outercxt for the whole of its inner loop; for each inner key, locate the inner posting set, count the AND of (outer group, WHERE sources, inner group) through lion_count_sources_cached(), release the inner set, and emit (outer key, inner key, count) when the count is above zero. At most two group pins exist at a time, however many distinct values either column has, which is the pin budget of §9 - and none while a row is up with the executor: the outer set gives its pin up then and is a NOPIN copy for the rest of its loop, whose pairs the inner sets carry (§15, “Paused and finished counts”). NULL groups are each column’s reserved NULL entry (§14) and come out like any other; count(col) of a group column is 0 in THAT column’s NULL group, which is why there are two target-list kinds for it.

The inner keys, and why only the keys (lion_load_inner_keys()). The inner index’s entries are read ONCE per relation and only their KEYS are kept, in innercxt. An entry’s head block or INLINE payload would be worthless without the buffer pin that goes with it (§9), and holding one pin per distinct inner value for the length of the scan is exactly what the pin budget forbids - so each pair re-locates its inner posting set with lion_posting_set_lookup(), one bucket page, and the bucket count is sized to the index’s entry count, so that lookup is a shared-buffer hit for every index small enough for the cost model to have chosen this plan. The alternative - walking the inner index’s entry scan once per OUTER group - needs no memory but re-reads every bucket page of the inner index outer_entries times; it is kept as the FALLBACK for when the keys do not fit the work_mem budget (checked every 256 keys with MemoryContextMemAllocated()), because the §16 lesson is that a plan-time bound is only as good as estimate_num_groups. inner->key == NULL selects that path, and then the inner key lives in the per-pair context like a single-column group’s does. The walk itself does not: it outlives every pair of its outer group, and the contexts its position and batch live in are created under whatever context is current when it begins, so it begins in the query’s context. It used to begin inside the per-pair context, whose reset before the next pair deleted them under the walk - a leaf copied into freed memory, and the contexts deleted twice at the end (2026-09-27 review; test/sql/group2_lowmem.sql runs the fallback, plain and partitioned).

Cost (lion_cost_count_rel()). Three terms on top of §10’s:

  • one cpu_tuple_cost per (outer, inner) PAIR, for the inner lookup and the per-pair bookkeeping;
  • the INTERSECTION, which is what decides. ANDing two containers costs about the members of the smaller of them, so one pair costs about Min(rows/outer_entries, rows/inner_entries) member steps and the sum over all outer_entries × inner_entries pairs is exactly Min(outer_entries, inner_entries) × rows - independent of which column drives;
  • and both sides' container bookkeeping at every pair, pairs × (containers(outer group) + containers(inner group)), which is what makes a pair of widely SPREAD groups expensive even when their intersection is empty: two groups whose rows are scattered over the whole heap have a container at nearly every container key and the merge steps through all of them.

A WHERE source under either grouping is priced as §22 prices a probed source, which for a grouping means one read of it and nothing per pair: the pairs probe it far more often than it has pages, and the copy §9 makes of it on its second use is probed in memory. So none of the three terms above moved when §22 recalibrated the pages, and neither did any cardinality this section accepts or refuses.

Measured (2026-09-21, 200k rows of a 100-byte-wide table, uncorrelated columns, assert build), node against the sequential aggregate: 20 × 2 groups 5.6 ms against 37.3 ms, 200 × 2 14.9 against 39.3, 200 × 20 60.7 against 36.8, 20000 × 2 169.8 against 51.6, and 20000 × 200 far worse. The model chooses the first two and refuses the rest, which is where the measurements say the crossover is. Note the deviation from this section’s sketch, which expected 200 × 20 to win: on UNCORRELATED data it does not, because the member term above is 20 × 200,000 = 4M container-member steps against 200,000 heap tuples for the sequential scan. Clustered data would change that - a posting set confined to a few container keys makes most pairs' merges terminate at once - and the model does not model correlation, so it refuses those too. That is conservative, and it is the open item this section leaves.

Partitions work exactly as §16’s single-column case: one partial aggregate per (outer, inner) pair per partition, streamed as each partition is counted, with core’s Finalize HashAggregate on top combining them; both grouping columns must be hashable. LION_PRIV_PARTS carries the inner index Oid per partition next to the outer one. Note that a partitioned plan leaves BOTH index Oids in LION_PRIV_OIDS invalid, so nothing in the executor may decide “is there a second group column?” from an Oid - groupattno2 != 0 is the test, and getting that wrong was a real bug: the inner group’s source was left out of the count and every pair came back with the outer group’s count.

EXPLAIN prints both indexes and both columns: Lion Indexes: ix_b (b), ix_a (a) and Group Key: b, a, in outer-then-inner order. test/sql/pushdown.sql covers the results against the ordinary plan (as a multiset both ways round) with and without a WHERE clause, with NULLs in either column, with count(col) of either, under an OR restriction, on a dirty heap and a clean one, on a partitioned table, and pins the plan choice for 20 × 2, 200 × 2, 200 × 20, 20000 × 200 and 20000 × 2 with nothing disabled.

21. Sorted key directory (format version 4, implemented)

Replaces the hash-bucket entry directory of §4 with a B-tree of entries keyed by the index key.

Why. The hash directory had three measured costs: an index created empty kept its 64 buckets forever (100k unique keys: 0.104 ms vs 0.321 ms per 100-key IN after growth), a large IN list was one random bucket page per value, and entry iteration (GROUP BY, IS NOT NULL, verify) was in hash order, so ordered output and range-bounded entry walks were impossible. A B-tree directory grows by splitting, makes a sorted IN list a near-sequential leaf walk, and gives GROUP BY output in key order (the Sort above the node disappears when the query’s ORDER BY is the group key).

Structure. Meta page (block 0) points at a root and records the root’s height and the number of dirpages the directory has; nbuckets is gone (the field is kept at zero for layout stability). Directory pages carry LION_PAGE_DIR (internal) or LION_PAGE_BUCKET (leaf; the flag name is kept so verify/stats/page-inspection tools keep working), plus LION_PAGE_ROOT on the current root and LION_PAGE_INCOMPLETE_SPLIT on a page whose split has not finished. The special area gains level and a leftlink (24 → 32 bytes after MAXALIGN, which is why every item on every page moves and the version has to be bumped). Leaf pages hold LionEntryTuple items in key order, exactly as bucket pages did; internal pages hold downlink pivots, LionEntryTuple-shaped with LION_ENTRY_DOWNLINK and the child block in the header’s last word, so the key helpers apply unchanged. Every non-rightmost page’s FIRST item is its high key (LION_ENTRY_HIGHKEY, key only), and every internal page begins with a LION_ENTRY_MINUSINF downlink, with no key, that compares below everything (“Height” below). Reserved entries sort first (NULL < EMPTY < EMPTYDIM < EMPTYINNER < NULLELEMS < value) via a kind prefix in the comparison, not in the stored key, so they live on the leftmost leaf and are reached by the ordinary descent. Multi-key classes: the key type is the STORAGE type (§17), which is what is compared.

An entry tuple is capped at LION_MAX_ENTRY_SIZE, which is the largest item a page can hold minus the largest pivot (LION_MAX_PIVOT_SIZE, MAXALIGN(DOUBLEALIGN(LION_ENTRY_SHORT_HDRSZ) + LION_MAX_KEY_SIZE) + sizeof(ItemIdData)), because a leaf that holds ONE entry must still have room for the high key a split would give it. A payload that would exceed it spills onto container pages exactly as one that exceeds inline_limit does. (Not in the first draft of this section, which did not notice that a 2000-byte key with a 4096-byte inline payload plus a 2000-byte high key is 40 bytes over a page.) (Nor did the first build: ambuild spilled at inline_limit alone, so an INSERTer could leave a posting set that INSERT had spilled but REINDEX, VACUUM FULL or a restore kept INLINE and could not write - 3900 rows of one 1990-byte key made a 6092-byte entry. Build and INSERT now take the bound from one function, lion_inline_max(); the 2026-09-23 review found it.)

Entry tuples (format version 10)

Every item of the directory - a leaf entry, a high key, a downlink - is a LionEntryTuple (lion_entryfmt.h, which needs nothing but c.h so that test/unit/entry_test.c checks it against the offsets given here). There are two header forms, chosen by the flags:

short header, 12 bytes: INLINE entries, high keys and downlinks
    0   uint16  flags
    2   uint16  keylen          bytes of key data (0 for a key-less entry)
    4   uint16  attno           key column (§24); 0 on the minus-infinity downlink
    6   uint16  ncontainers     INLINE: containers in the payload; 0 on a pivot
    8   uint32  ntids / child   INLINE: TIDs in the payload; downlink: the child block;
                                high key: InvalidBlockNumber

CHAIN header, 28 bytes
    0   uint16  flags
    2   uint16  keylen
    4   uint16  attno
    6   uint16  zero
    8   BlockNumber head        root of the posting tree (§22)
    12  BlockNumber tail        its last leaf (the append hint)
    16  uint64  ntids
    24  uint32  ncontainers

then the key, at the end of the header - or at the next 8-byte boundary with LION_ENTRY_ALIGNKEY - then, with LION_ENTRY_POSITIONS, the 8-byte positions extension (LionEntryPosExt: pos_root, pos_len, §17) at the next 4-byte boundary, then the INLINE payload, unaligned, to the end of the item. A CHAIN entry ends where its payload would begin, and its item length is exactly that (lion_page_entry_fetch() and verify() check it). An item is placed on a MAXALIGNed page offset, so the header’s fields are aligned; nothing after them is padded beyond what is said here. LionEntryHdrSize(), LionEntryKeyOffset(), LionEntryExtOffset() and LionEntryPayloadOffset() compute the offsets from the flags and the key length, and the counts and links are read and written only through lion_entry_ntids(), lion_entry_ncontainers(), lion_entry_head(), lion_entry_tail() and their setters, since they live in different places in the two forms.

So an INLINE entry of an int4 key costs 16 bytes before its payload, of an int8 key 20, and a CHAIN entry of an int4 key is 32 bytes in all.

The key is stored at its type’s width. A by-value key is typlen bytes (1, 2, 4 or 8), written from the Datum’s low bytes and read back as fetch_att() would, sign-extending 2- and 4-byte values; a by-reference key is as lion_store_key() describes (§4). A by-reference key of a type whose alignment is ’d' - interval, timetz, a composite or range of such - is placed at an 8-byte boundary, which the flag LION_ENTRY_ALIGNKEY records: the opclass functions read the key in place and may rely on the type’s alignment. Every other key is at most 4-byte aligned, which the 12- and 28-byte headers give, and a by-value key is read with memcpy(). The flag is a function of the column (lion_key_layout_flags()), every entry and pivot of a column carries it or none does, and verify() checks that; a pivot copies it from the entry it was made from.

The short form’s counts are enough. An INLINE payload is at most LION_MAX_ENTRY_SIZE bytes, so it holds fewer than 65536 containers of at least 8 bytes, and fewer than 232 TIDs since each holds at most 32768. Both are static assertions next to lion_entry_set_ntids(), which refuses a count that does not fit its field as corruption rather than truncating it. A CHAIN entry’s counts are 64 and 32 bits wide. Turning an INLINE entry into a CHAIN one (lion_entry_chain_shape()) rebuilds the header in the other form and moves the key and the extension with it.

No entry stores its hash. The order of §21 has the hash as its fourth term, and it is a function of the key, so the comparison computes it from the stored key with the column’s hash function when it needs it: when proc 4 ties, which in an ordered column is a run of equal keys and happens once or twice a descent; and at every step in an unordered column, where it stands in for the proc-4 call an ordered column makes. Reserved entries and summaries take 0 for it. The owner stamp of §18 on posting and position tree pages is hash_bytes() of the stored key bytes for a value entry and 0 for any other (lion_owner_stamp()), which needs no opclass and lets verify() and every reader check a page’s owner from the entry in hand. Storing either would cost four bytes on every entry to save a hash function call that the ordered descent almost never makes.

The format is version 14 (LION_VERSION_V14: the EMPTYDIM and EMPTYINNER entries of §17, and NARROW items without a meta page flag, §38): these entry tuples, with keys of up to 2047 bytes (§4), under internal pages that each begin with a minus-infinity downlink (“Height” below), and the pages VACUUM takes out and hands out again (“Page deletion” below, §22 “Empty leaves”). An index of an older version is refused with the REINDEX hint: its entries would be read at the wrong offsets, its long jsonb keys looked for in the wrong form, (version 11) its directory may be of any height, or (version 13) its empty arrays with dimensions are in the EMPTY entry and its meta page may carry the NARROW flag this format does not have.

An index of another format

Refused by its readers, such an index was still planned with, and the planner reads lion indexes for its estimates whether or not the plan uses them - a column’s n_distinct from the meta page (§33), a range’s entries, the directory’s pages (§27). So once a build of another format was installed, every query that so much as estimated a column such an index covers failed with the index’s error until it was rebuilt: a count by another column, through the count pushdown’s look at the indexed one, a GROUP BY of it, a join on it. The hook that follows get_relation_info() - get_relation_info_hook, and from PostgreSQL 19, which replaced it, build_simple_rel_hook - now leaves such an index out of rel->indexlist (lion_drop_other_format(), lion_am.c), as the planner leaves out an index that is not valid - the hook’s documented use, removing an index: no path of core’s or lion’s reads it, and the table’s queries run without it, by core’s plans. The backend warns of it once (an OID list in TopMemoryContext), so that a query slower than it was says why. Its readers refuse it as before - an insert into the table, VACUUM, lion_index_verify() and the SQL functions - so that a table written to cannot go on without its index unnoticed, and a REINDEX gives it back to the planner.

  • Which index. lion_index_other_format() (lion_meta.c) reads the meta page as lion_read_meta() does - a special area of another size still read for its magic and version - and says yes only to lion’s magic with a version not this build’s: a damaged meta page is not another format, and its readers' errors stand. Not a partitioned index, which has no storage, nor a hypothetical one.
  • Its cost. One pin and a share lock of the meta page per lion index of each relation planned, none for an index whose state the backend has built (rd_amcache): that state was read through the version check, and a library of another format is loaded only by a new backend.
  • Loading. When the library is not preloaded it is loaded inside the get_relation_info() that opens the first lion index, and _PG_init installs the hook before that call reaches it: the table that brought the library in is the first the hook sees.

test/sql/corrupt_items.sql, section 10: an index whose meta page says version 13, in a new backend - the warning, the count, a GROUP BY and a join on the column planned without it, an insert refused, and after the REINDEX the index planned again.

The order

KEY COLUMN (§24), then kind (MINF < NULL < EMPTY < EMPTYDIM < EMPTYINNER < NULLELEMS < VALUE),
then proc 4 under the column's
collation, then the hash (computed from the key, "Entry tuples" above), then a bytewise
comparison of the stored datum

The leading key-column term is §24’s; a single-column index has one value for it and everything below reads as it was written. It comes FIRST so that each column’s entries are one contiguous run: a lookup is an (attno, key) descent, an entry scan bounded to one column is the leaf walk from its first entry to the first entry of the next column, and an item of another column is settled by that one comparison without calling a single opclass function - which is what lets one descent cross the entries of columns whose key types it knows nothing about. The minus-infinity downlink carries column 0 and therefore still sorts below everything.

The kind and what follows it are the prefix within one column. Two entries whose prefixes tie are candidates for being the same key, and only the opclass EQUALITY decides: a descent lands on the first item of the prefix run and scans it - across right links if it spans pages - applying strategy 1. That is what makes citext’s 'Alice' and 'alice' ONE entry: citext_cmp returns 0 and citext_hash agrees, so they tie, and citext’s = says they are the same key. The bytewise tail exists so that the order is TOTAL even for a type with no btree opclass at all, where the prefix is only (kind, hash); it is never what decides that two entries are distinct, because entries in one prefix run are found by equality.

(Deviation from the first draft, which said the order was simply “kind, then cmpproc, then hash, then memcmp”. Stopping at the bytewise tail would put citext’s two spellings at two different positions and therefore in two entries - the very thing the section requires a test for. The run scan is what reconciles the two, and it is nbtree’s own handling of duplicate keys.)

A search key whose stored form is not available - every lookup, which has a Datum and not the bytes, and every cross-type lookup, which cannot produce them at all - compares as the SMALLEST member of its own run, which is exactly what positions it at the run’s start. The INSERT path does have the bytes (it has just built the entry tuple), and uses them to place the new key at its exact position, which is what keeps the on-disk order total.

Every rule below is per key column (§24): each column has its own opclass, so its own key type, hash, equality, comparison, collation, multi-key extraction and its own answer to “ordered?”. LionState is that per-column state and LionIndexState holds the meta page and the array of them; lion_get_state() hands back column 1, which is all a single-column index has.

The ordering source. Every default opclass gains support proc 4, the btree comparison of the KEY type: the type’s default btree opclass’s support function 1, named in the SQL script (bttextcmp, numeric_cmp, …; enum_cmp for anyenum, citext_cmp for citext, bttextcmp for tsvector_ops, whose STORAGE type is text). array_ops has none - its STORAGE type is the polymorphic anyelement - and falls back to the element type’s default btree comparison through lookup_type_cache(TYPECACHE_CMP_PROC_FINFO) at lion_fill_state(), exactly as its hash does, and under rule 2 below: its keys are compared with the element type’s own default equality, which is the one that borrowed comparison agrees with. lionvalidate() accepts proc 4 as OPTIONAL, with signature (keytype, keytype) → int4, and is the one support function allowed to be cross-type.

A type with no btree opclass at all - xid, cid - has no ordering: LionState.ordered is false, the directory is ordered by (kind, hash, bytes), which is a complete order but not the type’s, lion_index_stats() reports ordered = false and the count pushdown claims no pathkeys.

Three rules decide what the directory is ordered by (lion_fill_state()), and all three say the same thing: the order the BUILD lays entries out in and the order a SEARCH descends must be one and the same, and both must agree with the opclass EQUALITY, because the directory holds exactly one entry per equality class.

  1. An opclass WITH proc 4 is ordered by it.
  2. An opclass WITHOUT proc 4 may BORROW the key type’s default btree comparison only when its own equality operator IS the key type’s default btree equality (compare the strategy-1 operator OID against get_opfamily_member() of the type’s default btree opfamily). Otherwise it is UNORDERED. (Deviation from the first draft, which borrowed unconditionally. The borrowed comparison can be FINER than the opclass’s equality, and then the two members of one equality class sort to two different positions while the directory keeps only one entry: the case-insensitive lion_lower_ops of test/sql/pushdown.sql stores 'A' and 'a' in one entry, bttextcmp separates them, and a lookup for the spelling the entry was NOT created with descended past it and found nothing. An unordered directory ties wherever the hash ties, which is exactly where the run scan applies the opclass equality, so the same class works.)
  3. Either way the ordering needs a < OPERATOR THAT SORTS WITH THE SAME FUNCTION, because ambuild sorts through the SortSupport an operator names (its tuplesort did, before §24 “Build”). The key type’s default < qualifies only when the comparison is the key type’s default one; a comparison of the opclass’s own is looked for in the btree opfamily that uses it as its BTORDER_PROC (a pg_amproc scan, once per relcache build, and only for an opclass that names a comparison no built-in one does). The candidate is verified with get_ordering_op_properties(), which is the same catalogue path PrepareSortSupportFromOrderingOp() takes, so “the sort will use this function” is not a guess. An ordering the build cannot reproduce is NO ordering: the index is unordered instead, which is still correct. (Deviation from the first draft, which took the type’s default < (TYPECACHE_LT_OPR) whatever the comparison was, so an opclass with a comparison of its own sorted one way at build time and searched the other: every key ended up where no search looked for it, and verify() said so.)

The ordering must agree with strategy 1: two keys the comparison calls equal must be equal to the opclass. The directory holds one entry per equality class and finds it by scanning the run the comparison ties, so a comparison FINER than the equality (one that separates two equal keys) would create two entries for one class. Rule 2 above is what keeps a BORROWED comparison inside that promise; a proc 4 an opclass names itself is still the author’s responsibility, and the AM cannot detect a bad one.

The order is the index’s, not the catalog’s (2026-09-24 review of §28). The three rules above decide how a BUILD lays the directory out; they used to be applied again at every relcache build, from the catalog as it stands, and nothing stored the answer with the index. The catalog can change under an existing index: a btree opclass created later whose support function 1 is a lion opclass’s proc 4 makes that proc 4 sortable (rule 3), so a directory built in HASH order was read in value order - a BETWEEN 100 AND 2000 counted 10 rows for 19,010, a = 1234 0 for 10, and an insert put its entry where no later search looked - and dropping that btree opclass does the reverse. A proc 4 added to a family by ALTER OPERATOR FAMILY ... ADD FUNCTION is a loose member that can be dropped and replaced by another function, and a borrowed comparison (rule 2) follows the key type’s default btree class, which can change too.

So the build RECORDS the order on the meta page, in three words of what was reserved space (order_flags, ordered_cols, order_ident; format version unchanged, as for §25’s wal_mode): which key columns are ordered, and a hash of the comparison each ordered one was built with - of what the function RUNS (lion_proc_ident()): prosrc, which is the C symbol of an internal or C function and the body of a SQL or PL one, and probin, the library of a C function. Neither an Oid nor a name would do. pg_upgrade keeps an index’s files but recreates its user functions under new Oids; a schema-qualified name made ALTER EXTENSION citext SET SCHEMA - citext is relocatable, and its type and citext_cmp move with it - brick every citext lion index until REINDEX (the first version of this fix did exactly that, the coordinator’s review caught it); an unqualified one would do the same to ALTER FUNCTION ... RENAME. The source survives all three and still catches the common ways of changing a comparison: a loose proc 4 swapped for another function, a string body replaced by CREATE OR REPLACE, a borrowed comparison now taken from another default btree class. The argument types are left out on purpose: they are the key type, which the index fixes, and a moved or renamed type is the same type. A RENAME of a user opclass’s proc 4 therefore changes nothing, and replacing its body is an ERROR until REINDEX, which is the intended pair.

It is a best-effort guard, not a guarantee (the 2026-09-24 external review of 118f623 found two gaps, both fixed below, and the limits are stated here). A function’s source does not include what it CALLS: a string body that calls another user function, and that function replaced later, compares differently with an unchanged source; so does a C function whose shared library is swapped under the same symbol. Nothing short of knowing what a function does could catch that, and core’s btree has exactly the same exposure - PostgreSQL’s rule is that changing the behaviour of an operator class’s function requires a REINDEX of every index that uses it - so that is this index’s rule too. The guard catches the common, direct changes; it does not certify the rest.

  • SQL-standard bodies are never ordered by. A RETURN function keeps its parsed body in prosqlbody with an empty prosrc, so replacing an ascending body with a descending one left the identity unchanged and an existing index answered 0 for 10. The tree cannot be hashed as it stands - it embeds Oids that pg_upgrade does not keep for user objects - and neither can its deparse, which depends on the session’s search_path, so two sessions would disagree. A normalised hash that replaced every user-object Oid by a stable identity would work and is far more code than the case deserves. So a BUILD whose comparison has a SQL-standard body lays that column out in HASH order, with a NOTICE naming the column and the fix (a string body or a C function). That is chosen over an ERROR because a hash-ordered column is still a correct and fully usable index - equality descends by hash, ranges take §28’s test-every-entry walk, only the count pushdown’s pathkeys are lost - and no later change of the comparison can make a hash order wrong; refusing the CREATE INDEX would only take the index away. A body that BECOMES SQL-standard later is seen, since prosrc changes to the empty string. The one flip left is an index built before the order was recorded (order_flags = 0) with a SQL-standard comparison: it is read from the catalog, which now says hash order; REINDEX it.
  • A backend that already has the index open checks too. The state is cached in the index’s relcache entry (rd_amcache), and CREATE OR REPLACE FUNCTION invalidates pg_proc, not the index, so an open backend went on reading an ascending directory with a descending comparison (0 for 10, even with prosrc changed). A syscache callback on PROCOID, registered once per backend, now only counts pg_proc invalidations - it runs while invalidations are processed, where no catalog may be read - and lion_get_index_state() checks the cached state’s OWN comparisons (the functions its FmgrInfos call) against the recorded identity whenever the count has moved since it last passed; a state that passes is good until the next change. A comparison the catalog would resolve differently today but this state does not call cannot hurt it, and the next state built is checked in full by lion_fill_index_state(). The check is skipped while the caller holds a buffer lock or is in a critical section (InterruptHoldoffCount, CritSectionCount): it reads the catalog, which is no business of a page change half-way through, and the next call made with nothing held does it. The check therefore runs at the next use of the index in that backend after the change is seen. A query already running when another session commits the change is not protected: its plan and its FmgrInfos were set up before, and a SQL function’s cached plan follows the new body at its next call - the same window a btree scan has, closed by the same rule. test/isolation/order_ident_cache.spec counts in a backend, replaces the comparison from another (a string body, and a SQL-standard one), and requires the REINDEX error in the first backend and the right answer once the body is restored.

lion_fill_column_state() still resolves the comparison from the catalog, but the RECORDED bit decides whether the column is ordered: a directory built in hash order stays in hash order whatever btree opclass appears later, and one built in value order is read in value order without the sort operator, which only the build’s sort of the keys ever needs. If an ordered column’s comparison no longer resolves at all, or resolves to a function with a different source than the recorded one, opening the index is an ERROR with a REINDEX hint rather than a quietly different order (within the limits of the guard, above). Readers and writers use the same stored order, so backends whose relcache entries were built at different moments cannot disagree any more. REINDEX asks the catalog again. ambuildempty() records the order of the empty directory the same way. The meta page reaches WAL as it always has - the build’s bulk write and ambuildempty() log it as a full page image in either WAL mode, and a later split’s meta update carries the whole struct - so replay needs nothing new. lion_index_verify() already checks every leaf and every pivot against the order the index is READ in, which is now the recorded one. An index built before the record existed has order_flags = 0 and is read as it was, from the catalog; REINDEX records its order. test/sql/ordering.sql is the review’s repro both ways round (hash order kept after the btree opclass appears, value order kept after it goes), with inserts in between and verify, a swapped loose proc 4 refused, a renamed proc 4 accepted and a re-bodied one refused, a SQL-standard comparison built hash-ordered (NOTICE) and still exact after its body is reversed, and a citext index that keeps answering exactly (equality, ranges, inserts, verify) after ALTER EXTENSION citext SET SCHEMA - moved back afterwards for the tests that follow.

A relcache flush must not free the state, and must not make the write path read the meta page (2026-09-24, found under debug_discard_caches, which flushes on every catalog read; ordinary invalidation traffic can do the same at any lock acquisition, only rarely). Two things went wrong. A flush of an index’s relcache entry pfree()s rd_amcache, and rd_amcache WAS the LionIndexState: every caller that took the state and then read the catalog before it was done - the order check itself, a scan resolving a cross-type probe, lion_index_verify() comparing text keys under a collation - held a pointer into freed memory; the order check read a garbage identity and refused every index, and verify crashed. rd_amcache is now a HANDLE (LionAmCache) on a state allocated beside it in rd_indexcxt, so a flush frees only the handle and a pointer taken earlier stays valid - no longer the entry’s current state, and the next lion_get_index_state() builds a new one, as it always did after a flush. That holds because a flush of an OPEN index entry with its support info loaded is, on every release from 16 to 20, the in-place reload (RelationReloadIndexInfo(): the index branch of 16’s and 17’s RelationClearRelation(), 18-20’s RelationRebuildRelation()), which frees rd_amcache and keeps rd_indexcxt; the context goes only with the entry, in RelationDestroyRelation(), which asserts a reference count of zero. So the rule is that a state is valid while the caller holds the index open - every caller does, and none keeps one across index_close() - and a stale state is a correct one: its content is what was built, only no longer the entry’s. A flush therefore leaves one state behind in rd_indexcxt per flush of an open index until the entry is destroyed, as it already did for the column states. And lion_wal_mode(), asked for when a record begins - in a split, with the meta page held EXCLUSIVE - rebuilt the state after a flush, which reads the meta page: a second lock on a buffer the backend already holds, an assertion failure on a cassert build and a wait for ever on a production one. The mode never changes for a relfilenode, so it is remembered per relfilenode the first time a meta page is read (lion_index_meta_wal_mode()), and the write path never reads one. test/sql/discard_caches.sql runs counts, ranges, bitmap and plain index scans, a GROUP BY, a count(DISTINCT), an insert and verify on text, int, citext and a custom ordered opclass under debug_discard_caches = 1 where the build allows it (assert builds) and without it elsewhere, with one expected output.

The opclass’s functions run under a leaf’s share lock. A descent’s binary search, the run scan and the range walk of §28 call proc 4, the equality, and - on an unordered column - the range operator itself while holding a directory leaf SHARE-locked. A SQL-language function that reads the same index would ask for that leaf’s lock again from the same backend, and buffer LWLocks have no deadlock detection: an EXCLUSIVE request queues behind our own share lock and the backend hangs. Core’s own btree has the same exposure with its support functions. Only a superuser can create an operator class, so this is a property of the opclasses one installs, not an attack surface; the shipped classes call C functions that read no relation.

Cross-type searches (int4col = 123::int8) need an ordering of a STORED key against a value of another type, and it is resolved ONCE for the single-value lookup, the batched one AND the bitmap scan (lion_probe_init() / lion_probe_find() in lion_set.c, declared in lion_count.h and used by lion_scan.c), because all three walk the same tree and a path that descended where another scans would read the directory in an order it is not in. The outcomes, in this order:

  • an UNORDERED column (rule 3 below can make one even when its family names comparisons): the hash and the cross-type equality only. No comparison of any kind is used on a directory in (kind, hash, bytes) order - the family’s cross-type proc 4 included - and the family’s cross-type hash agrees with the stored keys' by the family’s contract, so the descent is exact;
  • the family’s cross-type proc 4 (btint48cmp and friends; integer_ops, float_ops and datetime_ops, the families with cross-type equality, carry them): descend as usual, by proc 4 ALONE - the hash term is left out of the search key and the run of entries proc 4 ties is scanned with the equality, because datetime_ops‘ types do not hash compatibly (§28, “Cross-type date and timestamp”);
  • otherwise a BINARY coercion of the value to the key type (find_coercion_pathway() returning COERCION_PATH_RELABELTYPE: the same bytes, varchar to text): the value becomes one of the index’s own and hash, equality and ordering are all the index’s own, which is consistent by construction;
  • otherwise the leaves are walked with the cross-type EQUALITY (lion_dir_find_by_scan()), which is correct and linear.

(Deviation from the first draft, which described only the walk and let lion_posting_set_lookup_many() call the descent directly: an IN list of int8 values against an int4 index whose family had no cross-type proc 4 descended a value-ordered tree comparing hashes and returned nothing at all. test/sql/directory.sql §12 builds such a family.)

(Three further deviations, from the 2026-09-22 review. (1) The coercion step used to take any IMPLICIT cast function too. Implicit is not lossless: text → name truncates to 63 bytes, so for a family with name = text equality and no cross-type ordering a long text that is no stored name became one and was counted. PostgreSQL cannot say that a cast is a bijection, so only a binary coercion is taken; §16 of directory.sql. (2) The bitmap scan resolved its own comparison and used the family’s cross-type proc 4 without asking whether the column was ordered, binary-searching a hash-ordered directory in value order; it now goes through the shared resolution; §15. (3) See Readers, below, for how a LIST of cross-type values is sorted.)

Operations

  • Lookup: descend from the root with a binary search per page, SHARE locks, releasing the parent BEFORE locking the child. (Deviation from the first draft, which said “lock coupling parent→child as nbtree does”. nbtree does not couple downwards - _bt_relandgetbuf() unlocks the page it came from first - and the reason is not thrift: a split holds the CHILD while it locks the parent, so coupling downwards would close a deadlock cycle that buffer locks have no detector for. What makes the un-coupled descent safe is the right links: a searcher that lands on a page whose high key no longer exceeds its key moves right, which is where the split put the items.) The leaf is taken in the caller’s mode directly (the descent knows it is at level 1), so no lock is ever upgraded. The leaf pin remains the §9 pin for INLINE sets.
  • The root is cached in LionState and validated by the LION_PAGE_ROOT flag, which a root split clears on the page it demotes - nbtree’s BTP_ROOT trick. A lookup therefore costs no meta-page visit at all until the root really moves. A root split can also land between reading the meta page and locking the root it names, and then that page is demoted too. lion_dir_get_root() reads the meta page again for as long as the height it names keeps growing - a root split raises it in the very record that demotes the old root, and nothing lowers it - and refuses as damage only a meta page that names the same height again with a page that is not the root. It used to allow one refresh, which made a root split racing it a spurious “is not the root page” ERROR (2026-09-27 review). nbtree’s _bt_getroot() descends from the demoted page instead, which would do for a search but not for the split repair’s lion_dir_find_parent(), whose parent may be on the new level above it.
  • Insert of a new entry: descend with the leaf EXCLUSIVE; scan the prefix run for the key; if it is absent, walk back to its exact bytewise position on that leaf and insert. On no room, split. Find-or-create is one serialised operation: the directory holds one entry per equality class, so no second writer of the same key may run its lookup between this writer’s unsuccessful lookup and its insert. On one leaf that is automatic - the leaf is held from the lookup to the insert. A prefix run that SPANS pages (which needs an opclass whose comparison ties for distinct entries: an unordered class whose hash collides, as test/sql/directory.sql §11 and test/isolation/dir_insert_race.spec build) is handled by the guard: the leaf the run is entered from - the leaf a descent for the run’s prefix lands on, which the lookup started on - is NOT released when the scan steps right; the scan couples its steps (next page locked before the current one is released) and, if the key is absent, gives up the leaf it ended on and hands the guard back to the insert, which walks right from it under the same coupling to the key’s exact bytewise position, finishing any split it steps into, and inserts there. Every INSERT of a key of that prefix - one that creates it or one that adds to an existing entry of the run - descends to the guard first, so while it is held nobody else can look the key up, and the insert needs no second existence check. (VACUUM re-finds entries with an exact key and may land further right, but it only ever rewrites or deletes an entry, never creates one.) The guard is a fixed point while it is held: it cannot split (it is locked), a page to its left can only split into pages whose high keys are still below the prefix, and a descent always ends on the first leaf whose high key’s prefix is at or above the search key’s. (Deviation from the first draft, which RELEASED the leaf and re-descended with the exact key and inserted without looking again: two writers of one new key could both miss and both insert, and the rows under the entry no lookup returned were lost to counts and scans. verify() now reports two entries of one equality class, byte-identical or not, as corruption.) Lock order. The guard and the page the walk is on are both leaves, taken left to right - the order every directory writer already uses (lion_dir_place_again(), a split’s old right sibling) - and nothing ever waits for a leaf to its LEFT while holding one; a descent holds nothing, an ascent (a split’s parent insertion, a repair) holds leaves and waits only for internal pages and the meta page, and nobody holding those waits for a leaf. So a writer holding the guard and splitting the target page further right is nbtree’s “hold the child, lock the parent” with one more leaf held on the left, and it cannot close a cycle. Container pages still come after directory pages, and the guard is released before any of them is touched on the update path. Injection point lion-dir-add-entry-spanning fires with the guard held, between the unsuccessful lookup and the insert.
  • Split: allocate the right sibling (lion_alloc_buffer with reuse = true is fine; directory pages are never posting-set roots), rebuild both halves from a private copy of the page with the new item inserted, cut at half the bytes - or, when the page is rightmost and the item goes at the very end, put only the new item on the right and walk the cut back only as far as the high key the left page now needs, so an ascending key sequence fills its leaves completely (without that walk-back the cut is rejected outright, because the page being split is full, and an ascending build lands at 50%: measured 12.2 MB against 6.2 MB for 100k ascending bigint keys) - set the high keys and the sibling links, flag the left page INCOMPLETE_SPLIT, and write ONE generic record for (left, right, old right sibling, meta page) = 4 buffers. Then insert the downlink into the parent, and clear the flag in a THIRD record. (Deviation from the first draft, which had two records with the flag cleared “in a second record” together with the downlink, as nbtree does. GenericXLog takes at most four buffers and a parent SPLIT already needs four of its own, so the child cannot join that record. Three records are safe because the repair is idempotent: it looks for the downlink before inserting one. The left page is held EXCLUSIVE from the first record to the last, which is the property nbtree’s README says must hold - two writers that both saw the flag would both insert the downlink.) The meta page is in the record because it counts the directory’s pages, which is what lets the cost model tell a directory page from a container page without reading the index. The two walk-backs can pull in OPPOSITE directions and leave no cut at all, and that is no longer an error (2026-09-22 review of this section). An 8 KiB page has room for one item plus one pivot (6056 + 2068 = 8124) but not for two items plus a pivot, so a page holding two large entries under a large high key would have no two-way cut for a third large item between them. The split then runs in two steps: the page is split as it STANDS, which always has a cut - k = 1 puts one item plus its high key on the left and a suffix of what the page already held plus its old high key on the right, and both were on the page a moment ago - and the item is then placed again on whichever half now owns it. That terminates, because every split moves at least one item off the page and a page holding one item always takes a second. With today’s size caps the shape is in fact unreachable - a fresh entry is a little over 2 kB at most on a page and a replacement grows by at most a few dozen - but the caps are three independent constants and none of them should be load-bearing for whether an INSERT errors out. A posting-tree internal page needs none of this (§22): its pivots are fixed size, so the middle cut always fits.
  • Root split is ATOMIC instead: left, right, the new root and the meta page are four buffers, so the meta page never names a root that does not exist and a root is never flagged. The old root loses LION_PAGE_ROOT in the same record, which is what makes every other backend’s cached root detect the change.
  • The repair rule: any WRITE descent that lands on a page with INCOMPLETE_SPLIT finishes the split before touching the page, and so does lion_dir_find_parent() when it lands on such a parent. Finishing means: take the high key as the separator, find the parent by descending with a key the page itself holds (or, for a page VACUUM has emptied, by scanning the level from its leftmost page), insert the downlink unless it is already there, and clear the flag. Injection point lion-dir-split-incomplete fires between the first record and the second; test/recovery/run.sh phase 1c crashes the server there and proves the next insert repairs it.
  • Entry rewrite in place: as today, on the leaf, EXCLUSIVE. A grown entry that no longer fits splits the leaf and is placed by the split, instead of spilling onto a container page as it had to with the hash directory.
  • Delete (VACUUM, §18): entries are deleted with PageIndexMultiDelete, which compacts. (The PageIndexTupleDeleteNoCompact of §18 existed to keep entry offsets stable for readers parked on a page; a sorted directory cannot offer that anyway - an insert in the middle of a leaf shifts every offset after it - so the readers changed instead, below, and the compaction reclaims the line pointers.) A leaf VACUUM empties is then taken out of the tree, and its page used again (“Page deletion”, below).
  • Ordered iteration: leftmost leaf, then right links, SHARE lock one page at a time, pin held while an INLINE set from that page is being counted exactly as the bucket walk did.

Height

Every internal page begins with a key-less minus-infinity downlink (format version 12), as nbtree’s first data item on an internal page has no key. The key that downlink would carry is the page’s lower bound, which the page’s left sibling holds as its high key and the parent as the downlink to the page; a descent that reached the page has a key at or above it, so the first child is the one to take for anything below the second downlink, which is what lion_page_downlink() did anyway. A split of an internal page writes the right half’s first downlink without its key - the key goes up as the separator, and is the left half’s new high key - and so does the build. Finishing a split, and verify’s confirmation of a page its walk did not reach, search for a key the page holds, which on an internal page is its second downlink’s (lion_dir_first_keyed()).

What it buys is room. A pivot of a LION_MAX_KEY_SIZE key costs P = 2068 bytes of a page’s 8136, so an internal page holds a high key and three downlinks only when one of them has no key. With every downlink a full pivot a non-rightmost internal page holds two, a split of three makes halves of one and two, and pages of one downlink take over the directory: 25,000 hashed tsvector lexemes inserted in md5 order make 50,383 internal pages over 16,533 leaves and a directory 36 levels tall.

The rule, and the bound it gives. An internal page split leaves at least LION_ABS_MIN_DOWNLINKS = 2 downlinks on each half (lion_dir_choose_split()), and such a cut always exists. With m the minus-infinity downlink’s 20 bytes, m + 3P fits a page, so a page with two downlinks and a high key takes a third without splitting: a split has n >= 4 items, the new one included. The left half at cut k costs L(k) = s(0) + … + s(k) (items up to k, and a high key copied from item k, which costs what item k does), rising in k; the right half R(k) = m + s(k+1) + … + s(n-1) + the old high key, falling. L(2) and R(n-2) are at most m + 2P; take the largest k <= n - 2 whose left half fits: if it is n - 2 both fit, and otherwise L(k+1) is over the page, and since the page held everything but the new item, R(k) < m + s(new) + s(k+1) <= m + 2P. The build gives every internal page two downlinks too, but may leave one on the rightmost page of a level; a root has two when it is made.

VACUUM takes downlinks out (“Page deletion”), which can leave a page with one, so the bound counts the pages MADE at a level - by a split, a root split or the build - rather than the pages there now. For one level let Φ be the sum over its pages of max(d - 2, 0), d a page’s downlinks. A downlink added without a split raises Φ by at most one; one added by a split lowers it by one (the page had n - 1 >= 3, a term of n - 3, and its halves have terms adding up to n - 4); a downlink taken out, or a page of one going, never raises it; a root split makes a page of two, a term of 0; the build makes pages of terms d - 2, but for the rightmost of the level, which may have one downlink. Φ starts at 0 and never goes below it, so A + 1 >= 2M, with A the downlinks ever added to the level and M the pages made at it. Each downlink added names a page made at the level below, and each such page gets one, so every level has had at least 2M - 1 pages made where the level above it had M, and the level below the root at least two. A directory of height h has had at least 2k-1 + 1 pages made at level h - k, and at least 2h-1 + 1 leaves.

Without page deletion every page made is still there, and as a relation has fewer than 232 blocks, h <= 31 = LION_MAX_DIR_HEIGHT. With it pages are used again, and 32 levels take more than 231 leaves made over the index’s life - a leaf split for every half page of entries added, some 8 TB of them. The meta page check refuses a taller directory as damage, and a root split that would pass it refuses itself before it writes anything, with a hint to REINDEX, which builds the directory anew at the height its keys need. verify() checks the page rule: every internal page’s first downlink is minus infinity, and it has a downlink.

The leaves are not bounded the same way and need not be: a leaf holds a high key and two entries of maximum size, so random inserts leave some 1.5 entries a leaf, and the leaves only add to the count the height is bounded by.

Measured (25,000 rows of to_tsvector('simple', md5(g::text) || repeat('x', 2013)), inserted one statement at a time into an empty index; and one jsonb document of 20,000 top-level keys of 2,000 characters):

height leaves internal size time
tsvector, 4,000 rows 9 2,628 1,924 36 MB 1.3 s
tsvector, 25,000 rows 11 16,533 12,090 224 MB 8.2 s
tsvector, 25,000 rows, REINDEX 9 12,504 6,256 147 MB 1.9 s
jsonb document 11 21,962 14,465 285 MB 4.3 s

With every downlink keyed the same inserts made heights 24 and 36 (79 and 523 MB), and the REINDEX height 13 (195 MB).

GIN holds the same documents in a few MB because it compresses its index tuples, which repeat a character 2,000 times here; that is a question of key size (§4), not of the directory’s shape.

Offsets are not names any more

The hash directory only ever APPENDED to a bucket page, so (page, offset) named an entry for its whole life, and two readers relied on it: lion_entry_scan_next(), which gives up its lock between two entries, and VACUUM’s pass 2, which refers to the entries pass 1 found while the leaf is unlocked. A sorted directory inserts in the MIDDLE of a leaf, which shifts everything after it, and splits it, which moves the upper half away. Both readers therefore changed:

  • The entry scan resumes at a KEY: “the first key above the last one I returned”. Nothing is skipped or returned twice, because splits move entries only rightwards onto a page the walk has not passed and their keys are still above the last one returned; a deleted entry is simply gone, and only an EMPTY posting set is ever deleted, so its group had nothing the scan’s snapshot could have counted. An entry INSERTED behind the walk is missed, which is the same freedom the bucket walk had. test/isolation/dir_split_scan.spec parks a GROUP BY between two entries and splits the leaf under it. (Since §28’s “One read per leaf” the scan resumes at a key once per LEAF rather than once per entry: it copies what it wants out of a leaf under one lock and hands the copies out. The key rule is unchanged; only how often it is applied.)
  • VACUUM carries the entry’s KIND and KEY next to the offset, and re-validates before every use: one entry per key, so equal stored bytes mean the same entry. A hit at the remembered offset is the common case, a scan of the page catches an insert that shifted it, and anything else means the entry moved to another leaf and is found again by a descent - with nothing else held, because taking a directory lock while holding a container page would close exactly the cycle §11’s waiting rule exists to avoid. The final delete step validates the same way and simply skips an entry that has moved; the next VACUUM gets it.

Bulk build

(Since 2026-09-27 the build sorts no TIDs at all - §24 “Build” appends each key’s codes as the scan delivers them and sorts only the distinct keys - so the tuplesort this subsection describes is gone. It is kept for the record, and what it established still holds: the sort that is left leads with the key, the reserved entries come first, the hash is compared as a uint32, and a collision’s keys are ordered at flush time.)

The tuplesort’s sort keys are (key ASC NULLS FIRST, [kind], code) for an ordered opclass and (hash, kind, code) for an unordered one, so the entries come out in directory order. Neither LEADS with kind, although the directory order does, and that is deliberate: a reserved entry has no key at all, so its NULL sorts before every value with NULLS FIRST, and for an unordered opclass the reserved entries hash to LION_NULLKEY_HASH = 0 which is the minimum - so both give exactly the (kind, key, hash) sequence. Leading with kind gives the same ORDER and is much slower: tuplesort compares the leading key from a datum it precomputed and every further key by fetching the attribute out of the tuple, and kind is the same value for every real row, so every comparison would fall through to a fetch. Measured on one million distinct bigint keys: 1.43 s of sort with kind leading against 0.19 s with the key leading, and 2.04 s against 1.18 s for the whole build (HEAD’s hash-directory build of the same index is 1.40 s). The three fetched columns are for the same reason the fixed-width ones, first in the tuple descriptor, so that their offsets are cached.

kind is a sort key whenever the leading one can TIE across kinds: for a multi-key opclass, which has key-less entries as well as a NULL one and none has a key to tell them apart, and for every UNORDERED opclass, whose leading key is the hash. The grouping pass starts a new entry whenever the kind changes, so a real key that hashes to 0 interleaving with the reserved NULL rows - code by code, which is how the sort leaves them - wrote the NULL entry out several times over and IS NULL then answered from whichever of them the descent reached first. An ordered non-multikey class needs no such key: its only reserved kind is NULL, whose sort key is NULL, and no real key is. (Not in the first draft, which added kind for a multi-key class only. test/sql/directory.sql §11 gives an xid class a deliberately coarse hash so that every value collides with the reserved entry.)

The hash column is int8, zero-extended: it is a uint32 and the directory compares it as one, and sorting it as int4 would put everything above 231 first - which for an unordered opclass IS the order, and verify() catches it.

A hash COLLISION is ordered at flush time. The sort brings the tuples of one hash together but says nothing about the several distinct keys inside it, so the grouping pass opens a builder per key in the order the first TID of each happened to arrive. lion_flush_builders() sorts the open builders with the directory comparator - (kind, proc 4, hash, stored bytes), the same order lion_cmp_entry() applies to two entries on a leaf - before writing any of them out. There is one builder unless the hash function collides, so the sort costs nothing in the normal case. (Not in the first draft, which flushed them in creation order: the leaf items came out unsorted, which breaks the binary search and lets the key-based resume of lion_entry_scan_next() skip entries. verify()’s “strictly increasing within a page” check is what catches it.)

Pass 1 of the old build is GONE. It existed only to count distinct keys so that the bucket count could be sized; the one remaining pass counts them as it groups, so the build reads the sorted data once instead of twice and the tuplesort no longer needs TUPLESORT_RANDOMACCESS.

Leaves are filled left to right at fillfactor (reloption, 10..100, default 90; to whichever side of it the last entry lands nearer, below), and the internal levels are built bottom-up in the same pass, nbtree’s _bt_buildadd shape: one open page per level, written out when the item that does not fit arrives - that item’s key is the page’s high key, and the page’s own first key is the downlink it hands to the level above. The first page of every level gets the minus-infinity separator. The level whose last page is also its first is the root. An empty table gets a single leaf that is also the root.

fillfactor is about the LEAVES only. An internal level is packed to LION_NONLEAF_FILLFACTOR = 70, nbtree’s own separate non-leaf fill, and takes at least LION_ABS_MIN_DOWNLINKS = 2 downlinks whatever the fill target says (it aims for 3 and settles for 2 when the keys are too large - three LION_MAX_KEY_SIZE pivots plus a high key do not fit a block). Each internal page’s first downlink is written as minus infinity and counted at its own size, as an INSERT’s split writes it (“Height”). Without that floor, fillfactor = 10 and 512-byte keys leave ONE downlink on an internal page, every level is then as large as the one below it, and lion_build_finish_dir() climbs for ever without reaching a root. It now also checks the invariant it relies on - each level strictly smaller than the one below - and errors out rather than looping. (Not in the first draft, which applied the leaf budget to every level and said no floor was needed.)

The leaf fill lands on the nearer side of the budget (2026-10-11). A leaf is written out when the next entry would take it past fillfactor - unless taking the entry leaves the page over the budget by less than it would be short of the budget without it, and the page still holds the entry and a high key (lion_build_leaf_crosses(); the entry’s own key stands for the next page’s first, which becomes the high key, on both sides of the comparison). For entries small against a page this moves the fill by less than half an entry, so fillfactor keeps its meaning for them: a unique int8 column of 500,000 keys builds leaves 10.2% free at the default 90. For large INLINE entries it is what decides the fill. On a 20,000-key column over 5M rows every entry is some 1.55 kB: four take 76% of a page and a fifth 95%, and “never past 90%” stopped every leaf at four, 22.8% of every leaf empty. The fifth is nearer the budget (5% over it, against 14% short), so the leaf now takes it. The internal levels keep their own rule (LION_NONLEAF_FILLFACTOR and the downlink floor above). A leaf filled past the budget has less room for an insert that grows one of its entries; such an insert splits the leaf, as it does on any full leaf.

Measured (freshly built, the review’s free-space-by-page-kind query over pageinspect):

index size before size after leaves before leaves after leaf free before after
c20k, 5M rows, 20,000 keys 39.3 MB 32.1 MB 4,988 4,075 22.8% 5.6%
docs.tags (text[]), 500k rows 9.2 MB 8.5 MB 447 362 23.8% 6.1%
docs.tsv (tsvector), 500k rows 13.4 MB 12.6 MB 951 853 17.7% 8.3%

What it costs is the room inserts had. 4,000 rows inserted into the c20k table after the build grow the new index by 224 kB (some 28 leaves split) and leave the old one as it was; 40,000 - two for every key, so that nearly every leaf gets one - split nearly every leaf of the new index, which ends at 43.7 MB against the old one’s unchanged 39.3 MB. An index of large entries that takes many inserts across its keys after the build is better built at a lower fillfactor, which this rule honours as it does 90.

The high key is accounted for with nbtree’s “last item becomes the high key” rule. A page’s high key is a copy of the FIRST key of the page to its right, which is the item that did not fit; keys are variable length, so one of LION_MAX_KEY_SIZE arriving behind a page full of short ones needs room the page does not have, and reserving for the incoming key alone (what the first draft did) overflowed the page and failed the build outright. lion_build_level_flush() therefore moves trailing items to the next page until the high key fits, and the first of THEM supplies it - a pivot copy of a key that was already on the page, which is never larger than the item it came from, so the reserve is exact by construction. test/sql/directory.sql §8 puts a 2047-byte key behind short ones at every distance from a page boundary, at the default fill and at 100.

Readers

lion_posting_set_lookup() is a descent. lion_posting_set_lookup_many() sorts the values into the directory order and locates them in ONE left-to-right leaf walk, stepping right while the next value is at most LION_LOOKUP_WALK_MAX = 8 pages ahead and descending again when it is further - so a dense list pays for the leaves it crosses and a sparse one for a descent each, instead of either always. Duplicates are dropped twice over: bytewise between neighbours (free, and it saves the lookup) and then by comparing the located entries' STORED KEYS with the index’s own equality, which is what §15’s disjoint sum needs and is stronger than the (page, offset) identity it used to use (offsets move now).

The walk only steps right, so the sort must BE the directory order. Values of the index’s own type (or binary-coerced to it) are sorted with the column’s comparison, or by hash for an unordered column, whose directory leads with the hash. A cross-type list resolved through the family’s cross-type proc 4 cannot be sorted by that function - it compares a stored key with a value, not two values - so it is sorted with the family’s OWN proc 4 for (value type, value type), the family’s statement of how it orders that type; a family without one gets no walk, and every value descends by itself. The value type’s DEFAULT btree order is never used: it is the directory order only when the opclass happens to sort that way, and against a reverse comparison every value but the first was looked for to the right of where it lives (directory.sql §14). A list whose resolution came out as “walk the leaves” takes that fallback per value, as the single lookup does. (Deviation from the first draft, which sorted every cross-type list with the value type’s default comparison.)

A walk that followed a prefix run across a page boundary descends again for the next value. It then stands to the right of where the run begins, and the next value may be a hash collision of the same run stored on a page it has passed; otherwise the leaf a lookup lands on is the one its run begins on, so a value sorted after it is there or further right. (Not in the first draft: a list over a coarse-hash unordered class lost every value stored left of where the previous one was found, directory.sql §17.) lion_entry_scan and lion_emit_all_keys walk from the first leaf of THEIR key column to the first entry of the next one (§24), and IS NOT NULL, verify and stats walk the leftmost leaf and then the right links, so their output is in key order for an ordered opclass.

EXPLAIN ANALYZE reports Directory Pages Read, the leaves and internal pages the node read, which is what test/sql/directory.sql uses to prove that a thousand-value IN list costs one pass over the leaves its values live on rather than a descent each.

The FK-side join’s lookups take the same order one key at a time (LionLookupWalk, §27 “Lookups in key order”): a count runs between two of them and reads the heap, so no directory page may stay locked from one to the next, and the walk keeps its PLACE instead - the leaf, its right sibling and a copy of its high key - and reads the leaf again by its block number. That rests on the same invariant the list walk does: a page’s lower bound never moves.

A link read from a directory page is data, and a damaged one is an ERROR (INDEX_CORRUPTED), not a walk (2026-09-25 review). verify() checks all of the above offline; a reader checks, per page it reads and at the cost of a comparison each, only what it would otherwise follow blindly:

  • a block number it is about to read is valid (lion_dir_readbuf()). InvalidBlockNumber is P_NEW, and ReadBuffer() on PostgreSQL 16 to 18 EXTENDS the relation when asked for it: a root downlink set to 0xFFFFFFFF made every SELECT and INSERT that descended through it grow the index by a block before the page check refused the new, empty page. A block past the end needs no test of its own (ReadBuffer() fails on it), and asking for the relation’s size at every step of every descent would cost a system call each;
  • an internal page has a downlink where the binary search points (lion_dir_downlink_block()): a page with no data items made lion_page_downlink() answer one past maxoff, and the descent read a line pointer past pd_lower;
  • a page reached through a downlink is exactly one level below the page holding it, and a right sibling is at the page’s own level (lion_dir_check_level()). A directory page keeps its level until it is reused, which waits until no scan can hold a link to it (“Page deletion”, below), so this is exact rather than a heuristic, and it is what makes a descent terminate: a downlink to the page itself, or to one above it, used to send it round for ever;
  • a non-rightmost page has the high key every reader takes as its first item without looking;
  • the parent’s right sibling that lion_dir_downlink_present() reads during a split repair is a directory page at the parent’s level, like every other page this file reads;
  • an entry’s attno names a key column the index has before lion_search_key_exact() takes that column’s state, which lion_column() only Asserts: past the end of ix->cols the comparison would have called whatever function pointers it found there;
  • (2026-09-27 review) a page reached through a link is not one this backend already holds, and that is checked BEFORE the lock (lion_dir_held_link()), because a buffer content lock is neither reentrant nor watched by the deadlock detector: a second lock on a page one holds waits for ever, with every lock held. The walks that hold a page while they lock the next are the coupled steps through a prefix run (the page stepped from and the find-or-create guard), a split’s old right sibling (the page being split and its new sibling), lion_dir_place_again() (the left half), lion_dir_downlink_present() (the parent and the child being repaired) and lion_dir_find_parent(), whose caller holds the child from the descent to the end of the scan; a right link or downlink to one of those used to hang the backend; and
  • (the same review) a split’s old right sibling is a directory page at the split page’s level before the split record writes its leftlink. It was not checked at all, so a damaged right link had that write land on whatever page it named.

A block number past the end is not caught here: it fails in ReadBuffer(). A damaged key datum used to be let through as well, as “the same risk every index AM takes with its own keys”, until two damaged bytes of a text key, 01 01 - the header of an INDIRECT TOAST pointer - had the first descent that compared against it detoast through whatever the page held, a SIGSEGV (2026-09-29 review). A stored key has exactly one form, the one lion_store_key() writes: typlen bytes for a by-value type and for a fixed-length one, a plain 4-byte varlena header whose length is the key’s (PG_DETOAST_DATUM() gives every stored key one) - or, for a key too long for that, an inline compressed header of a method this build knows whose raw length is past the limit (§4, “Long keys”) - a cstring whose only NUL is its last byte, and at most LION_MAX_KEY_SIZE bytes in any case. So every key that comes off a page is checked for that form before any opclass function sees it (lion_entry_key(), lion_check_key(), lion_state.c) - in the descent’s comparisons, the prefix-run scans, the entry and summary walks, the posting sets' stored keys that LionOrdered and the GROUP BY counts emit, the planner’s probes and the insert and VACUUM paths - which is a few comparisons next to the function call it guards, and verify() requires the same form (lion_verify_keylen(), which used to accept a short header too; test/sql/corrupt_items.sql). Every step of a descent and of an uncoupled walk right checks for interrupts BETWEEN the pages, with no content lock held. It used to check just after locking the next page, where the lock holds interrupts off, so a descent round a cycle of downlinks (the third case of the test below, before the level check refused it) ignored statement_timeout and pg_terminate_backend() alike and only SIGKILL stopped it. And EVERY walk right is bounded (LionRightWalk, lion.h): a walk that only moves right never passes a page twice, since no page leaves a level and a split puts its new page to the right, so a walk that has taken more steps than the index has blocks is going round a cycle and is refused with INDEX_CORRUPTED. The size is only asked for once a walk is 1024 pages long, and asked again whenever the walk outgrows it, since the index grows meanwhile; before that a step costs a counter and two comparisons, which refuse at once a right link to the page itself and one to the meta page. The 2026-09-27 review bounded the walks that can NOT be cancelled, because they hold a page at every point - the coupled steps through a prefix run, and lion_dir_find_parent()’s walks right, under the child - and left the others to “walk until cancelled”. The 2026-09-29 review found what that meant: one right link pointed back into its own level sent every count, bitmap and plain scan, lion_index_count(), LionOrdered and VACUUM round the cycle until statement_timeout, and autovacuum, which has none, round it for ever, relaunched every cycle. So now every walker steps through it - the descents' moves right and lion_dir_step_right(), which takes the walk as an argument, the leaf walks of the scans (LionLeafWalk, LionOrderWalk in both directions), of the count engine (LionEntryScan, the lookup walks) and of the planner’s endpoint probe, the walks of a posting set’s leaves (§22) and VACUUM’s walks of both (§18), which therefore fail the VACUUM with the ERROR, as a damaged nbtree does. test/sql/corrupt_walk.sql damages a right link into a cycle, to the page itself and to the meta page, on disk, and runs every one of those walkers under a timeout. test/sql/corrupt.sql damages a freshly built index’s root on disk in the first three ways of the list above and checks that queries and inserts fail with INDEX_CORRUPTED and leave the relation’s size alone; against the code before that review the first grew the index by a block per statement, the second answered from a line pointer past pd_lower, and the third never returned. The cases the 2026-09-27 review added are not in that test yet, and neither is the root-split race above, which needs two backends and a split to land between one’s meta-page read and its lock.

Planner

When the grouped column’s index is ordered, the GROUP BY is a single column driven from that index (not the IN-list driver, not two columns, not a pinned single group), the table is not partitioned and the column is NOT NULL, the CustomPath gets pathkeys for the group column, so an ORDER BY <group col> above it needs no Sort.

Three conditions are subtler than they look:

  • the NOT NULL requirement is not conservatism: the reserved NULL entry sorts FIRST and ORDER BY col means NULLS LAST, so a nullable column would be claimed in an order the node does not produce.
  • the pathkeys are built for ASCENDING order from the KEY TYPE’s own <, and not from the query’s own SortGroupClause.sortop: standard_qp_callback() rewrites the GROUP BY clause’s sort operators to match the query’s ORDER BY when it can, so ORDER BY k DESC hands back a DESCENDING clause and building pathkeys from it would claim an order the node does not produce. (That bug was real and test/sql/directory.sql pins the DESC plan.)
  • and the index’s ordering must BE the key type’s default btree ordering, not merely some ordering, because that is the order an ORDER BY asks for. A partitioned table gets none: each partition is ordered, but the Finalize HashAggregate on top destroys it.

What goes away

nbuckets, lion_bucket_of, bucket sizing in ambuild, the bucket-chain warning and LION_BUCKET_PAGES_WARN, lion_clamp_buckets, lion_bucket_nentries. The buckets reloption is still accepted and ignored, with a NOTICE (“buckets is ignored since format 4”) when it is SET - not when the relcache reads it back. max_bucket_pages, nbuckets and bucket_pages in lion_index_stats() become directory_height, leaf_pages, internal_pages and ordered. The §17 cardinality guard’s insert-side estimate becomes “the entries on this leaf times the number of leaves” instead of “the entries in this bucket times the bucket count”; both are crude by design.

Page deletion

VACUUM takes a directory leaf it has emptied out of the tree, together with the internal pages above it that have no other child, and the pages are used again: a table whose oldest rows are deleted settles at a steady size, and a walk to the low end of a column reads live keys rather than a chain of empty leaves. The protocol is nbtree’s (_bt_pagedel()), in lion_dir_delete.c; lion_vacuum_leaf_page() calls it on each leaf it has finished with.

What goes. A leaf with no items but its high key that is neither the root nor the rightmost leaf, and is not flagged INCOMPLETE_SPLIT. With it go the pages above it that have one downlink, up to the highest such page, the TOP of the subtree that goes. The subtree’s downlink must not be the last on its parent, and the downlink after it must name the top’s right sibling; otherwise the leaf stays, empty and linked, for a later VACUUM. The root and the rightmost page of every level are never deleted, so the directory never loses a level.

Stage 1, one record holding the parent and the leaf. The subtree’s downlink is pointed at the top’s right sibling, and the downlink after it, that sibling’s own, is deleted: the right sibling takes the subtree’s range over, and its lower bound moves down to the subtree’s. The leaf is marked LION_PAGE_HALF_DEAD and rewritten to hold a single MINUS INFINITY high key, and its owner_head names the top, the next page to unlink (InvalidBlockNumber when the leaf is the whole subtree).

Stage 2, one record per page, the top first and the leaf last. The page is unlinked from its left and right siblings and marked LION_PAGE_DELETED. It keeps its kind, level and links and a single MINUS INFINITY high key, and holds its safexid, ReadNextFullTransactionId() at that moment, in minckey (low half) and maxckey (high half). The leaf’s owner_head moves down to the next page. A META record then lowers dirpages by the pages that went: the meta page comes after the parent in a split’s lock order, so it is not in the records above. The page each stage rewrites is registered as one its record initialises (LION_WALBUF_INIT, REGBUF_WILL_INIT): redo zeroes it and replays INIT + ADD, and the record takes no full-page image of it. Registered as an ordinary change, it would take one at every rewrite: the rewrite zeroes its LSN, which is then older than any checkpoint.

Readers need almost nothing new, because every page that leaves the tree has a MINUS INFINITY high key: a descent that reaches it through a downlink it read before, or a walk through a link it read before, moves right, to the page that took its range over. The descents that follow a downlink without comparing a key - to the leftmost page of a level - step past DELETED pages by hand (lion_dir_past_deleted()), and the walk to the left (lion_dir_step_left(), nbtree’s _bt_walk_left()) skips them and takes the page whose right link names the page it came from.

The first downlink of an internal page has no key (“Height”), so stage 1 changes no key when it lowers the bound of the page right of a subtree. When the subtree’s downlink is its parent’s first, the minus-infinity one, it stays key-less and names the right sibling, whose range then begins at the parent’s lower bound; otherwise the sibling’s range begins at the separator the subtree had. At an internal level the sibling’s own first downlink is minus infinity already, so unlinking the leftmost page of a level rewrites no item of it.

What a deletion leaves of its parent is one downlink fewer, and a parent of two then has one. A page of one downlink does not split before it has gained two more, so it does not weaken the bound of “Height”, which counts the pages made over the directory’s life; when its last child empties, the page goes with it, as a subtree of single downlinks.

Reuse. VACUUM’s sweep hands a DELETED directory page to the free space map once its safexid is older than every snapshot, and lion_alloc_page() checks it again before it takes it, nbtree’s rule (§18, “Reuse”): a scan that can still hold a link to the page began before the page was unlinked. Every reader that keeps a block number across pages - the ordered walks (§30), the lookups in key order (§27), a paused walk of §29 - holds a snapshot while it does. Taking such a page first writes a recovery conflict for the standby, whose queries can hold the same links: REUSE in rmgr mode (§25), nbtree’s XLOG_BTREE_REUSE_PAGE in generic mode, whose redo is the same conflict and nothing else.

An interrupted deletion - an ERROR or a crash between the records - leaves a HALF_DEAD leaf linked in its level with no downlink, and possibly some of its subtree still to unlink above it. Readers move past it, inserts into its old range go to its right sibling, verify() accepts it (below), and the next VACUUM finishes the deletion when it meets the leaf. test/isolation/dir_page_delete.spec stops VACUUM after each stage.

Locking. One VACUUM at a time works on an index, so two deletions never meet, and verify(), which takes the same lock, never sees one in progress. Stage 1 holds the leaf EXCLUSIVE and locks the parents upwards one at a time, as a split’s ascent does; stage 2 holds the leaf and then locks the left sibling, the page and its right sibling, left to right at the page’s level, as a split does. Neither waits for a lock below a page it holds, and neither ever ERRORs on what a concurrent writer did: a page that is not as expected leaves the leaf for a later VACUUM. Before stage 1 the leaf’s predicate locks move to its right sibling (lion_predlock_move(), §9), which takes over the range they cover.

verify() accepts a DELETED page only outside the levels it walks, and a HALF_DEAD page only as an empty leaf, not rightmost, with a MINUS INFINITY high key. An internal page needs one downlink, not two. A page of a level that no downlink names must be the top of a subtree being deleted: a chain of single downlinks down to a HALF_DEAD leaf whose owner_head names it. The pages of such a subtree bound nothing - their range is the right sibling’s already - so their high keys are compared with nothing.

The empty leaves of a LIVE key’s posting set go too, by a protocol of one record each (§22, “Empty leaves”).

Locking summary

Directory pages: nbtree’s rules - no coupling downwards, move right when the high key no longer exceeds the search key, splits hold left, then right, then the old right sibling, then the meta page, and an ascent holds the child while it locks the parent. The insert path’s guard (Operations, above) adds only more left-to-right holding at the leaf level: the leaf a spanning prefix run is entered from stays locked while the writer couples rightwards through the run and places the new entry, splitting it if need be. Posting pages: the same rules again (§22), with no old-right-sibling and no meta page in the record, and with all WRITERS of one key serialised by the very directory leaf below. The lock order directory page → posting page is preserved, and the one place that used to break it - VACUUM re-finding a moved entry - is done with nothing held. VACUUM’s two-pass protocol addresses the LEAF that holds the entry where it used to address the bucket head.

Format

LION_VERSION 4 at the time this section was written; §22 bumped it to 5 in the next wave, and a version 4 index is refused with the same REINDEX hint for the reason given there. verify() checks the tree level by level from the leaves up: page kinds and level numbers, sibling links, the high key present iff the page is not rightmost, keys strictly increasing within a page and the last key below the high key, the high key of a page not above the first key of the next, every downlink of a level naming exactly the pages of the level below in that order (which is “every leaf reachable from the root exactly once and the leaf right-link chain equals the in-order sequence”), the leftmost downlink of each level being minus infinity, each separator at or below its child’s own first key, each child’s high key at or below the next separator, and INCOMPLETE_SPLIT pages reported as a WARNING with the repair hint - or as an ERROR when the page is the rightmost of its level or HALF_DEAD, where no split leaves the flag (§7). At the leaf level it also checks that no two entries of one prefix run - the same column, kind, comparison and hash, across page boundaries too - are equal under the opclass equality: two entries for one key would leave the rows of the one no lookup returns uncounted, and the ordering checks alone only catch byte-identical twins (directory.sql §13).

Not done in this version

A backward scan (leftlink exists on DIRECTORY pages and verify() checks it, but nothing reads it yet - a future amgettuple will; §22 turned out not to need it, and posting pages therefore keep no left link at all); parallel build (done since, §24 “Build”); and online deduplication of a prefix run that spans pages, which an opclass with a comparison coarser than its equality could in principle produce.

22. Per-key posting tree (format version 5, implemented)

Replaces the linked chain of container pages per CHAIN entry (§4) with a B-tree over container keys (ckey), GIN’s posting-tree shape.

Why. A chain can only be walked. Every intersection therefore streams the dense sets in full: the 3-column AND at 5M rows visits 570 containers for a 590-row answer, and dirty-heap grouping re-walks chains per group. With a tree, the AND is driven by the most selective set: for each of its containers, the other sets are probed by ckey (descend, or step right from the last position). Cost tracks the selective side. Inserts into the middle of a chain stop being a linear walk from the head (the churn tests' mid-chain inserts).

Structure. The entry’s head is the posting-tree ROOT - or, for a set of at most 16 leaves, its FIRST LEAF: such a set has no internal level at all, and its leaves are one rightlinked chain from the head (below, “A small set has no internal level”). No separate root format, as in GIN. Container pages keep their layout (items sorted by first ckey, minckey/maxckey, owner stamps, growth slack, DELETED marking) and use the level §21 had already put in the special area: leaves are level 0, internal posting pages level 1 and up. They carry LION_PAGE_CONTAINER like the leaves - the level is what tells the two apart - so the owner check, the DELETED marking and the leak sweep apply to them unchanged. Right links stay at the leaf level, so every sequential walk (scans, single-set counts, VACUUM) is what it was: the leaves are still one rightlinked list in ckey order. leftlink stays unused on posting pages, which is what keeps a split inside the four buffers a GenericXLog record allows.

An internal page’s items are LionPostingPivot { uint32 ckey; BlockNumber child; }, 8 bytes, so a page holds 679 of them. The convention is §21’s, not GIN’s: ckey is a SEPARATOR - a lower bound on the child’s subtree - and the FIRST item of every non-rightmost internal page is its HIGH KEY, a strict upper bound with child = InvalidBlockNumber. Separators are non-decreasing rather than strictly increasing (see the repair below). The leftmost downlink of a level is 0, which is minus infinity for an unsigned ckey. Leaves carry no high key - they hold containers, not pivots - and use maxckey instead; the next paragraph is why that is enough.

Writers of one key serialise on that key’s directory leaf, and everything below rests on it. Every path that changes a posting tree - aminsert, an INLINE spill, VACUUM’s apply and regrow steps - holds the directory leaf that carries the entry EXCLUSIVE while it does (§5, §11, §21), and VACUUM re-validates that the entry is still on the leaf it holds before every write. So no split of a posting tree can be in flight while another WRITER descends it, and a write descent needs no move-right at the leaf level at all: the separators say exactly which leaf owns a container key, which is also what keeps “a leaf’s keys are all below its right sibling’s” true when the key lands in a gap. Only READERS race with splits, and a reader that lands on a leaf whose maxckey no longer reaches its key moves right, which is where the split put the items.

Operations. - Descent by ckey with SHARE locks, releasing the parent BEFORE locking the child - not lock coupling, which is a deviation from this section’s first draft and the same one §21 records: a split holds the child while it locks the parent, so coupling downwards would close a cycle buffer locks have no detector for. The leaf is taken in the caller’s mode directly, so no lock is upgraded. lion_chain_find_page() is that descent (lion_posting_search() in the new lion_posting.c); the tail hint stays for appends and is now decided with the EXCLUSIVE lock the insert needs anyway, without descending at all, when the tail is still the rightmost leaf. (The hint serves only the in-place fast path; anything that may split the page descends - see the addendum “an unfinished split and the writers that do not descend”.) - Leaf split: as today (items at and after the insert position move to a brand new page N linked right; when the new items still do not fit on P they get a second new page M linked between them), plus a downlink insert into the parent. Each new page’s LEFT neighbour is flagged LION_PAGE_INCOMPLETE_SPLIT in the split record and held EXCLUSIVE until its downlink is in and the flag cleared, so a P → M → N split flags both P and M and inserts M’s downlink first - a descent cannot reach M before M has one. Records: (P, M, N, entry leaf) = 4 buffers for the split, one for each downlink, one tiny one for each flag. In a set with no internal level there is no parent: the split is its one record, and nothing is flagged. Injection point lion-posting-split-incomplete fires between the first record and the downlink; test/recovery/run.sh phase 1d crashes the server there and proves the next writer repairs it - an append as much as a descent: the addendum “an unfinished split and the writers that do not descend” below covers the two writers that reach a leaf WITHOUT descending. - A small set has no internal level (LION_POSTING_FLAT_LEAVES = 16). A root over a handful of leaves is a page of a few dozen bytes, one per key that has more than one leaf, and on an index of many mid-sized keys those pages were about a tenth of the index. So up to 16 leaves the head is the first leaf and the rest follow it by right links, with nothing above them: - a WRITER finds the leaf that owns a container key by walking the chain from the head - the last non-empty leaf whose minckey is at or below the key - which is what a descent would have found (lion_posting_flat_route(); writers of the key serialise, so the chain cannot change under the walk). A split is one record and flags nothing: there is no downlink to post. - the split that would make a 17th leaf first PROMOTES the set (lion_posting_promote()): the push-down below, with a downlink to every leaf in the same record. The head’s items go to a new leaf that takes the head’s place in the chain; the root gets that leaf with separator 0, then each other leaf with its minckey as separator (an EMPTY leaf, which only VACUUM leaves, one past the last key of the leaf before it, so that those keys still route there; never below the separator before it). Then the split goes on as a split of a tree leaf. When the page that overflowed is the head, the caller’s items are placed on the new leaf exactly as after a root push-down (injection point lion-posting-pushdown-child between the two); otherwise the overflowing leaf stays where it is, the head is locked as well with that leaf still held, and the leaf is split after the promotion (injection point lion-posting-promoted between the two). A crash at either point leaves a whole tree - a root over every leaf, nothing flagged - and the next writer splits the leaf; test/recovery/run.sh phase 1g crashes the server at the second and checks that, and test/isolation/posting_split_repair.spec (D and F) stops both with an ERROR. - READERS walk right from wherever they stand, as they always could: a seek into a set with no internal level never descends, however far it goes (below, “Cursors”). - VACUUM’s descent is the head itself, and its walk is the chain’s. - the bulk build holds up to 16 finished leaves back and writes them as the chain (below, “Bulk build”); a VACUUM spill of 2 to 16 leaves logs the head first as an internal page with no downlink, which no sweep takes for a live root, writes the other leaves, and makes the head the first leaf in the last record, with the entry (§18). - verify accepts a level-0 head with right siblings, and refuses more than 16 of them, a flagged leaf among them, and a right sibling of an internal root. Position trees (§17) take the same shape, and the planner’s posting height is 0 up to 16 leaves. Measured below (“The sets without an internal level, measured”). - Root split is a PUSH-DOWN, and this is the deviation that matters most from the first draft, which had it allocate a new root and rewrite the entry’s head. The root block never moves: its items go to a brand new child and the root block itself becomes the level above, holding one downlink to that child (nbtree and GIN both keep their root in place). Three things fall out of it. head never changes, so the entry is never rewritten for a root split and the four-buffer budget is never the binding constraint - the push-down is (root, child, entry leaf) = 3, and the entry is only there because tail moves. “A head block is never recycled as a head” (§18) stays true without any new rule, so owner_head remains the identity a reader holding nothing but a head block can validate against. The promotion of a set with no internal level (above) is this same push-down with a downlink to every leaf of the chain. And the child is a verbatim copy of the old root, so the operation that overflowed simply runs again at the same offset on the child - the root lock is dropped for that window, because the child’s own split has to take the root to insert ITS downlink and buffer locks are not reentrant, which is safe precisely because writers of one key serialise. The push-down’s record carries the entry AS IT IS ON THE PAGE with only tail moved - not the caller’s copy, whose counters already count the items being placed, because those go onto the child in the NEXT record, which can fail (addendum “a root push-down logs the counters its page holds” below). - VACUUM: the leaves are visited in ckey order via right links exactly as before, with a cleanup lock on every one of them (§11); pass 2 starts by DESCENDING from head to the leftmost leaf, and the descent takes a cleanup lock on every page it passes, the root first (lion_vacuum_descend()). (Deviation from this section’s first version, which walked straight to the leftmost leaf and said internal pages “are NOT cleanup-locked on the ordinary walk and need not be”. They hold no TIDs, but the head of a set with no internal level is a LEAF a reader may have copied containers from, and the promotion turns it into an internal page with those containers on a child VACUUM would then clean without ever asking for the reader’s pin - §11 and test/isolation/count_root_pushdown_race.spec.) Internal pages are never empty while the set lives, so the leak sweep leaves them alone. Freeing a whole set frees its internal pages too, bottom up, under ConditionalLockBufferForCleanup like the leaves; a set with no internal level is its chain, freed from the head. Internal pages are not deleted while the set lives; a leaf the walk leaves empty is taken out of the set once the walk is over (“Empty leaves” below). A root that is pushed down while VACUUM is walking it: VACUUM finds a page that is no longer a leaf, and descends again. That is safe for the reason §11’s split hole is safe - the cleanup lock it holds on the old root waited for every pin on the page those containers came from, so no reader can hold a stale copy of them, and the page they moved to did not exist a moment ago. An internal page’s own split changes nothing here: it moves pivots, not TIDs, to a brand new page at the same level, and a page’s level never changes except the root’s; the leftmost downlink of every level stays where the descent looks for it. An INTERNAL root’s push-down likewise moves only pivots. - Cursors (LionSetCursor) gain seek(ckey): from the current leaf, step right while the leaf’s maxckey is below the target and the walk is short (LION_POSTING_SEEK_STEPS = 2 pages, about where a descent’s height reads and binary searches become cheaper), otherwise descend from the root - in a tree: a set with no internal level is only ever walked right, however far, since a descent would read the same chain from its start; then position at the first item that can hold the target. A sparse segment covers a RANGE of container keys, so a seek into one skips its PAIRS rather than the item. A materialized set is an array and is binary-searched. (Deviation: an INLINE set is NOT binary-searched, as the first draft said. An inline payload is a sequence of items packed without padding and carries no offsets to search; the seek skips forward over item headers instead, which costs what the sequential walk it replaces costs - one forward pass over the payload - and an offset index built to allow a binary search would have to make that very pass to build itself.) - The merge is a leapfrog join. One source is walked sequentially and the others are probed at the container keys it produces: the driver is the positive source with the fewest members, read off the entries' own ntids, and only the driver steps past a container key that has been counted - the others are left standing and are sought forward on the next round. Leaving them standing is what makes the probe a probe: stepping every source by one first would cost each of them a container at key + 1 that the seek is about to skip anyway. The same holds inside an AND node of the expression evaluator (§17), where only the driver steps. Union and single-set counting keep the sequential walk. The probes are ordered and abandoned early (§25): the non-driver sources are sorted by ascending members (the same ntids), and a container key is abandoned - the remaining sources neither sought nor read - the moment the running intersection empties or a seek lands past the target. That is the executor half of the open item below, and it is measured there. An AND node is the same leapfrog (lion_leapfrog(), one function for both), and it is what a lion index scan’s AND of its quals' sets is (§29.11, “The AND, as the count makes it”). And a union it meets is built at a key only where that is the cheaper way to AND it; elsewhere the intersection’s members are looked up in the union’s containers (§29.11, “Unions probed”). - The §9 pin discipline is unchanged. A leaf stays pinned until the container taken from it has passed the visibility-map check, and a seek releases the previous leaf’s pin only at that same point: the two callers of seek are the merge’s “this container key is missing from some set” branch and an AND node’s wind-forward, which skip container keys rather than step through them with lion_ecursor_next(), and for the same reason - nothing of those container keys reaches the visibility map, so no answer rests on them. The early exit of §25 does not touch it either, for the same reason and one more: a container key that is abandoned contributes nothing to the answer, so no visibility-map question is asked about it and there is nothing to discharge for any page it touched; and the sources that were not sought are not stepped, so they simply KEEP the pins they were standing on. A pin too many never makes a count wrong - it only makes VACUUM wait. - Owner validation (§18) applies to internal pages as well, and to the LEVEL: a leaf’s right link always names another leaf, so a page above level 0 reached through one is treated exactly as a page whose owner no longer matches - the end of the set. lion_posting_search() and lion_posting_search_level() end the set there: a descent started again would follow a damaged right link back to the same page for ever. - A link read from a posting page is data, as §21’s “Readers” says of the directory, and the posting tree checks what its descents and walks would otherwise follow blindly (lion_posting.c, “POSTING-TREE LINKS ARE DATA”). Every failure is an ERROR, INDEX_CORRUPTED: - a pivot’s line pointer is one whole pivot inside the page before it is read, and a non-rightmost internal page has its high key (lion_posting_pivot_at(), lion_posting_highkey_at(), which the binary search, the high key and the repair’s scans read them through) - a damaged line pointer taken on trust would put the “pivot” up to 32 kB past the page; - a downlink names a valid block (lion_posting_downlink(), which VACUUM’s descent uses too), and so do the head and tail an insert reads off its entry (lion_insert_lock_chain_page()): InvalidBlockNumber is P_NEW, and ReadBuffer() EXTENDS the index when asked for it; - a child is one level below its parent and a right sibling at its page’s level. Only the root changes level (the push-down), and the root is nobody’s child or sibling, so this is exact. VACUUM’s descent, lion_posting_level_start() and lion_posting_scan_for_downlink() had no level check at all. lion_posting_search() retries a mismatch as it retries the one real race, a head pushed down between its two locks, and COUNTS those restarts: more than LION_POSTING_MAX_HEIGHT - more push-downs than a set’s root can have in its whole life - is the ERROR. Finishing an unfinished split is not counted, because each one clears a flag for good and writers of the key serialise. Before the bound, a downlink to the root or to the page itself restarted the descent for ever; - a page reached through a link is not one the backend already holds, checked before the lock: the insert’s head and tail against the directory leaf it holds, a flagged page’s right sibling against the page itself, and every walk of the split repair - lion_posting_level_start(), the scan for a downlink, lion_posting_downlink_present(), lion_posting_adopt() - against every page the repair holds, however far up and sideways it has recursed (LionPostingHeld, the addendum “a repair inside a repair” below). A link from inside the tree to the entry’s directory leaf is caught only when it is the head or the tail, a double fault this does not chase; - and a walk right along a level ends (LionRightWalk, lion.h): a walk that only moves right never passes a page twice, since no page leaves a level of a live set and a split puts its new page to the right, so one that has taken more steps than the index has blocks is going round a cycle of right links, and is refused. The index’s size is asked for only once a walk is 1024 pages long. A right link to the page itself or to the meta page is refused at the step; the second would otherwise END a reader’s walk without a word, since block 0 does not claim the set and a page that does not is what a freed one looks like (§18), and the count would come out short. This holds for the READERS' walks of the leaves too (§21, “Readers”): lion_emit_chain(), LionOrderWalk, the count cursor over the whole of its life, seeks and all (a cursor only moves forward, and a seek lands no further left than it stands), the private copy of lion_posting_set_materialize(), and VACUUM’s pass 2, its regrow walk and its freeing of a set’s levels. Unbounded, they would go round a cycle until cancelled, and VACUUM for ever under autovacuum.

“For ever” meant UNCANCELLABLE on every writer’s path, which is why the bounds, and not the checks for interrupts, are what matter there: a writer holds the key’s directory leaf throughout, a held content lock holds interrupts off, and every other writer of the leaf, VACUUM and the checkpointer queued behind it. The descents' checks for interrupts now run between pages, with none of the descent’s own locks held, which is what lets a READER be cancelled. - Bulk build: leaves are written left to right as before, and the internal levels are built bottom-up in the same pass, one open page per level, through the bulk-write API (nbtree’s _bt_buildadd shape, which is what §21’s directory build already does). A set of up to 16 leaves has no internal level: the build holds its finished leaves back, and writes them as a chain whose first leaf is the head once the set ends. The moment a 17th leaf is needed the build reserves a block for the ROOT and re-stamps the held leaves with it - their images are still in memory, and every page of a set has to carry the root’s block as its owner (§18) - and writes them with their downlinks. A leaf is filled to posting_fillfactor (reloption, 10 .. 100, default 90, nbtree’s leaf default and the directory’s fillfactor), the chain of a set without an internal level and the leaves of a tree alike: the build closes a leaf when the next item would take it past that share of the page (lion_builder_leaf_full()), counted as PageAddItem() spends it, the MAXALIGNed item and its line pointer, and a leaf always takes its first item. At 100 that is exactly “the next item does not fit”, the rule every build had before the option existed. Below 100 it leaves the room the first wave of writes after a build needs: a bulk-built container has no slack (§4), so the first member an insert adds INSIDE it - the new row version of an UPDATE that stays on the old row’s page, as it does on a heap below fillfactor 100 or on a page VACUUM made room on - rewrites the item longer, and on a packed leaf that is a split, of nearly every leaf by the time the wave has touched every container. Only rows the heap appends land in new containers at the end of their sets and need no room in the leaves before them, so an index on an append-only or read-only table can be built at 100 and save the room. The default is 90 because a table that is updated at all writes inside its containers sooner or later, and the room costs about a tenth more posting pages at build where its absence costs a split of nearly every leaf under the first wave. Measured below (“Posting leaves with room, measured”).

Empty leaves

A key that keeps getting rows while its oldest ones are deleted empties the LOW end of its posting set: the leaves of its oldest TIDs. VACUUM takes a leaf it leaves empty out of the set while the key lives on (lion_posting_delete.c), so that readers do not walk it and its block goes back to the free space map. It is nbtree’s page deletion cut down to what a posting set needs - one record per leaf, because a set’s internal pages stay.

  • What goes. Pass 2 notes every leaf its walk leaves empty, with the leaf it came from. When the walk of the set is over - and unless the whole set goes with its entry in the final step (§18) - each one is taken out in a window of its own with the key’s directory leaf held EXCLUSIVE, which every writer of the set holds too (rule 2), so the set cannot change shape meanwhile and an insert waits for one leaf at a time. Never taken out: the root (a set without an internal level has its first leaf as its root); the rightmost leaf, whose range has no right sibling to go to and which is the entry’s tail; a leaf whose split, or whose left neighbour’s split, never finished; and in a tree, a leaf whose downlink is its parent’s last, whose range would have to go to a page under another parent. Such a leaf stays, empty, and the key’s next rows in its range go into it.
  • The record. L’s left neighbour (found from the leaf the walk came from, or from the leftmost leaf after a root push-down; there is none for a tree’s leftmost leaf) gets L’s right link. In the parent, found from where the neighbour’s last key routes, L’s downlink is pointed at L’s right sibling R and R’s own downlink - the next one on the page - is deleted: R takes L’s range over, with L’s separator as its lower bound, as nbtree’s right sibling does. L itself is flagged LION_PAGE_UNLINKED and keeps everything else: its kind, level, owner stamps, right link and empty body. Its leftlink, which a posting leaf does not otherwise use, holds the low half of the next transaction id (lion_page_get_safexid() puts the epoch back as the one that places it at or below the next transaction id). One record of up to three blocks, UNLINK in rmgr mode (§25), in both WAL modes an ordinary change of three live pages.
  • Readers. Nothing in them changed. A reader that held a link to L - the right link of a leaf it copied (§29.5), or a downlink it is about to follow - lands on an empty leaf of its own set, and empty leaves were always walked past: maxckey 0 is below every key, so a descent moves right, and a walk finds no item and follows the right link to R, which is where L’s range went. Items only ever moved right, and still do.
  • Reuse. A sweep (§18, “Reuse”) that finds L unvisited and UNLINKED records it in the free space map once its transaction id is behind every snapshot, nbtree’s rule, as it records a DELETED page. The VACUUM that took L out records it at its end when the horizon has passed it by then, and the next one otherwise: one with rows to remove, or one without them once such pages are more than a twentieth of the index (§18, “Reuse”). The allocation that takes it checks the horizon again and announces the reuse to a standby first (§25), whose queries may hold a link for longer. Until then the page is exactly what it was, an empty leaf, on a standby as on the primary. lion_index_stats() counts it among the deleted_pages.
  • verify() reports an UNLINKED page reached from a live entry, and one that is not an empty posting leaf; an unreferenced empty one is a free page, as a DELETED one is.

Measured on four keys of 100,000 rows each (g % 4), twelve rounds of deleting the oldest 100,000 rows, VACUUM, inserting 100,000 new ones and VACUUM, the heap reusing its space: a bitmap scan of one key reads 25 index pages after the last round where it read 36 without the unlinking, and the index settles at 154 pages from the fifth round on, where it settled at 142 - the leaves a round takes out wait one round for the horizon, while the rows that refill their range arrive in the same round.

The sets without an internal level, measured

Bulk builds on the benchmark’s synthetic tables (2026-10-11, PostgreSQL 18): fact (5M rows) on c200, 200 keys of about 25,000 rows, and docs (500k rows) on tags (text[]), tsv (tsvector) and tsv WITH (store_positions = true). No posting set of these indexes has more than 16 leaves, so every root went; there are as many leaves as before.

index                  pages before -> after   posting roots   position roots   size
fact (c200)                    2015 -> 1815      200 -> 0             -           16 MB -> 14 MB
docs (tags)                    1179 ->  977      202 -> 0             -          9432 kB -> 7816 kB
docs (tsv)                     1712 -> 1508      204 -> 0             -           13 MB -> 12 MB
docs (tsv, positions)          3920 -> 3515      205 -> 0         203 -> 3        31 MB -> 27 MB

The roots that went were 98-99% empty: a handful of downlinks each, 9 on c200 and fewer than 3 on the docs columns. The three position trees left with a root have more than 16 leaves.

Posting leaves with room, measured

PostgreSQL 18.6 release build, rmgr mode, wal_consistency_checking off; a synthetic 5M-row table on a heap at fillfactor 90 with six lion indexes on columns of 2, 20, 200, 2,000, 20,000 and 1M keys spread by a hash of the row number. One wave updates one row in a hundred, spread evenly, to the next key of the 20-key column, which inserts into every index (the update is not HOT) at a TID inside a container the build wrote. WAL is the index records' bytes per updated row, with and without full-page images (the wave starts after a CHECKPOINT); “after” is after the wave. The SPLICE operation of §25 is in every arm.

posting_fillfactor          100                  90                   80
20 keys   size        10.8 -> 18.8 MB       12.0 -> 12.0 MB      13.5 -> 13.5 MB
          leaf splits      969                    0                    0
          WAL/row     370.8 B (163.8 B)     290.5 B (83.1 B)     290.9 B (83.1 B)
200 keys  size        13.1 -> 25.2 MB       14.8 -> 14.8 MB      16.4 -> 16.4 MB
          WAL/row     514.1 B (278.1 B)     335.3 B (94.0 B)     335.5 B (93.9 B)
2,000     size        32.9 -> 50.0 MB       32.9 -> 32.9 MB      35.2 -> 35.2 MB
          WAL/row     863.7 B (393.5 B)     620.5 B (104.3 B)    620.0 B (104.3 B)
six indexes           158.1 -> 196.4 MB     160.9 -> 162.1 MB    166.7 -> 167.9 MB

At 100 the wave split 969 of the 20-key column’s some 1,300 leaves (3,854 B a SPLIT record, plus a downlink and a flag record each) and grew the index by three quarters; at 90 nothing split anywhere and the six indexes end 17% smaller, for 1.8% more at build. The 2,000-key column builds to the same size at 90, because its sets are two leaves each whatever the fill; the room moves to the first leaf. 80 buys nothing more for a wave of this size. The 20,000- and 1M-key columns keep their sets INLINE and are not affected; the INLINE growth is §4’s.

Cost model (lion_cost_count_rel())

The leapfrog changes two terms, and both say the same thing: only the DRIVER is read end to end.

  • Containers. An AND source’s containers term is the selective source’s container count times the number of sources (probes), instead of the sum of all sources' containers. OR leaves and an IN list that drives the groups keep their own term - neither is an AND source - and the GROUP BY driver’s term is unchanged, because what changed in the executor is the intersection of the WHERE clauses and nothing else.
  • Pages. The driver pays its whole share of the index’s container pages, as every source did before the posting tree. Every other source pays only for the pages its probes touch, and never for more than that share: it is sought, not walked.

Which source drives is decided here as the executor decides it, from the MEMBERS (lion_run_merge() sums the entries' ntids) and not from the containers - a union of k sets lies in up to k times as many containers as it has container KEYS, and counting those would hand the merge to the wrong source. The sources are the ones the executor merges: one per OR restriction (a union is ONE source, §19), one per positive clause outside them, and none for an IN list that drives the groups, which is not intersected with anything (§15). What the driver then costs the others is its container KEYS, of which there are never more than the heap has.

How many pages a probed source reads. One seek is a descent - one internal page per level plus the leaf the container key lives on - or, when the key is a page or two ahead, a walk right, which the seek takes only while it is no dearer than the descent it saves (LION_POSTING_SEEK_STEPS, 2 pages). So

pages = Min(walk, probes x (height + 1))

where walk is the source’s share of the index’s container pages, what it cost before this section. height is not stored anywhere the planner can reach - the meta page carries the DIRECTORY’s height, not a per-key posting tree’s - so it is derived from the fanout the tree is built with (679 pivots to a page), which is exact for a bulk-built tree, and is 0 for a set of up to 16 leaves, which has no internal level (lion_posting_height()). A probe into such a set walks its chain from where the cursor stands; an FK join’s probe into one is priced as a walk of half the chain (lion_plan_fkjoin_cost.c). A probed source is charged at the interpolated page cost of lion_heap_page_cost() against its own index’s size, as §15’s IN list lookups are, so a probe into an index far larger than the cache is still random I/O.

Measured against the executor, counting buffer accesses on the benchmark’s one-million-row fact (release build, 2026-09-22): c2 = 1 walks its 151 container pages and touches 154 buffers. Probed by c1m = 12345, which has one container key, it touches 2 - the root and the leaf, which is where height + 1 comes from. Probed by c20k = 77, which has a container at about 40 of the heap’s 301 container keys, it touches 118, against the 100 this charges and the 154 a walk takes: 40 descents plus the page or so of stepping each, since 40 keys leave a gap of under four leaves and the seek walks two of them before it gives up and descends. The estimate is therefore about right at both ends of the range and a little optimistic in the middle, where the leaves the seek crosses and abandons are what it does not count.

An IN list inside an AND is k sets sought k times over, and an OR leaf is sought like any other set (lion_ecursor_seek() descends into each arm), so the bound applies per set of the source. A GROUP BY probes the WHERE sets once per group, so probes is counted over all the groups together - which is many times more probes than a WHERE set has pages, so the formula charges one read of each WHERE set and a grouped count is priced exactly as it was. That is also what it really costs: the sets a GROUP BY intersects with every group are copied out of the index on their second use and probed in memory after that (§9), so nothing but the first read is ever paid.

Measured (2026-09-22, 1M rows of the benchmark’s fact, 19231 heap pages all-visible, release build, nothing disabled - which plan the model picks IS the measurement; times are the median of five EXPLAIN ANALYZE runs of the node and of the best plan without it):

count(*) WHERE ...                     cost before -> after   chosen      node / other ms
c20k = 77 AND c200 = 17 AND c2 = 1       168.8 -> 117.6       BitmapAnd     0.25 / 0.88
c20k = 77 AND c2 = 1                     161.3 -> 110.1       node          0.25 / 0.18
c200 = 17 AND c20 = 3 AND c2 = 1         186.2 -> 186.2       node          0.51 / 4.41
c200 = 17 AND c2 = 1                     165.7 -> 165.7       node          0.60 / 9.75
c200 = 17                                  8.8 ->   8.8       node          0.15 / 0.43
c200 IN (17,18,19) AND c20 IN (3,4,5)    733.5 -> 733.5       node          2.19 / 9.73
c200 = 17 OR c20 = 3                     165.6 -> 165.6       node          0.41 / 26.2
GROUP BY c200                            906.5 -> 906.5       node          3.98 / 87.0
GROUP BY c20 WHERE c200 = 17             335.9 -> 335.9       node          3.16 / 8.33

Only the two ANDs with a SELECTIVE source move, by a third; the IN lists, the OR, the NULL shapes, both GROUP BY forms and the two-column grouping are priced to the cent as they were, because in every one of them the driver is either the only source or has a container at nearly every container key. The quick benchmark at one and five million rows confirms it from the other side: no case changes plan and every case is at its baseline within run-to-run noise.

The CPU of the leapfrog is lion_merge_cpu_cost() (§10, “The units”; the fit is in §31): the driver’s containers and members, a probe and the running intersection’s members per other source at each of its keys - and the k'th source is sought only at the keys where the ones before it left a row, 1 - exp(-lambda) of them, lambda the rows the intersection is expected to hold at a key. That is §25’s early exit: the executor seeks the third source of c20k = 77 AND c200 = 17 AND c2 = 1 over 5M rows at two of the 219 container keys of c20k = 77 and abandons the rest once c200 = 17 is empty (217 “Probes Avoided”), so the count costs 59 against a BitmapAnd’s 237 and takes 0.03 ms against 2.8, where a probe - and a descent of its posting tree - at each of the 219 would price it at 777. The probed pages of lion_probed_pages() are counted over the probes that happen, too.

These terms are shared with the lion index scan (§29.11, “One price for the AND of sets”). A WHERE source’s lookup, the pages a walk of its sets reads, its containers and a list’s union are lion_cost_set_clause()’s, what the driver walks and the others' probes touch is lion_cost_set_pages()’s, and the directory leaves the lookups read - one each, and no more of an index than its directory has - lion_cost_leaf_pages()’s; lion_cost_set_and() puts them together with lion_merge_cpu_cost() exactly as an ungrouped count does, and lioncostestimate() charges that for a scan’s AND of two sets or more. The rows the recheck is priced for (lion_probe_rel_rows()) are corrected by the intersection probe where the WHERE sets of one index are correlated (§29.11, “Correlated sets”). It measures what the open item below says nothing in the model could see - whether a count’s sets are independent or correlated - but its answer is not yet used for the merge’s own 1 - exp(-lambda).

The open item: the node is charged for pages, its competitor for tuples

c20k = 77 AND c200 = 17 AND c2 = 1 is still NOT the node’s, and it should be: 117.6 against the BitmapAnd’s 62.1, for a count the node answers in 0.25 ms against 0.88. The remaining gap is not the count of pages - 100 against the 118 buffer accesses measured above - it is what a page is worth here. The node’s 100 pages are a resident part of a 306-page index (ix_c2 is read by every query that touches c2), charged at seq_page_cost, which stands for a page read from a DEVICE; what the BitmapAnd is charged is CPU per TID, 4957 of them at cpu_index_tuple_cost, for work that measurably takes three times longer. Per microsecond of real time the node is charged about thirty times what its competitor is, and no honest count of pages closes that.

Two ways out, neither taken here:

  • the executor could stop reading those pages. DONE in §25’s wave, and it did what this said it would: probing in selectivity order and abandoning a container key as soon as the accumulator empties takes this query from 112 buffer accesses to 11 at one million rows, and 0.095 ms to 0.040 ms. What it did NOT do is close the gap in the ESTIMATE, and §25’s “also” item records the experiment that says why: a discount built from a survival fraction per level is right when the columns are independent and ten times optimistic when they are not, and nothing lion_cost_count_rel() can see tells the two apart. So pages = Min(walk, probes x (height + 1)) stands, as an upper bound the executor can now beat.
  • or the model could price a resident index page as a buffer hit. lion_heap_page_cost() already argues residency from effective_cache_size, but its floor is seq_page_cost, because for HEAP pages the competing plan reads the same pages and the comparison is fair. For the container pages of a small hot index it is not fair, and a third rung below seq_page_cost would say so. It would have to apply to walked pages as much as to probed ones, which moves every estimate in this model, so it needs its own pass over every pin in test/sql/pushdown.sql - the 20000-group refusal of §10 and the 200x20 refusal of §20 are the ones to watch, since both are refusals the node deserves. TAKEN in §39 (“Resident index pages”, 2026-10-01): lion_index_page_cost(), pg_lion.resident_page_cost, walked and probed pages alike, for the custom paths' counts.

Format: LION_VERSION 5. (Deviation: this section planned to share §21’s version 4, because the two were meant to land in one wave; §21 shipped first, so the bump is separate. The page HEADER did not have to change - §21 had already given container pages a level - but a version 4 posting set of more than one page is a flat rightlinked chain with no root above it, which this code cannot descend and therefore cannot write to. Opening one is the existing ERROR with the existing REINDEX hint.)

verify(): per key, the tree is checked level by level from the leaves up, exactly as §21’s directory is. Page kinds, levels and owner stamps; the leaf right-link chain and ascending ckeys within and across leaves, with minckey/maxckey describing the items; on internal pages a high key present iff the page is not rightmost, separators non-decreasing and below the high key, the leftmost downlink of each level being minus infinity, every downlink of a level naming exactly the pages of the level below in that order - which is the same statement as “the leaf right-link chain equals the in-order leaf sequence” - each separator at or below its child’s own first container key, and each child’s own upper bound below the next separator. INCOMPLETE_SPLIT pages are a WARNING with the repair hint, as §21’s are, and an ERROR on the rightmost page of a level, which no split leaves flagged. lion_index_stats() gains posting_internal_pages and max_posting_height; container_pages counts leaves only. lion_index_posting_root(idx, key) is a test helper that returns one key’s root block, which is how the regression test asserts that a root split did not move it.

The one shape the regression suite cannot reach

A split of an INTERNAL posting page. A page holds 679 downlinks, so an internal split needs a key with 680 leaves, and a leaf covers at least one container key of 64 heap blocks: about 340 MB of heap for one key, which make installcheck has no business building. test/sql/posting_tree.sql says so where it stops, and the path was exercised by hand instead:

CREATE TABLE bigtree (id int, k int, pad char(60));    -- ~100-byte rows, so 64 heap blocks
INSERT INTO bigtree SELECT i, i % 2, '' FROM generate_series(1, 3600000) i;   -- hold ~5000 rows

3.6M rows of about 100 bytes give 44,500 heap pages and 695 container keys, and two alternating keys put 2,500+ members at each of them - a BITSET, so exactly one container per LEAF, 694 leaves per key. Built by CREATE INDEX and grown from empty by 3.6M inserts, the two come out identical and both verify clean: 1388 leaves, 6 internal pages, height 2 (per key: a root, two level-1 pages because 694 downlinks do not fit on one, and its leaves). That is the internal split, the level-1 root push-down and the bulk build’s second internal level, all three.

What §21 left §22 (written after §21 was implemented, kept for the record)

  • The entry’s head is written under the leaf’s EXCLUSIVE lock, in the same GenericXLog record as the page that made it change - which is exactly what a posting-tree root split needs. Every writer of a key holds the directory LEAF that carries its entry from the descent to the last record (lion_insert_one() takes it EXCLUSIVE and releases it at the end; VACUUM takes it per window). lion_put_entry() / lion_replace_entry() are the only ways head ever changes, they take the caller’s open GenericXLogState, and an entry never changes size once it is a CHAIN entry, so the write cannot fail. In the end the root push-down made this moot for the root split itself - head never changes at all - but it is still what lets tail travel with the same record, and it is the property that makes writers of a key serialise.
  • The buffer budget is the thing to watch. The directory’s own split already uses all four buffers of a GenericXLog record (left, right, old right sibling, meta page), and the meta page is in there only to keep dirpages exact for the cost model. A posting-tree split that also wants the entry leaf has three buffers of its own left, which is why §21 put the “clear the incomplete flag” step in a third record rather than joining it to the parent insert: the same trick - an idempotent repair driven by a page flag - is what §22 uses, and it is cheaper than a custom resource manager. Posting pages keep no leftlink, which is what buys the fourth buffer for the entry leaf.
  • lion_chain_find_page() is already the only place a ckey is turned into a block, and its two callers (lion_insert_lock_chain_page() and the verifier) pass the entry’s owner stamp/head/tail, so replacing the walk with a descent changed one function.
  • Container pages have a free level field in the special area since §21 (the directory needed one and the special area grew to 32 bytes either way), so §22 needed no further format change to the page header - only the internal posting page’s item layout.
  • The §11 proof text now says explicitly which pages are cleanup-locked and why (“every page that can hold a TID”), so §22’s internal posting pages fall under the same sentence: they hold no TIDs, and cleanup-locking them is optional rather than load-bearing. (Wrong for the ROOT, which is a page that can hold TIDs until it is pushed down; see the VACUUM bullet above and §11. Kept as written because this list is the record of what §21 handed over.)

Order of work. §21 first (it changes where entries live; the posting tree hangs off the entry and is independent of the directory shape), §22 second. Multicolumn indexes (USING lion (a, b, c)) come after both and mean one directory holding each column’s keys as independent posting sets (order-insensitive, like GIN); a composite-tuple key is deliberately not offered: a query with one fixed shape is btree’s job.

Measured (2026-09-22, 1M rows, 12352 heap pages all-visible, assert build)

fact(c2, c10, c200, c20k, c200_clustered), the same binary pair measured in one session. The number that moves is CONTAINERS VISITED; the wall times at this scale are a few hundred microseconds either way and are dominated by everything but the merge.

count(*) WHERE ...                        containers before -> after   ms before -> after
c10 = 3 AND c200 = 17 AND c2 = 1                579 ->  579            0.414 -> 0.435
c10 = 3 AND c200 = 17                           386 ->  386            0.315 -> 0.319
c20k = 77 AND c200 = 17 AND c2 = 1              424 ->  140            0.243 -> 0.203
c20k = 77 AND c200 = 17                         231 ->   93            0.149 -> 0.149
c200_clustered = 7 AND c2 = 1                    11 ->    5            0.157 -> 0.158
c200 = 17 (one set)                             193 ->  193            0.197 -> 0.199
GROUP BY c200                                 38600 -> 38600          12.40 -> 12.46

The three-column AND on UNCORRELATED dense columns cannot improve and does not: c2, c10 and c200 each have a container at every one of the heap’s 193 container keys, so every key is common to all three and 3 x 193 is the minimum any algorithm can read. The gain is the whole point of the section and shows up exactly where the section said it would - when one source is SELECTIVE in container keys: c20k = 77 has 48 of the 193, and the AND goes from reading both dense sets in full to probing them 48 times each. A clustered key is the extreme of the same thing. The wall times at this scale are a few hundred microseconds either way and are dominated by everything but the merge; the containers are what the section changes.

Writes and maintenance, same session:

mid-chain insert of 20k rows      662 / 692 / 731 ms   ->   155 / 134 / 142 ms
portfolio build, 8 indexes            8348 ms          ->       8221 ms  (163 -> 165 MB)
VACUUM, portfolio (bench/vacuum_micro.sh)
    wall / index WAL / index records   764 ms / 59 MiB / 10155
                                   ->  760 ms / 59 MiB / 10245
VACUUM, churn cycles 1..3          119 / 114 / 123 ms  ->   121 / 116 / 115 ms
GROUP BY c200 on a 5% dirty heap        99.5 ms        ->       95.6 ms

The mid-chain insert is the write-side point of the section, and it is the largest single win here: finding the page for a container key in the middle of a 277-page posting set used to be a walk from the head - about 140 pages, each SHARE-locked, inside the window that holds the entry’s directory leaf - and is now a two-level descent. 4.8x, on a 4M-row table with a DELETE + VACUUM in the middle of its id range and 20,000 rows inserted into the freed heap pages.

The index grows by 2 MB over the portfolio (163 -> 165 MB): one internal page per posting set with more than one leaf. The build and the VACUUM are unmoved, which is what they should be - neither does anything the section changed except write and walk one more page per multi-leaf set.

§22 addendum: the entry-page retry re-checks the page type

VACUUM’s pass 2 holds a container page’s cleanup lock while it filters it, then needs the entry page for the counters. If an insert holds the entry page, VACUUM lets the container page go (waiting for a directory leaf while holding a container page would invert the lock order), takes the entry page blocking, and retries the cleanup lock. In that unlocked window the insert that held the entry page may have overflowed this set’s head and pushed it down (§22), leaving an INTERNAL page at the block VACUUM is about to re-filter. The retry therefore re-checks LionPageIsPostingLeaf after reacquiring the cleanup lock and restarts the descent exactly as the first acquisition does; filtering an internal page as a leaf would read pivots as containers (an assertion on a cassert build, dead TIDs left behind on a release build). Injection points lion-vacuum-page-filtered and lion-vacuum-entry-busy make the window deterministic; test/isolation/vacuum_retry_pushdown.spec parks VACUUM on either side of it while an insert pushes the root down, and asserts that every dead TID is gone afterwards (ntids = rows) and verify() is clean.

§22 addendum: an unfinished split and the writers that do not descend

The repair rule of this section is that a WRITE DESCENT finishes every unfinished split it meets. Two writers reach a leaf without descending at all:

  • An append, through the entry’s tail (§5 INSERT step 4). A split of the set’s last leaf makes the new right half the tail in its FIRST record, so after a crash between the two records - or an ERROR there, such as running out of disk while the downlink insertion splits the parent - an append that went straight to the right half would never bring a descent near the flagged page. When the right half filled, its own split would look for its downlink in the parent, find none and fail with “no downlink for block N at level 1”, and so would every later split of the tail, since each would leave the next tail without a downlink too: an append-only key would never take another row.
  • VACUUM’s regrow (§18), which reaches the page a grown container lives on by walking right links under cleanup locks - it may not jump there (§11; rule 1 of lion_vacuum.c). On the RIGHT half, which has no downlink, its split’s parent lookup would fail as above, and VACUUM could not complete. On the flagged LEFT half, a split that did not first finish the old one would link the new page between the two halves of the old split, give the NEW page a downlink and clear the one flag that remembers the old right half, which would then have no downlink for good. The leaf chain would stay intact, so sequential readers would be right, but a descent - which is how a seek probes a set - would never reach that page again.

Three rules cover both, all three after nbtree:

  1. The append hint serves only changes that stay on the page. lion_insert_chain() uses the tail for the in-place member inserts of §4 and nothing else; a member that needs the general path, which rewrites the item and may split the page, lets the tail go and descends with forwrite. The right half has no downlink, so its keys route to the flagged left half, and the descent finishes that split on its way down. nbtree’s rightmost-leaf fastpath has the same limit: _bt_search_insert() uses its cached block only when the tuple fits without a split.
  2. A page with no downlink finds out why. lion_posting_find_parent(), on a miss, walks the child’s level rightwards from the page the separators route the child’s key to, finishes every flagged page it passes, and looks again (lion_posting_adopt()); failing that it scans the parent level from its leftmost page, because a route key can lead past a downlink whose separator an emptied neighbour shares. nbtree’s _bt_getstackbuf() finishes the incomplete splits it meets in the same spirit, though only ever one level up, since nbtree reaches every page it splits by a descent. The walk locks pages to the LEFT of the child the caller holds, which §5’s lock order otherwise forbids. It is safe for the reason the whole posting tree rests on: every writer of the key holds the entry’s directory leaf, so nothing else can hold a page of this tree while it waits for one to its right - readers hold one page at a time, and VACUUM, when it holds a posting page without the directory leaf, asks for the leaf only conditionally. When the flagged page’s right sibling is the child itself, the separator is read off the page the caller already holds instead of locking it a second time.
  3. A split first finishes the page’s own unfinished split (lion_split_and_place()), as lion_dir_place() does for the directory; nbtree’s _bt_insertonpg() refuses to touch such a page at all. Finishing touches the parent and the page’s flag, not its items or its right link, so the split that follows proceeds exactly as it would have.

Rule 1 makes the repair early - the first general-path insert after the crash does it - and rules 2 and 3 make it certain for a writer that did not descend, which means VACUUM’s regrow, on either half. What rule 1 costs is the descent’s locks, one per level, and only on the general path: never for a BITSET, nor for a NARROW but when a member widens it (§38), and for an ARRAY, a RUN or a sparse segment once each time the item’s growth slack (§4: an eighth of it, 8 to 64 bytes) runs out - every 32 members of an ARRAY past 512 bytes. Measured on the assert-enabled development build, 300,000 single-row inserts into a dense key, 500 keys of sparse segments and 20 keys of arrays showed no difference beyond the run-to-run noise of that shared machine (±30%).

Tests: test/isolation/posting_split_repair.spec cuts a split short with lion-posting-split-incomplete set to ‘error’ inside a subtransaction - which leaves on disk exactly what a crash between the two records leaves - and then (A) appends, (B) makes VACUUM regrow on the right half, which has no downlink, and © on the flagged left half. Each ends with verify() clean and the index agreeing with the heap; without the three rules A and B fail with “no downlink” and C trips an assertion. test/recovery/run.sh phase 1d appends after its real crash.

§22 addendum: a root push-down logs the counters its page holds

An insert adds its member to ntids in its private copy of the entry before placing it, and VACUUM’s regrow subtracts the TIDs it filtered out of a container it has not written back yet. When the placement pushes a leaf root down, the items are placed by the NEXT record after the push-down’s, the child’s split, which allocates pages and can fail or be cut short by a crash. Had the push-down’s record written that copy - the new tail and the new counters - an ERROR at lion-posting-pushdown-child in between would leave the entry claiming one TID more than its containers hold after an insert, and fewer after a VACUUM, and every later VACUUM would subtract the regrown container’s dead TIDs once more, so the drift would never heal. So the push-down logs the entry as it stands on the page with only tail moved; the counters travel with the record that places the items, as §5 says they must. test/isolation/posting_split_repair.spec cuts a push-down short (D) in an insert and (E) in VACUUM’s regrow, and checks verify() after each.

§22 addendum: a repair inside a repair

The split repair’s walks refuse, before the lock, a link to a page the backend already holds: a content lock is not reentrant, and a second one waits for this very backend for ever - uncancellably, with the key’s directory leaf held. The pages a walk’s own caller holds are not enough to check, because the repair RECURSES: up a level when the page that takes a downlink is flagged itself or has to split, and sideways when a page has no downlink and lion_posting_adopt() finishes the split to its left - whose right half may have no downlink either. Two unfinished splits on one level (two crashes between the records, or two ERRORs in the parent insertions) are enough: adopt(C) holds C, meets the flagged F on its walk and finishes it, and F’s own missing downlink runs adopt(F) inside, with C still held. A walk that stopped only at F would not do, for a route key can land RIGHT of the page that holds it - separators are only non-decreasing: a split repaired with both halves empty shares its left neighbour’s, and an internal split can cut a run of equal ones - so a walk that started past F would go on to C and lock it a second time.

Two rules, both in lion_posting.c:

  1. The held pages travel with the recursion. LionPostingHeld is a list with one link per frame of lion_posting_finish_split_ext(), on that frame’s stack: the page whose split it finishes. Every walk of the repair looks each page up in it before the lock - the root and the moves of lion_posting_level_start(), the scan for a downlink, lion_posting_downlink_present()’s look at the parent’s sibling, both walks of lion_posting_adopt() - and a held right sibling whose first key a leaf repair needs as its separator is read off the held page.
  2. lion_posting_adopt() walks twice. First SHARE and finishing nothing, from where the route leads, to learn whether its child lies ahead at all; only then again, from the first flagged page that walk saw up to the child, finishing. A single walk that finished as it went would also finish, to no purpose, every flagged page RIGHT of the child whenever the route led past it - a child’s missing downlink is only ever caused on its left - and could so nest a repair whose own child lies right of a page it would walk through. A repair finishes splits only left of its child, so the pages nested adoptions hold on one level strictly decrease, each left of the one around it, and a walk that meets a held page other than its own child has passed its child exactly as a walk that reaches the end of the level has: it stops there, without the lock, and the next attempt starts from the leftmost page, where the child comes before all of them. Nothing either walk passes changes in between: finishing a split clears a flag on the child’s level and writes only above it, and no other writer is in the tree.

Why neither rule turns a sound index into an ERROR or a wait. The recursion only climbs or stays on its level, so every page the repair holds is on the level being walked or below it, and never the root (only a flagged page goes into the list, and the root’s split is a push-down, which flags nothing; the push-down’s unlocked window is therefore never refused). lion_posting_level_start() returns a held page of its target level as the answer, unread, for the caller to judge, and refuses one above that level; the scan for a downlink and downlink_present() walk the level ABOVE the child, where the repair holds nothing but the parent it has found, so a held page there is damage (ERRCODE_INDEX_CORRUPTED); and adopt() treats a held page as the end of the level. So every such ERROR is a link a sound tree does not have, and the one legitimate meeting - nested adoptions on a level - is a stop. Neither rule adds to the lock order: the walks take pages in the modes and directions the rest of the repair does, and skip the held ones and the flagged pages right of the child. The cost is the first walk of adopt(), a SHARE read of the pages between the route and the child, on a path that only runs after a crash left a split unfinished.

Not covered: the leaf split’s second page outside the repair - lion_split_and_place() holds M, EXCLUSIVE, while it finishes P, and P while it finishes M, and neither is in the list, because lion_posting_put.c does not pass them. Neither is reached: M has no downlink and lies right of P, where every walk of that repair stops (P is held); and finishing M finds the downlink finishing P has just placed where M’s first key routes - in a sound tree no later separator is at or below that key - without adopting. A link to the directory leaf stays the double fault the “link is data” bullet above does not chase. Nor does the suite reach the shape: two unfinished splits on one level, with a route past the page, need two injected failures in one key’s tree, and posting_split_repair.spec injects one.

§21 addendum: binary coercion requires the same equality function

The cross-type probe shortcut (relabel a binary-coercible probe to the key type and use the key type’s own hash/equality/ordering) is taken only when the family’s cross-type strategy-1 operator is implemented by the same function as the key type’s own strategy-1 operator; otherwise the family has stated a different equality (e.g. bpchar =~~= text with text semantics on a bpchar index, where ‘x’ and ‘x ’ differ) and the probe walks the leaves with that equality. test/sql/directory.sql §18 shows the shortcut answering 100 where the family and the seqscan answer 0.

Before any of the outcomes above, a value of the column’s OWN type needs no resolution - and for a class declared on a POLYMORPHIC type the class’s input type does not say what that is. enum_ops is FOR TYPE anyenum: a scan key names its member by that type (sk_subtype = anyenum), but the elements of m IN ('a', 'b') are of the column’s enum, because the operator is polymorphic and the parser leaves the array as it is (make_scalar_array_op()). For a polymorphic class lion_probe_init() takes the column’s actual type, read from the key column (LionState.typid), for its own type as well as the class’s, with domains looked through on both sides. With keytype == opcintype the only way to say “own type”, every path that probes with the ARRAY’s element type - a plain index scan’s set tree and its LIST batches (§29.3, §29.4), a multicolumn bitmap scan (§24) - would look up an (anyenum, mood) member, find none and raise “type mood cannot be compared with index”; only the single-column bitmap scan, which probes with sk_subtype, would work. A different enum is not the own type (its OIDs mean nothing to this column) and is refused. lion_range_add() makes the same step for range bounds (§28). test/sql/keytypes.sql runs every scan shape against a sequential scan on an enum column, a list longer than a batch included.

23. Backlog (not urgent; ordered by when they should happen)

  • Prune on access in the count’s heap recheck (PostgreSQL 19+) - implemented. The argument that on-access VM setting keeps “a set container bit on an all-visible page is exactly one visible row”, checked against 19’s and 20devel’s pruneheap.c, and the implementation notes, are in §11’s last subsection (“On-access pruning sets the visibility map too”); the claim holds, so the pages core already cleans on access on 19 were never a correctness problem either. lion_recheck_heap_heap() calls heap_page_prune_opt() once per heap block it reads, with rel_read_only decided when the LionCount node starts (the SQL-callable counts pass false). Tests: test/sql/prune.sql (after HOT updates the first pushed-down count leaves the pages all-visible on 19+ and the second rechecks nothing; a GROUP BY meets pages an earlier group cleaned; exact answers everywhere) and test/isolation/count_prune_race.spec (a count parked with its container pinned while VACUUM waits for the pin and core and the count prune around it; exact answers). Limits: helps only HOT updates (non-indexed columns) on pages that are nearly full (pruning’s own free-space heuristic); deletes and indexed-column updates leave LP_DEAD items and still need VACUUM. Measured motivation: the 2026-09-23 PostgreSQL 19 focused run had its dirty pages cleaned by the untimed correctness scan before timing (all but one heap block skipped via the VM; 1.95 ms vs 79.7 ms on 18 for the 5M dense count), so that run’s “dirty” rows are clean measurements. Page-at-a-time visibility for dirty pages was considered alongside and deferred past v1: it re-implements HOT-chain visibility outside heapam for a gain on 16-18 and on non-HOT churn only.
  • Insert batching, only if the custom rmgr leaves hot-key throughput short. A GIN-style pending list was considered and rejected: GIN’s per-row cost is the number of extracted keys (30 posting trees per tsvector row), which batching amortises; ours is one posting-set update per scalar row, and the measured bottleneck is the generic-WAL page copy inside the lock window (unlogged control: 643 → 868 tps only), which a pending list moves rather than removes. It would also put TIDs outside the pinned-page protocol of §9, forcing every count to scan and snapshot-check an unordered list (pg_roaring_index’s count path pays exactly this) and adding a merge phase to VACUUM. If batching is still needed after the rmgr: per-key deferred appends kept inside the entry tuple (a small sorted TID tail merged into the containers on the next insert that finds it full, or by VACUUM), which stays inside the pinned-page protocol and keeps counts exact. Multi-key classes are the one place a real pending list might pay; revisit only with a measured tsvector ingestion case. (2026-10-27: bit slices (§51) are the other place, some 50 posting-set inserts a row; the deferred tail is designed for them there, measured and not started - §51, “Insert cost, revisited”.)
  • FK-side join pushdown: implemented, see §27. GROUP BY dim.attr over a fact table joined on a lion-indexed FK column. The sketch here - per dimension group, the union of the member keys' posting sets ANDed with the fact filters - became a count per DIMENSION ROW emitted as a partial aggregate, with core’s Finalize Agg doing the grouping: the join count is additive over dimension rows, so no per-group key set (and no memory bound on one) is needed, and the “subquery-produced key set” is the dimension side’s own plan, run as the node’s child.
  • count(DISTINCT k) in the pushdown: implemented, see §26. Both shapes (count(DISTINCT k) over the WHERE, and per GROUP BY g) with the rules this item set: existence tests with early exit, NULLs never counted, the grouping-equality check, multi-key columns and partitioned tables declined, mixed target lists, HAVING through plan.qual, pairs charged for shape 2, and a distinct-count variant of count_vacuum_race (count_distinct_vacuum_race.spec). It was done before the FK-side join pushdown, not after it as first planned.
  • PGXN packaging (done 2026-09-23 up to the release steps; before the Citus/TimescaleDB work). Distribution through the PostgreSQL Extension Network is how the two environments below will install it, so it comes first. Done:
    • META.json (PGXN Meta Spec 1.0.0; validate_pgxn_meta from PGXN::Meta::Validator 0.16 says OK, and CI runs it): distribution pg_lion 0.3.0, release_status testing (beta), license postgresql, provides pg_lion and pg_lion_citext, PostgreSQL >= 16.0.0 as a build and runtime prerequisite (the minimum since eb1579e, src/lion_compat.h), citext runtime- recommended and test-required (only the companion needs it, and its control file’s requires = 'pg_lion, citext' enforces that at CREATE EXTENSION), resources and tags.
    • Versions: see the addendum “Extension versions” below. A release’s extension version is META.json’s semver, 0.3.0; between releases every change to an install script is a new X.Y.Z-devN with an upgrade script to it.
    • make dist: git archive of the committed HEAD into pg_lion-<version>.zip (META.json, README, LICENSE, DESIGN, Makefile, control and SQL files, src/, test/{sql,expected,isolation, unit,recovery}, test/rmgr-check.sh, dev.sh; no .local, bench, review evidence or build products). Built and tested from the extracted archive, outside the checkout, against 16, 18, 19beta4 and master from source and against the PGDG packages 16.15 and 19~beta3 (non-cassert, unpacked from the .debs): unit, installcheck and installcheck-rmgr all green. Two Makefile changes for packaged installs: PG_CONFIG falls back to pg_config on PATH when .local/pg does not exist, and the unit tests link pkglibdir first, because Debian keeps the server’s own libpgcommon.a there and libdir holds libpq-dev’s, which is of another major.
    • CI (.github/workflows/ci.yml): PGDG packages for 16, 17, 18 and 19 beta (the injection-point specs are skipped there, since packaged servers have no injection_points module); REL_19_STABLE and master from source with cassert and injection points, cached per upstream commit, failing if any spec is skipped, rmgr run with wal_consistency_checking; and a dist job (validate, make dist, build and test from the zip). Each server job runs unit, installcheck and installcheck-rmgr through ./dev.sh with LION_SOCK/LION_PORT. Checked with actionlint 1.7.7 and shellcheck 0.10; it runs on pull requests, pushes to main, and by hand (workflow_dispatch) on any branch.
    • pg_upgrade: works across majors, and found a preload bug (§23 addendum below).
    • The WAL resource manager id: 170, reserved for pg_lion on https://wiki.postgresql.org/wiki/CustomWALResourceManagers, is pg_lion.rmgr_id’s default (LION_RMGR_ID, src/lion_wal.h). The GUC stays so a site can move out of a collision. The rule for changing it (WAL written under one id replays only under that id, so only after a clean shutdown with every standby and archive consumer caught up; pg_upgrade carries no WAL, so an upgrade may change it freely) is in docs/REFERENCE.md under wal_mode. Remaining:
    • Publishing: a PGXN account and upload of the zip, and a README install section (pgxn install pg_lion --testing while release_status is testing, which is what the client needs to see a release that is not stable).
    • Version scripts: done 2026-10-28, see the addendum “Extension versions” below. Any upgrade that leaves existing indexes needing REINDEX - a format-version bump, or a support function or operator changed under an index - must say so in the release notes (an upgrade script’s NOTICE is not seen: the server runs extension scripts with client_min_messages at WARNING or above). (The seven pre-18 opclasses of the addendum below need no repair there: lionvalidate() accepts their hash functions on every major.)
    • Hook coexistence in the matrix - done 2026-09-23, see “Hook coexistence” below.
  • Citus and TimescaleDB compatibility - verified 2026-10-12 against Citus 14.2 and TimescaleDB 2.30.2 on PostgreSQL 18.6; the findings are in the addendum “Citus and TimescaleDB” below. These are the environments the extension is most likely to run in. Verify, with a test matrix run against each:
    • Citus: distributed and reference tables with lion indexes (CREATE INDEX propagation via the normal DDL path; the opclasses, reloptions and the citext companion must be creatable on workers); the count pushdown must run on the worker shard queries (the coordinator sees a per-shard count/GROUP BY fragment, which is our single-table shape) and must be safe under Citus’s use of the create_upper_paths_hook and set_rel_pathlist_hook (chain, never replace); shard rebalancing (indexes on moved shards rebuilt correctly); citus.enable_repartition_joins; citus_columnar tables are a separate item, below.
    • citus_columnar — secondary; a bonus after bare Citus and TimescaleDB are verified (a table access method, not a planner layer). Two facts drive it: it has no visibility map and its MVCC is stripe-level, so the heap-skipping count is unavailable as is; and its TIDs are synthetic (stripe row numbers), with offsets far beyond the heap’s per-page maximum, so the 9-bit offset encoding of §2 cannot represent them and lion_check_key_offset() errors at build. Two levels of support, in order: (1) correctness — a lion index on a columnar table must either be refused with a clear error or work through the ordinary bitmap path with a TID encoding chosen per table AM (a wider offset field for non-heap AMs; containers cover fewer “pages” then), while the count pushdown declines the table via the table-AM check the heap recheck already uses; (2) value — a count pushdown for columnar needs a visibility source in place of the VM: columnar’s stripe metadata records fully visible stripes and stripes with deletions, which is a coarser equivalent; the §9 interlock would have to be re-derived against columnar’s own vacuum before any stripe is counted without a row visit. Level (1) should hold before release (an index on a columnar table must not corrupt or error obscurely); level (2) is optional and decides whether lion is useful on columnar rather than tolerated. Level (1) done (2026-09-23), as the refusal: ambuild refuses every table AM whose routine is not the heap’s, with a clear FEATURE_NOT_SUPPORTED error, and the count pushdown and the SQL counts decline or refuse such a table too; the rules, the reasons and the tests are in §2. Verified against a copied-heap test AM (test/modules/lion_hooktest), not against citus_columnar itself, which still has to be run once when the Citus matrix exists. The same rule applies to a TimescaleDB chunk kept in another table AM (its hypercore AM): the index fails to build on that chunk rather than skipping it, and whether that is the behaviour wanted there is for the TimescaleDB item below to decide.
    • TimescaleDB: hypertables (indexes created per chunk through Timescale’s DDL hooks; the pushdown’s partitioned-parent path in §16 sees a hypertable as an inheritance parent with chunks as children — confirm the AppendRelInfo mapping and the rte->inh handling; Timescale also installs planner hooks that must be chained with ours), compressed chunks (no lion index on compressed chunks; the node must decline or the planner must not offer it for those children — compressed chunks are a different table AM), continuous aggregates (their materialized hypertables are ordinary hypertables), timescaledb.enable_chunk_append, and retention drops (a dropped chunk takes its indexes with it; nothing to do but test). Time-bucketed columns are a natural lion key (time_bucket(...) as an expression index is NOT supported — only plain columns — so this means a stored bucket column; document that).
    • Both: hook chaining order (load order via shared_preload_libraries vs LOAD-on-first-use), the pg_lion.* GUC prefix under their GUC validation, EXPLAIN output through their custom nodes, parallel-plan interaction (we are parallel-unsafe; their planners must respect it), and the supported PostgreSQL major versions (they lag master; this decides which release to target).
  • Range/zone-map opclass family (last; limited value): bucket a continuous column into value ranges, one posting set per bucket; a range predicate becomes the sum of the fully covered buckets' cardinalities plus a heap recheck of the two edge buckets' rows; GROUP BY width_bucket(...) is a header read per bucket. ORDER BY is not served (bitmaps deliver heap order). A bitmap zone map, not a btree substitute.
  • Plain index scans: implemented, see §29. amgettuple on every major (16 .. 20): the scan streams the bitmap path’s own set algebra one container at a time, drops every pin between calls under an MVCC snapshot and keeps the batch’s page pinned under any other (exclusion constraints become possible). Its follow-ups, in the order the user set on 2026-09-24:
    • a CustomScan that walks a BTREE on the ORDER BY column and tests each TID against lion’s exact WHERE set, stopping at LIMIT n (PG16+, DESC and multi-column order from the btree): designed in §30 (LionOrdered), which builds its set from the scan’s source of §29.8;
    • ordered lion scans (amcanorder, 18+): deferred, §29.8 records what they would take;
    • index-only scans (amcanreturn, §29.9) and backward scans (§29.10).
  • Range predicates on the scalar classes: see §28. <, <=, >=, > and BETWEEN as a bounded walk of the sorted directory, in the bitmap scan and - bounding the driving entry walk - in the count pushdown. Its follow-ups, in order of value:
    • DONE (2026-09-27, §32 “A range as a source”): a range on a column that does NOT drive the count (g, count(*) ... WHERE <range on k> GROUP BY g, the commonest shape of all; a range under an OR; a range as an FK-join fact filter) is a union SOURCE, collected per container key into one private set, past the memory into a temporary file - or walked at every count where the planner finds that the cheaper. Left: a walked OR arm (inclusion-exclusion over the range’s pieces), and the bitmap scan of a range reading summaries;
    • min(k) / max(k): walk from either end of the column to the first entry with a visible row (§26’s existence test), printing its key under the value-representation contract; max needs the backward walk §21’s leftlink was kept for;
    • DONE (2026-10-28, §28 “Cross-type date and timestamp”): cross-type date/timestamp members (datecol < now()), with their equality;
    • carry the recheck batch across the entries of a range SUM, as §15’s disjoint sum carries it across a list’s sets: the entries' counts are only added up, so their dirty TIDs may be rechecked together, one visit per page per batch, instead of per entry (§28’s measured 68,384 visits for 37,133 pages, and a model that has to charge them). (Done a LEAF at a time for small entries since §28’s “Counting a walk”; entries counted one by one still flush their own);
    • a plan-time estimate of a range’s entries that does not rest on n_distinct and the histogram alone (§28, “The cost of a summed range”): descend to both bounds and estimate the leaves between from the internal levels, as InnoDB’s records_in_range does;
    • DONE (2026-09-27, §32): summary posting sets per key bucket, so that a range is the union of the summaries of the buckets it covers whole plus exact work at its two edges (format 7, opt-in with the summaries reloption). Left: splitting a middle bucket that inserts grow (REINDEX rebalances today), per-column control of which columns are summarized, and auto as the default once its insert overhead is judged acceptable;
    • DONE (2026-09-27, §10): a STABLE bound expression in the count pushdown (ts > now() - interval '1 day', the commonest range of all). The node takes any expression an index scan would take as a run-time key, for every clause kind and not only a range bound, and evaluates it once per scan beside the Params.

§23 addendum: Extension versions (2026-10-28)

The SQL objects of an installed extension change only through ALTER EXTENSION ... UPDATE, so every change to an install script needs a new version name and an upgrade script that makes the same change to an installed copy; without one, an installed copy keeps whatever it was installed with (an old OUT row type, a missing function, a removed one, a class without its newer strategies) while the library underneath it moves on.

The rules (test/upgrade/lint.py checks them in CI, upgrade.yml):

  • Each extension has one install script, <ext>--<default_version>.sql; older versions are reached through upgrade scripts <ext>--<from>--<to>.sql, and a chain of them leads from every version that has one to the current one.
  • default_version is X.Y.Z at a release and X.Y.Z-devN between releases, where X.Y.Z is the release being prepared. META.json’s version and each provides.<ext>.version are that X.Y.Z, without the -devN; provides.<ext>.file is the install script, <ext>--<default_version>.sql.
  • Changing an install script that the base branch has under the same version is refused: rename it to the next -devN (git mv), set default_version and META.json’s file, and add <ext>--<old>--<new>.sql doing the same change with plain DDL - CREATE OR REPLACE FUNCTION, DROP FUNCTION and CREATE FUNCTION where the arguments, the result or the OUT columns change, ALTER OPERATOR FAMILY ... ADD for new strategies and support functions, CREATE OPERATOR CLASS for new classes. An upgrade script, once on main, is never changed or removed.
  • A release turns X.Y.Z-devN into X.Y.Z by renaming the install script unchanged, with an upgrade script between the two that holds only comments.

The extensions are versioned separately: a change to pg_lion’s SQL does not bump pg_lion_citext.

test/upgrade/run.sh (also make upgradecheck) proves the scripts: for every version of every install script in the git history from test/upgrade/floor on, it installs that script under its version, creates an index on every operator class it has (so that the update meets families in use), runs ALTER EXTENSION UPDATE, and requires test/upgrade/catalog.sql’s dump of the extension - its members, every function’s signature, result, volatility, parallel safety, strictness, ACL and comment, every operator family’s operators and support functions, the classes and their defaults, the initial privileges - to equal a fresh install’s, every index to REINDEX, and the objects the update left in the extension’s schema outside any extension to be exactly those its WARNINGs say it detached. The one difference it allows is whether a class holds a member or it is loose in the family: ALTER OPERATOR FAMILY ... ADD can only add loose ones, and that only changes what DROP OPERATOR CLASS takes with it. A script for another major (before 2026-09-24 they named hashbool(), 18’s) or naming a C function the library no longer has is skipped and reported; any other install failure fails the check, and the newest earlier one must install.

0.3.0 starts afresh (2026-11-01). Each extension’s SQL is one install script, <ext>--0.3.0.sql, with no upgrade script from any earlier version, and the on-disk formats start again at 1: the lion meta page’s LION_VERSION and lionvec’s LIONVEC_VERSION. An index written by an earlier build is refused with a hint to REINDEX it (§21, “An index of another format”), and an extension installed by one is dropped and created again. test/upgrade/floor names 0.3.0: run.sh skips install scripts in the history older than it, and lint.py makes no comparison with a base branch whose version is older, so the rules above apply from 0.3.0 on.

§23 addendum: hook coexistence (verified 2026-09-23)

What pg_lion installs into the server, all of it from _PG_init() (lion_am.c, lion_plan_hooks.c, lion_ordered.c, lion_bm25_scan.c, lion_selfuncs.c, lion_wal.c): create_upper_paths_hook and set_join_pathlist_hook; set_rel_pathlist_hook three times over - LionOrdered (§30), LionBm25 (§39) and the plain-scan repricing with the row correction (§33, §40), installed in that order, so the repricing runs first -; get_relation_stats_hook and get_index_stats_hook (§33, §41); and nothing else among the hooks - no planner, executor, ProcessUtility or object-access hook -; a custom WAL resource manager, only while shared_preload_libraries is processed; its GUCs, with the pg_lion prefix reserved by MarkGUCPrefixReserved(). Every hook saves the previous value and calls it: all but one FIRST, adding their own path after (lion_create_upper_paths(), lion_ordered_set_rel_pathlist()), so whichever extension was loaded before pg_lion has already added its paths when pg_lion adds its own; the plain-scan repricing calls it LAST, so that the plain paths are repriced before anything compares against them. An extension loaded after pg_lion reaches pg_lion through its own chaining, when it chains: Citus does not, for two of these hooks (the addendum “Citus and TimescaleDB” below). Nothing is ever uninstalled: core has not unloaded a library since PostgreSQL 15 (no _PG_fini), and _PG_init() runs once per process, so there is no reload-order case beyond the two load orders. No chaining bug was found.

make hookcheck (test/hook-check.sh, in CI on every server job) checks it against test/modules/lion_hooktest, a test-only module that chains the same hook, counts its calls, notes whether a LionCount path was already in the grouped rel when the previous hook returned (which tells which of the two hooks was installed first), registers a resource manager of its own when preloaded (id 128, RM_EXPERIMENTAL_ID, by default; pg_lion’s is its reserved 170) and reserves its own GUC prefix. Against the dev cluster, restarted with the options on pg_ctl’s command line:

  • nothing preloaded, lion_hooktest LOADed and pg_lion then loaded on first use, by the planner in the middle of planning the first query over a lion index; and the other way round;
  • shared_preload_libraries = 'pg_lion,lion_hooktest' and 'lion_hooktest,pg_lion', each on its default id (pg_lion 170, lion_hooktest 128): both resource managers registered, an index built in rmgr mode, written, vacuumed and verified next to the other;
  • in each of those four, the other hook runs on every query, sees pg_lion’s path exactly when it was installed after pg_lion (6 times per test file, or 0), LionCount is in the plan for a plain count, a GROUP BY and a partitioned GROUP BY, the answers equal those with the pushdown off, and SET pg_lion.<anything else> and SET lion_hooktest.<anything else> are errors; since §30 the module chains set_rel_pathlist_hook as well, and its hook finds the LionOrdered path in the base rel exactly when pg_lion’s was installed first (twice per test file, or 0), with the ordered answer equal to the one with pg_lion.enable_ordered_scan off;
  • both preloaded on id 170 (lion_hooktest.rmgr_id = 170, another extension taking pg_lion’s id): the postmaster refuses to start with core’s failed to register custom resource manager "lion_hooktest" with ID 170 / Custom resource manager "pg_lion" already registered with the same ID - the collision the pg_lion.rmgr_id GUC exists to move out of;
  • two restarts that have nothing to do with the hooks and are here because only a restart sets them: shared_buffers = 128kB, the counts of §15’s pin budgets on sixteen buffers; and, on 16 alone, old_snapshot_threshold = 60, under which no lion index is read (§9);
  • HOOKCHECK_FULL=1 also runs the whole regression and isolation suite in both preload orders (done once, on 18: green).

Green on 16, 17, 18, 19 and master. What this does not show is how Citus or TimescaleDB use the hook - whether either replaces paths, or plans the query elsewhere, before pg_lion sees it; that is still the Citus and TimescaleDB item’s to test.

§23 addendum: Citus and TimescaleDB (verified 2026-10-12)

Run against Citus 14.2.0 (the first release that accepts PostgreSQL 18; 13.4 refuses it) with citus_columnar, and against TimescaleDB 2.30.2, both on PostgreSQL 18.6 with lion at 87c0404; the scripts, outputs and full reports are kept outside the repository. Every answer - more than 2,000 query runs under Citus over distributed, reference, partitioned and co-located tables in every planner mode and all three preload orders, 186 query-and-variant runs under TimescaleDB in three load arrangements - equalled the same query with the pushdown off and with sequential scans only, and lion_index_verify(idx, true) passed on every shard and chunk index after every operation: index DDL (reloptions, CONCURRENTLY, REINDEX, timescaledb.transaction_per_chunk), shard moves (logical under concurrent writes, and block_writes), rebalance, splits, create_distributed_table_concurrently, undistribute_table, repartition joins, chunk compression, decompression and recompression, reorder_chunk, merge_chunks, split_chunk, continuous aggregate refresh, retention, and a worker crash replayed under the resource manager. pg_lion’s own regression and isolation suites pass against a Citus coordinator and against a server preloading TimescaleDB. What the matrix found:

  • Citus does not chain two hooks. Its shared_library_init.c assigns set_rel_pathlist_hook and set_join_pathlist_hook without saving the previous value, so under shared_preload_libraries = 'pg_lion,citus' lion’s LionOrdered, LionBtreeScan, LionBm25, LionSemiJoin/LionAntiJoin and plain-scan repricing never run; the count pushdown, on create_upper_paths_hook, does, and answers are right. 'citus,pg_lion', and lion loaded on first use next to a preloaded Citus, work in full. Documented as “list citus first” (README, REFERENCE); the fix belongs in Citus.
  • The index-name SQL functions are node-local. lion_index_count(), lion_index_facets(), lion_bm25() and lion_bm25_score() open the index they are named, and Citus rewrites the table of a distributed query to its shard but not a regclass literal, so on the coordinator and on every worker they read the distributed table’s empty shell index: 0, no rows, and a BM25 score of 0 for every row. The ORDER BY lion_bm25_score(...) LIMIT n path (§39) plans on a shard’s own index only when the shard index is named. Documented; run_command_on_shards() is the way to call them.
  • Citus rewrites avg(col) to sum(col), count(col) per shard, and §37’s aggregates over keys decline that pair, because count(col) is answered as a count with col IS NOT NULL, a WHERE. An extension of §37 that returns a column’s non-NULL weight sum beside its sum would answer it; not built.
  • A shard copied by a move, rebalance or split has no visibility-map bits until VACUUM, so the pushdown rechecks every row there and the planner prefers a sequential scan of that shard.
  • TimescaleDB plans a hypertable as an old-style inheritance parent, which lion_count_path_rel() declines (§16 takes declarative partitions only), so no count over a hypertable uses the pushdown: TimescaleDB’s chunk-wise partial aggregation runs over per-chunk lion bitmap scans instead, 2 to 9 times slower than the same count written as a UNION ALL of one LionCount per chunk. Extending §16 to inheritance children is the follow-up with the most value under TimescaleDB. Its obstacles: the parent may hold rows of its own (core’s expansion lists it as a child, TimescaleDB’s does not), a child may be a foreign table (tiered chunks), and a partially compressed chunk keeps its lion indexes while its heap holds only the uncompressed rows, so the path must decline unless every live child is a plain heap with scan paths of its own, and it has to compete on cost with the chunk-wise paths TimescaleDB adds to the same upper rel.
  • TimescaleDB 2.30 plans a SELECT ... LIMIT n over a hypertable through its DeferredChunkAppend with the root’s rte->inh cleared: it builds the root’s index list, replaces the root’s pathlist with that node, and only then calls the previous set_rel_pathlist_hook. Lion’s base-rel paths saw a plain table with lion indexes and added LionOrdered paths over the empty root; they lost on cost at every setting tried, so no wrong rows were returned, but nothing but cost kept them out. lion_rel_is_unexpanded_parent() (lion_plan_hooks.c) now declines a relation that has inheritance children but is planned without them - in the count pushdown, LionOrdered, LionBtreeScan, LionBm25 and the FK-side joins -; FROM ONLY parent of a plain inheritance tree loses lion’s custom scans for it, which core’s scans of the same indexes replace (test/sql/only_parent.sql).
  • Compressed chunks keep their lion indexes, and lion plans nothing on them; a mix of compressed and uncompressed chunks counts right. A chunk in a non-heap table access method makes the hypertable’s CREATE INDEX ... USING lion fail with §2’s error rather than skip the chunk, which is the right outcome: a skipped chunk would be an index that silently misses rows. (The hypercore access method is gone from 2.30; the case was run with lion_hooktest’s copied-heap AM.)
  • The expression index on time_bucket() builds on every chunk and is not used for counts on the hypertable (§41’s rule for partitioned and inheritance parents); a stored bucket column is.
  • TimescaleDB-side: it does not reserve its timescaledb. GUC prefix; its SkipScan ignores PostgreSQL 18’s disabled_nodes (one plan diff in the distinct_unique test inside a TimescaleDB database, same answer); a build that fails under transaction_per_chunk leaves an invalid parent index whose chunk indexes keep being created.
  • Not covered: majors other than 18 (Citus 14.2 and TimescaleDB 2.30 support 15 to 18); level (2) of the citus_columnar item above; Citus HLL distinct counts, MERGE, CDC, tenant schemas, a replication factor above 1; TimescaleDB space partitioning, tiered chunks, hierarchical continuous aggregates; the injection-point specs, the recovery harness and the soak under either.

§23 addendum: pg_upgrade with lion indexes (verified 2026-09-23)

pg_upgrade restores the catalogs with pg_dump --binary-upgrade and copies the index files as they are. Nothing in the on-disk format (version 6: meta page, directory, entry and container pages, WAL mode recorded per index) depends on the server major, so the new server’s meta-page check accepts them and no REINDEX is needed. Run 16 -> 19beta4 and 18 -> master (20devel), both clusters with shared_preload_libraries = 'pg_lion', over a 200k-row table with ten lion indexes (int, text, citext, bool, date, bytea, int[], a multicolumn one, and one each built with wal_mode = rmgr and generic), after deletes, VACUUM and a dirty update:

  • lion_index_verify(idx, true) passes on every index after the upgrade, entry and TID totals are unchanged, and the LionCount answers (equality, GROUP BY, multicolumn, citext, arrays) are identical before and after. Inserts, deletes and VACUUM on the new server, then lion_index_verify again, a LionCount answer checked against a sequential scan, and a new index build all pass. Both WAL modes keep working: WAL does not cross pg_upgrade, so the resource manager id does not matter to it.
  • Bug found and fixed: with the library preloaded, pg_upgrade failed at “Checking database user is the install user” with access method "lion" does not exist. The planner hooks run in every database, and lion_get_am_oid() looked the access method up with missing_ok = false, so any aggregate with a WHERE or GROUP BY in a database without the extension (template1 here) raised an ERROR. It failed cleanly, but it also broke every such database of a server that preloads pg_lion for rmgr mode. The lookup now returns InvalidOid there, and no index or operator matches; test/sql/noextension.sql covers it under installcheck-rmgr.
  • Upgrading from 16 or 17 to 18 or later keeps the seven opclasses that the install script creates through lion_create_opclass_pre18() (bool, bytea, date, timestamptz, xid, xid8, cid) on their pre-18 substitute hash functions (hashchar, hashvarlena, hashint4, timestamp_hash, hashint8), because the dump carries the old catalog rows. The indexes are correct: the substitutes compute the same hash, which is why core accepted them before 18. But amvalidate() reported those seven as invalid on the new server, since lionvalidate() accepted the substitutes only below 18, and the function cannot be changed in place: ALTER OPERATOR FAMILY ... DROP FUNCTION 1 refuses because the opclass requires it. Fixed at merge: lionvalidate() now accepts the substitutes on every major (they are hash-identical), so an upgraded index validates as it did before the upgrade. 18 -> master has no such difference; amvalidate is clean there.

24. Multicolumn indexes: independent per-column key sets in one relation (format version 6, implemented)

CREATE INDEX ON t USING lion (a, b, c) builds one relation whose directory holds, for each column, that column’s keys as independent posting sets. It is order-insensitive by construction: a query on any subset of the columns intersects the matching columns' posting sets, exactly as separate single-column indexes would, and it is GIN’s multicolumn model (amcanmulticol = true). A composite tuple key is deliberately NOT offered: a query with one fixed shape is btree’s job.

Why one relation rather than n indexes: one build pass over the heap instead of n (n tuplesort inputs from one scan), one VACUUM pass, one relcache entry and one planner IndexOptInfo, and a natural home for the count pushdown to find every column’s posting sets without matching n indexes.

Directory. The directory key is (column number, key): the directory comparator compares attno first, and every entry tuple carries its attno in a uint16 of its header (§21, “Entry tuples”). Reserved entries (NULL, and the key-less ones of §17) exist per column. The leading attno term is compared before any opclass function is called, which is what lets one descent walk past the entries of columns whose key type it knows nothing about; the minus-infinity downlink of an internal page carries attno 0 and still sorts below everything.

Each column has its own opclass, so the cached state became two structs: LionState is ONE key column - key type, collation, hash, equality, ordering, ordered, multi-key extraction, plus its attno and a pointer back to the index - and LionIndexState holds the meta page and the array of them. lion_get_state() returns column 1, so every caller that predates this section compiles and means what it meant. A LionSearchKey carries the column it is searching and that column’s state, which is what makes the comparator a pure function of (item, search key).

Lookup: (attno, key) descent. Entry scan for a GROUP BY: from the first entry of that attno to the first entry of attno+1 - a descent to (attno, MINF), a bounded leaf walk that stops at the first entry of the next column, still sorted for an ordered opclass. IN lists: per column.

Build: one heap scan feeding one accumulator PER KEY COLUMN, drained into the shared directory in attno order (“Build: appends, not a sort” below; until 2026-09-27 it was one tuplesort per key column). (Deviation from the first draft, which said “one tuplesort of (attno, kind, key, hash, code)”. One tuplesort needs one tuple descriptor and one sort operator per sort key, and the columns of a multicolumn index have DIFFERENT key types, so there is no single key column to describe. n sorts from one scan is what this section’s own rationale asks for, it compares nothing across columns, and it keeps each column’s sort keys exactly what §21 chose for that column. The accumulators kept all three properties.) The directory order leads with the column, so the columns' entry runs laid end to end are already sorted and the bottom-up level builder of §21 never learns that more than one column exists.

Insert: one aminsert call produces n key inserts, each its own posting-set update under its own entry’s directory leaf, exactly as a multi-key opclass already produced several. VACUUM is unchanged (posting sets are per entry); it carries the entry’s attno next to its kind and key in the identity check that survives a leaf split. Stats: lion_index_stats() returns ONE ROW PER KEY COLUMN with a leading attno; entry- and posting-set-derived counters are that column’s own and relation-wide ones (the directory’s shape, free and deleted pages) are repeated on every row. A container page is attributed to a column through its owner_head stamp (§18) and the attno of the entry that owns that head, accumulated per posting set and folded in at the end; a single-column index skips that and attributes everything to column 1. verify() checks the attno of every entry (in range, never below the entry before it), at most one of each reserved entry per column, and - with heapallindexed - every heap row under every key of every column.

Scans (amgetbitmap): the scan keys arrive with sk_attno; each key resolves to its column, every set qual of every column is answered (§5 SCAN step 5: several on one column are ANDed like the columns) and the trees are ANDed through the expression evaluator of lion_expr.c - the same evaluator a multi-key query’s AND/OR tree goes through, because “the sets of a and the sets of b” is an AND of set trees whatever produced them. A qual the sets cannot express is DROPPED and the TIDs are marked for recheck, which is always correct because the bitmap heap scan re-applies the original quals: IS NOT NULL (the complement of a set), a multi-key query the extractor answers with LION_QMODE_ALL, and a range beside its own column’s sets. IS NULL per column is that column’s reserved entry. A RANGE column (§28) is a walk and not a set: its entries are ANDed with the other columns' answer a container at a time, a window of that answer per walk (§28, “Bitmap scans”).

amoptionalkey is TRUE (it was false). Without it the planner refuses any path that does not constrain the FIRST index column, which for independent key sets is meaningless - WHERE b = 1 on (a, b) would not use the index at all. GIN sets it for the same reason. It also lets a PARTIAL index whose predicate the query implies be scanned with no quals at all, and liongetbitmap() answers that by emitting every row one key column holds, the reserved NULL entry included, with no recheck.

Count pushdown (implemented, §10). lion_find_roaring_index() returns the index AND the key column for a heap column - any key column, not just the first - and the WHERE clause / GROUP BY / IN / OR / multi-key rules are all per (index, key column): every opfamily[0], opcintype[0] and indexcollations[0] became [i], every lookup and entry scan its _col form, and lion_extract_query() and the datumCopy() of a printed key take lion_index_column_state(index, attno). A query constraining a, b and c of one multicolumn index becomes three posting-set sources from one index (the same as three indexes, but located through one relcache entry), and a two-column GROUP BY may take BOTH columns from one index, which is this section’s natural use.

The column does NOT travel in custom_private. The executor derives it in lion_open_relation() from the index it really opened and the clause’s heap attnum (lion_index_col_for()), because a partition’s index may order the same columns differently from the parent’s - and may number the heap column differently too, which is resolved by name (lion_heap_attno_in()). So the planner has only to be right about WHICH index. The executor matches the column by the planner’s own rule: the first key column on that heap column whose opclass is multi-key exactly when the use needs one - a multi-key clause does, every other clause and every driving column needs a scalar one. An index may list one heap column twice, (tags array_ops, tags <a whole-array class>), and the two columns hold different entries, elements under one and whole arrays under the other; the first match on the heap column alone used to be taken whatever its opclass, so count(DISTINCT tags) counted elements, and the reversed index handed tags @> '{1}' to the scalar column, whose state has no extraction function to call (2026-09-27 review). lion_extract_query() now refuses a scalar column’s state with an error instead of an Assert. The one thing the executor cannot derive is the driving column of a sum-over-all (§14), which has no group column at all: it is the column of the first IS NOT NULL clause, which is exactly the clause the planner drove from. Two IS NOT NULL clauses over one index made that distinction load-bearing - the driver’s own clause is DROPPED and the other column’s NULL set must still be subtracted, and before this the two were told apart by the index Oid alone.

The cost model treats each column’s set as it treats a single-column index of that column: the directory and container page terms are scaled by that column’s share of the relation’s entries, estimated from the heap columns' n_distinct (the bullet in §10 has the formula and its limitations). EXPLAIN prints idx.col for a multicolumn index and nothing for a single-column one. The AM-side API for all of it is lion_index_column_state(index, attno), the attno a LionPostingSet now carries, and the _col forms of lion_posting_set_lookup{,_many,_null} and lion_entry_scan_begin; the wrappers without a column are that column being 1.

One thing the node must NOT do is sum across columns. The disjoint-sum short-circuit of §15 counts the entries of a union separately and adds them up, which is only the count of the union because the entries of one SCALAR key column are disjoint; two COLUMNS of one index are not, and lion_sources_disjoint_sum() refuses a source whose sets differ in attno for that reason. Nothing in the pushdown can build such a source today - only an IN list is marked disjoint, and one clause is one column - but an OR across two columns of one index is exactly the shape the guard is about, and test/sql/multicolumn.sql pins it from both ends: the count equals the union and not the sum, and EXPLAIN ANALYZE reports no posting set summed.

Not in scope: INCLUDE columns (amcaninclude is false and core refuses them) and per-column reloptions. amcanmulticol limits: 32 columns (INDEX_MAX_KEYS, core’s own error); a NULL in one column does not affect the other columns' entries. (Deviation: expression columns and duplicate columns were listed as out of scope and turned out to need no code at all - the column number an entry carries is the INDEX column’s, never a heap attribute’s - so both work and test/sql/multicolumn.sql pins that rather than an error.)

Format: LION_VERSION 6. Reading a version 5 index by treating attno 0 as 1 was considered and rejected: the two formats would then be told apart by a field that means “column one” in one of them and “no column at all” in the other. A version 5 index is refused with the same REINDEX hint every earlier version gets.

Measured (2026-09-22, 1M rows of the quick-v3 scalar heap, assert build)

One 3-column lion index on (c2, c200, c20k) against three single-column lion indexes on the same columns and against three btrees, plus 10,000 inserts with each portfolio in place:

portfolio build size insert 10k rows
lion, one 3-column index 3.01 s 15532032 (15 MB) 0.41 s
lion, three single-column indexes 3.07 s 15572992 (15 MB) 0.41 s
btree, three single-column indexes 1.48 s 21135360 (20 MB) 0.12 s
btree, one 3-column index 1.99 s 30711808 (29 MB) -

The two lion portfolios are the same index in one relation or in three: the same entries, the same posting sets, the same page count to within 40 KB (three meta pages and three roots instead of one), and neither the build nor the insert path can tell them apart, which is the point - the win is one heap scan, one VACUUM, one relcache entry and one IndexOptInfo, not a denser index. What a 3-column BTREE costs by comparison is the composite key it stores: twice the size of all three lion columns together, for an index that answers one leading-column order.

(Numbers from an assertion-enabled server, so they are comparable with each other and not with a release build.)

Not done in this version

Per-column reloptions. A cost model that knows a column’s real ENTRY count: the page terms are scaled by the column’s share of the heap columns' n_distinct, which is the right order of magnitude for a scalar column and an underestimate for a multi-key one (one entry per lexeme, not per value), and the directory HEIGHT is still the whole relation’s. Reading a column’s entry count off the meta page - one counter per column, maintained by build and by insert - would remove both approximations, and lion_index_stats() already computes it the expensive way.

Build: appends, not a sort (2026-09-27, lion_spool.c)

Why. A wide multicolumn index over tens of millions of rows took over an hour to build, on one core with the disk idle, and ~70% of it was tuplesort comparisons: comparetup_heap_tiebreak, attribute fetches, and strcoll() under an en_US libc collation. The sort already led with the key (§21 “Bulk build”), but Lion is for low-cardinality keys, so nearly every comparison TIED on the key and fell through to fetching and comparing the codes; a libc collation other than C gets no abbreviated keys, so the key comparison itself was a strcoll() every time. (The report proposed to stop leading with kind first; that was already the case, and what was left of the sort’s cost was the codes, which the change below takes out of the sort altogether.)

Nothing sorts the TIDs. A serial heap scan delivers them in ascending order, so each row’s code is APPENDED to its key’s entry in a per-column accumulator, and only the DISTINCT keys are sorted: K log K comparisons for K keys, where the tuplesort made N log N over all N (key, row) pairs. That is GIN’s shape (ginBuildCallback(), BuildAccumulator); a posting set is then written exactly as before, by the same builders, fed the same codes in the same order. (One change on that side: the context a key’s builders are made in now has a first block that holds all four of their buffers, so resetting it after every key no longer hands blocks back to malloc - a fifth of the first build measured below, whose timestamps are all distinct keys.)

  • The accumulator of a column is a simplehash table found by the opclass hash (through murmurhash32(), since nothing says an opclass hash spreads its low bits) and the opclass equality, the same pair the directory uses. An entry holds the key’s stored bytes and its codes as varbyte DELTAS, the first counted from zero - one or two bytes a row for a dense key - and a key with one code keeps it in the entry and allocates nothing. The reserved NULL (§14) and EMPTY (§17) entries are two more entries outside the table, which is all “build them outside the sort” takes once there is no sort.
  • Two exceptions to “ascending”, both handled. A heap-only tuple is reported under its HOT chain’s ROOT offset while the scan walks the page in physical order, so the codes of one heap page can arrive out of order - within that page only. Such a code goes in as a zero delta (no real delta is zero) followed by the code, the entry is marked unsorted, and it is read back one heap page at a time, each page’s codes sorted (at most 1 << LION_OFFSET_BITS). A synchronized scan could start mid-table and wrap around: the serial scan asks for none (allow_sync = false, as GIN’s serial build does) and the parallel one is told none (phs_syncscan = false, set before any participant starts, which is when the heap AM reads it); should the block number go down anyway, every column is spilled at that point, so no run straddles the jump, and the merge below orders codes across runs whatever their order.
  • The key an entry is written with is the one its smallest code came with. It matters for an opclass whose equality is coarser than its bytes ('Alice'/'alice' under citext, 1.0/1.00 as numeric, the SQL-function classes of test/sql/directory.sql §9), and it is what the sort wrote, since it put a key’s smallest code first. A HOT root offset can bring a smaller code late; the entry then takes that row’s key.

Memory is one budget for all the key columns, not an even share each: a boolean column costs a few bytes a row and a column of unique keys the most, and each takes what it needs. The budget is maintenance_work_mem (divided between the participants of a parallel build), and it counts what the columns' entries take - a column’s stored positions (§17) included - and the sorted list of them a spill reads them through, not the empty table and first block every column has whatever the budget - counting those made a 64kB budget on a wide index spill every column at every heap page. It is checked when the scan moves to the next heap page and after every 1024 keys (LION_CHECK_KEYS), since one row of a multi-key column can bring any number of keys, and when it is exceeded the LARGEST columns are spilled until half of it is free: sorted into directory order and written to a logical tape as one RUN, and their memory reset. A run can therefore end in the middle of a heap page or of a row, and the codes a key has on that page are then split between two runs - out of order between them, when a HOT root offset came late. Nothing depends on runs ending at page boundaries: each run holds a key’s codes in ascending order, and the merge orders a key’s codes across its inputs whatever their order. (Until 2026-09-28 the budget was checked only when the heap page changed, and a page of rows with large arrays or tsvectors could add millions of entries between two checks.) Block sizes (the columns' contexts, the arena that entries and keys are carved from) scale with the budget, so a small one is not gone in one block.

The budget holds while the columns are written out too. When the scan is over, the entries of every column not written yet can still be in memory, and a column’s merge needs read buffers, the build’s summaries of that column (§32) take up to a quarter of maintenance_work_mem, and a colliding group (below) sorts its codes, all at once. So before a column is merged the largest columns - that one included - are spilled until what is left in memory, the merge’s buffers and the summaries' share fit the budget together (lion_spool_merge_column()), and a colliding group’s sort gets what is left over. (Before, all of that stacked up to about twice maintenance_work_mem.)

A run is a sequence of records - kind, flags, hash, key length, the key’s stored bytes, then the codes as varbyte deltas in chunks of at most 8kB - and an end marker. A record whose successor in the same run has the same PREFIX (the kind, then the comparison or, for an unordered opclass, the hash) is flagged TIED, which happens when an unordered opclass’s hash collides.

The merge (lion_merge_column()) takes a column’s runs and what is still in memory, k-way through a binary heap ordered by that prefix - the key type’s SortSupport comparison for an ordered column, the hash for an unordered one - and pops every input that ties with the smallest as one GROUP:

  • one key, held in one record by each input (the normal case): its codes are streamed from the inputs in ascending order. The rule is to take the input with the smallest code and stay on it while its codes stay below the runner-up’s, which makes the runs of a serial scan, whose codes follow one another, a concatenation, and a parallel build’s participants, whose codes interleave one block chunk at a time, one comparison per chunk rather than per code;
  • several keys (a collision, within a run or across runs): every record of the group is read once, its key matched to the group’s distinct keys by the opclass equality and its codes put into a tuplesort as (code, distinct key) pairs, in whatever memory the budget has left; the keys are then handed over together in the directory’s full order, their codes interleaved in code order - which is how the sorted build fed the builders of one hash, so their pages are allocated in the same order too. A merge that writes a run (a pass over too many runs, or a parallel participant’s export) does not gather them at all: it copies the group’s records to the run as they are, all of them TIED, and the merge that reads the run gathers them. (Until 2026-09-28 every record’s codes were copied into arrays of 8-byte codes, and again per distinct key: one large key that collided with another took gigabytes.) Matching costs an equality call per record and distinct key, which for an opclass whose hash collides freely - one group for the whole column - is quadratic in the column’s keys; so is its accumulator’s hash table, and so is every lookup in its directory.

More runs than a merge pass reads at once (maxorder, the buffers of one pass taking at most half the budget) are merged in passes first. The distinct keys of a spill are sorted through the key type’s SortSupport, abbreviated where the type offers it (text under C or ICU, numeric, …) and given up as tuplesort gives it up, when the converter’s own estimate says it does not pay. That sort is where leading with the key still pays: a C or ICU collation gets abbreviated keys there.

Parallel build (PostgreSQL 17 and later, amcanbuildparallel; 16’s index_build() asks for workers only for btree). nbtree’s and GIN’s shape: the leader sets up a parallel heap scan, a shared fileset and one TapeShare per participant, launches the workers and takes part itself - always, as nbtree’s and GIN’s leaders do: parallel_leader_participation is Gather’s and Gather Merge’s setting, and no parallel index build of core’s reads it. Each participant scans its block chunks into an accumulator of its own, with maintenance_work_mem divided between them, spills as a serial build does, and at the end merges each key column’s runs into ONE tape for that column (lion_spool_export()). A column’s runs and its tape share a file of the shared fileset, so the tape re-uses the blocks the runs free as the merge reads them. The write that follows is split by column, since no two columns share a key (below). Workers get their own IndexInfo, so expression and partial indexes work, and report reltuples, index tuples and a broken HOT chain through shared memory; how many there are is plan_create_index_workers()’s decision (max_parallel_maintenance_workers, the table’s parallel_workers, 32MB of maintenance_work_mem per participant, parallel-safe index expressions and predicate). (Considered and not done: aligning the scan’s block chunks to container boundaries, 64 heap blocks at 8kB, so that each container’s codes come from one participant. The chunk size is the heap AM’s (table_block_parallelscan_startblock_init()), not something an index AM can set, and the merge’s “stay on the smallest input” already costs one comparison per chunk; alignment would buy nothing measurable.)

The write is split by column (2026-10-20). Once every participant’s tapes are there, the workers take the columns one by one, the largest left first (by the size of the participants' files of it), while the leader writes the first: whoever writes a column imports the participants' tapes of it and merges them as above, so merging, grouping codes into containers, posting and position trees and summaries all run in parallel. The directory, the meta page and the one bulk writer stay the leader’s, and so do the options: every participant scans and writes with the copy the leader took when the build began (lion_build_options(), in the shared state), not with its own relcache’s, which an ALTER INDEX ... SET committed since - it does not wait for CREATE INDEX CONCURRENTLY, whose lock on the index is RowExclusiveLock - would have changed. A worker writes its column with the builders a serial build uses into a file of its own instead of the index: its blocks are numbered by a counter of its own from 0, each finished page goes to the file with its number, and so does each entry tuple, which a serial build would add to the directory there and then; every record says how many blocks the worker had handed out by then. The leader then takes each column in turn: one nobody has taken yet it writes itself, straight into the index, and one a worker took it REPLAYS once the worker is done. Two columns are the leader’s own and no worker takes them: the first, which it can start on at once, and the largest when it is at least as large as all the others together - a worker writing that one would leave the leader replaying it long after the rest are written, at more than the workers saved on them, where the leader writing it itself starts on it as soon as the columns before it are in. At each record of the file it first gives the worker’s blocks handed out since the last record the next numbers of its own counter, then adds the entry to the directory, which opens directory pages exactly where a serial build would - a serial build’s counter interleaves a column’s tree blocks with the directory pages its entries open, and nothing else - so every block gets the number a serial build gives it, and once the few block references a worker writes are renumbered (a tree page’s rightlink, owner stamp and downlinks; an entry’s head, tail and position tree root) the index is the serial build’s, page for page, which the build tests check byte for byte. The leader’s replay costs it the directory and the page writes, and their WAL: timed by hand, about an eighth of what the column cost the worker for a column of a few keys, about half for a column of a key a row, where the directory is most of the index. Memory: each process writing a column takes its share of maintenance_work_mem, and so do the column’s summaries (an eighth of it, and as much again for the tuplesort a large bucket goes to); the leader takes all of it for a column it writes once no worker can still be writing one, as for an index of one column. The workers' files take temporary space up to the size of the columns they write, each deleted once the leader has replayed it.

  • Measured (PG 18.6, 4 cores, maintenance_work_mem 256MB, 3 workers, 3M synthetic rows, each build the mean of two or three runs alternating with the build before the split):
    • seven scalar columns of 10 to 2M keys (191MB index): 5,353 ms before, 4,340 ms after, 1.23x;
    • sixteen scalar columns of 2 to 3M keys (354MB index): 8,549 ms before, 5,985 ms after, 1.43x;
    • a tsvector column and the seven after it: 76.4 s before, 72.4 s after, 1.06x, the workers writing the seven while the leader writes the tsvector column;
    • the seven and a tsvector column after them (77.7 s before, 76.1 s after), and the tsvector column alone (68.1 s, 69.2 s): no change beyond the runs' own spread of about 5%. The tsvector column is most of the write, and one process writes it either way.
  • Not done: splitting one column by key range, which an index of one large column (a single tsvector, say) would need to gain anything: the summaries are cut across the whole column and would have to be stitched. Nor do the workers write their pages into the index themselves: the bulk writer is one process’s, and the block numbers are only known once the leader has replayed the columns before.

Progress (pg_stat_progress_create_index, lionbuildphasename()). The build reports its phases numbered on from core’s initializing, as nbtree’s are: scanning table, whose blocks the heap AM reports; merging runs (workers), a parallel leader’s own export once its share of the scan is done (lion_spool_export()) and its wait for the other participants'; writing columns; and copying columns (workers), the leader’s wait for and replay of a column a worker wrote. Through the write, tuples_total is the (key, code) pairs the spools took (lion_spool_ntids(): a row’s code once for each key column, once for each key of a multi-key column, and once for each slice a sliced column puts it in, §51), each of which the merge hands out once, and tuples_done those written: lion_build_emit() counts them and reports every 64k, when it checks for interrupts, and at the end of each column, and a replayed column adds the count its worker kept (LionBuildColumn) once it is in. An assertion checks at the end that the two agree. The scan’s blocks are cleared at the switch to the write, as nbtree clears them. Only the process whose build the view shows reports, a serial build or a parallel one’s leader, as in core’s builds; a worker counts the codes of the columns it writes and reports nothing.

The index is the one the sorted build wrote, page for page. The same keys come out in the same order with the same codes and the same stored bytes, and the builders and the directory are unchanged, so every block is allocated in the same order. Checked against the tuplesort build by comparing relation files block by block (LSN and checksum masked) over a corpus of every opclass shape - scalar and multi-key, NULLs and empty arrays, an unordered class whose hash collides into posting trees, a coarse-equality class, HOT chains, partial and expression indexes, dead rows, empty and all-NULL tables - built in memory, at 64kB (up to a few thousand runs, merged in passes), in parallel, in parallel at 64kB, and by REINDEX CONCURRENTLY serially and in parallel. test/sql/build.sql keeps the in-memory, spilled and parallel builds of such a corpus byte-identical to each other (unlogged tables, whose pages carry no LSN) and checks each column’s keys, TIDs, NULL and key-less rows against the heap.

Measured (2026-09-27, 2M rows each, assert-enabled PostgreSQL 18.6: ratios, not absolute numbers)

CPU seconds (utime + stime from /proc) of the backend running CREATE INDEX, and for a parallel build of its workers too (the postmaster’s reaped children); maintenance_work_mem 512MB.

index before after, serial after, 2 workers after, 3 workers
5 columns: enum, bool, int (2,000 keys), unique timestamptz, text (50 keys, C) 18.80 s 3.34 s 3.54 s (2.04 + 1.50) 3.79 s (1.94 + 1.85)
11 columns, en_US libc text of 3 to 50,000 keys and one unique, bool, int, two text[] 87.28 s 22.67 s 24.42 s (11.66 + 12.76) 22.77 s (8.39 + 14.38)

Parallel figures are the leader’s CPU plus the workers'. Wall clock went 19.2 s → 3.4 s → 2.1 s (2 workers) for the first index and 90.9 s → 23.2 s → 13.7 s (2 workers) → 10.9 s (3 workers) for the second, on a machine whose other load kept all four cores busy. What the workers cannot take is the leader’s share: the merge of every participant’s tape and the one write pass. At maintenance_work_mem 64MB the two indexes spill 3 and 43 runs and take 3.15 s and 23.74 s serially: a spill costs about what writing and reading the codes once costs, which is little next to a sort.

The unique text column is most of what is left of the second index: alone it takes 16.9 s under en_US against 6.5 s under C and 2.7 s for a unique int column, the difference being the strcoll() calls of sorting two million distinct keys, which a libc collation cannot abbreviate. Both indexes came out byte-identical to the tuplesort build’s, serially and in parallel.

25. Custom WAL resource manager (replacing generic WAL)

Why. Every page change is logged through GenericXLog today: register the buffer (an 8 KB image copy), modify, then GenericXLogFinish() diffs image against page and writes the delta (or a full image for a new page). Measured (§5): a single-TID insert moves ~48 KB of memory inside the lock window, and an unlogged control lifts hot-key throughput only from 643 to 868 tps, so the copy, not the logging, is the cost; mid-chain inserts carry 3x btree’s WAL (quick-v3: 66 vs 25 MiB per 10k rows at 1M). A custom resource manager logs the bytes that changed with no image, no diff, and a lock hold that is one memcpy long. It also lets replay take the locks the §9/§11 argument needs on a standby (a cleanup lock when applying a removal), which would let the standby count from the visibility map again instead of rechecking every TID.

Status: implemented (src/lion_wal.[ch], and every write path in the extension converted). The resource manager is OPTIONAL and per index: a cluster can hold indexes of both kinds, a server without the preload keeps writing generic WAL, and REINDEX moves an index from one to the other. The subsections below say what was built, where it deviates from the paragraphs above, and what it measured.

Registration. RegisterCustomRmgr(rmid, &lion_rmgr) from _PG_init; requires shared_preload_libraries = 'pg_lion' (the API refuses registration after startup). rmid from a GUC pg_lion.rmgr_id (default: one of the reserved custom ids; document the collision rule and how to check pg_get_wal_resource_managers()). Without preload the extension keeps working on generic WAL: the rmgr is optional, chosen at index creation and recorded on the meta page (wal_mode), so a cluster can mix; an index created in rmgr mode refuses to be written by a backend whose server did not register the rmgr (ERROR with the preload hint) but can still be read. (Implemented so, with one exception: an unlogged or temporary index writes no WAL, and is written anywhere.)

The shim (lion_wal.[ch], implemented)

Four calls, and every write path in the extension goes through them:

LionWalState *lion_wal_begin(Relation index);
Page  lion_wal_register_buffer(LionWalState *, Buffer, int flags);
void  lion_wal_op(LionWalState *, Page, uint8 op, OffsetNumber off,
                  uint16 aux, const void *data, Size len);
void  lion_wal_finish(LionWalState *, uint8 info);      /* and _abort() */

lion_wal_register_buffer() hands back the page to work on: the SCRATCH IMAGE in generic mode, where everything is exactly as it was before this section, and the BUFFER’S OWN PAGE in rmgr mode. lion_wal_op() describes one change and is a no-op in generic mode, where the byte-wise diff describes it already. So a call site is written once and reads the same in both modes; what forks is four lines in one file. lion_wal_register_data() exists for record-level payload and is used for the record header alone.

The rule a call site must follow, and the deviation it forced. In rmgr mode lion_wal_begin() opens a CRITICAL SECTION, because the page is modified in place and an ERROR between the first modification and XLogInsert() would leave a page in shared buffers that no record describes. Everything fallible therefore has to happen BEFORE the record opens - which includes looking at the line pointers the record is going to move, overwrite or delete items by (§3: lion_page_check_items(), lion_page_check_alone(), lion_page_entry_fetch()), and the header of a page it only adds an item to (§3: lion_page_check_header()). Three consequences, all of them deviations from what this section assumed:

  • Allocating a page is two steps: lion_alloc_page() (fallible: the free space map, or extending the relation) and lion_wal_init_buffer() (registers the page the caller already has with REGBUF_WILL_INIT and initialises it). Every split therefore works out how many pages it needs BEFORE it opens its record.
  • A container split rehearses its own deletion. lion_split_and_place() asks “do the new items still fit on P once the moved ones are gone?” of a private copy of the page that the deletion is rehearsed on, and allocates the second new page M before the record when the answer is no. A copy rather than arithmetic because PageIndexMultiDelete() compacts, and what a compaction recovers is exactly what the arithmetic would have to guess at. A split copies a page once; the hot path copies nothing at all, which is the whole point.
  • lion_wal_abort() is legal only while nothing has been modified, which it asserts. All four callers qualify: each is a PageIndexTupleOverwrite() that tests before it writes. The “can’t happen” elog(ERROR)s that remain inside a record become PANICs in rmgr mode, which is the honest answer for a page that changed under a lock this backend holds.

Nothing above changes generic mode, where GenericXLogFinish() opens its own critical section.

(2026-10-27: a fifth call, lion_wal_begin_ext(index, maxblocks), opens a record of more than LION_WAL_MAX_BLOCKS (4) buffers in rmgr mode - up to LION_WAL_MAX_BLOCKS_WIDE, 8, the bits of the header’s masks - and makes room for them in the WAL insertion state (XLogEnsureRecordSpace()) before the critical section, since that may allocate; lion_wal_max_blocks() says what an index gets, 4 in generic mode, which is GenericXLog’s own limit. Only the batched adds of a row’s bit slices ask for it (§51, “Insert cost, revisited”), and their lock argument does not rest on the four: every posting leaf after the first is only tried. Every other record keeps to four.)

The record catalogue as implemented

An rmgr record’s payload is, per registered block, a packed stream of eight-byte operation headers {op, off, aux, len} each followed by len bytes. One lion_redo_apply() executes the stream; the record TYPE (the top four bits of xl_info) names the operation for pg_waldump/pg_walinspect and decides which blocks replay takes a cleanup lock on. That is a deviation from this section’s first draft, which gave each record type a payload struct of its own: twelve bespoke redo handlers would have been twelve places for a replay bug, where an operation stream is one, and wal_consistency_checking checks the one.

The main data is a four-byte header, {initmask, cleanupmask, nvisit}: one bit per block for the first two, and the number of barrier ranges that follow the header (below, VACUUM_VISIT). It travels in the record’s own data rather than a block’s so that redo can read it even when every block carries a full-page image.

record          info  buffers  payload
ITEM_SET        0x00  1-8      CONTAINER_ADD or SPARSE_INS on the container
                               page, MINMAX if a segment's range grew, and
                               the entry tuple on the directory leaf.  The
                               hot-key insert record: ~64 bytes.  A row's
                               bit slices (§51, "Insert cost, revisited"):
                               the same on up to seven container pages
                               (three in generic mode), then the leaf with
                               a SETBYTES of each CHAIN entry's counter
                               and a DELTA of each INLINE entry.
ITEM_REPLACE    0x10  2        DELTA (the ranges of the item at its allotted
                               length that differ from what the page holds)
                               or SPLICE (the same with the bytes an insert
                               moved up said once) + MINMAX + the entry.
                               98 B for a dense container whose whole item
                               is 1658; a rewrite that changes more than
                               half the item falls back to REPLACE, which is
                               the whole item at its allotted length with
                               its slack zeroed.
ITEM_ADD        0x20  1-2      DELETE (when replacing) + ADD per item +
                               MINMAX + the entry.  Also every record of an
                               INLINE spill.
ITEM_DELETE     0x30  2        MULTIDEL + MINMAX + the entry.  CLEANUP LOCK
                               on the container page.
PAGE_INIT       0x40  1        INIT + SPECIAL.  A page initialised before
                               anything links to it: the root of a set a
                               spill makes (lion_entry_spill()) and of a
                               new position tree (lion_postree_create()).
                               Every other page is initialised inside the
                               record that links it in.
SPLIT           0x50  2-4      Both halves as INIT + ADD/ADDMANY + SPECIAL,
                               the old right sibling's SPECIAL, and META or
                               the entry.  Covers the directory split, the
                               root split, the container split and the
                               posting-tree root push-down.
DOWNLINK        0x60  1        ADD of one LionPostingPivot into a parent.
SPLIT_CLEAR     0x70  1        FLAGS: clears LION_PAGE_INCOMPLETE_SPLIT.
META            0x80  1        NDISTINCT: the key counts of §33, which
                               VACUUM and ANALYZE write on the meta page,
                               and the bulk deletes of §37, which the end
                               of every ambulkdelete call counts in it.
                               The meta page's other fields change only
                               inside a directory split, whose record
                               carries the META operation.
VACUUM_PAGE     0x90  1-2      MULTIDEL + REPLACE per shrunk item + MINMAX +
                               the entry, or a batch of REPLACEs on a
                               directory leaf's INLINE entries.  CLEANUP
                               LOCK on the page the TIDs leave.
PAGE_DELETED    0xA0  1        DELETED with the safexid, which is LOGGED and
                               not recomputed at redo.  CLEANUP LOCK.
ENTRY           0xB0  1        ADD, REPLACE or MULTIDEL of entry tuples on a
                               directory leaf.  CLEANUP LOCK when VACUUM
                               deletes entries.
VACUUM_VISIT    0xC0  0        The standby barrier on its own: the header,
                               the relation's RelFileLocator and `nvisit`
                               block ranges, no registered block.  Written
                               only when VACUUM has collected
                               `pg_lion.vacuum_barrier_ranges` (1024)
                               ranges with no removal record to carry
                               them.
UNLINK          0xD0  2-4      A page taken out of its tree.  A directory
                               page deletion (§21, "Page deletion"):
                               stage 1, the parent's rewritten downlink
                               and DELETE of the next, and the leaf as
                               INIT + ADD of its MINUS INFINITY high key,
                               HALF_DEAD; stage 2, the left and right
                               siblings' SPECIAL (and the right one's
                               first downlink made MINUS INFINITY when the
                               page was the leftmost of its level), the
                               page as INIT + ADD, DELETED with the
                               safexid, and the leaf's SPECIAL.  An empty
                               posting leaf (§22, "Empty leaves"): the left
                               neighbour's SPECIAL, the leaf's SPECIAL,
                               UNLINKED with its transaction id, and in a
                               tree the parent's rewritten downlink and
                               DELETE of the next.
REUSE           0xE0  0        A DELETED page, or UNLINKED posting leaf,
                               about to be used again: the
                               RelFileLocator, the block and its safexid,
                               no registered block.  Redo resolves the
                               recovery conflict with that horizon, as
                               nbtree's XLOG_BTREE_REUSE_PAGE does, which
                               is what generic mode writes instead.
                               Written only when the standby needs it
                               (`XLogStandbyInfoActive()`).

The operations are: INIT, SPECIAL (the whole 32-byte page special area), ADD, ADDMANY, REPLACE, DELTA, SPLICE, SETBYTES, MULTIDEL, DELETE_NC, DELETE, MINMAX, FLAGS, META, DELETED, CONTAINER_ADD, SPARSE_INS, NDISTINCT (§33). (2026-10-27: SETBYTES, replayed by every build since this section but written by none, is now the counter of a CHAIN slice entry in a row’s batched record - the byte or two of its ntids that changed (§51, “Insert cost, revisited”). It names its bytes, so nothing about masking changes, and wal_consistency_checking checks it under the recovery harness’s slices.)

DELTA is what a rewrite of one item costs now, and it is the one operation whose payload is a function of the page as well as of the record. It says: the item at off becomes aux bytes long, and here are the ranges ({uint16 at, uint16 len, bytes}) in which it differs from the item that is there now, taken as the BASE IMAGE truncated to aux bytes or zero-extended to them. The writer computes those ranges against exactly that base before it overwrites the item - lion_wal_save_item() copies the image, the PageIndexTupleOverwrite() happens, lion_wal_op_replace() does the comparison - so replay reconstructs the writer’s item byte for byte and wal_consistency_checking proves it does. Redo patches the item where it lies when the length has not changed, and otherwise rebuilds it in a scratch buffer and puts it back through the same PageIndexTupleOverwrite() the primary called, which is what keeps the page identical when an item moves.

The allocated length may change, and that is the point of aux: an item that has used up its growth slack (§4) is rewritten LONGER, and the new slack is zeroed on both sides so it costs nothing in the delta. Every call site that overwrites an item uses the pair - a container item, an entry tuple on a directory leaf, VACUUM’s shrunken items and INLINE payloads - and the fallback is automatic: when the ranges would come to more than half the item, the operation is dropped and a plain REPLACE takes its place, so an ARRAY that turns into a BITSET still goes in whole. Nothing about masking changes, because the bytes outside the named ranges are the bytes the page already had.

SPLICE is a DELTA with bytes inserted (lion_waldelta.[ch]). A member added in the MIDDLE of an ARRAY moves every member above it up by two bytes, and a DELTA can only say that by repeating every byte that moved: half the item on average, which is what the first insert into a bulk-built container cost, since such a container has no slack and takes the general path (§4). SPLICE’s payload starts with {uint16 at, uint16 inslen, inslen bytes}, which are inserted into the base image at byte at before it is cut or zero-extended to aux bytes, and the DELTA ranges follow, applied to that. One member added to an ARRAY is then the two bytes, where they go and the new count: 12 bytes of payload where the DELTA carried every member above the new one. The same shape is a run added to a RUN (4 bytes), a pair added inside a sparse segment (a varint and the re-encoded one after it), and any of those inside an INLINE payload, where everything after the item moves up too. The writer does not say which it did: lion_wal_op_replace() hands the old image and the new item to lion_delta_encode_item(), which computes the DELTA and, unless that is already as small as a SPLICE can be, looks for an insertion of 1 to 8 bytes - the smallest at from which the new item is the old one moved up - and keeps whichever is smaller, DELTA on a tie, or neither when both exceed half the item. Redo decodes either with lion_delta_decode_item() into its scratch buffer and puts the item back as for a DELTA; a payload that does not fit the item the page holds (a splice point past it, a range past the new length) is a PANIC, because the record was not written against that page. The encoder and the decoder are pure functions of their bytes, which test/unit/waldelta_test.c round-trips over every container shape, random images and malformed payloads, and wal_consistency_checking checks on every record of the suite.

Measured (PostgreSQL 18.6 release build, rmgr mode, 5M rows, six lion indexes on synthetic columns of 2 to 1M keys, one wave of 50,000 updates that move one row in a hundred to the next key of a 20-key column, on a heap at fillfactor 90 so that the new versions land on the pages the old ones were on; bytes of index WAL without full-page images):

index record before after
20 keys ITEM_REPLACE 629.8 B 96.9 B
20 keys all records, per updated row 439.8 B 163.8 B
200 keys ITEM_REPLACE 152.1 B 97.2 B
20,000 keys ENTRY (INLINE) 523.1 B 165.7 B
1M keys ENTRY (INLINE) 77.9 B 74.8 B

What is left on the 20-key column is the splits of its full leaves, which the posting_fillfactor reloption of §22 is for.

Two operations log the writer’s CALL rather than its bytes, and that is where the hot record’s size comes from. Adding a member to a sorted ARRAY shifts every element above it, so the bytes that change run to the end of the item - up to 4 KB - while “add member lo to the item at offset off” is four. CONTAINER_ADD calls lion_container_add() and SPARSE_INS calls lion_sparse_insert() at redo, on the same item, which is deterministic for the same reason every other operation is and is what wal_consistency_checking checks. Everything else logs bytes.

Offsets are always explicit. A writer that calls PageAddItemExtended() with InvalidOffsetNumber logs the offset it GOT, so replay never has to reproduce the line-pointer search - a page whose unused line pointers differ would otherwise diverge silently.

A directory split logs BOTH halves, which is a deviation: this section preferred logging only the moved half, and that is what a CONTAINER split does (P keeps its items, N and M get theirs). A directory split cannot, because both halves are rebuilt from scratch and the only page the other half could be derived from is the very page being re-initialised - whose pre-image a full-page image would destroy. Both halves are registered REGBUF_WILL_INIT instead, so neither pays for an image at all and the record is the page’s live bytes.

One record type is not enough to decide the cleanup lock, and wal_consistency_checking did not find that one - reading the code did. VACUUM’s REGROW step (§18: a container that grew while it was being filtered and no longer fits its slot) re-places the item through the general machinery of the page layer, the same code an INSERT goes down, so the record it writes is an ordinary ITEM_REPLACE or ITEM_ADD or SPLIT - and it removes TIDs. The fact is a property of the CALLER, not of the operation: VACUUM holds a cleanup lock on that page throughout. So lion_wal_removal_begin()/_end() bracket that window and every record written inside it marks its FIRST registered block - which on every one of those paths is the container page - as needing a cleanup lock at redo. It travels that way rather than through five function signatures because five signatures would have to carry a fact that only one caller in the tree knows.

The same holds for an INLINE spill that VACUUM’s filtering causes, and there the page TIDs leave is NOT the first block (2026-09-22 review). A filtered payload can outgrow its entry (a RUN that loses every other member becomes an ARRAY), and pass 1 then spills it through lion_entry_spill() - the INSERT path’s code, which registers the posting set’s new root first and the directory leaf last, with no cleanup mark. When every changed entry of the leaf spilled, the cleanup-marked VACUUM_PAGE record pass 1 had opened was abandoned and the spill’s records were the only ones that said the entry’s dead TIDs had left the leaf - so replay took no cleanup lock on it, and a standby reader pinning that leaf for its INLINE copy was overtaken (test/recovery/run.sh phase 3, “spill”: 200 for 100). lion_wal_removal_begin(Buffer also) therefore names the buffer as well: inside the window, every record marks its first block AND any block that is also, and pass 1 wraps each spill in a window naming the leaf. lion_wal_finish() then puts the cleanup blocks FIRST in the record (existing pages before pages the record initialises), whatever order the writer registered them in - the block ids are assigned only at XLogRegisterBuffer(), so this is a permutation of the shim’s state - which keeps §11’s waiting rule for the startup process without asking the shared placement code to know about VACUUM. Every record VACUUM writes is now one of: VACUUM_PAGE/ITEM_DELETE/ PAGE_DELETED/ENTRY with its cleanup mark set by VACUUM itself, or a record of the shared placement code written inside a removal window (regrow, spill); the re-descent that finds a moved entry writes nothing.

Since the 2026-09-25 review a spill VACUUM causes may write MANY records (§18, “The spill”): one per leaf, whose only block is the new leaf, and the root with the entry last. All of them are inside the window. A leaf record’s first block is the leaf it initialises, so it is marked too and replays with RBM_ZERO_AND_CLEANUP_LOCK - a lock on a page nobody can have pinned, which costs nothing - and the first of them carries whatever barrier VACUUM has collected. That is before the removal, which is in the last record, and replay applies them in that order, so the barrier still comes first. Each record registers one or two buffers, whatever the number of leaves.

The standby BARRIER, and why the cleanup mask alone is not enough (2026-09-22 review). ambulkdelete cleanup-locks every page that can hold a TID in chain order, and every page of each posting tree’s descent (§11), whether or not it removes anything there; that is what stops it from overtaking a reader whose TIDs a split, a root push-down or a spill has since moved off the page it pins. A page it locks and leaves unchanged writes NO record, so replay never waited for a standby reader’s pin on it, and the removal replay did wait on was on the page the TIDs had moved TO. Phase 3 of test/recovery/run.sh got 50000 for 48548 (split) and 1000 for 500 (push-down) from exactly that. So VACUUM hands every page it cleanup-locks without writing to it to lion_wal_visit(), which keeps a list of block RANGES (consecutive blocks merge), and lion_wal_finish() attaches the list to the next record whose cleanupmask is not empty: nvisit {uint32 start, uint32 count} pairs right after the header. Redo cleanup-locks every block of the list, one at a time and releasing each, with nothing else held, before it takes the record’s own blocks - only while InHotStandby, since there is no reader to wait for in crash recovery. This is nbtree’s old XLOG_BTREE_VACUUM lastBlockVacuumed in its economical form: no record of its own in the common case, eight bytes a range, and the record type VACUUM_VISIT only for a list that grows past the limit with nothing to ride on. Replay meets the barrier for a page before the removal that could hurt a reader pinning it, because VACUUM visits the page before it removes anything from a page right of it and the record is written after the visit; pages visited after the last removal of an ambulkdelete are dropped (every dead TID of the cycle has been removed by then, and a TID that moved off a page later in the walk moved onto a page later still).

Redo takes only the barrier blocks that are in shared buffers (2026-09-27 review). A reader that still pins a page it copied containers out of keeps that page’s buffer from being evicted, so a block the buffer mapping does not have has no reader to wait for; redo looks each one up there (lion_redo_resident_buffer(), the lookup PrefetchSharedBuffer() makes before it starts a read of a block it misses) and pins it with ReadRecentBuffer() only when it is there and valid. A reader that pins the block after the lookup copies the page as replay has left it, which is the position of a reader that pins a resident block the moment the barrier lets go of it. Before, the barrier read every block with XLogReadBufferExtended(), and the ranges of one sparse VACUUM cover nearly the whole index: its replay read nearly all of it, synchronously, a block at a time.

Redo. lion_redo() reads the header, applies the barrier if there is one, then takes each registered block in the order the record lists it, with RBM_ZERO_AND_LOCK for a block in initmask and get_cleanup_lock = true for a block in cleanupmask, and applies that block’s operation stream when the action is BLK_NEEDS_REDO. Blocks that need a cleanup lock are always FIRST (the shim reorders them, above), so the wait happens with no other buffer lock held - §11’s waiting rule, applied to the startup process. Replay takes a posting set’s pages BEFORE the entry’s leaf - the reverse of the primary’s order (§5), because the writer locks the leaf first and registers it last, when the counters go into the record - and generic_redo() takes a generic record’s blocks in the same order and holds them to the end of the record as lion_redo() does. That is safe against standby backends only as long as none of them holds a directory page while it waits for a posting page: a scan walks the leaves one shared lock at a time, a descent releases the parent before locking the child, and §11’s rule for readers already forbids taking a directory lock while holding a container page. (Deviation, found by the 2026-09-29 review: this paragraph said that much and called the order safe, and lion_index_verify() was the exception - on a standby it walked every posting set with the entry’s leaf held SHARE, and its heapallindexed lookups descended a set with the leaf held. The startup process, holding a container page and waiting for the leaf, and the backend, holding the leaf and waiting for the container page, waited on buffer content locks for good. verify() now holds nothing across a set on a standby (§7), and test/recovery/run.sh phase 4 runs it there beside a write load with a watchdog on replay.) Registering the leaf first was not done and would not remove the rule: a removal record’s cleanup block goes first whatever the registration order (above), and WAL already written keeps its order. The records that name more than one block, in the order redo takes them:

ITEM_SET, ITEM_REPLACE, ITEM_ADD         container page, entry leaf
ITEM_SET of a row's bit slices           up to seven container pages (three in generic
                                         mode), each of another set, then the entry leaf
ITEM_DELETE, VACUUM_PAGE (VACUUM)        container page (cleanup), entry leaf
SPLIT of a posting leaf                  P, N (new), M (new, between them), entry leaf
SPLIT pushing a leaf root down           root, new child, entry leaf
SPLIT pushing an internal root down      root, new child
SPLIT of an internal posting page        P, new right sibling
SPLIT of a directory page                P, new right sibling, old right sibling, meta page
SPLIT of the directory root              old root, new right sibling, new root, meta page
ITEM_ADD ending an INLINE spill          new root, entry leaf - but entry leaf, then new
                                         root, both cleanup, for VACUUM's spill in rmgr
                                         mode (the reorder above)

VACUUM’s regrow and the spill its filtering causes write theirs inside a removal window, which cleanup-locks their first block (and the spill’s entry leaf); that changes the order only where it says so. Every other record names one block.

rm_mask() masks the page LSN and checksum, the hint bits, the free space between pd_lower and pd_upper, and the MAXALIGN padding after every item.

That last one is the one thing this section got right for the wrong reason and is worth spelling out, because wal_consistency_checking found it on its very first run and nothing else would have. PageAddItemExtended() reserves MAXALIGN(size) bytes and copies size of them; the bytes in between keep whatever the page held there before. Core’s own access methods do not care, because a core index tuple is MAXALIGNed by index_form_tuple() and there is no padding - but a lion entry tuple is a header, a key and a payload BYTE COUNT, none of them padded to MAXALIGN, and a container item’s size comes out of its header, so both routinely end on an odd boundary. On the primary the padding holds whatever the page held; on a standby the page may have arrived as a full-page image, whose HOLE is restored as ZEROES, and the two pages then differ in bytes no record ever described and no reader ever reads. The first failure was a 54-byte entry tuple on a directory leaf, two bytes wide.

The growth SLACK inside an item (§4) is a different thing and is NOT masked: an item is always logged at its ALLOCATED length with its slack already zeroed (lion_item_zero_slack()), and the three operations that change an item without rewriting it leave every byte outside the range they name untouched on both sides. Masking it would hide a real divergence. Generic mode never had either problem, because a byte-wise diff of the whole image covers padding and slack alike - which is the general shape of what this wave traded away.

Standby (implemented)

With cleanup locks taken at redo for removals, the hot-standby count uses the visibility map again for rmgr-mode indexes. lion_count.c decides it per COUNT, not per index: cx.in_recovery is now RecoveryInProgress() && !lion_sources_all_rmgr(...), because a container of an intersection carries the dead TIDs of every source it came from and the interlock has to hold for all of them - ONE generic-mode source loses it. The argument is written out next to the code and is §9’s with “VACUUM” replaced by “the startup process”: while the backend holds a pin on the page it took a container from, replay cannot have removed a TID from that page, so it cannot have replayed the heap record that set any of that container’s heap pages all-visible. §11’s split hole closes the same way - a page is only ever linked in immediately right of the page whose split created it, and replay applies those records in the order the primary wrote them - but only with the barrier above. (Deviation, found by the 2026-09-22 review: this paragraph first claimed the split hole closed by itself. It does not: the primary’s VACUUM passes the page the reader pins without writing to it, and replay used to take no lock on it. Until the barrier existed an rmgr-mode standby could count a dead row from a copy whose TIDs had moved; the shipped rule stays “trust the map in rmgr mode” because phase 3 below now proves the barrier under wal_consistency_checking. WAL written by a binary without the barrier carries none, and a standby replaying such WAL keeps the old exposure until it has replayed past the upgrade.)

The reader pays for it the way a primary reader does: replay WAITS, which is a recovery conflict resolved by max_standby_streaming_delay rather than a wrong answer. test/recovery/run.sh phase 2 covers both halves in both modes - a generic-mode standby must report blocks_skipped = 0 and recheck every TID, an rmgr-mode one must report blocks_skipped > 0, and the REPEATABLE READ standby session whose rows the primary deletes and vacuums must be either preserved (hot_standby_feedback = on) or cancelled by a recovery conflict, never silently answered with a different number. Phase 3 attacks the pin itself: a standby reader parks at lion-count-containers-pinned (containers copied, page pinned, map not consulted), the primary moves the copied TIDs off that page - a leaf split, a root push-down, a VACUUM-made INLINE spill - and VACUUMs, and the standby has max_standby_streaming_delay = -1. In rmgr mode replay must be seen BLOCKED on the reader’s pin (the startup process waiting for a cleanup lock) and the reader, released then, must answer the count its snapshot sees; in generic mode replay catches up and the reader must still be right, because it rechecked every TID. The split case runs with pg_lion.vacuum_barrier_ranges = 1, so its barrier travels in stand-alone VACUUM_VISIT records; the push-down and spill cases carry it on removal records. All three answered wrong before the barrier and the removal window of the spill (50000/48548, 1000/500, 200/100).

The aggregates over keys (§37) are the one reader that trusts entry HEADERS with no pin held, and on a standby they never do: in recovery every entry is counted, in either mode, so neither the pin nor the barrier is asked of them there. The meta page’s bulkdeletes, which tells them on a primary that a bulk delete finished under the walk, is replayed all the same - it rides in the META record’s NDISTINCT operation, which redo copies whole - so a promoted standby and a recovered primary carry the number the primary had.

A standby query can hold a link to a directory page the primary deletes (§21, “Page deletion”) just as a primary scan can, and the primary’s safexid test does not see the standby’s snapshots. The record that hands the page out again carries the safexid as a conflict horizon - REUSE, or in generic mode nbtree’s REUSE_PAGE - and its replay cancels the standby queries older than it, unless hot_standby_feedback has kept the primary from reusing the page in the first place.

Registration and migration (implemented)

RegisterCustomRmgr(lion_rmgr_id, &lion_rmgr) from _PG_init(), and ONLY while process_shared_preload_libraries_in_progress - so a library that is merely LOADed does nothing. The pg_lion.rmgr_id GUC is PGC_POSTMASTER and is defined in the same guarded block, because core refuses to define a postmaster-level GUC after startup (pg_stat_statements returns from _PG_init() at exactly this point and for exactly this reason). Default 170 (LION_RMGR_ID), the id reserved for pg_lion on the wiki page the core documentation names. The GUC stays - the list is only a convention, and the point of a reserved range is that a site can move out of a collision - and pg_get_wal_resource_managers() is where to look. Once an index has been written in rmgr mode the library must stay preloaded for as long as WAL that mentions it may be replayed, which is the rule core states for every custom resource manager.

wal_mode lives in the META PAGE, in reserved[0], so LION_VERSION stays 6 and a version-6 index written before this section has a zero there, which is exactly “generic”. The reloption wal_mode is auto | generic | rmgr with auto the default: auto means rmgr when this server registered the manager and generic when it did not, which is what lets one CREATE INDEX script work on a cluster that preloads the library and on one that does not. Asking for rmgr by name on a server without it is an ERROR with the preload hint, at CREATE INDEX (from lionoptions()) and again at build. READING an rmgr-mode index works on any server; WRITING one without the manager ERRORs in lion_wal_begin(), which is the one place every write path passes through, with the preload hint and the REINDEX alternative - unless the index is unlogged or temporary, whose writes insert no record at all, so that nothing about them needs the manager (2026-09-27 review: they used to be refused all the same). REINDEX is what changes an index’s mode. lion_index_wal_mode(idx) reports it.

A VACUUM that has dead entries to remove from such an index is a write too, and stops at its first one with the same error - autovacuum included, every time it comes back to the table, so the table’s dead rows accumulate while the error shows only in the server log. The hint names both ways out: the preload, or REINDEX with wal_mode = generic (which auto also gives on a server without the preload). test/recovery/run.sh phase 1f checks the insert, the VACUUM and the REINDEX.

VACUUM is the one path that could write without lion_wal_begin() (2026-09-25 review): the standby barrier’s stand-alone VACUUM_VISIT record is inserted by lion_wal_visit_flush() directly, and lion_wal_visit() looked at the index’s wal_mode and RelationNeedsWAL() and nothing else. So a VACUUM of an rmgr-mode index on a server without the preload that removed nothing - a partial index none of whose rows died - collected its visited pages all the same and, past pg_lion.vacuum_barrier_ranges, wrote them out under lion_rmgr_id, which without the preload is the GUC’s boot value 128 and not a manager this server has: pg_waldump showed them as custom128 records of an unknown type, and crash recovery or a standby stops with a FATAL (“resource manager with ID 128 not registered”) at the first of them. Now lion_wal_visit() collects nothing unless the manager is registered, and the flush refuses to insert without it. Visiting changes no page, so it is not refused the way a write is; the barrier is not needed either, because the only records it could ride on are rmgr-mode removals, which this server cannot write - a VACUUM that has something to remove from such an index still stops at its first write, with the hint. What one that has nothing to remove would write all the same - the meta page’s counts, a directory leaf left empty or HALF_DEAD, a set’s empty leaves, an entry an earlier VACUUM emptied and left, the leaks its sweep finds - it leaves alone (lion_vac_may_write()), for a VACUUM on a server with the manager. test/recovery/run.sh phase 1f vacuums an rmgr-mode partial index on a server restarted without the preload, then crashes it: no custom-manager record may be in the range, and recovery must succeed. Before that it inserts into and vacuums an UNLOGGED rmgr-mode index there, which must work.

Testing (implemented)

  • make installcheck runs the whole suite in generic mode, which is what a server without the preload gives, and make installcheck-rmgr restarts the dev cluster with shared_preload_libraries = 'pg_lion' on pg_ctl’s command line, runs the same suite, and restarts it back. Nothing is written into postgresql.conf, so an interrupted run leaves no trace. WAL_CONSISTENCY=1 adds wal_consistency_checking = 'pg_lion'.
  • test/sql/walrecords.sql drives every write path and reads the WAL back with pg_walinspect. It has TWO expected files, because the point is that the output differs: walrecords.out is the generic run and walrecords_1.out the rmgr one, and pg_regress accepts either. It uses pg_walinspect and not pg_waldump because pg_waldump does not load the module and therefore prints the manager as custom170 (by its id) and every record type as UNKNOWN; pg_walinspect runs in the backend, where rm_identify() and rm_desc() are ours. One trap, found the hard way and now closed in test/rmgr-check.sh: pg_ctl restart REUSES the options in postmaster.opts, so restarting after an -o start keeps the preload and leaves the dev cluster in rmgr mode - and the next plain make installcheck then silently tests the wrong mode and passes, because walrecords.sql’s two expected files make either answer acceptable. Both directions stop and start instead, and the restore prints how many resource managers are registered afterwards.
  • test/recovery/run.sh takes --mode generic|rmgr and a repeatable --conf. make recovery-check-rmgr is --mode rmgr --conf "wal_consistency_checking = 'pg_lion'", and THAT is where the “replay reproduces every page” proof actually lands: wal_consistency_checking only compares during REPLAY, so turning it on for a primary that never replays proves nothing. The harness’s standby replays everything phases 1 and 2 write, and every crash of phase 1 replays its own range; a mismatch is a FATAL in the startup process. Phase 3 (the pinned standby reader, above) makes a standby of its own and replays VACUUM_VISIT and barrier-carrying records under the same check.

Measured (2026-09-22, release build, alternating arms, one binary and one cluster)

The A/B lever is shared_preload_libraries: the benchmark cluster is restarted with and without it, so wal_mode = auto resolves to rmgr or generic and nothing else differs. Arms were alternated within every series. The box was NOT idle (the regression suite ran throughout), so the millisecond columns carry about +/-10% and the WAL columns - which were byte-identical across every repeat - do not. Times are from a -O2 build and are therefore NOT comparable with the assert-build numbers elsewhere in this document; the WAL bytes are, and the generic arm reproduces §5’s “before” to within noise, which is the cross-check that says so.

8-client hot-key burst, 2 key values, synchronous_commit on
                  tps            p95
  generic         765            14.8 ms
  rmgr           1275             9.9 ms
  btree          1632             7.8 ms
  no index       2336-2371         -        (identical in both arms)

... and by client count (tps / p95)
                  1 client      4 clients     8 clients
  generic        456 / 2.9     702 / 8.7     761 / 15.0
  rmgr           515 / 2.7    1165 / 4.1    1313 /  9.5
  btree          522 / 2.6    1185 / 4.0    1600 /  8.2

The hot-key ceiling of §5 is broken, and by more than §5 predicted was there to break. §5 measured an UNLOGGED control at 868 tps and concluded that the 8 KB page copy, not the logging, was the cost. rmgr reaches 1275-1313 tps - PAST that unlogged ceiling, at 66-72% of btree where generic is 47% - which says the copy was indeed the cost and that removing it also removed what the unlogged control still paid (GenericXLog copies and diffs the page whether or not it writes the record). At four clients rmgr is level with btree.

INSERT 10k rows into the 1M-row, 8-index portfolio
                  after a checkpoint        steady state
  generic         922 ms / 88.10 MiB        853 ms / 53.53 MiB
  rmgr            254 ms / 64.53 MiB        142 ms / 29.21 MiB
  btree           169 ms / 33.31 MiB        104 ms /  6.75 MiB

WAL per insert record (10,000 single-row inserts, one index, second batch)
  column   path        generic              rmgr
  c2       container   121.6 B  1.16 MiB    114 B ITEM_SET  1.62 MiB
  c20k     INLINE     2918.7 B 27.83 MiB    422.4 B ENTRY    4.03 MiB
  c1m      INLINE     1682.2 B 16.04 MiB    124.1 B ENTRY    1.18 MiB

8-index portfolio build      generic 4077 ms / 57.47 MiB
                             rmgr    4051 ms / 57.47 MiB
                             btree   2653 ms / 59.25 MiB
portfolio index size         generic 70,852,608 B = rmgr 70,852,608 B

Two of those rows say something this section did not expect.

The build does not move at all, and cannot. wal_mode has no effect on CREATE INDEX: ambuild writes through the bulk-write API (§5 step 4), which logs whole pages with log_newpage-style records and never touches the shim. 57.47 MiB in both arms, to the byte. That is worth saying out loud so nobody expects build WAL to move; lion’s build already writes less WAL than btree’s.

The container path writes MORE WAL, not less: 1.16 -> 1.62 MiB per 10k inserts, +40%. The cause is visible in the record histogram: 9,631 of the 10,000 inserts are a 114-byte ITEM_SET, which is the record this section was designed around and is 8 bytes cheaper than the generic delta - but 361 of them take the general path and emit an ITEM_REPLACE averaging 1658 bytes, which is the whole rewritten container item, where GenericXLog’s byte diff of the same operation cost about the same 122 bytes as everything else. §25 specified ITEM_REPLACE as “(block, offset, newlen, bytes)” and that is exactly what was built; for a 1.6 KB container it is the wrong trade. It costs no latency - the record is written outside the lock window that matters and the burst numbers above are what they are - but it is a real byte regression on dense keys, and the fix is known: an item that is being rewritten at the SAME allotted length differs from its predecessor in a handful of bytes, so ITEM_REPLACE should either carry a delta or be split into a shrink/grow plus an ITEM_SET. It was the first thing to do in this area and it is done: LION_OP_DELTA above, and the re-measurement below.

The INLINE path is where the win is, and it is the win §25 predicted for a feature it did not build: 2918.7 -> 422.4 B and 1682.2 -> 124.1 B per insert, 7x and 14x, purely from logging the rewritten entry tuple instead of a page diff that covers the whole tail of the leaf. INLINE entry slack (the “also” item below) would cut it again - and it does, in both arms, below.

Not measured, and honestly so: the quick-v3 mid_insert at 1M rows and the VACUUM micro-benchmark. Both harnesses build their own cluster in ways that have no hook for shared_preload_libraries (bench/comprehensive/run.py writes its own postgresql.conf; bench/vacuum_micro.sh points at the dev cluster), so neither can be run in rmgr mode without changing them. mid_insert is dominated by the same INLINE entry rewrites that dropped 7x above, so it should fall well below its 66 MiB, but that is a prediction and not a measurement.

(One trap for whoever repeats this: pg_ctl restart re-uses the previous postmaster’s options out of postmaster.opts, so a plain restart after a preloaded start stays preloaded and the “generic” arm silently is not one. Pass -o "", or stop and start. test/rmgr-check.sh had this bug and now stops and starts in both directions and prints how many resource managers are registered afterwards.)

Measured again after the delta and the INLINE slack (2026-09-22)

Same box, same release prefix, same 1M-row cluster and the same two arms; the A/B swaps only the installed pg_lion.so (base is the commit above, new is this one) and restarts the cluster with or without the preload. bench/write_micro.sh grew LION_WM_PGBIN and LION_WM_PRELOAD so that this is one command per cell. WAL bytes were byte-identical across repeats; the millisecond columns are medians of 2-5 runs on a box that was not idle.

WAL per insert record (10,000 single-row inserts, one index, second batch)
  column  path        generic  before -> after   rmgr  before -> after
  c2      container   121.6 B  -> 121.6 B        114.0 B ->  79.0 B  ITEM_SET
                                                1657.8 B ->  98.4 B  ITEM_REPLACE
  c20k    INLINE     2918.7 B  -> 173.0 B        422.4 B -> 180.5 B  ENTRY
  c1m     INLINE     1682.2 B  -> 106.8 B        124.1 B ->  82.0 B  ENTRY

... per 10k-row batch, the same thing in MiB of index WAL
  c2       generic 1.16 -> 1.16     rmgr 1.62 -> 0.76
  c20k     generic 27.83 -> 1.65    rmgr 4.03 -> 1.72
  c1m      generic 16.04 -> 1.02    rmgr 1.18 -> 0.78
  (btree on c200, for scale: 0.63 MiB, 64 B per leaf insert)

INSERT 10k rows into the 1M-row, 8-index portfolio (before -> after)
                after a checkpoint            steady state
  generic    893 -> 935 ms  88.10 -> 90.82   851 -> 768 ms  53.53 -> 9.48 MiB
  rmgr       285 -> 289 ms  64.53 -> 51.68   143 -> 130 ms  29.21 ->  9.08 MiB
  btree      150 ms         33.31            93 ms           6.75 MiB
index size after 20k inserts: 70,967,296 B in BOTH arms, to the byte

8-client hot-key burst, 2 key values, synchronous_commit on, rmgr mode
                   tps          p95        WAL/row   of which the index
  before          1328         9.31 ms      238.8 B      166 B
  after           1290         9.64 ms      159.0 B       86 B
  btree           1642/1644    7.9  ms      143.1 B       70 B
  no index        2394/2461    3.9  ms       72.6 B        -
(the index share is the row minus the no-index control, which is the same
heap in every arm).  The btree and no-index arms moved under 1% between the
two runs, so the 3% the lion arm lost is probably real: it is what the delta
comparison costs inside the critical section, against half the index WAL.

The +40% is gone and the container path is now cheaper than generic WAL. 1.62 -> 0.76 MiB against generic’s 1.16: the 361 general-path rewrites fell from 1658 to 98 bytes, which is 16.8x, and the 9,631 hot-key records fell from 114 to 79 because the entry tuple that travels with every one of them is now a delta of its ntids counter instead of a copy of the whole 40-byte tuple. That second number is the one to remember about DELTA: it pays on the SMALL records too.

The INLINE slack pays most where the rmgr could not reach: generic mode, where the record is a diff of the page and the old code moved every entry after the one it grew. 2918.7 -> 173.0 B on c20k is 16.9x and brings generic-mode INLINE inserts within 2.7x of btree’s leaf insert. In rmgr mode the same change is 2.3x (422.4 -> 180.5), because the record was already the entry alone; what is left is a sparse segment’s own shape - inserting a pair shifts both the ckeys[] and the los[] array inside the payload, so the delta covers the tail of both. An operation that replayed lion_sparse_insert() against an unaligned payload (redo would have to copy it out, call the library function and copy it back) would cut that to a dozen bytes, and it is the obvious next step; it is not done here. It is also why generic (173.0 B) is now marginally CHEAPER than rmgr (180.5 B) on c20k: a byte diff of a page describes that shift in fewer bytes than four-byte range headers do. (Since format version 9 a segment is a stream of delta varints (§13), so the shift is gone without a new operation: a pair appended above the last one - every row the heap appended - rewrites the header’s count and length and adds one varint, two or three bytes, at the end of the stream, and the DELTA against the old entry covers those and the entry’s own counters. On the 5M-row table of §12, 50,000 appended rows, the ENTRY record of c20k is 78 bytes a row, against 576 with the two arrays, c1m’s 79 against 88 and skew’s lion WAL as a whole 80 against 158. A pair that lands in the middle of a segment, a row placed in space VACUUM freed, rewrites the stream from there to its end, at most 255 bytes.)

Two costs, both small and both real. The steady-state portfolio insert is unchanged to slightly faster in rmgr mode (143 -> 130 ms) - the delta comparison runs inside the critical section, so it is written to skip equal bytes eight at a time, and the page-tail memmove it removes is worth more than it costs - but the first batch after a CHECKPOINT writes 2.7 MiB MORE in generic mode (88.10 -> 90.82), which is full-page images: entries that carry slack fill a leaf a little sooner, so a batch touches a few more pages the first time round. The index itself is the same size, to the byte, after the same inserts.

Slack is written by an insert, so in generic mode the FIRST insert into each entry or container of a bulk-built index still moves everything after it on its page. The build_slack reloption has the build write the slack an insert would (§4, “Slack at build time, measured”): on a wave of scattered updates after a build it takes generic-mode WAL from 1.1-2.4 KB to 65-435 bytes a row, at 4-46% more index at build. It is off by default in both modes.

Also in this wave (§23 items that need the same lock-window work):

  • INLINE entry slack: DONE, in the wave after this one, and §4 carries it (see “Growth slack inside an INLINE payload”). It took the shape this section predicted - the zero-terminator convention of §18, no header field, no use of §24’s two spare bytes - and one thing it did not: the record is not an ITEM_SET with a SETBYTES, it is the ordinary ENTRY record with the DELTA operation above, because that operation had to exist anyway for ITEM_REPLACE and it describes this change exactly (the counters in the header, and the payload from the insertion point up). lion_insert_inline() still rebuilds the payload in a work buffer and still decides everything before the record opens; what changed is the LENGTH it writes the entry at. When the new payload fits the bytes the entry already has, the allocated length does not change, so PageIndexTupleOverwrite() moves no other entry on the leaf - which is where the generic-mode 16.9x comes from - and when it does not, the entry is written with fresh slack, bounded by inline_limit, by the page’s free space and by LION_ENTRY_SLACK_BOUND.
  • Early exit in the AND merge: DONE (lion_run_merge() in lion_count.c). The non-driver sources are ordered by ascending members - the same ntids §22 picks the driver from - and the merge seeks and folds them in that order, abandoning a container key the moment the running intersection is empty or a seek lands past the target. The sources that were not sought are not stepped and keep their pins. §9 is untouched, and here is why: an abandoned container key contributes nothing to the answer, so no visibility-map question is asked about it, so there is no obligation to discharge for any page it touched; a pin too many never makes a count wrong, it only makes VACUUM wait. The pages of sources that WERE sought are released by lion_ecursor_seek(), the same window at the same kind of key as before. A Probes Avoided counter is in LionCountStats and in EXPLAIN ANALYZE, and test/sql/pushdown.sql pins three shapes: one where the two selective sets are provably disjoint (200 divides 20000) and the third source is never sought, and one where one set provably contains the other (20 divides 200) and nothing may be skipped, before and after the heap is dirtied. Measured at 1M rows: c20k = 77 AND c200 = 17 AND c2 = 1 goes from 112 to 11 buffer accesses and 104 to 70 containers, 0.095 to 0.040 ms; a four-source AND goes from 127 to 14 buffers. Two-source ANDs and every GROUP BY are unmoved, because there is no third source to skip - §22’s “118 buffers probed for c2” is the two-source shape and was never going to move. The all-dense three-source case reads 4% MORE buffers (262 -> 273) while visiting 80 fewer containers, which is LION_POSTING_SEEK_STEPS = 2 biting: a source left standing falls further behind and its next seek pays a descent where it used to step one leaf right. It is a wash in wall time and the threshold was not tuned for it. The §22 cost formula was deliberately NOT changed, and the measurement is the reason. The candidate - multiply probes by an expected survival fraction per level - predicts the new behaviour accurately when the columns are independent and is 10x optimistic when they are not, and the planner cannot tell the two apart from the inputs lion_cost_count_rel() has. The decisive pair, one table, two queries with IDENTICAL marginal statistics and the same estimate of 245.25: k1 = 17 AND k2 = 17 AND d = 1 where k1 and k2 are the same column avoids 0 probes and reads 236 buffers, while k1 = 17 AND k2 = 18 avoids 223 and reads 13. A survival fraction built from the marginals charges ~24 probes for both; for the disjoint query that is right and for the correlated one it is ten times light. So pages = Min(walk, probes x (height + 1)) stays as it is: after this section it is an UPPER BOUND the executor can only beat, and it is still exact for the second source, which is probed at every driver key. If a discount is ever wanted, the defensible form is to take the survival fraction from clauselist_selectivity() over the conjunction of the already-folded clauses rather than from a product of marginals - that is the one number in the planner that consults extended statistics, so a user who declares the correlation gets the full charge and one who does not gets the independence assumption the rest of the planner already runs on. Keep the Min(walk, ...) cap either way, and re-check every pin in test/sql/pushdown.sql (§10’s 20000-group refusal and §20’s 200x20 refusal are the ones to watch), because it moves every AND estimate.

Measured: what the standby barrier costs (2026-09-22)

bench/vacuum_micro.sh --only portfolio (1M rows, eight indexes, DELETE ... id % 100 = 1, one VACUUM) on a private cluster of the assert build, alternating arms of the binary before the barrier (HEAD 453fa32) and after, same data directory, same settings:

                       VACUUM ms        index WAL        index records
rmgr     before         686, 678        51 MB (50 FPI)       10,245
rmgr     after          688, 649        51 MB (50 FPI)       10,245
generic  before         780             59 MB (50 FPI)       10,245
generic  after          775             59 MB (50 FPI)       10,245

No record is added: at a 1% delete nearly every page VACUUM visits is one it changes, and the 1,420 visited-but-unchanged blocks ride on 1,255 of the existing removal records as 10,128 bytes of ranges - 0.02% of the index WAL - with no stand-alone VACUUM_VISIT record at all. A sparse VACUUM is the unfavourable case: DELETE ... id % 10000 = 8 (100 rows) then VACUUM (INDEX_CLEANUP ON) wrote 1,046 VACUUM_PAGE records, 749 of which carried a barrier of 7,488 blocks in 11,960 bytes, 0.15% of its 7.8 MB of index WAL, and still no stand-alone record. The generic arm writes no barrier (a generic-mode standby rechecks every TID, §9) and pays only for the cleanup locks of the posting-tree descents, which are not measurable here. Replay pays one buffer lookup per barrier block and one uncontended cleanup lock per barrier block in shared buffers, and only in hot standby.

26. count(DISTINCT k) in the pushdown (v1, implemented)

Formerly the §23 backlog item of the same name. Two shapes, with k - and g - scalar key columns of lion indexes (any column of a multicolumn one, §24; since 2026-10-19 of a partial index too, in an attempt for it, §45: the walk of k’s entries holds the count to its rows - the uniqueness proof below still asks for a whole btree index):

  1. SELECT count(DISTINCT k) FROM t [WHERE <pushdown quals>]
  2. SELECT g, count(DISTINCT k) FROM t [WHERE <pushdown quals>] GROUP BY g

What is counted. A scalar index puts every row under exactly one entry of k - its value’s, or the reserved NULL entry (§14, and §15 for the argument) - so the distinct non-NULL values of k among the rows the WHERE selects are exactly the non-NULL entries of k whose posting set, intersected with the WHERE sources (and, in shape 2, with the group’s set), holds at least one row visible to the snapshot. The NULL entry is never counted (count(DISTINCT k) ignores NULLs), and the reserved EMPTY entry of §17 cannot occur because multi-key columns are refused. Shape 1 is therefore §10’s GROUP BY-k walk with each group’s count replaced by an EXISTENCE test and the groups summed into one row; shape 2 is §20’s nested loop over (outer g, inner k) with each pair’s count replaced by an existence test and the inner loop summed into one row per outer group.

The existence test (lion_exists_sources_cached(), lion_count.c)

The same merge as a count - the same sources, the same set algebra, the same visibility-map check per container, the same recheck batch and visibility cache - with one difference: it stops as soon as the answer is known.

  • After every container the merge has put through the visibility map (the VM check itself, lion_count_container_vm(), is unchanged) the test asks whether anything has been counted. A member on an all-visible block settles it at once, and the recheck TIDs the container queued, if any, are dropped unread.
  • Otherwise the container’s dirty-block TIDs are rechecked right there - the ordinary lion_recheck_flush(), called at a container boundary, which is a heap-block boundary - and one visible row settles the test.
  • Only when the sets run out is the answer no. A group whose rows are all on dirty pages therefore rechecks container by container until its first visible row and never more than one container past it; a group on an all-visible heap costs the first container of its intersection and nothing else.

The disjoint sum of §15 and the single-set path stop the same way, between two sets and between two containers respectively. The heap recheck and the visibility cache are called, not changed.

Why §9 and §11 cover existence exactly as they cover counting. An existence test is a count compared with zero, computed from the same containers read the same way; the only change is that the count stops growing early, and stopping reads less, never differently. Every “yes” is one of the two things a count adds a row for: a member of a container on a block the visibility map reported all-visible WHILE the page that container was copied from was still pinned - the rule of §9, which the unchanged lion_count_container_vm() enforces: VACUUM cannot have finished ambulkdelete on that page, so it cannot have set all-visible on a block holding a dead TID of that container - or a TID the heap recheck found visible under the snapshot, which needs no pin at all (§9 step 5). Every “no” is a count of zero. Dropping queued recheck TIDs after a VM hit discards candidates that could only have added to an answer already known to be positive. So the §11 VACUUM rule needs no addition. test/isolation/count_distinct_vacuum_race.spec proves it the way count_vacuum_race.spec proves it for a count: the key whose rows are all dead comes first in the entry walk, the distinct count parks at the existing lion-count-containers-pinned injection point with that key’s container pinned, the VACUUM that would remove those rows and mark their pages all-visible is seen waiting for the pin, and the answer - for both shapes - excludes the dead key and includes the key deleted after the snapshot.

Executor

  • Shape 1 (lion_distinct_relation()): walk k’s entries - or, when the WHERE pins k itself with k = c or k = ANY (list), that clause’s located sets, exactly as §15’s GROUP BY driver does (the clause is then not intersected: an entry intersected with a union of disjoint entries that contains it is the entry), so k = c answers 0 or 1 in one test. A k IS NOT NULL clause on the driving column is dropped and the NULL entry skipped, as for §14’s sum-over-all. One test per non-NULL entry, one row out.
  • Shape 2 (lion_next_group_distinct()): for each outer g entry, first the group itself - does g ∩ WHERE hold a visible row? A group with none is not emitted, and a group whose rows all have k NULL is emitted with count(DISTINCT k) = 0, as SQL requires. Then one test per non-NULL inner key: the inner keys are read once per relation and each inner set is located per pair, which is §20’s pin budget (two group pins at a time). One row per group. The outer set and the WHERE sets are the ones every pair reuses, so §9’s materialization applies to them as in §20. An IN list on g does not drive the groups here (the entry scan does), which keeps the output in g’s directory order and the pathkeys of §21 valid.
  • Mixed target lists. count(*) and count(col) (under §14’s rules) may stand beside count(DISTINCT k), and so may count(k) - the number of non-NULL k - which a nullable k would not otherwise allow. Whether a test must COUNT rather than stop early is decided from the target list’s kinds: in shape 1 any other count makes every entry test a full count (the total is the sum over all entries, the NULL entry included - §15’s disjointness again - and count(k) the sum over the non-NULL ones); in shape 2, count(*) and count(g) make the GROUP test a count and count(k) makes the PAIR tests counts. A pure distinct count is existence tests only.
  • A GROUP BY the planner folded to constants (WHERE g = 3 GROUP BY g) is shape 1 under §10’s single-group rule - no row when nothing matches. Whether a row matches when every matching row has k NULL is one more existence test, on k’s NULL entry, made only when no distinct value was found.
  • HAVING on the distinct count is the plan.qual of dff300f unchanged: the distinct count is a column of the node’s scan tuple like any other count.

Planner acceptance (lion_try_count_path())

  • Each count(DISTINCT x) must be count(any) of one argument that is a plain column of the relation (relabels stripped), with no FILTER and no ORDER BY, not variadic and not split, and every distinct aggregate of the target list and the HAVING must name the same column with the same equality and collation. count(DISTINCT (a, b)), count(DISTINCT a + 1), other DISTINCT aggregates and two different distinct columns decline here, and are counted at a finer grain instead (§10, “DISTINCT aggregates over a finer grouping”).
  • k must not be a GROUP BY column, and there may be at most one GROUP BY column (so at most two driving columns, LION_MAX_GROUPCOLS); more GROUP BY columns go the same way.
  • k goes through lion_collect_targets() as a driving column, so every per-column rule of §10 applies to it unchanged: a SCALAR lion index (a multi-key one is never found, so count(DISTINCT tags) - which counts distinct ARRAYS, not elements - declines); the index’s collation must be the DISTINCT’s (exprCollation() of the argument, which is what nodeAgg sorts and compares with, so count(DISTINCT t COLLATE "C") over a default-collation index declines); and strategy 1 of the index’s opfamily must be the DISTINCT’s own equality operator (SortGroupClause.eqop of aggdistinct, compared by Oid - the grouping-equality check of the 2026-09-20 review, finding 3), so a coarser opclass declines and citext_ops over citext is accepted. No value of k is printed, so the representation gate of finding 4 does not apply.
  • Partitioned tables decline: distinct counts are not additive across partitions, and §16’s shape (partial aggregates added up by a Finalize Agg) would add them. With a GROUP BY the per-partition (g, k) groups are what is counted instead, and the Finalize Agg merges them for core’s Agg to count (§10, “DISTINCT aggregates over a finer grouping”). (A column unique over the whole table is not walked at all, below.)
  • Everything else - WHERE clause kinds, OR, IN lists, Params, HAVING, privileges, RLS - is §10 as it stands. A query with no WHERE clause at all is accepted: k’s entries drive it.

custom_private gains LION_PRIV_DISTINCT (IntList: the distinct column’s attnum, or empty) in front of the target-list kinds - shape 8, LION_PRIV_NMEMBERS 11 - and the target-list kinds gain LION_TL_COUNT_DISTINCT and LION_TL_COUNT_DISTCOL (count(k)). k’s index travels in the existing Oid slots, the driving one in shape 1 and the inner one in shape 2; groupattno2 stays 0, because k is not a grouping column, and the executor calls the inner column innerattno.

A unique column (2026-09-28)

A benchmark counted SELECT g, count(DISTINCT id) FROM t WHERE <lion filters> GROUP BY g with id the table’s primary key. Nothing above answered it: id had no lion index, and with one the (g, k) loop would have tested one entry per ROW for every group - far more than the count it is. For a column no two rows share a value of, count(DISTINCT col) over any set of rows is count(col) over the same rows, and so it is taken now (lion_agg_distinct_unique()): such an aggregate is not k at all but one more count of the target list, which §14’s rules for count(col) then answer - count(*) where col is known non-NULL (declared NOT NULL, through attnotnull on 16 and core’s notnullattnums from 17, which leaves out a NOT VALID constraint on 18 and later; or a clause of the WHERE that is strict on col, or col IS NOT NULL), the group’s count or 0 when col is the group column, and 0 under col IS NULL. Where none of those holds - a nullable unique column nothing proves non-NULL - the aggregate is k as before and is walked if col has a lion index. For a proven one nothing is walked, priced or printed as a Distinct Key: the path is §10’s count with that count in the target list, and costs what count(*) costs.

The proof is the one §27 takes for a dimension key, lion_column_is_unique() (lion_fkjoin.c), over the relation’s indexlist - the valid indexes a plan may rely on under this snapshot, which leaves out one that CREATE INDEX CONCURRENTLY has not finished:

  • a btree index that is UNIQUE and on exactly one key column, which is col itself (not an expression over it, not col among others);
  • enforced immediately. A DEFERRABLE constraint is checked at commit, and inside a transaction that has deferred it two rows may share a value: test/sql/distinct_unique.sql makes one, and the count has to come out one less than the rows;
  • not partial, even under a WHERE that implies the predicate. The predicate proof would hold (the rows counted are rows the index covers) but is not taken in v1, as core’s join removal does not take it for proofs that must hold before any join;
  • not hypothetical (an index an advisor only pretends exists);
  • whose opfamily has the DISTINCT’s own equality (SortGroupClause.eqop) as its equality strategy, and whose collation is the DISTINCT’s (exprCollation() of the argument) when that is a collation at all: a key unique under “C” is not unique under a case-insensitive one. count(DISTINCT t) over a unique index on t COLLATE "C" declines, and count(DISTINCT t COLLATE "C") is counted.

Uniqueness among the table’s rows is uniqueness among any snapshot’s: a unique index lets no second live row with a key in until the first is deleted by a committed transaction, so no snapshot sees both - the guarantee core’s own join removal and unique joins rest on.

Partitioned tables. A partitioned parent’s indexlist holds its PARTITIONED indexes (16 and later), and a unique one of those is a proof for the whole table: it has to contain every partitioning column, so a single-column one is on the partition key, which keeps equal values in one partition, whose own unique index keeps them apart - the proof core’s join removal takes from the same list. A unique index on each partition is not one, and the test makes the values repeat across partitions to show it declines. Without a GROUP BY the partitions are counted and summed as for count(*) (§16). With one it still declines: core never marks an aggregate with a DISTINCT splittable (hasNonPartialAggs, so GROUPING_CAN_PARTIAL_AGG is unset), and §16’s partial counts under a Finalize Agg are the only form a partitioned GROUP BY takes.

Nothing is recorded about the proof in the plan, as for core’s uses of it: dropping the index or the constraint, and ALTER COLUMN ... DROP NOT NULL, change the table’s catalog rows and so invalidate its relcache entry, and every cached plan that reads the table - its OID is in the plan’s relation list - is made again before it runs. The test prepares a generic plan, drops the unique index (then the NOT NULL) under it, inserts duplicates (then NULLs), and checks that the next EXECUTE is planned without the count and counts them once.

The plan tells the two kinds of count(DISTINCT) apart by their column: the one LION_PRIV_DISTINCT names is walked, any other is a count, LION_TL_COUNT or one of §14’s kinds (lion_plan_custom_path()). The member’s shape is unchanged: a plan an earlier build made never holds a DISTINCT aggregate of another column, and the kinds are decided at plan time. So one column is never both: a distinct count proved unique under one collation beside a walked one of the same column under another declines. Beside a walked k of another column the proven one is simply one more count, and makes the tests of shape 1 counts as any other count does.

Not done: a multi-column unique index whose other columns the WHERE pins to one value, the predicate proof of a partial one, and the FK-side join (§27), whose count(DISTINCT) is core’s own aggregate over the node’s rows already.

Cost

lion_cost_count_rel() prices shape 1 as §10 prices a GROUP BY over k - one entry scan, and the per-group container work for every entry, because every entry is tested whether or not the WHERE leaves it a row (the entries are n_distinct(k) over the whole table) - and shape 2 exactly as §20 prices the (g, k) nested loop: one cpu_tuple_cost per PAIR, the intersection term Min(|G|, |K|) x rows and both sides' container bookkeeping per pair. That is what charges pairs and not groups, and what makes a large |G| x |K| lose to the sorting aggregate core builds for a distinct count - the decline the backlog item asked for, with the cost model as the guard, as in §20. On top of that:

  • A fixed cost per test, LION_DISTINCT_TEST_COST (50 x cpu_tuple_cost), for every entry test, group test and pair test: a merge set up and torn down, and for a pair the inner set’s lookup. On the release build a test of the walk over 20,000 one-container entries of a 5M-row table costs 1.0 us, which is 50 cpu_tuple_cost at the 500 units a millisecond of §10’s “The units”; the assert build measured 2 us with nothing to intersect and 3.4 us with a WHERE set to seek. Without it, a walk of 1000 entries of k under a WHERE that leaves 500 rows was chosen, and took 3.4 ms against the bitmap scan and sort’s 0.56. In shape 1 it is charged per entry WALKED, not per value the node emits: until 2026-10-01 it was the emitted values, the distinct k the WHERE leaves, which ClickBench’s count(DISTINCT UserID) WHERE MobilePhoneModel <> '' (5M rows, release PostgreSQL 18) put at 281,060 tests where the walk made 3,413,988 - 275k cost units against the bitmap scan’s 437k, 9.1 s against 0.7. Charged per entry it costs 1.85M and is left to core.
  • The merges are §10’s (lion_merge_cpu_cost(), 2026-09-27): each entry’s own set is a source of its test, which drives whenever it is the smaller, so a test over a selective WHERE costs its entry’s containers and not the WHERE’s share. With the old per-container charge count(DISTINCT c20k) WHERE c200 = 17 over 5M rows cost 12,427 and took 341 ms; it now costs 404,500 - which overprices it (341 ms is 170,500 at 500 units a millisecond: the existence test finds its row sooner than lion_exists_fraction() expects) and refuses it, as its time says it should be.
  • The early exit is discounted only where it is real. In shape 1 each entry test reads the share of its intersection lion_exists_fraction() expects: a test whose intersection is expected to hold s >= 1 rows over D containers stops after about D/s + 1 of them, and one expected to be empty (s < 1) reads everything, as a count would. The recheck candidates are scaled the same way, in both shapes, because a test rechecks one container at a time and stops with them. A pair test of shape 2 stops the same way, and its MERGE is discounted by the same share (since 2026-10-11, below); what the early exit does not shrink is charged per pair on top: the test’s fixed cost and the lookup of the inner set, one descent of the inner index’s directory (LION_DESCENT_COST a level, as a single-key clause’s lookup). Until then the pair terms were not discounted at all, on the argument that a pair’s time is its lookup and its merge setup far more than its members - measured per pair, an existence test took 6.5 us against 7.7 for §20’s count of the same 200 x 50 pairs, and discounting the merge alone chose those pairs at 10,566 against the sort’s 10,694, for 45 ms of CPU against 27. That holds on a table of a few container keys and fails on a large one, where the full merge grows with the table and the test does not.
  • Tests that must COUNT (the mixed target lists above) are charged in full.

Measured with the final model (100k rows, uncorrelated columns, warm cache, assert build - ratios, not absolute numbers; test/sql/distinct.sql pins the rows marked *):

query LionCount core plan chosen
count(DISTINCT k50) * 0.68 ms 18.4 ms node
count(DISTINCT k1k) 2.04 15.7 node
count(DISTINCT g2k) 3.26 16.0 node
count(DISTINCT k50) WHERE g8 = 3 0.90 2.69 node
count(DISTINCT k1k) WHERE g200 = 3 3.16 0.39 core
g8, count(DISTINCT k50) * 5.41 51.9 node
g8, count(DISTINCT k1k) 35.8 60.0 node
g200, count(DISTINCT k50) * 57.5 37.8 core
g2k, count(DISTINCT k50) 401 40.8 core
g200, count(DISTINCT k1k) 693 44.1 core
g2k, count(DISTINCT k1k) * 5699 39.2 core

Every choice in the table is the faster plan. The open item is the one §20 records: the model does not model correlation, so clustered data, where most pairs' merges end at once, is priced as if it were not.

The pair test’s early exit (2026-10-11). The optimization review of 2026-10-11 measured SELECT c20, count(DISTINCT c200) FROM fact GROUP BY c20 over 5M rows (96,196 heap pages, 1,503 container keys, uncorrelated columns, release build of PostgreSQL 18): the node ran in 19-24 ms and was priced 961,710, nearly all of it the 4,000 pairs' full merges at 240 each across every container key, against core’s sort at 808,355, which took 2.6 s. The walk visited 18,132 containers for its 4,020 tests, 4.5 a test. Priced with the merge discounted by lion_exists_fraction() and the per-pair lookup added, it costs 6,862 and is chosen (12 ms); c200, count(DISTINCT c20) 5,754 (16 ms against 2.7 s); with WHERE c2 = 1, which no lion plan had been chosen for even with sequential and bitmap scans off (it was built, at 1.22M, against an index scan and sort at 475k), 8,265 (18 ms against core’s 1.8 s). What must still lose does: c20, count(DISTINCT c20k) (400,000 pairs of 12 rows) is priced 1.59M against 808k (it runs in 2.7 s against 3.8: the model is conservative there), c200, count(DISTINCT c20k) 100M, and on the 100k-row table above g200, count(DISTINCT k50) 16,003 against 10,694, g2k, count(DISTINCT k1k) 3.6M. That table’s columns are functions of i modulo their counts, so most of its pairs are empty and are read to the end, which the model, assuming independence, does not see: the per-pair lookup is what keeps 200 x 50 there with core. On 200k rows of hashed columns, g200, count(DISTINCT k50) (10,000 pairs) is now chosen, 20 ms against the sort’s 79. test/sql/distinct.sql pins a 4,000-pair shape over 87 container keys (lion_dtw) that the full merge priced at 48,509 against 15,611.

The grouped distinct does not take the decoded walk of §34: the walk decodes every row at every key the WHERE holds (88 ms for the same 4,000 pairs as GROUP BY c20, c200), where the existence tests read a few containers a pair; and its partial (g, k) counts would need core’s count(DISTINCT k) above them to deduplicate.

EXPLAIN

Lion Indexes names k’s index like a group index - ix_k (k) first in shape 1, after the outer group’s index in shape 2 - then the WHERE indexes; Group Key is g alone; a new line Distinct Key: k says which column is counted. With ANALYZE, Distinct Keys Tested counts the entry and pair tests made.

Tests

test/sql/distinct.sql: every query against a forced sequential scan with the pushdown off, as a multiset both ways round - NULLs in k and in g, deletes on dirty pages and after VACUUM, IN lists on k itself and on another column, k = c, k IS NOT NULL, an OR, Params under a generic plan, the mixed target lists, HAVING, a folded GROUP BY, citext, a multicolumn index (shape 2 out of one index) - and the declines: an array column, a partitioned table, a collation mismatch, a coarser opclass, an unindexed column, two distinct columns, k as the group column and two GROUP BY columns. The plan choice for shape 2 is pinned at a small and a large pair count, and EXPLAIN ANALYZE shows the early exit reading fewer containers than the same walk with count(k) beside it.

test/sql/distinct_unique.sql does the same for a unique column, and says for each query whether it was counted, walked or declined: a primary key with and without a GROUP BY, under every clause kind, HAVING, a folded and a two-column GROUP BY, a generic plan, beside a walked non-unique column; a nullable unique column, walked unless the WHERE rules its NULLs out, and as the group column; the proofs that do not hold - an expression, a two-column unique index, an expression index, another collation, a partial index (with duplicates outside its predicate) and a deferrable constraint (with a duplicate made inside a deferring transaction); a dirty heap and VACUUM; the cached plan replanned after its unique index and then its NOT NULL are dropped; and a partitioned table with per-partition unique indexes over repeating values, then a primary key. Every query there runs with LionOrdered (§30) switched off beside core’s scans, and so does group_coalesce.sql: a btree on the DISTINCT or GROUP BY column gives core an aggregate over LionOrdered’s presorted rows, which enable_indexscan disables as it disables core’s index scans (§39, “Core’s switches”), so that it still competes wherever core’s other scans are switched off too, and which is the cheaper plan for a walk over a column with an entry per row (count(DISTINCT u) of a nullable unique u, GROUP BY u). From 17 core counts an index on ANY grouping or DISTINCT key as useful for grouping, not only one on the leading key, so the same queries meet that competitor more often there than on 16. Left on, it decides the plan by cost and says nothing about whether the node was built; the first CI run of the test tripped over exactly that, on shapes the node answers.

27. FK-side join pushdown: GROUP BY dim.attr over a fact table joined on a lion-indexed FK (v1, implemented)

Formerly the §23 backlog item of the same name; the semi and anti joins (EXISTS, IN, NOT EXISTS) and the fact filters collected once were added on 2026-09-27, below, and count(DISTINCT), parallel plans and the forward semi join over a non-unique key after them; the lookups in key order on 2026-09-28, and after them what a key’s time is made of, the per-key path and the cost of a key (“Where a key’s time goes” and the two sections after it), counts beside the aggregates that need the dimension rows (“Every aggregate over the node’s rows”), a GROUP BY of the join key, the fact’s or the dimension’s (“Grouped by the join key”), a partitioned fact table (“A partitioned fact table”), and a dimension that is itself a join of a table and the tables semi- or anti-joined to it (“A dimension that is a join”); on 2026-09-29, a run that probes the fact filters and collects them part way through once probing has cost what collecting would (“Probed, then collected”), the semi and anti join offered as a join path of its own, whose rows are the outer side’s (“The semi and anti join as a join path”), and a GROUP BY of a fact column, each key counted once per group of it (“Grouped by a fact column”); on 2026-10-19, a partial lion index on the fact table whose predicate the fact’s restrictions imply (§45): read for the FK lookups, the fact group or the fact filters, with the restrictions it implies left out, when the key’s lookup, the fact group’s entries or a fact filter read from it hold every key’s count to its rows (lion_fkjoin_setup()). The shape is the star-schema aggregate

SELECT d.attr, count(*)
FROM fact f JOIN dim d ON f.fk = d.pk
[WHERE <pushdown clauses on f>] [AND <anything on d>]
GROUP BY d.attr [HAVING ...]

and its ungrouped form, SELECT count(*) FROM fact f JOIN dim d ON f.fk = d.pk WHERE d.attr = c, a semi-join-like count over the fk keys of the matching dimension rows. The ordinary plan reads every fact row the fact filters leave, probes the dimension for each and aggregates; the posting sets already hold the fact rows of every fk value, partitioned by value.

What is computed, and why it is exact

Let D be the dimension rows that survive the dimension-side quals under the query’s snapshot, and for one of them, d, let F(d) be the fact rows that satisfy the fact filters and f.fk = d.pk under the join’s operator. The inner join’s count(*) for a group G of d.attr is the number of join PAIRS (f, d) with d in G, which is sum over d in D ∩ G of |F(d)|. So the node never builds the per-group key set the backlog item sketched:

  • it runs the dimension side as an ordinary child plan (below) and, for EACH dimension row, looks d.pk up in the fact’s lion index on fk and counts that one posting set ANDed with the fact filters - the §9 count of f.fk = <value> AND <fact filters>, with the value taken from the dimension row instead of from the query;
  • it emits one PARTIAL aggregate per dimension row whose count is not zero - (the dimension columns the query needs, partial count) - and core’s Finalize Agg above it groups those rows by d.attr with d.attr’s own equality and adds the counts, exactly as §16’s partitioned GROUP BY hands its per-partition partials to it.

Consequences, each of them a rule rather than an optimisation:

  • No per-group union, so no memory bound to enforce on one. The node holds one dimension row, one fk posting set (located, counted and released per row, like a §20 inner pair’s) and the fact filters' sets for the whole scan. The grouping state lives in core’s Finalize HashAggregate, which spills under hash_mem (§16’s argument, unchanged). A dimension of a million rows is not refused for memory; it is refused by the cost model, because a million lookups lose to a hash join.
  • Grouping equality and value representation are core’s. d.attr is never read from an index: its value comes out of the dimension’s own tuples through the child plan, and the Finalize Agg groups it with the SortGroupClause the parser chose. Neither §10 finding 3 (the driving index’s equality is the grouping equality) nor finding 4 (the value-representation contract) applies, because no index key is printed and no index drives the groups. Any grouping over dimension columns - GROUP BY d.a, d.b, GROUP BY upper(d.name) - is therefore fine: the node’s projection computes it per dimension row from the child’s columns. A VOLATILE one is refused, since evaluated per dimension row instead of per join row it would mean something else.
  • HAVING belongs to the Finalize Agg, as for a partitioned GROUP BY (§16): any HAVING core can evaluate over the grouped target is accepted, as long as every aggregate in it passes the count test below.
  • NULLs. A NULL d.pk joins nothing (the join operator must be strict) and is skipped without a lookup; a NULL f.fk lives in the fk index’s reserved NULL entry (§14), which no lookup of a value reaches. A NULL d.attr is a group like any other, formed by the Finalize Agg.
  • A dimension row with no fact rows emits nothing, so an inner join’s absent group stays absent; the ungrouped form’s plain Finalize Agg then answers 0 for an empty input, as count(*) over an empty join does. A GROUP BY the planner folded to constants (WHERE d.attr = 3 GROUP BY d.attr) gets a zero-column sorted Finalize Agg, as core builds itself, so an empty join has no row.

Uniqueness of the dimension key

The sum above is the join count whatever d.pk holds - a duplicated key is two dimension rows, each counted with its own multiplicity, which is exactly what the join does - so, unlike the backlog sketch’s union (where two rows of one group with one key would have been counted once), this design does not need uniqueness to be RIGHT. v1 required it anyway, as a scope decision (the ordinary star schema has it, and a non-unique key was kept out of the tested surface); since “A dimension that is any join” an inner join takes any dimension, and the proof is asked only where a rule below rests on it. The cost model prices the child’s rows, a lookup each, whatever they hold.

The proof is lion_column_is_unique() (§26’s unique column takes the same one): a single-column, UNIQUE, immediately enforced, non-partial btree index on the dimension’s join column whose opfamily contains the join operator as its equality strategy and whose collation equals the join clause’s input collation (or the type is not collatable). It is written against the index list rather than through core’s relation_has_unique_index_for(), because before PostgreSQL 19 that function does not compare collations (its own XXX): a unique index under "C" does not make a join under a case-insensitive collation unique. Lifting the rule is a costing change, not a correctness one.

The proof is therefore a COSTING AND SCOPE GUARD ONLY, and nothing about correctness rests on it (2026-09-23 review): because the per-dimension-row sum is exact with duplicates, a unique index that is later dropped, becomes invalid, or is deferrable and violated inside the current transaction can make the node do more lookups than it was priced for, never give a wrong answer. The one exception is a GROUP BY of the key, whose Agg takes each of the node’s rows as a group of its own (“Grouped by the join key”): it relies on the proof as core’s unique inner join relies on its own, and with the same guarantees - an index that is valid, enforced immediately and not partial when the plan is made, and whose DROP invalidates the plan. So does the uniting of a batch’s keys under a GROUP BY of a fact column, whose union counts a fact row once however many dimension rows share its key (“Grouped by a fact column, the keys united”).

It guards the inner join’s GROUP BY of the key, its keys united, and the dimension-side LEFT JOIN to the fact. A semi join’s count is not a sum over join pairs, and the forward one over a key that nothing proves unique is counted over the dimension’s DISTINCT keys instead (“Forward semi joins over a non-unique key”, below), which makes the question moot there.

The join semantics the lookup must reproduce

The count for one dimension row is the §10 count of f.fk = v with v = d.pk of that row, and a lookup answers that clause exactly when the clause is one the pushdown already accepts:

  • the join operator is strategy 1 of the fk index’s opfamily, the compared type (d.pk’s) has a cross-type equality and a hash function in that family, and the collations agree - exactly lion_match_index() for a LION_CLAUSE_EQ clause, which the join key is fed through, per relation. Cross-type keys (int4 fk against int8 pk) work wherever int4col = 8::int8 works, and an opclass on fk whose equality is not the join operator never matches;
  • the operator is strict, so a NULL key joins nothing;
  • the join is the ONLY join clause: one equality between a plain column of the fact rel and a plain column of the dimension rel, from an equivalence class or from joininfo. A second join clause (AND f.a < d.b, a composite key), a pseudoconstant qual anywhere in the query, an outer join (any join_info_list entry but the one semi or anti join of “Semi and anti joins” below, those inside a dimension that is a join, and the one LEFT JOIN of “The dimension LEFT JOIN the fact”), a LATERAL reference or a PlaceHolderVar all decline. An equivalence class with a constant (d.pk = 5) generates no join clause at all - both sides are restricted to the constant - and is left to the ordinary plan.

Visibility and the §9 interlock

  • The dimension side is an ordinary plan: the dimension rel’s cheapest total path, planned by core with its restriction clauses and its RLS security quals (with core’s leakproofness rules), carried as the CustomPath’s custom_paths child and run by the node under the query’s snapshot. Only rows that passed it are ever looked up. Privileges on both tables are the executor’s range-table check, as before; EXECUTE on the join operator, which the node evaluates in place of a hash or nested-loop join, is the node’s own check at executor startup (§9, “Privileges”). The fact side keeps §10’s rule: a fact rel with security quals (RLS) or TABLESAMPLE declines. A partitioned fact table is counted leaf partition by leaf partition, each a table of its own in everything this section says (“A partitioned fact table”).
  • The fact side is the §9 count, unchanged. Each dimension row’s fk set is located with lion_posting_set_lookup_col(), counted with lion_count_sources_cached() beside the fact filters' sources, and released before the next dimension row, so at any moment one fk set and the WHERE sets hold pins - and, when the keys are looked up in key order, the walk’s leaf, which is the fk set’s own leaf or one no count follows on (“Lookups in key order”). The pin budget of §15 applies to that lookup as to every single lookup (a NOPIN set is re-located, or covered by another source, by the count itself), and no argument of §9, §11 or §15 changes: the fk set is to this node what one group’s set is to §15’s GROUP BY driver. The visibility cache is shared across dimension rows as it is across groups, and each fk lookup takes the predicate locks of §9, “SERIALIZABLE”, as every lookup does.
  • One snapshot for both sides. The child and the count both read es_snapshot, so a fact row is counted for a dimension row exactly when both are visible to it - which is the join.
  • The WHERE sets' pins are held across the child - until the node returns its first row. The fact filters are located once per scan and keep their pins (§9) while the node calls ExecProcNode() on the dimension child between counts. A child that runs user code - a slow function in a dimension qual, a pg_sleep() - can therefore keep a manual VACUUM of the fact table waiting on its cleanup lock on those index pages until then (2026-09-23 review). That is the exposure every scan that holds a pin while the query runs has - a core index scan parked under a slow qual - and not a correctness issue: the VACUUM waits (autovacuum skips the page). From the first row on the WHERE sets are NOPIN copies, each count carried by its fk set’s pin, and the node holds no pin between rows or after its last (§15, “Paused and finished counts”). A plan that looks its keys up in key order gives them up before it reads its first batch of child rows instead (“Lookups in key order”), so its child never runs with them pinned; so does a plan whose fact filters were collected, as soon as the copy is made, since no count reads them after it; and so does a summed join (“The per-key path, end to end”), before each child row, since it hands up no row until the end.

Planner integration

create_upper_paths_hook at UPPERREL_GROUP_AGG, as today, with the input rel a JOIN rel of two base tables, plain ones or a partitioned fact table (“A partitioned fact table”, below) - or of a fact and a dimension that is itself a join (“A dimension that is a join”, below). lion_fkjoin_recognize() (src/lion_fkjoin.c) finds the one join clause and proves the dimension key unique, for each orientation in turn, and hands lion_try_count_path() a LionFkJoin. That function then runs over the FACT rel, unchanged for everything about the fact side - the WHERE analysis of §10/§14/§15/§17/§19, Params, index matching per clause - with the join key appended as one more LION_CLAUSE_EQ clause whose value expression is the dimension’s Var, so that lion_collect_targets() finds and checks the fk index for it like any clause. It differs only in what it lets through the target list:

  • the grouped target and the HAVING may reference only dimension Vars (in any non-volatile expression), the fact’s join column where it stands for the dimension’s key (“Grouped by the join key”), and count aggregates: count(*), and count(x) where x is a non-NULL constant or either side of the join key (non-NULL in every joined row). Other aggregates, window functions, SRFs, grouping sets and row marks decline, as for a single table, and so does a DISTINCT aggregate but the count(DISTINCT) of “count(DISTINCT)” below, which takes a path of its own - and so do the aggregates “Every aggregate over the node’s rows” answers over that path’s rows;
  • the aggregates must be splittable (GROUPING_CAN_PARTIAL_AGG) and every grouping column hashable. The node’s target is lion_make_partial_target() of the grouped target, and what goes into the grouped rel is create_agg_path(AGG_HASHED, or AGG_SORTED over zero columns for a folded GROUP BY, or AGG_PLAIN; AGGSPLIT_FINAL_DESERIAL) over the node.

Executor encoding: the join key clause travels in the clause lists like any other - its value, the dimension Var, lands in custom_exprs, where setrefs.c rewrites it into an INDEX_VAR reference to custom_scan_tlist, which is how EXPLAIN deparses it as fk = d.pk - but it is not a source and is never located with the WHERE clauses. A new member LION_PRIV_JOIN names it and the child-plan column that carries its value; the node reads the key straight from the child’s tuple, and the fk set occupies source slot 0 (the group slot, unused without an index-driven GROUP BY). custom_scan_tlist holds every dimension Var the target needs, the key’s, and the partial counts; a dimension column’s target-list kind is its position in the child’s target list. Shape marker 9, LION_PRIV_NMEMBERS 12.

Cost (lion_cost_fkjoin_rel()): the child’s total cost, plus per dimension row one directory descent (the leaf at lion_heap_page_cost()’s interpolated cost, capped by the directory’s pages) - or, where that is cheaper, a step of a walk of the leaves in key order (“Lookups in key order”, 2026-09-28) - a fixed per-count cost for a merge set up and torn down (LION_FKJOIN_COUNT_COST, 50 cpu_tuple_cost, the order §26 measured per test), the fk set’s containers - lion_containers_for(heap_pages, rows per fk value) - at §10’s two cpu_operator_cost each, a PROBE into each fact filter’s set at the container keys the two have in common (LION_FKJOIN_PROBE_COST, 30 cpu_operator_cost, measured below), the set’s chain pages (none when its entries are INLINE), and LION_FKJOIN_ROW_COST per row handed up - one a run when the target is counts alone (“The cost of a key, after the per-key path”); each fact filter’s lookup and chain once (located once, materialized on second use, §9); and §10’s recheck term for the fact rows the dimension rows reach - min(D × rows per fk value, fact rows) × filter selectivity × dirtyfrac candidates, on dirty pages fetched once per query. Nothing charges the whole fact heap: that is what the node exists not to read. The Finalize Agg is costed by core.

The probe term is the one that decides against the node, and it had to be measured: a probe (a seek of the materialized filter set and the AND of two containers) costs about ten times what §10 charges a container. Priced like a container, the node was chosen for ... WHERE f.x = 3 GROUP BY d.attr below at 157 ms against the ordinary plan’s 63; at 20 cpu_operator_cost it still was (23,378 against 28,057); at 30 it is refused, while the same query over a quarter of the dimension (39 ms against 67) is still chosen.

A fact filter that is a UNION - an IN list, or an OR across columns, whose leaves and their lists' elements are the union’s sets - is priced per SET, per count (2026-09-23 review). It is located once, but every count builds its k-way union again (§15): each sub-cursor is set up and positioned whether or not the fk set’s keys find anything in it, and the containers standing at the probed keys are merged. So every count pays LION_FKJOIN_SET_COST (80 cpu_tuple_cost) per set, measured - t IN (n values) over 300 dimension rows of a 1.5M-row fact took 57 ms at n = 30, 199 at 100, 599 at 300 and 1195 at 1000, 4 to 6 us per set per count - plus lion_merge_ops() of the whole source prorated to the share of the heap’s container keys its probes touch. Priced as one set, the review’s WHERE f.t IN (1000 texts) AND d.pk < 300 was chosen at 1.2 s against the nested loop’s 2 ms, and forced with d.pk < 2000 estimated 7,500 against 10,000 and ran 8.1 s against 38 ms; it is now refused (326,000 against 10,180), and so are the dense-list shapes where the per-count union is worst (x IN (1, 2) over 1000 dimension rows: 957 ms, refused, against 108). The rows of the table below are unchanged by it.

An IN list whose array is a PARAMETER (= ANY ($1)) used to be refused outright: its length is not known until the executor has it (§15), and a count that reads the filters does work per dimension row proportional to it - a 43,000-value array over 20,000 dimension rows ran for minutes in the review. Since the fact filters are collected once (“The fact filters, collected once” below) that work is done once per scan instead, whatever the list’s length, and the list is taken as a single table takes it (2026-09-27): evaluated once per scan and rescan, in every participant of a parallel plan, and so is any other array a stable expression computes (x = ANY (string_to_array(current_setting(...), ',')::int[])). The cost model prices it two ways (lion_cost_fkjoin_rel()): collected, once per scan, at estimate_array_length()’s guess - an expression’s array at its plan-time estimate, which the clause analysis puts in the clause (§10) - and probed, per dimension row, as at least LION_MAX_ARRAY_ELEMS values (a literal at its length). A wrong guess about a scan’s one-off work costs little; a wrong guess about a dimension row’s costs that many times over, which is what the review measured. So the copy is what is chosen wherever it is the cheaper - priced spilled where it is not expected to fit - and a copy that does not fit at run time spills to a temporary file (“The fact filters, collected once” below; it used to fall back to the probes, the very per-row work the guess was meant to avoid). IN ($1, $2) keeps its length at plan time and is priced like a literal list.

Measured (2026-09-23, prune slot’s PostgreSQL 20devel install - assert-enabled, so ratios and not absolute numbers; two million fact rows over 22,728 heap pages with a 40-byte pad, fk = a hash of the row number over 1..1000, x 10 values; a 1000-row dimension with 20 attr groups and a quarter of its rows region = 'eu'; lion indexes on fk and x; warm cache, best of five, max_parallel_workers_per_gather = 0). “Ordinary” is the plan the planner makes with the pushdown off - on 20devel a hash join over a sequential scan, or with x = 3 a partial aggregate below the join over a bitmap heap scan:

query all-visible heap: node ordinary every page dirty (5% of rows updated): node ordinary
d.attr, count(*) ... GROUP BY d.attr 53.8 ms 339 272 350
... WHERE f.x = 3 GROUP BY d.attr 157 (refused) 66 refused 73
... WHERE f.x = 3 AND d.region = 'eu' GROUP BY d.attr 39.8 68 refused 75
count(*) ... WHERE d.attr = 5 (50 dimension rows) 2.7 176 101 193

Bold is the plan the cost model picks. The dirty column is the recheck of §9 at work: every heap page holds an updated row, so every candidate TID is resolved against the snapshot, once per page per query through the visibility cache.

Parallelism: see “Parallel” below (2026-09-27). Before it the node was not parallel-safe, and its child could be a Gather below it, which it still can.

EXPLAIN:

Finalize HashAggregate
  Group Key: d.attr
  ->  Custom Scan (LionCount)
        Lion Indexes: fact_fk_idx (fk = d.pk), fact_x_idx (x = 1)
        ->  Seq Scan on dim d
              Filter: (region = 'eu'::text)

and with ANALYZE, besides §10’s counters, Join Keys Looked Up (dimension rows with a non-NULL key) and Join Keys Without Entry, and the counters and timings of “Where a key’s time goes” below.

Semi and anti joins (2026-09-27)

Two common count shapes could not be pushed down (2026-09-26), both over a fact table and a dimension selected by lion-indexable filters of its own:

-- forward: fact rows whose dimension row qualifies
SELECT count(*) FROM fact f WHERE <f filters>
  AND EXISTS (SELECT 1 FROM dim d WHERE d.pk = f.fk AND <d filters>)
-- reverse: dimension rows with at least one qualifying fact row
SELECT count(*) FROM dim d WHERE <d filters>
  AND EXISTS (SELECT 1 FROM fact f WHERE f.fk = d.pk AND <f filters>)

and their IN spellings. Both arrive as semi joins (pull_up_sublinks()), and they are answered differently.

  • The forward form needed nothing new. Its inner side is the dimension, and when the dimension’s key is provably unique for the join clause the planner turns the semi join into an inner join before the join search begins (reduce_unique_semijoins(), which deletes the SpecialJoinInfo): a fact row then joins at most one dimension row, so counting join pairs counts fact rows. What reaches UPPERREL_GROUP_AGG is the inner join this section was written for, and the EXISTS, the IN and the JOIN spellings get the same plan. The equivalence is core’s, made from core’s uniqueness proof; lion_column_is_unique() is asked afterwards, as ever, as a costing guard. (Before PostgreSQL 19 core does not compare collations in that proof - the XXX of “Uniqueness of the dimension key” - so an EXISTS over a key unique only under another collation is an inner join to core; the node then has no proof of its own, declines, and the ordinary plan is whatever core made of it.)
  • A non-unique key keeps its semi join, and has to: counting pairs would count a fact row once per duplicate. The node never treats it as an inner join. Its inner side is the DIMENSION, so the one shape it could take at first is the reverse one below with the roles exchanged - each fact row tested against the dimension’s posting sets - which needs a lion index on the dimension’s key, dimension filters that posting sets answer, and a lookup per fact row, which the cost model refuses whenever the fact table is the large one (tested). Core’s other way of running it, the dimension’s distinct keys joined as an inner join (JOIN_UNIQUE_INNER), is the one the node takes now, as a sum over distinct keys (“Forward semi joins over a non-unique key”, below).
  • The reverse form is new. Its rows are the OUTER side’s, each once, when it has a match. That is the lookup the node already makes per dimension row, with §26’s existence test in place of the count: the fk set merged with the fact filters and stopped at the first container that shows a row the snapshot sees. The partial count is 1 per matching dimension row and the Finalize Agg adds them up as before, so the grouped form (d.attr, count(*) ... GROUP BY d.attr) comes with it.
  • NOT EXISTS is the anti join, the same test inverted: a dimension row counts when its key is NULL (it joins nothing under a strict operator), when its key has no fk entry, when the test finds nothing, and - with no lookup at all - when a fact filter selects nothing. NOT IN is not an anti join (its NULL semantics differ), and on 16 to 18 never reaches the node as one; 19 makes one of it where it proves neither side can be NULL, where the two agree, and the node takes that as it takes NOT EXISTS’s (“The semi and anti join as a join path” below). The anti join the planner makes of LEFT JOIN ... WHERE f.fk IS NULL is declined.
  • Telling the two anti joins apart (lion_fkjoin_sublink_anti()). The LEFT JOIN ... IS NULL one carries the LEFT JOIN’s range table entry as its relid (ojrelid), which the parser made of the FROM clause (inFromCl). The NOT EXISTS one had no relid until PostgreSQL commit 0d6103e (“Give antijoins made from sublinks a relid”, in 19 and back-patched to the 16, 17 and 18 minor releases), which gives it a join entry the planner makes, not inFromCl, so that a qual on a PlaceHolderVar above it is not pushed into its inner side. So a NOT EXISTS is an anti join with no relid or with a relid whose entry is not inFromCl, on every release either side of the change, and no version test is involved. Its relid is no table: wherever recognition walks a join rel’s relids for tables it skips root->outer_join_rels, and the relid of the NOT EXISTS whose inner side is the fact belongs to the join of fact and dimension, not to the dimension.

Recognition (lion_fkjoin_recognize()): exactly one SpecialJoinInfo, JOIN_SEMI or JOIN_ANTI (that of a NOT EXISTS, above), whose minimal sides are one base relation each and equal to its syntactic sides; the fact is its inner (right) side and the dimension its outer side (a semi join is also taken the other way round, as “Forward semi joins over a non-unique key” below); the one join clause is found as for an inner join, from the equivalence class or joininfo. (Over three tables or more, the other special joins are the dimension’s own: “A dimension that is a join”.) A qual of a NOT EXISTS that names only the outer side (NOT EXISTS (... AND d.attr = 2)) stays a join clause of the anti join and is a second one, which declines; inside an EXISTS the planner moves it to the outer relation, where it is a dimension filter like any other. No uniqueness is asked: the count is per dimension row, so a duplicated key is two rows tested twice and counted twice, as the semi join counts them, and a NULL key is looked up by nobody. The aggregates are the inner join’s counts: count(*), count(1), and in a semi join count(d.pk) (a matched row’s key is not NULL); an anti join’s rows include the NULL keys, so there count(d.pk) is no count of rows. It used to decline; it is now a count of a column, answered over the anti join’s rows (“Every aggregate over the node’s rows”). The partial path’s row estimate is the join rel’s own.

Visibility is §26’s argument, unchanged: an existence test is a count compared with zero, computed from the same containers read the same way, and the per-dimension-row structure is this section’s. EXPLAIN prints Join Type: Semi or Join Type: Anti under Lion Indexes; an inner join prints no join type.

The fact filters, collected once (2026-09-27)

Measured first on the benchmark’s shapes (below: five million fact rows over 96,154 heap pages, fk a hash of the row number over 200,000 keys, 25 rows a key), the forward count with the fact filter doc @@ '(sen0 | sen1) & (jfd0 | jfd1)' (5.6% of the rows) and a three-clause dimension filter (10,601 dimension rows) took 8,030 ms through the node, which the model had chosen at 46,467 against the hash join’s 104,308 - and the hash join took about 500. Two errors compounded:

  • a multi-key clause was priced as ONE set, while its query is four lexemes, and every count builds the tsquery’s merge again at each of the fk set’s container keys: the per-count union the 2026-09-23 review found for IN lists, never priced for a multi-key clause. 10,601 counts read 1.35 million containers;
  • rows per fk key were tuples / estimate_num_groups(fk), which scales n_distinct down to the rows the relation’s own clauses leave: under the 5.6% filter the fk looked like 35.6 rows a key where it has 25, and a 2,000,000-key fk like 21.9 where it has 2.75.

Both are fixed in the price - lion_multikey_nkeys() counts the sets a multi-key query is made of, and n_distinct is taken over the whole table as lion_range_entries() takes it - but pricing the probes honestly only refuses the node. What makes it fast is not probing at all. The fact filters are the same for every dimension row, so the node merges them ONCE, over all of their containers - exactly the merge count(*) WHERE <f filters> makes, driven by the sparsest source and probing the others - and instead of counting the containers that survive it COPIES them into private memory (lion_sources_collect(), lion_count.c): the filters' intersection as one materialized posting set of no index entry. Each dimension row’s count is then its fk set ANDed with that copy, which a count seeks by a binary search in memory. The same forward count takes 135 ms (table below).

  • The copy is made at the first child row of each run, after the WHERE sets are located as they always are, in the node’s outer-group context; it is released on rescan and at the end, and a rescan with new parameters makes it again.
  • It is budgeted by a hash join’s memory, get_hash_memory_limit() - work_mem times hash_mem_multiplier - and not by a setting of its own: the copy stands where the ordinary plan’s hash table would, over the same rows, and is the same kind of memory; in a parallel plan each participant used to make its own, as each participant of a hash join below a Gather builds its own table (“Parallel” below), and the participants now make one together, in a Parallel Hash’s memory (“One copy per query” below). A copy that outgrows it SPILLS, as a hash join’s table would (2026-09-28 review): what it holds so far goes to a temporary file (LionSpill, lion_set_copy.c), and so does every container after it, in key order; memory keeps sixteen bytes a container - its key, where it is and its size - and a count reads the file a container at a time, a seek being a binary search over those keys. EXPLAIN ANALYZE prints Fact Filter Copies Spilled. It used to be abandoned there, and the run read the filters per count - for an IN list whose length the planner could not see (a parameter, priced at estimate_array_length()’s ten values) that is the union of every one of its sets built again for every dimension row, hour-scale for a long list over many rows, and exactly the plan the cost model had just refused. The temporary file honours temp_tablespaces, which are looked up before the merge starts (a lookup reads catalogs, which may process invalidations, and must not do so in the middle of a walk of the index); it belongs to the copy and is closed when the copy is released, and on an error with the resource owner it was opened under.
  • On a hot standby, where whether the map may be trusted depends on the WAL mode of every index read, no copy is made, and the run reads the filters per count, as before. The planner now knows it: in recovery lion_cost_fkjoin_rel() never prices the copy, and a plan is chosen for the probing it will do.
  • An IN list too long to open at once used to be refused a copy as well: a count takes such a list in batches (§15, “Bounded cursors”), each yielding every container key of its own, which would reach a copy out of order. A collection is never batched now (2026-09-27): the batches exist to keep a count’s pins, which a collection drops anyway, so the list is read as the windowed union any other union too wide for the open budget is (lion_plan_node(), lion_wide_fill()), a window of container keys at a time and every key in order. That is what makes a list whose length the planner could not see safe to collect, however long it turns out.
  • Nothing is asked of the visibility map or the heap while the copy is made, and nothing is pinned: its cursors drop each leaf once copied (cx->droppins), the visibility cache is not consulted, and the disjoint sum and the one-set shortcut of §15 are not taken - a copy wants the union itself, not a sum of counts. Filters that only subtract (IS NOT NULL alone) have nothing to intersect and are not collected.

Why a stale copy is safe is lion_posting_set_materialize()’s argument, which is §9’s: the copy may list a TID that VACUUM has since removed from its index, pruned from the heap and whose block it may even have marked all-visible - but the copy is only ever counted ANDed with a dimension row’s fk set, located under its own pin. A dead TID is then either gone from the fk set’s container, and out of the intersection, or in it, which means the fk set’s page was copied before VACUUM’s ambulkdelete got past that page; so VACUUM has not finished ambulkdelete on the fk index, so it has not set that block all-visible, so the TID goes to the heap recheck and the snapshot decides. The copy cannot lack a row the snapshot sees: it is made after the snapshot was taken, and a visible row was in every index before its transaction committed. An fk set located over the pin budget (NOPIN, §15) carries no interlock, and then nothing does: the merge finds no pinned positive source and rechecks every candidate (cx->novm), as any count does. lion_count_one_set() copies a collected set as it stands for a collection, which asks the map nothing: the copy of fact filters that are one range or one list already collected (§32), one source of one set, which the one-set shortcut takes (fkjoin_walk.sql pins it with the range source’s and the copy’s counters). It COUNTS one on its own as the merge counts a source nothing pinned carries: the map is asked nothing and every member is rechecked in the heap (cx->novm). The node arranges a carrier where it can - a filtered count’s run whose one source is a collected list is a sum over the driving walk (§15, “A list too long to locate at once”) - so that is the count’s fallback, exact wherever it is reached. test/isolation/fkjoin_vacuum_race.spec proves it the way count_distinct_vacuum_race.spec proves the existence test: the copy lists fk = 0’s dead rows, the first count parks at lion-count-containers-pinned with fk = 0’s set pinned, the VACUUM that would remove those rows waits for the pin, and the answers - semi join and inner join - exclude fk = 0 and include the rows deleted after the snapshot.

The choice between the copy and the probes is the cost model’s, per path, and the plan carries it: the join member of custom_private became {clause, kind of join, flags [, child column]} with LION_JOINFLAG_COLLECT (shape 12). EXPLAIN prints Fact Filters: collected once. Since 2026-09-29 a plan that probes may still collect part way through a run, once probing has cost what collecting would (“Probed, then collected”, below), which bounds what an estimate of the dimension rows far too low costs.

Cost, revisited (2026-09-27)

lion_cost_fkjoin_rel() prices both ways of reading the fact filters and keeps the cheaper:

  • probed, as before: per dimension row LION_FKJOIN_COUNT_COST, LION_FKJOIN_PROBE_COST per probe at the fk set’s container keys, and for a union - an IN list, an OR, and now a multi-key clause of several keys - LION_FKJOIN_SET_COST per set and its prorated merge, per count;

  • collected: once, the filters' merge over all of their containers (the driver’s containers, a probe of every other source at each of them, each union built in full) and LION_FKJOIN_COPY_CONTAINER_COST per container copied; per dimension row LION_FKJOIN_COPY_COUNT_COST, and per fk container LION_FKJOIN_COPY_PROBE_COST and LION_FKJOIN_COPY_MEMBER_COST per member it holds. The copy is sized as it is made, a container at its smallest: its members at two bytes each, or a bitmap no wider than the offsets of the relation’s pages - a NARROW (§38), 1,032 bytes at 120 rows a page - and a bitset’s 4 kB at most (lion_container_max_bytes()). Past the get_hash_memory_limit() the executor gives it - times the participants, for one shared - it is priced SPILLED: its pages written once at seq_page_cost, and every fk container a count looks up read back from the file at LION_SPILL_READ_COST (pg_lion.spill_read_cost). Never on a hot standby, which makes no copy.

    The read is measured (2026-10-19, assert-enabled build): the copy of x IN (1, 2, 3, 4, 5) over 3M rows, 1.5M of them, spilled at a 64 kB work_mem, against the same plan with the copy in memory: 20,000 dimension rows read 2.49M containers back, from 830,000 pages of the file, in 580 ms more (799 ms against 210 to 219) - 233 ns a container on a build that counts against the copy in memory at 287 units a millisecond, 135 ns at the reference’s 500: 27 cpu_operator_cost. At 256 kB the copy, 1.5M rows of 120 a page in NARROWs, is 400 kB and fits the 512 kB hash table, which a bitset’s 4 kB a container put at 1.6 MB, past it: the node takes the query in 204 ms where core’s hash join takes 907.

A copy the estimate said would fit and that does not is spilled, not abandoned (above), so the plan the estimate chose runs as collected whatever the fact filters turn out to hold; an IN list of a parameter that was estimated at ten values and holds a hundred thousand costs its one merge and a file, not a union per dimension row. That is why the probed price of such a list (the longest literal list, “An IN list whose array is a PARAMETER” in “Planner integration” above) is only the price of the probed PLAN: the review that found the fallback proposed pricing a list of unknown length at the worse of the two, which with the spill would refuse plans that run as collected. What still reads the filters per count is a run without a copy: a hot standby, which is priced so, and fact filters with a range too large to collect (§32), which is priced as walked by every count.

A directory descent costs LION_DESCENT_COST (120 cpu_operator_cost) a level, as every lookup of the count does (§10, “The units”). A semi or anti join’s existence test reads the share of the fk set lion_exists_fraction() expects - rows per key x filter selectivity visible rows spread over the set’s containers - of its containers, probes and recheck candidates, as §26’s walk does. The fk set’s containers are read and counted at §10’s LION_CONTAINER_COST and LION_MEMBER_COST a member, and a union filter’s sets are rebuilt at every count at LION_FKJOIN_SET_COST, which is §10’s LION_UNION_SET_COST.

Refitted on the release build (2026-09-27; 4-core VM, warm; §31). The node’s own CPU time - the node minus its dimension child, both timed by backend CPU time over DO loops, serial - on the 23 shapes of the table below (the fact filters collected in 20 of them) against its counters, fitted by non-negative least squares: 0.60 us a directory page read (the descents, 3 to 4 a dimension row), 0.14 us a container visited (the fk set’s and the probes into the filters), 84 ns a row of the fact filters collected, and nothing left per dimension row; median residual 9%, worst 89% (the thousand-row dimension, whose 2,000-row fk sets the fit prices by containers alone). At 500 units a millisecond that moved:

constant assert build release build
LION_FKJOIN_DESCENT_COST / LION_DESCENT_COST 25 cpu_operator_cost 120 (0.6 us a level)
LION_FKJOIN_PROBE_COST 30 cpu_operator_cost 80 (0.4 us; 0.46 measured over the 2,000-row sets of the thousand-row dimension)
LION_FKJOIN_COUNT_COST 50 cpu_tuple_cost 25 (the per-row constant the fit leaves at zero)
LION_FKJOIN_SET_COST 80 cpu_tuple_cost 100, as LION_UNION_SET_COST (2 to 6 us a set a count)
the fk set’s containers 2 cpu_operator_cost each LION_CONTAINER_COST + LION_MEMBER_COST a member
LION_FKJOIN_COPY_* 25 cpu_tuple_cost, 15, 3 and 20 cpu_operator_cost unchanged: the collected shapes come out at 374 to 595 units a millisecond with them

The model then prices the node at 230 to 717 units a millisecond of its own CPU (median 467), against 185 to 488 (median 294) before; the three shapes at 230 to 250 are the thousand-row dimension’s, whose collected plan the refitted model now prefers over the probed one they were timed in. The ordinary plans of the same queries run at 214 to 820 (hash joins 185 to 283, nested loops over a btree higher, since core charges their inner index pages as random reads). (The assert-build calibration found the collected paths at 150 to 390 units a millisecond and the probed ones at 106 to 177 there, a probe about 1 us; that build inflates lion’s own code more than core’s, which is why its constants were refitted.)

Measured (2026-09-27, PostgreSQL 18.6, assert build, so ratios and not absolute numbers; four cores shared with other work, medians of five, warm, all-visible, max_parallel_workers_per_gather = 0). The fact: 5,000,000 rows over 96,154 heap pages; dim_id over 200,000 keys (25 rows a key) and dim2_id over 2,000,000 (2.75 rows a key, the high-cardinality fk), both in random heap order and 1% NULL; kind 20 values, sen 6, doc a tsvector of three lexemes; lion indexes on each fk, on (kind, sen) and on doc. The dimensions: 200,000 and 2,000,000 rows, a primary key and a lion index on (status, country, size). “Ordinary” is the plan with the pushdown off: a hash join (or, for the reverse forms, a hash right semi/anti join) over a bitmap heap scan of the fact filter, a nested loop for the smallest dimension set. Bold is the plan the model picks now.

shape dimension rows ad9c155: the plan picked now: the node ordinary
forward, doc @@ '(sen0 | sen1) & (jfd0 | jfd1)'; EXISTS, IN and JOIN alike 10,601 node 8,030 ms (46,467) 135 (30,629) 533 (104,308)
forward, kind = 'k3' AND sen IN ('sen0', 'sen1') 10,601 hash join 440; node 467 at 2,315,300 112 (30,118) 494 (100,574)
forward, the tsquery, a narrower dimension 5,302 node 4,068 (26,739) 93 (21,139) 588 (104,054)
forward, no fact filter 10,601 node 51 (17,938) 52 (15,986) 963 (161,705)
forward, the tsquery, status = 'val1' alone 140,075 hash join 695 1,154 (257,150) 680 (108,455)
forward, the tsquery, 2,000,000-key fk 105,268 hash join 679; node 9,755 at 321,828 477 (111,406) 671 (120,256)
… 52,695 hash join 673; node 4,966 at 174,941 307 (73,425) 649 (117,839)
… 2,595 node 301 (24,399) 88 (24,199) 73, nested loop (62,356)
reverse, EXISTS or IN, the tsquery 10,601 not pushed down 112-162 (28,444) 655-721 (106,927)
reverse, kind = 'k3' AND doc @@ 'sen0 & jfd0 & jfr0' (0.2% of the fact) 10,601 not pushed down 128 (25,711) 27, nested loop (3,739)
reverse, the tsquery, GROUP BY d.size 41,951 not pushed down 307 (78,059) 592 (108,368)
reverse, the tsquery, 2,000,000-key fk 105,268 not pushed down 501-686 (111,178) 695 (120,581)
… 52,695 not pushed down 294 (73,312) 589 (118,165)
anti (NOT EXISTS), the tsquery 10,601 not pushed down 110 (28,419) 526 (106,902)
anti, kind = 'k3', 2,000,000-key fk 52,695 not pushed down 280 (68,418) 568 (118,021)

Where two figures are given for one plan they are the two five-run medians the same plan got as the cost model’s pick and forced; the machine was shared. The one wrong choice left is the smallest dimension set over the high-cardinality fk (88 ms against the nested loop’s 73), where core prices its own nested loop at 62,356.

§27’s own table, re-measured on this machine with the same layout (two million fact rows, fk over 1..1000 in random heap order, a 1000-row dimension):

query ad9c155 now ordinary
d.attr, count(*) ... GROUP BY d.attr 51 (5,400) 52 (5,275) 646 (55,920)
... WHERE f.x = 3 GROUP BY d.attr node 166 (29,622), refused node 152 (29,497), refused 125 (26,413)
... WHERE f.x = 3 AND d.region = 'eu' GROUP BY d.attr 43 (7,470) 37 (7,439) 95 (25,661)
count(*) ... WHERE d.attr = 5 2.7 (295) 3.0 (289) 200 (46,160)
count(*) ... WHERE f.x IN (1, 2) node 988 (1,044,553), refused node 176 (33,252), refused 159 (31,328)

The copy makes the IN list’s counts five times faster (988 to 176 ms), since its union is built once; the single-set filter gains little, because a probe into it was already a seek of one set. Every choice is unchanged. It took a per-member term to keep them so: priced per fk container alone, a count against the copy of sets that hold six rows a container (these) was underpriced, and the second and last rows were chosen at 17,183 and 18,252 - 1.2 and 1.1 times slower than the hash join. LION_FKJOIN_COPY_MEMBER_COST (3 cpu_operator_cost per member of an fk container) refuses them and moves no other choice.

A lookup per dimension row is what the node pays that a hash join does not, so a large dimension set is where it loses: 140,075 of the 200,000 dimension rows (status = 'val1' alone) is refused at 257,150 against 108,455 (1,154 ms against 680). The high-cardinality fk is cheaper per row - a set of 2.75 rows is two or three containers, and its entries are INLINE, so the only page a lookup reads is the directory leaf - and the node stays ahead of the hash join up to the largest set measured, 105,268 rows of the 2,000,000-row dimension (477 ms against 671), whose cost it prices within 8% of the hash join’s. The choice rests on the planner’s estimate of the dimension set; the fk’s cardinality is priced per key, from n_distinct over the whole table.

count(DISTINCT) (2026-09-27)

count(DISTINCT x) over the join, where x is the fact’s join column or a column of the dimension:

SELECT count(DISTINCT f.fk) FROM fact f JOIN dim d ON f.fk = d.pk
WHERE <f filters> AND <d filters>

SELECT d.region, count(DISTINCT d.attr) FROM dim d
WHERE EXISTS (SELECT 1 FROM fact f WHERE f.fk = d.pk AND <f filters>)
GROUP BY d.region

A DISTINCT aggregate has no partial form, so the node cannot hand it partial counts. It emits ROWS instead: one per dimension row that joins (for an anti join, one per dimension row that joins none), carrying the columns core’s grouping input needs, and core’s own Agg above it computes the aggregates over them, as it would over the join. The Agg’s input is sorted by the query’s group pathkeys: the GROUP BY columns followed by the DISTINCT argument core may have chosen to presort (PostgreSQL 16’s aggpresorted, whose DISTINCT then compares neighbours only), which is the order core’s own sorted Agg would be given. Which dimension rows join is the semi join’s question whatever the join type - an inner join’s row needs one visible fact row, not a count of them - so in this mode every join takes §26’s existence test.

Why it is the join’s answer:

  • A DISTINCT count does not see multiplicity. The join’s pairs (f, d) project onto x; for a dimension column, the values over the pairs are the values over the dimension rows that have at least one pair, each as often as it has pairs, and a DISTINCT keeps each once. The node’s rows, each matching dimension row once, give the Agg the same values.
  • The fact’s join column is a value the rows do not carry, but every fact value a dimension row d joins is equal to d.pk under the join’s operator. When the DISTINCT compares with that equality - its SortGroupClause equality operator and the join operator share a btree or hash opfamily (equality_ops_are_compatible()) and the aggregate’s input collation is the join clause’s - all of d’s fact values are one distinct value, and d.pk is it; so the node emits d.pk where the aggregate reads f.fk, which needs the two to be of one type (an int4 fk against an int8 key declines). The DISTINCT sees equality, never a representation, so §10’s value-representation rule has nothing to ask. In the target list or a GROUP BY, where it is printed, the fact’s key is taken under that rule as well (“Grouped by the join key”).
  • Uniqueness is not needed for the answer. Two dimension rows with one key are two rows whose equal keys the DISTINCT collapses, as it collapses the join’s pairs. (An earlier version of this section held that it would make uniqueness a correctness premise; that is true of counting the matching dimension rows, which is not what the node does.) The inner join is still recognised only when an index proves the dimension key unique (lion_column_is_unique(), above: core’s kind of proof, never statistics), so over a key without one it stays with the ordinary plan, as before; the semi and anti joins never asked for one.
  • NULLs, filtered and invisible rows. A NULL fk joins nothing; a NULL dimension column is a row whose value the DISTINCT skips, as it skips it among the join’s pairs; a NULL dimension key is a row of an anti join only. A dimension row the dimension filters remove, or the snapshot does not see, is never a row of the child.

What is accepted: count(DISTINCT v) of one plain column (under a relabelling cast), with no FILTER and no ORDER BY, and several of them over different columns (the Agg sorts for the ones it does not presort); any sortable GROUP BY of dimension columns and expressions of them, HAVING, ORDER BY and LIMIT above. A count of rows beside a DISTINCT count used to decline - the rows stand for dimension rows, not for pairs - and is answered now by rows that carry their counts (“Every aggregate over the node’s rows”, below); a DISTINCT of an expression, of a fact column other than the key, of another aggregate (sum(DISTINCT)), or under another collation still declines.

The node’s target is the join rel’s, the grouping input core would compute over the join, with the fact key’s Var mapped to the key’s child column (lion_plan_fkjoin_path()); the join member of custom_private carries LION_JOINFLAG_ROWS. Its cost is the existence tests' (“Cost, revisited”: exists), its rows the join rel’s estimate for a semi or anti join and the child’s for an inner one, and the Sort and the Agg are costed by core. EXPLAIN prints Join Rows:

Aggregate
  ->  Sort
        Sort Key: fk
        ->  Custom Scan (LionCount)
              Lion Indexes: fact_fk (fk = d.pk), fact_x (x = 3)
              Join Rows: the dimension rows with a match
              Fact Filters: collected once
              ->  Seq Scan on dim d
                    Filter: (region = 'eu'::text)

(the dimension rows without a match for an anti join). The sort key prints unqualified because an upper rel’s CustomScan carries no relids for EXPLAIN to name the fact by. Measured in the table of “Parallel” below: over ten thousand dimension rows the serial node takes 113 ms for the forward count(DISTINCT f.dim_id) and for the reverse count(DISTINCT d.attr), where the ordinary plans take 602 and 513.

Parallel (2026-09-27)

The node is parallel-aware. When the query may run in parallel - the grouped rel, the join rel, the fact rel and the dimension rel all consider_parallel, and the clause values are parallel-safe - it is offered, besides the serial path, over the dimension’s cheapest partial path below a Gather:

Finalize Aggregate
  ->  Gather
        Workers Planned: 3
        ->  Parallel Custom Scan (LionCount)
              Lion Indexes: fact_dim2 (dim2_id = d.id), fact_doc (doc @@ '...'::tsquery)
              Join Type: Semi
              Fact Filters: collected once
              ->  Parallel Bitmap Heap Scan on dim2 d
                    ...
  • Disjoint shares. The partial child gives each participant a disjoint share of the dimension rows (a parallel sequential scan’s blocks, a parallel bitmap heap scan’s pages, as core divides them). The node counts, tests or emits each of its rows exactly as the serial node does, and nothing else is divided, so every partial count or row comes from one participant and the answers are the serial node’s. Above the Gather, core’s Finalize Agg adds the partial counts (hashed for a GROUP BY, sorted over no columns for a folded one, plain without one); the rows of a count(DISTINCT) are sorted and aggregated above it by core’s Agg. (The forward semi join over a non-unique key divides its distinct keys instead, over the dimension’s whole plan, and shares a counter to do it: “Forward semi joins over a non-unique key”.)
  • Nothing is shared but EXPLAIN’s counters - and, since “One copy per query” below, the fact filters' copy. Each participant locates the fact filters (and its lists under its share of the list pin budget, §15), and used to make its own copy of them (lion_sources_collect()) under its own get_hash_memory_limit(), spilling it to a temporary file of its own when it did not fit. The copy was not built once in dynamic shared memory: its size is known only once the merge has run, after the Gather has sized its DSM, and one copy would make every worker wait for its builder (the barrier a Parallel Hash needs) to save a merge that took 0.8 ms for 1,500 containers of one set (“Cost, revisited”). A benchmark whose filters took far longer than that to collect is why it now is, in the query’s DSA (the size is no longer the DSM’s question), and collected by all of them together. The cost model charges the filters' lookups in full to every participant, the making of the copy divided among them, and the per-dimension-row work over one participant’s share of the rows, so that the path’s cost is one participant’s. The shared struct (LionJoinShared: a spinlock and sums) takes each worker’s counters when it shuts down (ShutdownCustomScan); the leader’s EXPLAIN ANALYZE prints its own plus the workers' - Join Keys Looked Up is the number of dimension rows with a key, whichever participants had them - and for Fact Filter Rows Collected the largest copy.
  • How many workers. The node’s work is its lookups and counts, not the dimension scan its child’s worker count was computed for: ten thousand dimension rows are a small scan, which core gives one worker, and a large join. So the Gather is given the workers compute_parallel_worker() would give a parallel scan of the fk index pages the lookups read - the index’s pages times the dimension rows' share of the fk’s distinct keys - or the child’s, whichever is more; the child’s parallel scan hands its rows to however many participants come. A parallel_workers setting on the dimension decides alone, max_parallel_workers_per_gather caps it as it caps core’s, and each participant’s rows are the dimension rows divided as core divides a partial path’s (get_parallel_divisor(), the leader’s share included under parallel_leader_participation). One worker and a leader that does not take part is no cheaper than the serial node, and is not chosen.
  • Snapshot, pins and locks. Workers run under the leader’s snapshot, so the child and the counts see what the serial node would. The §9 interlock and “Why a stale copy is safe” hold per participant: each pins its own fk sets beside its own copy. A worker opens the fact table with an AccessShareLock of its own, as core’s scans do in a worker (ExecGetRangeTableRelation()), and the indexes as every participant does. _PG_init registers the scan methods (lion_count_scan_register()): a worker reads the plan before it has planned anything.
  • Rescans. A rescan of the Gather - the inner side of a nested loop - ends its workers and starts new ones, each with a new node; the leader’s node rescans its child, whose shared scan state core reinitializes, and the counters' sums are kept across (ReInitializeDSMCustomScan leaves them), as the leader’s own are.
  • The serial node is parallel-safe when the same conditions hold and its child is, so a worker may run it whole, under debug_parallel_query’s single-copy Gather. Its counters then stay in that worker - core gives a node that is not parallel-aware no shared memory - and EXPLAIN ANALYZE shows zeros for them in that one configuration.

Measured (2026-09-27, the benchmark of “Cost, revisited”, same machine: four cores shared with other work, PostgreSQL 18.6 assert build, medians of five, warm, all-visible). “Serial” is max_parallel_workers_per_gather = 0, “parallel” 3; “node” is the pushdown with the join methods disabled, “ordinary” the pushdown off; cost in parentheses, wN the workers launched, bold the plan the model picks with parallel query allowed:

shape dimension rows node, serial node, parallel ordinary, serial ordinary, parallel
forward, the tsquery 10,601 137 (30,689) 79 w2 (20,774) 477 (104,485) 322 w3 (102,774)
forward, no fact filter 10,601 52 (15,951) 31 w2 (11,530) 832 (161,701) 373 w3 (119,377)
forward, the tsquery, status = 'val1' alone 140,075 1,169 (257,155) 448 w3 (107,152) 617 (108,646) 402-420 w3 (105,044)
forward, the tsquery, 2,000,000-key fk 105,268 497 (113,946) 208 w3 (74,535) 668 (120,610) 407 w3 (116,836)
reverse, the tsquery 10,601 111 (28,232) 73 w2 (19,720) 531 (107,180) 414 w1 (103,625)
reverse, the tsquery, 2,000,000-key fk 105,268 490 (113,710) 254 w3 (64,858) 647 (120,954) 378 w3 (117,337)
reverse, the tsquery, GROUP BY d.size 41,951 299 (76,277) 140 w3 (38,705) 571 (108,599) 411 w1 (104,780)
anti, the tsquery 10,601 107 (28,208) 71 w2 (18,719) 494 (107,156) 346 w1 (103,553)
count(DISTINCT f.dim_id), forward 10,601 113 (28,951) 72 w2 (20,469) 602 (105,393) 532, not parallel
count(DISTINCT d.attr), reverse 10,601 113 (28,927) 71 w2 (20,415) 513 (107,875) 432 w1 (105,155)
count(DISTINCT f.dim2_id), 2,000,000-key fk 105,268 507 (123,222) 195 w3 (83,811) 706-759 (121,557) 388 w3 (118,696)

Two workers bring 1.5 to 1.7 times, three 2 to 2.5, on four shared cores. The one set where the serial node loses to the hash join - 140,075 dimension rows over the 200,000-key fk - it still loses in parallel, by a tenth (448 against 402 ms, priced 107,152 against 105,044), and the model keeps the parallel hash join. With the child’s own worker count (one worker for ten thousand dimension rows) the first row took 94 ms and the third 650; the fk index’s pages give it two and three. The serial count(DISTINCT) over the 2,000,000-key fk is the one serial choice the model gets wrong here (the hash join at 121,557 against the node’s 123,222, 706 ms against 507).

One copy per query (2026-09-28)

Every participant of a parallel plan used to collect the fact filters for itself: the same merge over the same sets, the same copy, as many times as there were processes, and at the same time, so that each took longer than the serial collection did - Fact Filter Collect Time, summed, was several times the serial one and each participant’s share of it was spent before its first count. A benchmark’s collection was a large part of its parallel join’s wall time for that reason. Now the participants collect ONE copy, together, into the query’s dynamic shared memory, the way a Parallel Hash builds one table, and every one of them reads it (lion_shared_copy_collect(), lion_count_shared.c; LionSharedCopy).

  • When. A parallel-aware node whose plan collects the filters, in a Gather whose workers started. A plan run without them - none started, or no DSM could be made - runs in the leader, which makes a copy of its own within a hash join’s memory, exactly as a serial plan does; so does the serial node a worker runs whole under debug_parallel_query, which is not parallel-aware. Each participant still locates the filters for itself (under its share of the list pin budget, §15): collecting reads its own located sets, and every count its own located fk set.
  • Collecting, cooperatively. The heap’s container keys are cut into chunks - four for each participant the plan was made for, so that the work evens out, at most 64, at least 16 container keys each, the last one open-ended because the heap may grow - and each participant claims the next chunk nobody has from an atomic counter and collects it: lion_sources_collect()’s merge of its own located sources, sought to the chunk’s first key and stopped at its end (LionCollect.ranged). A windowed union (§15, “Bounded cursors”) starts its first window at the chunk’s first key and reads no child past its end; before it did, every chunk read its lists from the heap’s first key to its last. When a participant has no chunk left it waits at a barrier (the wait event LionFactFilterCopy from PostgreSQL 17, Extension before), and the last to arrive is ELECTED to index the copy: where every container is, its key, and the direct index where the keys are dense (“The copy, looked up by key”), in shared memory. The others wait for that, and every participant then reads the copy through a view of its own - its own array of pointers to the containers, eight bytes a container, and the files of the chunks that spilled opened read-only - and never writes it. A participant that starts late joins whatever phase the copy is in: it collects the chunks left, waits for the index, or just reads; one that never starts is waited for by nobody.
  • Memory. The copy’s containers are in the query’s DSA (es_query_dsa), whose size is not the DSM’s question - the reason the copy could not be built there when the DSM was the only shared memory (“Parallel” above). They may take what a Parallel Hash’s table may: a hash table’s memory (get_hash_memory_limit()) for each participant the plan was made for, the leader included - the Gather’s workers planned, plus one. A chunk is collected into its participant’s own memory first, at most what the copy has left of its memory, and then goes into the DSA if the room is still there when it is reserved, atomically; otherwise, and for a chunk that outgrew it while it was collected, into a file of the plan’s SharedFileSet, named for the chunk, as a Parallel Hash’s batches go to theirs - with where each container is in it, sixteen bytes a container, in the DSA. So a copy may be partly in memory and partly in files, and a count reads a container from its chunk’s file only when the container is there. The index - sixteen bytes a container and four for its key, and a direct index of at most sixteen - is the copy’s price of being read, as a spilled serial copy’s entries are, and is not held to the limit.
  • Why the counts stay exact (§9). The copy is the serial node’s, made in parts: every chunk is read after the query’s snapshot was taken, which is all “Why a stale copy is safe” asks of a copy - it cannot lack a row the snapshot sees - and each participant counts the copy only ever beside its own located fk set, which carries the interlock. The chunks cover every container key once, and a key’s containers are the merge’s at that key whichever participant collected them. A participant waits at the barrier with its located filters still pinned, as a serial collection holds them while it runs, and lets them go when the copy is done; a VACUUM held up by those pins waits for them with nothing else held (§9), so the wait closes no cycle - the participants it waits on need nothing of it to finish.
  • Rescans. A rescan of the Gather ends its workers and, before it starts new ones, calls ReInitializeDSMCustomScan, with no participant running: the leader lets go of its view (the rescan of its own node may come only after this, ExecReScanGather()), the copy’s memory is freed and its files deleted (SharedFileSetDeleteAll()), the barrier and the counters start again and the heap is cut into chunks again. The next run collects again, with the new values of whatever it depends on. A participant’s view goes when it shuts down, before the Gather detaches the memory it points into.
  • Errors and cancels. An error in a worker is rethrown in the leader, whose abort ends the other workers; a cancel reaches the leader, which does the same. A participant waiting at the barrier answers both, since a condition variable’s sleep checks for interrupts. The copy’s memory goes with the query’s DSA and its files with the DSM, whatever happened. Tried by hand on the repro below, with chunks spilling at a work_mem of 256 kB: statement timeouts of 3 to 120 ms, all cancelled cleanly, and workers terminated while they waited at the barrier - found by their wait event - which failed their query; the next query answered right, and no temporary file was left.
  • EXPLAIN ANALYZE. The counters keep their meaning. Fact Filter Rows Collected is the copy’s rows - the largest copy any participant made, which with one shared copy is the one they all read. Fact Filter Copies Spilled counts a shared copy once a run, when any of its chunks went to a file (the participant that indexes it counts it), where a copy of each participant’s own counts once for each. Fact Filter Collect Time is still summed over the participants, and is now each one’s chunks and its wait at the barrier - divided by the participants, about the collection’s elapsed time. Two counters are new, printed only when a copy was shared: Fact Filter Copies Shared, one a run, and Fact Filter Copy Chunks, the chunks the participants collected, summed.
  • Cost. lion_cost_fkjoin_rel() divides what making the copy costs - the filters' merge, the containers copied - among the participants, as it divides the dimension rows (lion_parallel_divisor()), and takes the copy for a parallel path when it fits a hash table’s memory for each of the Gather’s workers and the leader. The filters' lookups stay every participant’s in full.

Measured on the repro of “The copy, looked up by key”, with a column of 16,384 values of which an IN list of 400 keeps 2.4% of the rows (bench/fkjoin_perkey.sh --in 400 --force-node: a copy of 487,677 rows, collected serially in 38 ms), three workers and the leader; medians of interleaved runs on four cores that other work kept busy (a load of three to nine):

query Fact Filter Collect Time, summed cost
a copy each 289 to 377 ms 398 to 465 ms 94,310
one copy, 16 chunks 176 to 297 ms 73 to 152 ms 84,001

Serially nothing changed (38 ms to collect either way). With the 0.3% flag, whose copy of one set takes a millisecond to collect, the parallel query was within the noise either way. The sums of Join Count Time and Join Lookup Time came out higher with one copy, and the query faster, for one reason. With a copy each, the leader, which starts before its workers, was done collecting first and read the dimension’s rows into its first batch (walked lookups read theirs a work_mem at a time) before any worker was done: EXPLAIN (ANALYZE, VERBOSE) showed every worker’s Parallel Index Only Scan returning no row, so the parallel plan’s counting was the leader’s alone. With one copy every participant leaves the barrier when the others do, and the rows are divided among them (in two runs, 11,165 to 25,984 a worker, the rest the leader’s), so their counting times add up - on cores that other work shared - while the query’s goes down.

Forward semi joins over a non-unique key (2026-09-27)

SELECT count(*) FROM fact f WHERE <f filters>
  AND EXISTS (SELECT 1 FROM dim d WHERE d.k = f.fk AND <d filters>)

and f.fk IN (SELECT d.k FROM dim d WHERE ...), where nothing proves d.k unique. With no proof reduce_unique_semijoins() leaves the semi join a semi join, its inner side the dimension, and its answer is the fact rows whose key is in the SET of keys of the dimension rows that pass the filters: counting join pairs, as the inner join does, would count a fact row once per duplicate. Core runs it as a (hash) semi join, or as JOIN_UNIQUE_INNER - the dimension made unique on the key by a HashAggregate or a Sort and a Unique, then an inner join. The node does the second:

  • it runs the dimension’s plan as its child, as for every FK-side join, and sorts the keys of its rows, which puts equal keys next to each other; each is kept once;
  • it counts each DISTINCT key as the inner join counts a dimension row - the key’s fk set ANDed with the fact filters or their collected copy, a partial count - and the Finalize Agg adds the counts up.

Why that is the semi join’s count:

  • Distinct keys have disjoint posting sets. A fact row has one fk value, and two keys that are distinct under the join’s equality cannot both equal it: the sort’s btree family has the join operator as its equality, and equality in a family is transitive across its types (an int4 fk against int8 keys included). So every fact row with a match is counted once, under the one key it equals, and no other fact row is counted at all.
  • Which keys are distinct is the join’s own question. The sort is by the < of a btree family that has the join operator as its equality, for the dimension key’s type (get_ordering_op_for_equality_op(), the order core’s own sorted unique-ification of a semi join’s inner side takes), under the join clause’s input collation, and neighbours are compared with that family’s =. Two keys the join cannot tell apart are neighbours and one of them is looked up - which finds the other’s entry too, since the fk index answers the join operator (lion_match_index()). An operator no btree family knows has no such order, and the shape is left to the ordinary plan.
  • NULL keys are never sorted (a strict operator matches nothing to them), and a NULL fk lives in the fk index’s NULL entry, which no key reaches.
  • Visibility is the inner join’s: the child runs under the query’s snapshot, each distinct key’s count is §9’s, and the sort holds keys, never a posting set or a pin. The fact filters are located (and collected) before the child runs and keep their pins while it runs to its end, which is the exposure “Visibility and the §9 interlock” names for a child that runs between counts.

Recognition (lion_fkjoin_recognize()): the one semi join of “Semi and anti joins”, taken the other way round - the fact its outer side, the dimension its inner one - as JOIN_UNIQUE_INNER, core’s name for the plan. Where the dimension has lion indexes of its own, the reverse orientation with the roles exchanged is offered beside it and the cost model chooses (the tests' model takes the distinct keys). lion_try_fkjoin_path() treats it as an inner join over the distinct keys, LION_JOIN_INNER with LION_JOINFLAG_UNIQUE. What the query above the EXISTS can name is the fact’s: count(*), count(1), count(f.fk) (a matched row’s key is not NULL) and count(DISTINCT f.fk), whose rows (“count(DISTINCT)”) are then one per distinct key with a match, carrying the key - and, since “Every aggregate over the node’s rows”, its count too when a count stands beside it. A GROUP BY can only be of fact columns, which no FK-side join puts in its output (“Declined”), and declines - but for the key itself, each distinct key a group of its own (“Grouped by the join key”); so does the anti join the other way round, NOT EXISTS over the fact’s rows, which would count the fact rows WITHOUT a match.

A sort in the node, not core’s unique-ification as the child. Core’s path for it is create_unique_path() up to 18 and create_unique_paths() from 19, which builds other shapes (a presorted or incrementally sorted input, a two-phase unique-ification over a Gather) and different ones per major; a hashed one gives its keys in no order, and the parallel plan below needs every participant to produce the same sequence of distinct keys, which a sort by a total order gives. A datum sort of the key alone (tuplesort_begin_datum()) is what a Sort node over one column does and takes as long: 199,000 random int4 keys sorted in 21.5 ms, and in 20 ms by a Sort node over the same Seq Scan, against about 30 ms for the HashAggregate core would make of them. It sorts within work_mem and spills past it as a Sort node’s does.

Parallel. Unique-ification per participant is not global: with the dimension’s partial path as the child, as the other joins take it, a key whose rows went to two participants would be kept and counted by both, its fact rows twice. Three plans were measured (the table below):

  • serial;
  • unique-ified in the leader: the serial node over a Gather of the dimension’s partial path, which is what the node gets without anything built whenever a parallel scan is the dimension’s cheapest path. It divides only the dimension scan, which is the small part: 118.6 ms against the serial node’s 118.5 for the first row of the table, 98.4 against 94.7, 142.0 against 141.7, 435.9 against 441.5 and 460.2 against 434.9 for the 139,929 and 199,000 rows - no gain anywhere;
  • the keys partitioned (taken): every participant runs the dimension’s WHOLE plan - its cheapest path that may run in a worker, which is not its cheapest path when that is a Gather - and sorts all of the keys, so that all of them have the same sorted sequence of distinct keys, and they divide it by position in runs of LION_FKJOIN_UNIQUE_CHUNK (64), each claimed from an atomic counter in the Gather’s dynamic shared memory. A participant claims the run after its last one only once it has finished that one, so the counter never gives out a run behind the participant’s position, and every run is counted by exactly one participant, however many start and whether the leader takes part; a plan run without its workers counts every key in the leader, and a rescan’s ReInitializeDSMCustomScan sets the counter back to the first run. The child, the sort and the copy of the fact filters are every participant’s, as a hash join’s table below a Gather is, and are charged so; the counts are divided. 1.2 to 2.3 times faster than serial on four shared cores.

Every participant has to see the same rows, so the parallel path is not offered when a dimension qual has a volatile function in it (which could keep a row in one participant and drop it in another; an EXISTS with one is not pulled up to a semi join at all, an IN is). Join Keys Looked Up summed over the participants is then the distinct keys exactly, which the tests check, and over rescans of the Gather each run’s.

Cost. The child’s total cost and the sort’s are the node’s startup: LION_FKJOIN_SORT_COMPARE_COST (a quarter of a cpu_operator_cost) a comparison, N log2 N of them, and LION_FKJOIN_SORT_KEY_COST (6) a key for putting it in, taking it out and comparing it with the previous distinct key, plus one merge pass past work_mem - which the sort’s own bytes decide: three words a key and, for a key passed by reference, the key at its column’s width (lion_fkjoin_sort_cost(); until the 2026-10-18 review the child’s whole row width, and a copy of every key, which priced a sort that fits in memory as spilling - an overcharge only). Measured: 0.108 us a key to put in and sort 40,000 or 199,000 keys, and 0.023 us to take them out and compare them - about 13 cpu_operator_cost a key at the 250 units a millisecond the node’s other terms run at. cost_sort() charges 36 a key at 200,000, three times the time the sort takes (while cost_agg() charges core’s hashed unique-ification one a row), and it answers enable_sort, which this sort, no Sort node, does not. Then “Cost, revisited”’s per-key terms, over the distinct keys (estimate_num_groups() of the key over the child’s rows) - but not all of them. The keys of IN (SELECT k FROM ...) are any set’s, and only those that are fk values find an entry; the rest cost a descent and nothing more. How many find one is what core’s own estimate of the semi join says: the fact rows it leaves over the rows the fact filters leave of one fk value (the fact rel’s rows over the fk’s n_distinct). Before that term, fkjoin_nonunique.sql’s x = 3 against a 60,000-row dimension filtered to 600 rows - 300 distinct keys, 3 of them fk values - was refused at 1,069 against the hash semi join’s 872, running in 1.0 ms against 2.9; it is chosen at 462 now, and the same dimension unfiltered (30,000 distinct keys) is refused at 6,516 against 2,238 (32.5 ms against 17.6). The other FK-side joins take every dimension row to find an entry, as an fk into a dimension key is expected to, and are priced as before.

EXPLAIN:

Finalize Aggregate
  ->  Custom Scan (LionCount)
        Lion Indexes: bf_dim (dim_id = d.k), bf_doc (doc @@ '...'::tsquery)
        Join Type: Semi
        Join Keys: distinct, sorted
        Fact Filters: collected once
        ->  Bitmap Heap Scan on bd d
              ...

and in a parallel plan Parallel Custom Scan (LionCount) below a Gather, over the dimension’s whole plan. With ANALYZE, Join Keys Sorted (the child’s rows with a key; in a parallel plan what one participant sorted, summed over the runs), Join Key Sort Method, Join Key Sort Space Used and Join Key Sort Space Type (the sort as it went in the leader, when it ran one), and Join Keys Looked Up - the DISTINCT keys. The join member of custom_private carries the sort operator and collation (shape 13).

Measured (2026-09-27, the fact of “Cost, revisited” - 5,000,000 rows over 86,207 heap pages, dim_id over 200,000 keys in random heap order, 1% NULL - against dimensions whose keys repeat: bd, 200,000 rows over 50,000 keys (every fourth fact key, four rows each, 0.5% NULL) with status 70% 'val1', country 20 values and size 10 under a lion index, and a btree on the key; bdh, the same rows over 1,000 keys; and bdu, bd’s keys once each under no unique index. PostgreSQL 18.6 assert build, four cores shared with other work, medians of five after a warm-up: the backend’s CPU time for the serial plans, wall time for the parallel ones. “Ordinary” is the pushdown off, which before this change was the plan for every one of these; costs in parentheses, bold the plan the model picks, wN the workers planned. f is status = 'val1' AND country IN ('c1', 'c2') AND size < 7.)

shape dimension rows / distinct keys node, serial ordinary, serial node, parallel ordinary, parallel
doc @@ '(sen0 | sen1) & (jfd0 | jfd1)' (5.5%), bd under f; EXISTS and IN alike 9,776 / 8,848 117 (29,777) 585 (97,834) 82 w2 (20,842) 275 w3 (94,818)
kind = 'k3' AND sen IN ('sen0', 'sen1') (1.7%), bd under f 9,776 / 8,848 95 (29,211) 309 (95,353) 66 w2 (20,276) 316 (95,353)
no fact filter, bd under f 9,776 / 8,848 44 (16,748) 845 (156,354) 35 w2 (12,361) 305 w3 (112,384)
the tsquery, bd under status = 'val1' AND country IN ('c1', 'c2') 14,016 / 12,160 139 (34,837) 595 (97,460) 92 w2 (24,304) 255 w3 (94,159)
the tsquery, bd under status = 'val1' 139,929 / 46,957 442 (91,231) 701 (100,334) 190 w3 (47,861) 310 w3 (96,444)
the tsquery, all of bd 200,000 / 49,110 430 (92,233) 665 (99,878) 192 w3 (48,844) 338 w3 (95,602)
the tsquery, bdh under f 9,776 / 1,000 51 (11,697) 45 (94,802) not parallel
the tsquery, all of bdh 200,000 / 1,000 77 (17,515) 75 (96,119) not parallel
the tsquery, bdu under f 2,316 / 2,316 57 (12,660) 563 (94,396)
count(DISTINCT f.dim_id), the tsquery, bd under f 9,776 / 8,848 96 (28,654) 571 (98,660) 78 w2 (20,724) 272 w3 (96,458)

The ordinary plans are a hash semi join over the fact filter’s bitmap or index scan, or - with no fact filter, or a large dimension set - a hash join over a HashAggregate of the dimension’s keys, JOIN_UNIQUE_INNER; over bdh’s thousand keys a nested loop over that HashAggregate with a lion index scan of dim_id inside. That nested loop is the one plan the node does not beat: 1,000 keys are cheap to look up either way, and what the node adds is the fact filter’s copy (5.5% of the fact, about 35 ms) and, unfiltered, the sort of 199,000 keys. The model picks the node there, priced as its other plans are, against a nested loop core prices at 94,802 for 45 ms: 1.13 and 1.03 times slower. Over bd the node is 1.5 to 19 times faster serially and 1.6 to 8.7 in parallel, and with 139,929 dimension rows, which the inner join’s node loses to the hash join (“Parallel”), it wins here because it looks up the 46,957 distinct keys and not the rows. The workers are the fk index pages' (as for the other joins), so bdh’s thousand keys get none.

Lookups in key order (2026-09-28)

Each dimension row’s lookup was a descent of the fk index’s directory - the root, the internal levels and a leaf - in the order the child returned the rows, which is not the fk index’s. Over a high-cardinality fk the directory is large: at the default fillfactor a leaf holds about a hundred INLINE entries of a few rows each, so a million keys are some ten thousand leaves under a directory of height 2. A dimension set of tens of thousands of keys then descended as many times, through leaves spread all over that directory with no two consecutive lookups on the same one - a random read of a leaf per key wherever the directory is not in the cache, and each leaf read again for every key it holds. A benchmark found such a join slower than the hash join the planner chose in its place. What the model charged for those leaves - each read once, at the interpolated cost of a set that dense - was what a lookup in key order would read, not what the lookups did.

So the node looks the keys up in key order when the plan says so (LION_JOINFLAG_WALK, chosen by the cost model, below):

  • it reads the child’s rows a BATCH at a time, as many as work_mem holds - each row copied as a MinimalTuple, and for a forward semi join over a non-unique key each distinct key alone - and sorts the batch into the directory’s order: the probe’s own comparison of two key values, then the hash, which is the order lion_posting_set_lookup_many_col() sorts an IN list into (§21, “Readers”) and the order of an unordered column’s directory. Keys the directory cannot sort (a cross-type key whose family has no ordering for that type, lion_probe_init()) are looked up a row at a time, as before;
  • it looks the batch’s keys up in that order with one WALK of the leaves (LionLookupWalk, lion_set.c), and counts each key’s set exactly as the row-at-a-time loop does;
  • a key that appears again in the batch - two dimension rows with one key, which a semi or anti join counts once each - comes out of the sort next to the first, and takes its answer instead of a second lookup.

The walk keeps its place between keys: the leaf the last key was located on, that leaf’s right sibling and a copy of its high key, all taken while the leaf was locked. A key below that high key is on that leaf, or - if the leaf has split since - right of it, where the walk moves right as a descent does. A key at or above it is on the right sibling or further right, and while the keys have been close together the walk steps there, at most height pages (stepping over s pages reads s, a descent height + 1); otherwise, or past those, it descends. A descent that lands on the old right sibling says the keys are close again. A descent starts below the root where it can: each one remembers the level-1 page it took its leaf’s downlink from and a copy of that page’s high key (LionDirParent, lion_dir_search_from()), and the next descent starts there while its key is below that high key - the level-1 page and the leaf, two pages a key instead of height + 1. Only the first key under each level-1 page descends from the root. The argument is the leaf’s: a level-1 page stays at level 1 and its lower bound never moves, so a key at or after the last one is under it or to its right, and a split since then is crossed by the descent’s moving right. A key found by following a run of hash collisions across a page boundary leaves the walk to the right of where that run begins, and the walk forgets its place, as the list walk does. So a batch whose keys are dense reads each leaf it covers once, and one whose keys are sparse costs two pages a key.

The batch is sorted with the key type’s sort support where its btree family has one (the probe’s comparison is that family’s, BTSORTSUPPORT_PROC): the abbreviated key is computed once per row as it is copied into the batch and compared first, with the full comparison and then the hash breaking ties, and abbreviation is given up as tuplesort gives it up when it stops telling keys apart. The order is the one the probe’s comparison gives, so nothing below changes; a type with no sort support is sorted by calling the comparison as before.

Why the place may be kept. A directory page keeps its level and is not reused while the walk’s snapshot lasts, and splits and page deletions move keys only rightwards (§21, “Readers” and “Page deletion”), so a page’s lower bound never rises. A key that sorts at or after the last one is therefore on the remembered leaf or to its right. The remembered right sibling’s lower bound is the high key the walk copied, so a key at or above that is on the sibling or to its right - even when the leaf has split meanwhile and has a new right sibling in between, which holds only keys below that bound. A leaf read again by its block number is checked to be a directory leaf, as every link the readers follow is (lion_dir_read_leaf()), and every step right releases the page it leaves first and checks for interrupts with nothing locked (lion_dir_step_right()).

Why the counts are the join’s. Each row’s count is computed from the set a descent would have located - the walk finds the same entry - the same way and under the same snapshot, and the set is located, counted and released before the next row’s is located. Only the ORDER of the rows changes, and nothing above the node depends on it: the node claims no pathkeys, the partial counts are grouped and added up by core’s Finalize Agg whatever order they come in, and the rows of a count(DISTINCT) go through core’s Sort. A duplicate’s answer is its first occurrence’s because the count of one key under one snapshot is one number. In a parallel plan each participant sorts and walks its own share: the rows its partial child gives it, or the runs of distinct keys it claims (“Forward semi joins over a non-unique key”), which are claimed as the keys are read into a batch, in the same sequence as before.

Pins and locks (§9, §15 “Paused and finished counts”). No lock is held between two keys, and none while a set is counted, which is when the heap is read. The walk holds a pin on its leaf from one lookup to the next only when the key’s set is INLINE - whose own §9 pin is on that very leaf - or the key has no entry, when no count follows at all; a CHAIN set’s count may read its posting tree for as long as it takes, and the leaf is not held through it. Before a row goes up (lion_pause_run()) the walk lets its leaf go and reads it again by its block number for the next key. It does the same before each batch is filled, since the child runs then, and the WHERE sets give up their pins at that point as well: the row-at-a-time loop keeps them across the child only until its first row that joins, a batch would keep them across a whole batch of child rows. Every count after that is carried by its fk set’s own pin, as every count after the first row always was (“Visibility and the §9 interlock”); the §9 argument is unchanged, since each set counted against the visibility map is still counted under its own pin.

Cost (lion_cost_fkjoin_walk()), in LION_DESCENT_COST a page visited, from what the walk does: a descent (height + 1 pages) per batch; per row, the visit of its leaf, one page, and its place in the batch - the row copied in, its share of the sort and the slot it is handed back in - LION_FKJOIN_BATCH_ROW_COST; and between two keys leaves / keys leaves on average, stepped over a page each while that is at most the height, or past it a descent’s internal levels: one, the level-1 page, for a key under the last descent’s level-1 page, and the height for the first key under each, taken as one key in fanout / apart, with the level-1 page’s fanout taken as the directory’s mean, its pages to the 1/height. The batches are work_mem over the child row’s width plus LION_FKJOIN_BATCH_ENT_BYTES. The model takes the walk where it is cheaper than a descent a row, (height + 1) pages. LION_FKJOIN_BATCH_ROW_COST is priced as one page visit - its setting defaults to LION_DESCENT_COST’s (§31, “The settings”) - on the high side: putting a key into the datum sort of “Forward semi joins over a non-unique key”, sorting it and taking it out again measured at a fraction of what a page visit is fitted at, and a row’s copy and its slot are of the same order. High on purpose: the walk replaces the internal levels of a descent with it, so it is taken only where a descent crosses two internal levels or more and a batch holds more keys than there are leaves between them - never over a directory of height 1, whose descent is the root and a leaf. So no plan over an fk index of a few thousand keys changes, and the plans the suite pins are all of that kind. The leaves themselves are priced as before, each read once: that is what the walk reads, and what the descents read too while the directory stays in the cache. (Refitted 2026-09-29, “The per-key terms, refitted”: the walk’s counters show no leaf read again a key - 0.02 to 0.3 directory pages a key where the keys are dense - so a walked row is now its lookup on the leaf the walk stands on, LION_FKJOIN_LOOKUP_COST, and its place in the batch, LION_FKJOIN_BATCH_ROW_COST at 75, which together are the 740 ns a walked key measured; and the walk is taken over any directory where the batch’s keys are closer than a descent, the suite’s small fk indexes included.)

Prefetch. While the keys are close together the walk asks for the right sibling of each leaf it lands on (PrefetchBuffer()), after letting go of its lock, where the tablespace’s io concurrency is above zero: a leaf the next keys will want, read while the current key’s set is counted.

The fact filters' copy, probed per key. Each count of an fk set against the collected copy of the fact filters (“The fact filters, collected once”) is driven by the source with the fewer rows - the fk set, a few rows, against a copy of many - and seeks the copy with a binary search at each of the set’s container keys (lion_cursor_seek()), over the keys memory keeps of it when it is spilled; no count passes over the copy. What each count did read that it did not need was the copy’s FIRST container: a cursor is built standing at it, and a spilled copy read it back from its temporary file at every count, a second read of the file for every key. The spilled copy now keeps that one container in memory as well (LionMatSet.first), so a count reads the file once, for the container it is sought to. So does a GROUP BY’s spilled copy of its WHERE items (§10, “The WHERE sets, collected once”), which lion_sources_collect() makes the same way and every group’s count reads the same way; a range taken as a source (§32) is collected by a union of its own and is left as it was.

EXPLAIN prints Join Key Lookups: in index order under the join’s other properties, and with ANALYZE Join Key Batches, the batches of every participant and run. Directory Pages Read is the walk’s pages: about the leaves the keys cover plus a descent a batch, and a leaf again after each row that goes up, where a descent per key reads height + 1 pages a key.

Where a key’s time goes (2026-09-28)

A benchmark found the node slower than the hash join the planner had rightly preferred to it, by more than anything the cost model charges a key accounts for, and “Lookups in key order”, which took most of the directory pages off each key, did not change that. What the node’s time per key was made of - the child plan producing the dimension rows, the lookups, the counts, their reads of the collected copy and of the visibility map, pages read from disk - EXPLAIN could not tell apart. It can now. Every run keeps:

  • counters, always, each one addition where the event happens, summed over the participants of a parallel plan as Join Keys Looked Up is (LionJoinShared):
    • Join Child Rows: the rows the child returned - the keys looked up, and the NULL keys, which join nothing; for a forward semi join over a non-unique key every participant’s whole child;
    • Join Key Containers Read: the containers the counts read from the keys' own fk sets (LionCountStats.key_containers, the reads of the cursor over source slot 0’s one set);
    • Join Posting Pages Read: the pages of posting trees the counts read (the process-wide lion_posting_pages_read over each count, as Directory Pages Read is over each call) - the chains of fk sets too large to be INLINE, and the fact filters' when they are probed rather than collected. Zero when every set is INLINE, the high-cardinality fk’s shape; one or more a key otherwise, each a random read wherever the index is not in memory;
    • Visibility Map Checks and Visibility Map Pages Pinned: the containers whose heap blocks the map was asked about, and the map pages pinned to answer them;
    • with the fact filters collected: Fact Filter Copy Containers Read, Fact Filter Copy Seeks (the copy looked up at a key container’s key; a binary search, before “The copy, looked up by key”) and Fact Filter Copy File Reads (the containers read back from a spilled copy’s temporary file);
  • timings, only under EXPLAIN ANALYZE’s TIMING option, as core times a node only then (the executor’s es_instrument has INSTRUMENT_TIMER, in every participant): Join Child Time, Join Lookup Time, Join Count Time, Fact Filter Locate Time (since 2026-09-29) and, when the filters are collected, Fact Filter Collect Time, in milliseconds summed over the participants and over the leaves of a partitioned fact. Untimed, a phase costs a flag test and no clock read; timed, a key costs three clock reads and a child row two, which is what core’s instrumentation of the child already spends on it.

How to read them, for any workload:

  • Join Child Time against the node’s own time is the child’s share. The node’s actual time is per loop - a participant’s, in a parallel plan - where these are sums, so a parallel node’s are compared after dividing by the participants. The child’s own line prints its time too; this is that figure where the node’s counters are, and summed.
  • Join Lookup Time and Directory Pages Read over Join Keys Looked Up are a lookup: about a page a key walked in key order, height + 1 descended.
  • Join Count Time over the keys found (Looked Up less Without Entry) is a count, and Join Key Containers Read over them what one reads of its own set. Against the copy a count is bounded by those containers, never by the copy: it reads the copy’s first container when its cursor is built and one more per binary search, and it searches once for each of the key’s containers at most (since “The copy, looked up by key” it builds no cursor over the copy and reads the one container a lookup finds) - so Fact Filter Copy Containers Read is at most Join Key Containers Read plus the keys, Fact Filter Copy Seeks at most Join Key Containers Read, and Fact Filter Copy File Reads, of a spilled copy, at most the seeks. A count that is O(the copy) would show here as copy containers far above those bounds.
  • Join Posting Pages Read, and with BUFFERS the node’s shared reads less its child’s, are what the node itself read from disk.
  • Fact Filter Locate Time is the fact filters located - once a run, and once a leaf’s turn over a partitioned fact - and it is mostly a range among them collected into memory (§32, “A range as a source”); Range Sources Collected says how many ranges were collected. It was in no phase before 2026-09-29, and a semi join whose 90 ms were 87 of collecting a range showed none of it.
  • What the node took besides these phases - its batches, and handing its rows up - is its time less them.

An illustration from the synthetic data of test/sql/fkjoin_perkey.sql - 5,000 dimension rows of one region, 1,250 of them without an fk entry, every fk set two rows in two containers, the fact filter an eight-value IN list collected once:

Custom Scan (LionCount)
  Lion Indexes: lion_pkf_fk (fk = d.pk), lion_pkf_x (x = ANY ('{0,1,2,3,4,5,6,7}'::integer[]))
  Fact Filters: collected once
  ...
  Join Child Rows: 5000
  Join Keys Looked Up: 5000
  Join Keys Without Entry: 1250
  Join Key Containers Read: 7500
  Join Posting Pages Read: 0
  ...
  Fact Filter Copy File Reads: 0

and, for a filter whose rows alternate - x IN (0, 1, 3, 5, 7), a copy of 27 bitsets, where the eight values' runs of rows above are run containers that fit in 64 kB - at a work_mem of 64 kB, where that copy spills, Fact Filter Copies Spilled: 1 with file reads no more than the keys' containers and temporary blocks read no more than those reads and one.

The per-key path, end to end (2026-09-28)

With the counters above, what is left is to know what a key costs by construction. This is every operation between a dimension row leaving the child and its count being done, in order, with what it costs as the §10 units put it (a directory page visited about 0.6 us, a container read and counted about 0.14 us, on a warm release build) where the calibration covers it, and as estimated from what the code does where it does not; the fixes of 2026-09-28 are marked.

  1. The child’s row: whatever the dimension’s plan costs, core’s plan and core’s cost; now Join Child Time and Rows. In a plan whose dimension set is selected by a scattered bitmap heap scan over a large dimension, that is a heap page a row wherever it is not cached.
  2. Its place in a batch (in key order): the row copied as a MinimalTuple, its key’s hash, its share of the batch’s sort (log2 of the batch comparisons through the opclass) - a fraction of a microsecond; the batch’s memory is reset once a batch.
  3. The lookup: a walk step or a descent (§21), a page visit each. Row at a time it also RESOLVED THE KEY’S PROBE for every key (lion_posting_set_lookup_col() calls lion_probe_init()): nothing for a key of the column’s own type, but for a cross-type key (an int4 fk against an int8 key) the family’s cross-type equality, hash and comparisons looked up in the syscache, their FmgrInfos set up and the type’s length looked up - some microseconds a key, for the same answer every time. Fixed: the node begins its walk for every plan, which resolves the probe once, and a row-at-a-time lookup is a descent with it (lion_lookup_walk_descend()).
  4. The set located: an INLINE payload copied, the stored key copied, in the per-key context, which is reset first - O(the set).
  5. The count’s set-up (lion_count_sources_run()): the sources' trees, the pin-budget and batch decisions, two plans, two cursors - some twenty small allocations and decisions - and a MEMORY CONTEXT made for the count and deleted after it. That context came from core’s freelist, but its first block is 8 kB, and a merge’s two work containers and a spilled copy’s staging buffer are 8 kB chunks each: every count malloc’d two more blocks and freed them again, per key. Fixed: the count works in a scratch context the node keeps (LionVisCache.scratch), whose 64 kB first block holds an ordinary count, reset when the count is over - a few pointer resets instead of a context and two blocks made and freed.
  6. The visibility map: every count pinned the map page its first container needed and released it at the end - a buffer lookup and two atomic operations on the header of the one or two map pages of the heap that every count, and every participant of a parallel plan, pins. Fixed: the page stays pinned in the node’s cache from one count to the next (LionVisCache.vmbuf), between the node’s rows as well, as an index-only scan keeps its map page (ioss_VMBuffer), and is let go when the node is done, rescanned or ended or moves to another partition (lion_vis_cache_release_vm()). A map page is no index page and no part of the §9 interlock: VACUUM sets a bit under the page’s content lock and never waits for its pins, and only a truncation of the map would, under an AccessExclusiveLock this query’s own lock excludes. Visibility Map Pages Pinned counts what is left: a pin per map page a run needs.
  7. Per fk container: the copy’s binary search, log2 of its containers, and for a spilled copy one read of that container from the temporary file (a system call reading a block of it) - the first container stays in memory (“Lookups in key order”); the AND of the two containers; the visibility map’s mask of it. O(the set’s containers x log(the copy)), and nothing else of the copy is read. Since “The copy, looked up by key” below: a lookup of the copy at the container’s key, one read where its keys are dense, and for a container of a row or two a lookup of each row in the copy’s - O(the set’s containers).
  8. The heap recheck of candidates on pages the map does not vouch for, through the node’s visibility cache: O(the key’s intersection).
  9. The set released: its pin, its charge to the list budget.
  10. The row handed up: the per-tuple context reset, a virtual tuple, the projection, the return through the executor and its instrumentation, lion_pause_run() - which in key order lets the walk’s leaf go, so that the next key reads it again by its block number - and above the node the Finalize Agg’s transition; in a parallel plan also the Gather’s tuple queue, and the leader reads every participant’s rows and aggregates them alone. About as much as a count’s set-up again, none of it the key’s own work, and in a parallel plan serialized in the leader. Fixed where it can be: when the target list is counts alone (count(*) and its semi, anti and distinct-key forms, no dimension column to group by or print) the node adds the partial counts itself and hands up ONE partial row per run and participant (LION_JOINFLAG_SUM, chosen by the planner from the partial target) - what a Partial Aggregate below a Gather hands up. The Finalize Agg adds the participants' rows as it added the dimension rows': count’s combine is a sum. An empty join hands up nothing, as before, which is what a plain Finalize Agg answers 0 for and a GROUP BY folded to constants has no group for. A summed join fetches each child row with no pin held (lion_pause_run() before it, which has something to let go of only once), since it returns to the executor only at the end. A GROUP BY of dimension columns still hands up a row a dimension row: grouping by them is core’s equality, and a partial aggregate of its own would be core’s hashing (§10’s rules; “Declined”).

Looked for and not found, for any shape: no count reads the copy, or positions a cursor on it, other than at the key’s own containers; no count reads a spilled copy back from its file other than a container at a time, and never its first container; no per-key budget is computed from the copy - a count’s budgets are work_mem’s and the pin budget’s, a few instructions; no per-key catalog access but the cross-type probe of 3; no per-key parallel coordination - a participant claims a run of 64 distinct keys at a time with one atomic operation, and only in the forward semi join over a non-unique key; nothing per batch but its sort; and of EXPLAIN’s instrumentation, core’s clock reads around every row the node returns, which a summed join returns once, and the node’s own phase clock, three reads a key and only under TIMING. With the fixes a key costs its lookup, O(its fk set x log(the copy)), and a set-up of some twenty allocations from memory already there.

None of 2 to 10 is O(the copy), and on a warm cache their sum is the few microseconds the calibration of “Cost, revisited” found. A key that costs far more than that is paying for the child’s rows (1), for pages read from disk - by the lookups, the fk sets' chains (Join Posting Pages Read) or the rechecks - or, in a parallel plan, for its row’s trip through the Gather (10); the counters and timings of “Where a key’s time goes” tell which.

The copy, looked up by key (2026-09-28)

A benchmark’s summed FK-side join spent most of its time in Join Count Time, and there in the containers of the keys' own sets: its keys' rows lay about one to a container all over the heap, so each count read as many containers as its key had rows, and none of its time went to the terms “The per-key path, end to end” lists as O(the copy). Its fact filters' copy was dense in container keys, with a container at nearly every key of the heap. So step 7 was the whole of it, and it was profiled (perf, cpu-clock) on synthetic data of that shape: 20 million fact rows whose uuid fk runs over 667,000 keys by a hash of the row number, a flag kept by 0.3% of them (a copy of 58,819 rows, with a container at every one of the heap’s 1,992 container keys), and a dimension selecting one key in ten through a partial covering btree - 66,700 keys, 1.98 million key containers of one row each - run serially with the join methods disabled, PostgreSQL 18.6, release build (bench/fkjoin_perkey.sh makes it). Of the node’s time:

  • 33% was the copy’s binary search, 45% of Join Count Time: eleven steps a key container, each following the pointer to a container to read its key - a cache miss and a mispredicted branch a step;
  • 17% was the AND of the key’s one-row container with the copy’s: the general path galloped through the copy container’s thirty rows, built the result in a work buffer, optimized it and copied it out, for a result that was empty 997 times in 1,000;
  • 20% was the rest of the merge: the key’s own containers read (a sparse segment expanded a key at a time), the cursor over the copy sought, advanced and rebuilt, the two sources' keys compared and their order walked;
  • 6% was the counts' set-up, 0.3 us a key; the visibility map under 1% (only the containers whose AND is not empty ask it); the lookups and the batch’s sort the remaining quarter, outside Join Count Time.

Three changes, each measured on its own:

  • The copy indexed by its keys (lion_mat_index(), made once with the copy by lion_sources_collect() and never changed after, as the copy is not). The keys in an array of their own, four bytes a container, which a binary search reads without touching a container and without a branch; and where they are dense - at most four key slots a container - a DIRECT index, dir[k - first] the first container at or above key k, which answers a seek with one read (lion_mat_seek()). Both are memory of the copy: the collection’s budget makes room for the keys, the direct index is made only where what is left of the budget holds it, and a spilled copy, whose entries keep its keys already, gets the direct index alone - sixteen bytes a container at most, an entry’s size.
  • The AND of a container of at most three rows (lion_container_and()): each row looked up in the other container - a bit of a bitset, a branch-free binary search of an array’s rows or of a run container’s runs - and the result written straight into the destination. Three rows or fewer is an ARRAY in lion_container_optimize()’s choice (three in one run take fourteen bytes either way, and a tie goes to the ARRAY), so the result is the one the general path makes, byte for byte; a damaged operand’s row that is not above the last one written is passed over (§3’s untrusted containers), and test/unit/container_test.c checks every kind of other operand against its reference, the small one on either side.
  • The merge of a set and a collected copy (lion_run_merge_copy()): a count of two positive sources, the key’s set driving and the other a copy lion_sources_collect() made (LionMatSet.collected), needs no cursor over the copy: each of the set’s containers looks its key up in the copy and is ANDed with the container found there, and where the copy has none the set is sought on to the next key it has - the leapfrog of the general merge. §9 reads as it does in lion_count_container(): a container of the result is put to the visibility map before the set, which carries the pin, moves past it, and a key the copy has nothing at, or whose AND is empty, asks the map nothing. The copy holds no pin and is counted beside the set only, which is lion_sources_collect()’s argument, unchanged. A copy that drives - fewer rows than the key’s set - takes the general merge. So does a GROUP BY’s count against its collected WHERE sets (§10) take this one, when the group’s set drives.

Fact Filter Copy Seeks is still one a key container looked up, and Fact Filter Copy Containers Read the copy’s containers found there: fewer than before, since a count no longer reads the copy’s first container to build a cursor, and within the bounds “Where a key’s time goes” gives.

Measured on the data above, medians of seven, Join Count Time / the query’s time in ms, serially with the join methods disabled, and with three workers (Join Count Time summed over the four participants):

build serial three workers
before 333 / 445 319 / 268
the AND by lookup alone 240 / 339 255 / 190
the direct index alone 217 / 312 214 / 161
the keys' array and the AND, no direct index 195 / 281 209 / 160
the direct index and the AND 136-144 / 222-236 144-164 / 130-165
and the merge of a set and a copy 97 / 203 105 / 101

A key container went from 157 ns to about 40: some 10 for the key’s own container, 25 to 30 for the copy’s lookup and the AND. With a flag kept by 2% of the rows (a copy of 391,274 rows, 196 a container), 401 to 123 ms serially, the query 518 to 235 ms, and 359 to 136 ms with three workers. Over §27’s two million rows, whose fk sets hold six rows a container, the node is 8 to 12% faster (... WHERE f.x = 3 GROUP BY d.attr 124 to 113 ms, f.x IN (1, 2) 147 to 131): its ANDs merge, and only the seek went.

The accumulator, considered and not taken. For a summed join the alternative was to OR the keys' sets into one accumulator and AND it with the copy once, instead of an AND per key container. A prototype (not kept) measured what it could be worth at best: with ONE AND of the whole accumulator at the end, Join Count Time 70 to 75 ms against the 136 to 144 of the per-key path with the first two changes above. It cannot be had at that price:

  • As stated, it breaks §9. The accumulator is a collected set; once the keys' pins are gone, nothing carries the interlock for the TIDs it holds, and the copy is never counted beside anything else - where a collected set is otherwise counted beside a located set that carries the interlock, or on its own with every row rechecked in the heap, which is the very cost the map is asked to spare.
  • Keeping the interlock by keeping the pins: every set in the accumulator would hold its pin until the AND - an INLINE set its directory leaf, a CHAIN set every leaf its cursor crossed, which the cursor lets go of now as it moves on. The pins are bounded by the list pin budget (at most a thousand, shared by a parallel plan’s participants), so the accumulator would be ANDed every so many keys, and each AND reads all of it - a container per container key of the heap the keys touched, which keys scattered over the heap make all of them. With an AND every 2,048 keys the prototype took 102 ms; every 256, 328 ms - worse than before any of this.
  • Keeping the interlock by asking the map first: each key container asked about under its own pin as it is ORed in, the AND with the copy after. 141 ms serially, no better, and 234 against 158 with three workers: the map is asked about every container of every key, where the per-key path asks it only about the few whose AND is not empty.
  • And exactness: a key that appears twice is counted twice by the join and once by a union; the keys' sets are disjoint only for distinct keys, so every OR would have to test for an overlap.

So the node keeps its count per key, and the third change took Join Count Time to 97 ms without any of it.

Cost. LION_FKJOIN_COPY_PROBE_COST, the copy sought and ANDed per fk container, was 15 cpu_operator_cost (75 ns) while the seek was the binary search: about 115 ns an fk container of one row on the release build. The same is 25 to 30 ns now, and pg_lion.fkjoin_copy_probe_cost defaults to 5. LION_FKJOIN_COPY_MEMBER_COST stays 3: an fk container of more than three rows is still merged with the copy’s, and §27’s rows of six a container still run slower than the hash join (113 against 67 ms, 131 against 90), which it keeps refusing - at 5 and 1 the planner took the node for both, at twice the hash join’s time. Measured with the data above and §27’s two million rows, serially: every choice the model made at 15 it makes at 5. The flag of 0.3% stays with the hash join (171,372 against 152,381), which the node now matches in time (198 to 213 ms against 206 to 219) where it was 2.3 times slower; the 2% flag stays with the node (171,417 against 453,424; 184 to 202 ms against 460 to 495).

The cost of a key, after the per-key path (2026-09-28)

The model prices a key as the sum of what “The per-key path, end to end” lists, each term a named constant (so that each can be calibrated on its own), in the units of §10 - a cpu_operator_cost 5 ns and a cpu_tuple_cost 20 ns at 500 units a millisecond:

step constant charged what it stands for
2 LION_FKJOIN_BATCH_ROW_COST (120 cpu_operator_cost, LION_DESCENT_COST’s) a row, in key order its place in a batch
3 LION_DESCENT_COST a directory page visited a level of a descent, a step of the walk, a leaf
5 LION_FKJOIN_COUNT_COST / LION_FKJOIN_COPY_COUNT_COST (25 cpu_tuple_cost each) a key found a count’s set-up and tear-down, against the filters or their copy
7 LION_CONTAINER_COST (8 cpu_operator_cost) and LION_MEMBER_COST (0.15, at most LION_MEMBER_CAP a container) an fk container, a member the key’s own set, read and counted
7 LION_FKJOIN_COPY_PROBE_COST (5 cpu_operator_cost; 15 before “The copy, looked up by key”) and LION_FKJOIN_COPY_MEMBER_COST (3) an fk container, a member the copy looked up and ANDed there
7 LION_FKJOIN_PROBE_COST (80 cpu_operator_cost), LION_FKJOIN_SET_COST (100 cpu_tuple_cost) a probe; a union’s set, a count the same against the filters, probed
8 LION_RECHECK_TID_COST (1.5 cpu_tuple_cost), lion_heap_page_cost() a candidate, a dirty page the heap recheck
10 LION_FKJOIN_ROW_COST (10 cpu_tuple_cost, new) a row handed up the node’s side of handing it up

and once a run LION_FKJOIN_COPY_CONTAINER_COST (20 cpu_operator_cost) a container of the copy made, the filters' lookups and merge, and for the forward semi join over a non-unique key the sort (LION_FKJOIN_SORT_COMPARE_COST, LION_FKJOIN_SORT_KEY_COST). Pages of the fk index are core’s page costs: the leaves each once at lion_heap_page_cost()’s interpolation, the chain pages at seq_page_cost.

(The FK-side join’s terms of this table were refitted on 2026-09-29, with posting pages, a walked key’s lookup and a union’s sets priced on their own: “The per-key terms, refitted” has the values.)

The per-key constant. A count’s set-up, after the fixes, is some twenty allocations from memory the node already holds, two cursors built and the plan’s decisions: about half a microsecond, which is what LION_FKJOIN_COUNT_COST and LION_FKJOIN_COPY_COUNT_COST already charged - 25 cpu_tuple_cost, the value “Cost, revisited” left them at because its fit, whose rows each read three or four directory pages, could not tell a per-key constant from those pages. Where §10 and §26 could tell, the same machinery measured 1.0 us a count or test (LION_ENTRY_COUNT_COST, LION_DISTINCT_TEST_COST), each of which also made and deleted a memory context, pinned and released a map page and, when it merged two sources, malloc’d and freed two blocks: what the fixes of 5 and 6 took off. So the constants stay, now derived rather than left over: the executor came down to them. What was missing is the row a key hands up (10): a cpu_tuple_cost stood for the projection, the return through the executor and the pause - about 0.2 us from the operations, ten cpu_tuple_cost (LION_FKJOIN_ROW_COST), charged per row handed up, which is the path’s own row estimate: every dimension row for an inner join, the join’s rows for a semi or an anti join, the distinct keys with rows for the forward semi join over a non-unique key. A summed join hands up one row a participant: its path’s rows are 1, and it pays the constant once, and core’s Gather and Finalize Agg above it are charged for that one row, which is what they then handle.

Why the planner’s choices stay right. Against the model before, a join that hands up its rows is dearer by nine cpu_tuple_cost a row - a few percent of what the model charges a key - while what it does per key fell by the context, the blocks and the map pin (5, 6), some half a microsecond, 25 cpu_tuple_cost. A summed join is cheaper by one cpu_tuple_cost a key, and in a parallel plan by the Gather’s parallel_tuple_cost a key - core’s charge for rows that no longer go through it - while what it does per key fell by those rows' whole trip (10) as well. The model moves less than the executor did, and in the same direction: a choice it made for the node stays, and one it made against the node changes only where the node gained more than the model credits it with. For a small dimension set what decides is the once-a-run work - the child, the filters' merge and copy - which is unchanged; for a large one the per-key terms, which are the ones above. No choice the regression suite pins moves: the joins it prices near the hash join are ungrouped counts, which only get cheaper, and the grouped ones it prefers the node for are chosen by margins far above nine cpu_tuple_cost a dimension row. These are derived, not measured - no server ran with them - and “Where a key’s time goes” gives a refit what it needs: per phase, per key, per container.

The fact-driven alternative - left to core. When the collected copy of the fact filters is small against the dimension’s keys, the ordinary plan’s work is proportional to the copy - each fact row the filters leave read and probed into a hash of the dimension, or a memoized lookup of it - and the node’s to the keys: a lookup, a count and a row a key, most of them finding no row of the copy. Two ways of making the node fact-driven were considered, and neither is taken:

  • Walk the fk directory in key order once, merge-style against the sorted keys, and AND each fk set with the copy only where the copy has containers. That is the node as it is: the walk in key order reads each leaf the keys cover once (“Lookups in key order”), and a count’s AND already stops at the first of the key’s containers where the copy has none (the leapfrog merge, §22 and §25), so the work left per key is its lookup and a count’s set-up, which no merge-style walk removes; walking every entry of the directory instead of the keys' own leaves reads O(the fk’s keys), more than the keys' when the dimension set is the smaller side.
  • Drive the count from the copy. A posting set maps a key to its rows, never a row to its key, so the fk of a fact row the copy lists is in the heap alone: this is a bitmap heap scan of the copy’s rows - which core builds from the same lion indexes (§29) - feeding a hash semi join or a memoized index lookup into the dimension, which is exactly the ordinary plan. The node would reimplement core’s hash join over a lion bitmap scan, with core’s hashing and equality (§10’s finding 3), to gain nothing core’s plan does not have. (That memoized lookup is where lion helps since 2026-10-23: §40.10 makes it a nested loop’s probe of the dimension’s covering B-tree, index-only, over the set of the dimension’s own filters, the heap left out of the loop.)

So which side drives is the cost model’s choice between two plans, and what it needs is each priced for what it does: the node O(keys), per the table above, and core’s plan O(the copy’s rows), by core. That is what keeps the node from being chosen where the copy is small and the keys many, and chosen where the keys are few, whatever the copy.

Every aggregate over the node’s rows (2026-09-28)

A benchmark asked, in one statement, how many dimension rows join and how many join pairs there are - a CTE of filtered dimension rows, an inner join to the fact, and

SELECT count(DISTINCT d.pk), count(*) FROM fact f JOIN dim d ON d.pk = f.fk
WHERE <f filters> AND <d filters>

Each aggregate alone is pushed down; together they had no path at all. The counts are partial counts per dimension row that a Finalize Agg adds up, count(DISTINCT) needs the dimension rows and a plain Agg over them (“count(DISTINCT)”), and one core Agg is one or the other: the split is the Agg’s, and nodeAgg asserts that every Aggref under it has it. With the join methods disabled the plan was a disabled nested loop; the benchmark’s own plan was a nested loop blind to how many fact rows each key has, which timed out where the node reads each key’s posting set once.

So when any aggregate of the query needs the rows, every aggregate is answered over them:

  • The rows carry their counts. An inner join’s rows (and the forward semi join’s distinct keys) stand for dimension rows, not for pairs. The node emits them as for count(DISTINCT), and beside each its count of join pairs, n - the count the partial path computes for that row, where the rows alone take the existence test (LION_JOINFLAG_COUNTS). The column is a partial count(*) in the node’s target (a LION_TL_COUNT column, which the executor already fills), and EXPLAIN prints it as (PARTIAL count(*)), which is what it is.

  • The Agg’s aggregates are rewritten over n (lion_fkjoin_agg_kind(), lion_fkjoin_rewrite_mutator()), each in its place in the target list and in the HAVING:

    the query’s over counted rows why it is the join’s
    count(*), count(1), count of either side of the key lion_join_count(n) a group’s pairs are the sum of its rows' n; neither side of the key is NULL in a pair
    count(x), x any expression of dimension columns lion_join_count(n) FILTER (WHERE x IS NOT NULL) each of a row’s n pairs holds that row’s x
    sum(x), x such an expression of type int2 or int4 lion_join_sum(x * n) the sum over the pairs is the sum over the rows of x times n
    count(DISTINCT x) as it stands “count(DISTINCT)”
    min, max, bool_and, bool_or, every, bit_and, bit_or of such an x as they stand each is blind to multiplicity - min(v, v) is v, v AND v is v - and skips NULLs, so the rows give it the pairs' values, each once

    min and max are core’s aggregates with a sort operator, of every type, which planagg.c already answers as the first value in that operator’s order; every is bool_and by its SQL name. any_value would do as well and is left out (its answer is any row’s, which no test can pin). bit_xor, avg, string_agg, array_agg, the statistical aggregates and sum of anything else see multiplicity, and decline.

  • sum of an int8 or numeric column is left out. x times n of an int8 needs numeric arithmetic and a numeric sum, a row, where core sums the pairs in 128-bit integers: exact, but not cheap. An int2 or int4 x is exact in int8: the product and the sum fail with “bigint out of range” rather than wrap, and the one difference from core’s sum(int4) is which intermediate sum may overflow on the way to a total that does not - which the order of the pairs decides for core too (a parallel partial sum’s order is not a serial one’s).
  • Declined as before: a fact column other than the key, min or max of the fact’s key included (the dimension’s key answers the same and is taken; which of two keys the join calls equal to print is §10’s representation question), FILTER, ORDER BY or DISTINCT (but count’s) in an aggregate, and a volatile expression.
  • Semi and anti joins keep their rows. A semi or anti join’s rows are its result rows, each once, so every aggregate is computed over them as it stands and no count is carried: n would be 1, over which lion_join_count(n) is count(*). The one aggregate whose meaning differs is the count of the key in an anti join, whose NULL keys are rows: there it is no count of rows but a count of a column, which as it stands it is.
  • GROUP BY and HAVING are the Agg’s, as for count(DISTINCT): sorted by the group pathkeys when a DISTINCT needs them, and - when no aggregate is a DISTINCT one and the GROUP BY hashes - also hashed over the rows as they come, which needs no Sort; the cost model chooses. Grouped by the join key, each row is a group and the Agg neither sorts nor hashes (“Grouped by the join key”).

The aggregates that add n up. Two are wanted: one with count’s answer, an int8 that is 0 over no rows, and one with sum’s, NULL over no rows. They are the extension’s own, in the install script: lion_join_count(int8) is int8pl from an initial 0, and lion_join_sum(int8) is int8pl from NULL (the strict transition takes the first value as its state); both combine with int8pl and are PARALLEL SAFE. Against core’s sum(int8) with a cast back from numeric (and a coalesce(..., 0) for count’s 0):

  • Exactness: both are exact.
  • Overflow: n is never negative, so no partial sum of lion_join_count passes the total, and it fails exactly where count() itself would - count’s int8inc fails past int8 too. sum(int8) never overflows (its state is a 128-bit integer or a numeric) and the cast back fails at the same total: the same answer, later.
  • Parallel combining: both have a combine function. lion_join_count’s state is an int8 passed by value; sum(int8)’s is internal, with serial and deserial functions.
  • Cost: an int8pl a row, against int8_avg_accum’s 128-bit add into palloc’d state, a numeric made by the final function, a cast and, for count, a coalesce - more work for the same answer, and a less plain EXPLAIN.

So the extension’s own. They are found at plan time in the extension’s schema (lion_fkjoin_rewrite_init()) and must be objects of the extension - no namesake in another schema is taken - that the current user may EXECUTE, which the Agg asks at run time as for any aggregate (the node still asks what the plan it replaces would have asked, count() included: §9, “Privileges”). They are read from the catalog cache, not looked up by name as a query’s functions are: a lookup by a qualified name asks USAGE on the schema, and failed the planning of a user who had none, where running the aggregate asks no such thing. A database whose pg_lion was created by a script without them keeps the ordinary plan for these queries; the rows alone and the partial counts need neither.

The LionJoinAgg. The rewritten Agg cannot go into the grouped rel as it is. What core puts above a grouped path finds the query’s aggregates in its input’s target list by equal(): a Sort for ORDER BY count(*) looks for its key there (prepare_sort_from_pathkeys()) and, not finding it, would ADD a count(*) of its own to the projection-capable Agg - one that over the rows counts dimension rows, silently - and a projection above matches its expressions there (fix_upper_expr()) and fails. Nor can the grouped rel’s target be rewritten in place: core’s own paths share it. So the Agg path gets a target of its own, the grouped rel’s with each aggregate rewritten where it stands, and goes into the grouped rel below a LionJoinAgg (lion_add_join_agg_path()): a CustomScan whose target list and custom_scan_tlist are the grouped rel’s own, whose one child is the Agg, and which hands the Agg’s rows up unchanged. Column i of the Agg is column i of the query’s target, of the same type (checked when the plan is made), so the node has no projection to make - but for a target that names one aggregate twice, which setrefs.c points at the first - and returns the Agg’s own slot. Everything above it sees the query’s aggregates, everything below it the rewritten ones. It costs nothing, and it is registered with the scan methods (lion_count_scan_register()), since under debug_parallel_query a worker runs it.

Parallel. The rewritten aggregates are plain: a partial Agg would need partial states of count(DISTINCT), which it has none of. So the plan is the plain Agg above a Gather of the counted rows, as for count(DISTINCT): every row that joins goes through the Gather’s tuple queue, which core prices at parallel_tuple_cost a row, and the leader aggregates them all. A query of counts alone keeps the partial path, summed in the node (“The per-key path, end to end”), which hands up one row a participant and is cheaper for it.

Cost. The counted rows are priced as the rows of count(DISTINCT) but counted - the count’s terms of “Cost, revisited” where the rows alone take the existence test’s, since each row’s whole set is counted - with LION_FKJOIN_ROW_COST a row handed up; core prices the Sort, the Agg and the Gather, and the LionJoinAgg adds nothing. A query of counts alone is never offered this path, so its plan and its price are what they were. In fkjoin_mixed.sql the cost model takes the counted rows where, and only where, it takes the partial counts of the same query without its count(DISTINCT) or max: the Sort and the Agg over a few hundred rows add a few units to either.

EXPLAIN (VERBOSE):

Custom Scan (LionJoinAgg)
  Output: (count(DISTINCT d.pk)), (count(*))
  ->  Aggregate
        Output: count(DISTINCT d.pk), lion_join_count((PARTIAL count(*)))
        ->  Sort
              Output: d.pk, (PARTIAL count(*))
              Sort Key: d.pk
              ->  Custom Scan (LionCount)
                    Output: d.pk, (PARTIAL count(*))
                    Lion Indexes: fact_fk (fk = d.pk), fact_hot (hot = true)
                    Join Rows: the dimension rows with a match, each with its count
                    Fact Filters: collected once
                    ->  Seq Scan on public.dim d
                          Output: d.pk
                          Filter: d.kept

(the distinct keys with a match, each with its count for the forward semi join); count(d.name) prints as lion_join_count((PARTIAL count(*))) FILTER (WHERE (d.name IS NOT NULL)) and sum(d.small) as lion_join_sum((d.small * (PARTIAL count(*)))).

Grouped by the join key (2026-09-28)

A benchmark counts the join pairs per key, and then aggregates the per-key counts:

WITH a AS (SELECT d.pk FROM dim d WHERE <d filters>),
n AS (SELECT f.fk, count(*) AS n FROM fact f JOIN a ON a.pk = f.fk
      WHERE <f filters> GROUP BY f.fk)
SELECT count(*) AS keys, sum(LEAST(n, 3)) AS capped FROM n

Both CTEs are inlined. a is pulled up into the join, and n, which has a GROUP BY, stays a subquery that is planned on its own; its UPPERREL_GROUP_AGG is where the node is offered, as for any grouped join of two tables. The outer level is an ordinary aggregate over the subquery’s rows. GROUP BY f.fk with count(*) is exactly the node’s count per dimension row: one group per dimension row that joins. Before this change, the query with GROUP BY a.pk (the dimension’s key) was pushed down: partial counts, with a Finalize HashAggregate over them that hashed as many groups as it had rows. As written, with GROUP BY f.fk, it was not pushed down, because a fact column in the output declined (“Declined in v1”). GROUP BY f.fk, d.pk declined too, in either order: core keeps one of the two as the group key, since the join’s equivalence class makes the other redundant, but the target still names the fact’s column.

The fact’s key stands for the dimension’s. The rule is lion_fkjoin_agg_is_distinct()’s, with the grouping column in place of the DISTINCT argument (lion_fkjoin_fk_groups_ok()):

  • an inner join, or the forward semi join over its distinct keys;
  • one type on both sides;
  • a grouping equality compatible with the join operator;
  • the join’s collation.

Under those conditions the fact values a dimension row joins make one group, and the node emits the row’s key where the fact’s column stands (lion_plan_fkjoin_path(), as it does for the DISTINCT). A GROUP BY prints its column, and a DISTINCT does not, so one more condition applies (lion_fkjoin_fk_is_key()). The key has to BE each fact value it stands for, not only equal to it: the join operator has to be the type’s own equality, and that equality has to imply an identical representation under the join’s collation. That is §10’s value-representation rule, lion_type_equalimage(). numeric’s 1.0 and 1.00 are one group and print differently. If the node emitted the dimension’s 1.00 for a group whose fact rows all hold 1.0, it would print a value that no fact row has. So a numeric key declines (tested), and so do citext, text under a nondeterministic collation, bpchar and the floats. An expression of the column (GROUP BY f.fk + 1) declines. An expression that the target computes from the grouped column (SELECT f.fk + 1 ... GROUP BY f.fk) is the Agg’s. With the fact’s key grouped, the node emits partial counts, or rows with any of the aggregates “Every aggregate over the node’s rows” answers. min and max of the key still decline; the dimension’s key answers those.

Each row a group. When the grouping key is one the node’s rows never repeat, each row is a group of its own (lion_fkjoin_groups_per_row()). There are three such keys:

  • the dimension’s key of an inner join, which lion_column_is_unique() proved unique under the join’s equality and collation when the join was recognised;
  • the dimension’s key of a reverse semi join, where the same proof holds;
  • the fact’s key standing for the dimension’s. An inner join emits each dimension row’s key once. The forward semi join emits each of its DISTINCT keys once, having made them distinct itself.

In each case the key has to be grouped by an equality compatible with the join operator, under the join’s collation. Other grouping columns beside the key change nothing. No inner or semi join emits a NULL key. An anti join’s rows can hold several NULL keys, which form one group, so the anti join is not asked. When each row is a group, the Agg above the node is AGG_SORTED over the node as it is, with no Sort under it. This applies both to the Finalize Agg over partial counts and to the plain Agg over rows. A sorted Agg compares each row with the one before it, and when no two rows belong to one group, every row starts a group of its own, whatever their order. The Agg claims no order: its pathkeys are the node’s, which are none. A DISTINCT aggregate sees one value a group, which is as sorted as a presorted one needs. In a parallel plan each row comes from one participant, so the rows stay distinct across the Gather as well.

This is the first answer of the FK-side join that depends on the uniqueness proof for correctness. “Uniqueness of the dimension key” otherwise treats the proof as a costing guard. If two visible dimension rows shared a key, the per-row Agg would make two groups where there should be one. Core’s own plans rest on the same premise: a unique inner join stops at the first match. The proof is also of the same kind: a unique btree index that is valid, immediately enforced and not partial, under the join’s equality and collation, and whose DROP invalidates the plan. The forward semi join needs no index, because the node makes its keys distinct itself.

Cost. Core’s cost_agg() prices AGG_SORTED over its input as the input comes: a comparison per row and grouping column, the transitions, and cpu_tuple_cost a group, starting when the node starts. It charges nothing else: no hash table built before the first group, no spill when the table outgrows hash_mem, and no Sort. Before this change, the Finalize HashAggregate over the partial counts paid all of that for as many groups as rows, and the plain Agg over the rows paid for a hash or a Sort of every row, which grouped nothing. A grouping that is not per row is still hashed or sorted, and core prices it so, with nothing taken off. Such groupings include d.attr, d.pk + 0, a unique dimension column other than the key, the key under another collation, and an anti join’s key. fkjoin_bykey.sql pins both behaviours: the per-row Agg costs the same at 64 kB and at 64 MB of work_mem, while a hashed Agg over the same groups costs more at 64 kB.

Estimate. The per-row Agg’s groups are the dimension rows that join (lion_fkjoin_per_row_groups()). Of the n dimension rows the node reads (or the forward semi join’s distinct keys), those are the rows that at least one of the join’s p pairs falls on: n(1 - e-p/n) when the pairs fall at random, which is no more than n or p. That number is also capped by core’s own estimate of the groups over the join’s pairs, estimate_num_groups() of the grouping columns over the join rel’s rows. For the fact’s key, that estimate counts the fact’s distinct fk values. By itself, core’s estimate for GROUP BY f.fk over the join exceeds the dimension rows when the dimension is filtered. A semi join’s rows are already its result rows. The outer level of the CTE form sees this number as the subquery’s rows. In the test, 2,000 are estimated for 1,882 (the kept dimension rows, less those with no fact row). Under three fact filters, 574 are estimated for 517, where the node reads 3,000 dimension rows and core’s own estimate is 637. In the measurements below, 6,438 are estimated for 6,293 under x = 3, where the node reads 10,120 by its estimate (10,000 in fact) and core’s plan estimates 10,231.

Measured (PostgreSQL 18.6 packaged build, a VM shared with other work, warm, best of five; repeated runs agree within a third): two million fact rows with fk a hash over 200,000 keys (10 rows a key), x with 10 values and hot true on three rows in four, all under lion indexes; a 200,000-row dimension. The CTE form above, with WHERE d.attr < 5 leaving 10,000 dimension rows:

fact filter node ordinary (hash join, HashAggregate)
x = 3 28.2 ms (21,957) 77.0 (31,297)
hot 22.4 (22,406) 189.0 (53,044)

Over 100,000 dimension rows (attr < 50), the model keeps the hash join. That is right for x = 3 (140 ms against 157 for the node when forced). It is wrong for hot, where the hash join takes 572 ms against the node’s 180: a node cost of 147,307 against the hash join’s 60,964. The per-key terms of “The cost of a key” are dear at that many keys, which predates the per-row Agg and is not the grouping’s doing. Grouped by d.pk over all 200,000 dimension rows (hot, joins disabled), the per-row Agg takes 274 ms, against 344 for the same groups hashed (GROUP BY d.pk + 0). With work_mem at 4 MB, where the hash spills, the times are 279 and 381.

EXPLAIN:

Aggregate
  ->  Finalize GroupAggregate
        Group Key: fk
        ->  Custom Scan (LionCount)
              Lion Indexes: fact_fk (fk = d.pk), fact_hot (hot = true)
              Fact Filters: collected once
              ->  Seq Scan on dim d
                    Filter: kept

A GroupAggregate directly over the node, with no Sort between them, is the per-row Agg. The group key prints unqualified, as count(DISTINCT)’s sort key does.

A partitioned fact table (2026-09-28)

The fact side used to decline a partitioned table (“Declined in v1” said why: §16 opens one partition at a time). It is now counted the way §16 counts one: every dimension key is looked up in every live leaf partition’s fk index, and each leaf is a table of its own in everything the sections above ask of the fact side.

  • What is computed. A fact row lives in exactly one leaf, so the fact rows of a key under the fact filters are the disjoint union of each leaf’s own. Per dimension row, an inner join’s count is therefore the sum of the leaves' counts; a semi join asks whether any leaf has a match, and an anti join whether none has; a NULL key matches in none. The rows of count(DISTINCT), the counted rows under a LionJoinAgg (lion_join_count()) and the distinct keys of a forward semi join over a non-unique key are the same sums and existence tests, so everything above the node is unchanged.
  • Which leaves. The planner’s live leaves, after plan-time pruning, in its order - the targets lion_collect_targets() gives a count (§16) - each with its own fact filters less the ones its bounds imply (“Clauses the partition bounds imply”, §16): every leaf needs a lion index on fk, and one for each filter it keeps, or the query declines. None left: the planner knows the join is empty, and the node is not offered. The node has no Append and no PartitionPruneInfo, so a leaf that only RUN-TIME pruning would remove - one whose bound excludes ts >= now() - interval '...', or a generic plan’s kind = $1 - is counted too. It is exact: such a clause proves nothing at plan time (§16), so the leaf keeps it as a fact filter, and it selects none of the leaf’s rows. Mostly it is cheap: a leaf whose located filters select nothing at all (wheremissing) has no turn, and no key is looked up in it. But its indexes are opened and its filters located, and where they do select rows - a filter whose value run-time pruning would have used, beside one that does not narrow the leaf to nothing - every key is looked up and counted there for nothing. test/sql/fkjoin_partition.sql has such a leaf.
  • A batch at a time, a leaf at a time. The keys are read a batch at a time whatever the plan chose (“Lookups in key order”: the batches are work_mem’s), each key with an accumulator of its own (LionJoinEnt.acc), and each batch goes to every leaf in turn (lion_join_count_parts()): the leaf’s turn begins, its fk index’s walk begun, its fact filters located or its copy of them taken up again (lion_join_part_open()), every key of the batch looked up and counted and the count added into the key’s accumulator, and the turn ends - every set released, and its pins with them - before the next one’s begins (lion_join_part_close()). The leaves' relations stay open from the node’s Begin to its End (§16, “Why every leaf is open for the whole run”). An existence test does not look a key up again once one leaf has matched it. Where the plan walks the fk index in key order, the batch is sorted into the first leaf’s walk order and left alone for every leaf whose fk index orders keys the same way (LionWalkOrder, lion_walk_order_equal()) - one partitioned index’s leaves do - and sorted again for one that does not; where it descends for each key, the batch stays in the child’s order. Only when every leaf has had the batch are its rows handed up (lion_join_next_parts()), so the node never hands up a row in the middle of a leaf’s turn, and no leaf’s pin outlives its turn (§16).
  • Each leaf is its own table. Its own heap, visibility map and pins; the §9 interlock between its own fk set and its own filters; the pin budget of §15 per turn. Where the plan collects the filters, a leaf’s are located and copied at its first turn into a copy of its own (LionJoinPart), which its later turns count beside the fk sets they locate; the copy was made after the snapshot was taken and is only ever counted beside that leaf’s own located fk sets, which is all “Why a stale copy is safe” asks of it. The copy keeps, per multi-key clause, how the clause came out and the query it was located with - a tsquery’s prefixes expanded into THAT leaf’s index’s lexemes (§17), held for the run in the values' memory, not the turn’s - because a later batch builds the leaf’s row filter from them without locating the clause again (test/sql/fkjoin_partition.sql, “a prefix phrase, expanded per partition”). The copies are all held for the run, within one hash table’s memory between them - each leaf’s collection has what the ones before it left, and spills past it (lion_sources_collect()’s spill) - and are freed by a rescan. A plan that probes the filters instead locates a leaf’s again at each turn, since the leaf keeps nothing from one batch to the next.
  • Parallel. The shared copy is kept, one per leaf. The DSM holds a LionSharedCopy for each leaf, one after the other, each with its own barrier, chunks, DSA memory and SharedFileSet, and the memory a Parallel Hash would have (a hash table’s for each participant planned) divided among them by their heaps' blocks (lion_join_part_blocks()). Each participant takes its share of the dimension rows, a batch at a time, to every leaf in turn, and joins a leaf’s collection the first time it gets there; the participants that reach it collect it together, chunk by chunk, as “One copy per query” does for one table, and every one of them then reads it. A participant that never reads a dimension row never attaches. A rescan of the Gather frees and re-initializes every leaf’s copy. Private copies per participant, the fallback, were not needed: a shared copy is self-contained, so one per leaf is the one-table code, repeated.
  • Cost (lion_cost_fkjoin_path()). Each leaf is priced as a table of its own (lion_cost_fkjoin_rel() with the leaf’s Var, statistics and kept filters), with the keys that find rows there in proportion to the fk’s distinct values the leaf holds against the parent’s, and the leaves' costs added up; the recheck of each leaf’s dirty heap too. The plan’s two choices, walk or descend and collect or probe, are made once over the sums: a batch’s place is charged once a row, not once a leaf; collecting needs every leaf’s copy to fit in one hash table’s memory per participant together; and a probing plan’s locating of the filters is charged once a batch per leaf. Row estimates are the planner’s, as for a count (§16). Since 2026-09-29 each leaf is priced for the keys it looks up rather than for all of them: none in a leaf whose bound refutes its fact filters at their plan-time values, or whose filters are expected to select less than a row, and for an existence test only the share no earlier leaf has matched (§32, “What collecting a range costs”); and a range among the filters is estimated over the leaf’s own columns.
  • EXPLAIN has the Partitions line, Lion Indexes without index names and Implied by Partition Bounds, as a count’s (§16). The counters are summed over the leaves (and the participants): Join Keys Looked Up counts a lookup per leaf that had a turn - an inner join’s keys times those leaves, fewer for an existence test - Join Keys Without Entry a key per leaf it has no entry in, Fact Filter Rows Collected the rows of every leaf’s copy and Fact Filter Copies Shared one per leaf and run; and the timers, Fact Filter Locate Time every leaf’s locating of its filters at each of its turns.

Measured (2026-09-28, PostgreSQL 18, warm cache, four CPUs shared with other work; medians of five runs, ms). A synthetic star: a 500,000-row dimension with btrees on its filter columns that INCLUDE the key, and a 10,000,000-row fact LIST-partitioned by kind into five leaves, one kind sub-partitioned by range on ts, whose fk is skewed (a small set of hot keys holds many of the rows) and scattered over the heap, with a partitioned lion index over the filter columns and fk and a btree on fk; about 2 GB. The nested loop is into the leaves' fk btrees, reading the heap for the fact filters. Serial:

query (fact filters) dimension rows node nested loop hash join chosen
inner, kind = 'a', tags 250 hot keys 3.4 32 217 node
semi, kind IN ('a', 'b'), tags 420 3.4 4.9 359 node
inner, kind = 'a', tags 5,000 22 485 247 node
grouped, kind IN ('a', 'b'), tags 5,000 39 846 383 node
semi, kind = 'a', tags, ts range 5,000 62 236 87 node
grouped, kind = 'a', ts range 5,000 66 73 726 node
inner, kind = 'a', tags, ts range 25,000 105 283 103 hash join
semi, kind = 'a', ts range 420 41 2.7 1,045 node
inner, kind = 'a', tags, ts range 420 47 4.5 63 node

With two workers the node wins by as much where it won serially (35 ms against 113 and 139 for the 5,000-row inner join; 42 against 203 and 205 grouped) and is chosen over the parallel hash join of the 25,000 rows at 95 ms against 125. Its misses are all one: a range on ts whose value is now() less an interval, over a few hundred dimension rows, where a nested loop reads a few thousand heap rows from a warm cache. The node then collects the range in each leaf it keeps (§32’s “range as a source”), some two million ts entries from their summaries - 49 of the 54 ms of the 420-row semi join, none of it in the node’s timers, since it happens while the filters are located - and lion_cost_range_source() priced that as the walk of a summed range, a fraction of it. That is a plain table’s cost too (the same query on the one leaf: 26 ms against 5). With two workers the same holds of the 5,000-row joins with the range: the node, chosen at 60 to 70 ms, against a parallel nested loop’s 36 (semi) and 41 (grouped). (2026-09-29: the collection is priced now, at what its union does, and timed, in Fact Filter Locate Time; each leaf is priced for the keys it looks up. On a reproduction of this star the 420-row semi join is priced at 49,968 where it was 9,403 and goes to the nested loop, 2.4 ms against the node’s 97 - §32, “What collecting a range costs”. The nested loop’s warm cache, which its cost does not assume, is core’s.)

A dimension that is a join (2026-09-28)

Star queries over three tables timed out where the same queries over two tables ran in well under a second through the node. Two shapes, over a dimension d and two facts:

-- (A) a dimension semi-joined to two facts
SELECT count(*) FROM dim d
WHERE <d filters>
  AND EXISTS (SELECT 1 FROM f1 WHERE f1.fk = d.pk AND <f1 filters>)
  AND EXISTS (SELECT 1 FROM f2 WHERE f2.fk = d.pk AND <f2 filters>)

-- (B) a fact counted against a dimension narrowed by a semi join to another fact
SELECT count(*) FROM f2
WHERE <f2 filters>
  AND f2.fk IN (SELECT d.pk FROM dim d WHERE <d filters>
                AND EXISTS (SELECT 1 FROM f1 WHERE f1.fk = d.pk AND <f1 filters>))

The node was offered for a join of exactly two tables, so both went to the ordinary plan: a nested loop that walked a btree on a fact’s fk for each dimension row and read the heap to test its filter, a tsvector detoasted a row at a time. Core priced that loop at the fact’s average rows a key, where the dimension filter kept the keys that hold most of the fact’s rows.

The dimension may now be a join: ONE table d that carries the key, and other tables joined to it only as the inner side of semi and anti joins (EXISTS, IN, NOT EXISTS). (A) is then f2 semi-joined by the node to the dimension d ⋉ f1, or f1 to d ⋉ f2; (B) is f2’s count against the dimension d ⋉ f1. The node’s child is core’s plan of that join, and everything the node does with the child’s rows is what it does with a single table’s.

Why it is the query’s answer. A semi or anti join returns rows of its outer side, each at most once, and the tables of its inner side add nothing to them - neither rows nor columns the query above may name. So when every table of the dimension but d is inside the inner side of one of its semi or anti joins, the join rel of the dimension’s tables returns rows of d, each at most once: the dimension rows that pass d’s own filters and every one of those joins. The query is that rel joined to the fact by one join on f.fk = d.pk, and the node computes exactly what it computes for a dimension of one table, per row of the child: the count of the row’s key, or the existence test. What makes the order legal is core’s: each of the dimension’s semi and anti joins needs, on its outer side, only relations of the dimension (its min_lefthand, below), so it may be done before the join with the fact, and the join rel of the dimension is one core builds for that order. The node runs core’s cheapest path of it, under the query’s snapshot, as it runs a table’s.

Recognition (lion_fkjoin_recognize(), lion_fkjoin_try_fact()), from the SpecialJoinInfos, not from the shape of the join tree:

  • every special join in the join tree is a semi or an anti join (an outer join declines; an anti join made of LEFT JOIN ... IS NULL is taken inside the dimension and declines when the fact is its inner side: “A dimension that is any join”); every table of the join is a base relation with no LATERAL references and no TABLESAMPLE; and, as before, no PlaceHolderVar, no pseudoconstant qual and no LATERAL range table entry anywhere;
  • each table of the join is tried as the fact F - a plain or partitioned table with no RLS or TABLESAMPLE, as before - and the others are the dimension D. The join between them is the special join whose minimal and syntactic inner sides are F alone (the reverse semi join or the anti join, D its outer side); or a semi join whose minimal and syntactic outer sides are F alone and whose syntactic inner side is all of D (the forward semi join); or, when neither exists, an inner join;
  • every other special join is the dimension’s own and must not involve F: F in its minimal outer side (EXISTS (SELECT 1 FROM f1 WHERE f1.fk = d.pk AND f1.x = f2.y), a condition on the fact’s rows no posting set answers) or in its inner side declines. F may be in its syntactic outer side, which only says where the EXISTS was written;
  • the dimension’s own table d is a table of D in no inner side of those special joins; there may be several, inner-joined, and d is the one the key is taken from (“A dimension that is any join”). A table that only filters the fact is folded before D is formed (“A second dimension that only filters”). d is a plain, partitioned or materialized-view relation;
  • exactly one join clause between F and all of D - the equivalence classes' clauses between them and F’s joininfo, as before - and it is taken between F and d. A dimension that is a join may hold other members of the key’s class: with f1.fk = d.pk and f2.fk = d.pk, one class holds all three, and its clause between D and f2 may be f1.fk = f2.fk, the same rows by the class’s transitivity. The class is asked again for the pair (d, F), which names d.pk; that clause has to be of the same class, or be the same joininfo clause. The special join with the fact must have d in its minimal outer side (the inner one, for the forward semi join): an EXISTS on the fact nested inside another table’s EXISTS and correlated to it - EXISTS (SELECT 1 FROM f1 WHERE f1.fk = d.pk AND EXISTS (SELECT 1 FROM f2 WHERE f2.fk = f1.fk)) - gets the pair clause f2.fk = d.pk from the class, but its join is f1’s, and declines;
  • the child is find_join_rel() of D’s relations: the join search has made it when that order is legal, and nothing here assumes it - a join rel that is not there, or is proved empty, declines. Its target has to carry d.pk, which a join to the fact above it always needs, and every column of d the node emits (lion_fkjoin_child_has(): checked, not assumed); its cheapest total path must not be parameterized.

What is not required: the inner sides may hold anything core plans - a partitioned table (f1 in the benchmark), an inner join (EXISTS (SELECT 1 FROM f1 JOIN x ON ... WHERE f1.fk = d.pk)), a nested EXISTS - since only core reads them. Each table that qualifies as the fact is offered as one: in (A) f2 against d ⋉ f1 and f1 against d ⋉ f2, and the cost model chooses. Three facts each with an EXISTS are three offers.

Uniqueness. It is asked where it was asked, and of d: lion_column_is_unique() over d’s index list. A key unique in d is unique in the dimension’s rows, since they are rows of d each at most once - which the recognition above checks rather than assumes.

  • An inner join needs none for its counts (“A dimension that is any join”); its GROUP BY of the key asks it.
  • A reverse semi join and an anti join need none: they are per dimension row, and duplicated and NULL keys among the child’s rows are rows as before (tested over a non-unique dimension). GROUP BY the key of a reverse semi join is each row a group of its own only with the proof (“Grouped by the join key”), which holds for d’s rows here as it does there.
  • The forward semi join needs none: over a key nothing proves unique it is JOIN_UNIQUE_INNER, the child’s keys sorted and each counted once (“Forward semi joins over a non-unique key”). With the proof it is the inner join over the dimension’s rows: each fact row matches at most one of them, so counting pairs counts fact rows. That is reduce_unique_semijoins()’s equivalence, which core makes only when the inner side is one table and so never makes here; the node makes it on the same kind of proof. It saves the sort, and lets a parallel plan divide the child’s rows instead of running the dimension whole in every participant - which for a joined dimension is the join, run once a participant.

Everything on the fact side is unchanged: the fact filters, collected or probed; the lookups in key order and their batches; a partitioned fact’s leaves; the shared copy of a parallel plan; the §9 interlock, which is about the fact’s own sets. The dimension is a different child.

Parallel. The node over the dimension join rel’s partial path below a Gather, as over a table’s: a partial join path gives each participant a disjoint share of the join’s rows (core’s guarantee for every partial path, which a Gather over the join relies on too), so each dimension row is counted by one participant. The forward semi join over its distinct keys takes the join rel’s cheapest path that may run in a worker and runs it whole in every participant, which needs every participant to see the same rows: nothing volatile in the quals of any of the dimension’s tables, in the conditions of its semi and anti joins, or inside a table of it that is a subquery, a function or a VALUES list (lion_fkjoin_dim_volatile(); before, d’s quals alone). The workers are still counted from the fk index pages the lookups read and d’s own parallel_workers.

Cost. Unchanged: the child’s total cost - core’s price of the join rel’s path - plus the lookups, and the child’s rows are the join rel’s estimate. So the node inherits core’s estimate of the dimension’s own join. Where the fact left inside the dimension has the same skew the node was brought in for, core’s cheapest path of the dimension may be the very nested loop the node replaces one level up, priced at the fact’s average rows a key; the node’s cost is then as wrong as that path’s, and the node as slow as it. Nothing here corrects that: the child’s price is core’s. In the measurements below it did not happen - core’s cheapest path of the dimension semi-joined to f1 was a hash semi join over f1’s filtered rows - but the child was the larger part of every plan.

Taken: the counts, grouped by d’s columns or not, count(DISTINCT), the aggregates of “Every aggregate over the node’s rows” (counted rows below a LionJoinAgg for an inner join), GROUP BY the join key - d’s or the fact’s - over inner, reverse semi, anti and forward semi joins, with the dimension’s own semi and anti joins in any number, nested or not, and a partitioned fact. Declined: an outer join inside the dimension; a LATERAL reference, a PlaceHolderVar, a pseudoconstant qual; a second join clause between the fact and any table of the dimension; a semi join inside the dimension that involves the fact; the fact inside another semi join’s inner side (its join is that table’s); a dimension with no table the key may be taken from. (An inner join inside the dimension, a non-unique key and a partitioned d are taken since “A dimension that is any join”.)

EXPLAIN is unchanged: the child is the join, under the node, as core plans it. A forward semi join taken as the inner join prints no Join Type, as the one core makes of a single table’s does not.

Measured (2026-09-28, PostgreSQL 18.6 packaged build, shared_buffers 256 MB, a VM of four cores shared with other work; warm cache, all-visible, medians of five runs after one, in ms, cost in parentheses). Synthetic data, 3.2 GB: a 500,000-row dimension with a unique btree key and a lion index on its filter columns, the filter keeping 75,000 rows; f1, 10,000,000 rows LIST-partitioned by kind, one list sub-partitioned by range on ts, with a partitioned lion index over its filter columns and fk, and a btree on fk; f2, 5,000,000 rows of a plain table with a tsvector column, a lion index over it and fk, and a btree on fk. 80% of each fact’s rows are on the kept keys: 107 rows a kept key in f1 and 53 in f2, where the averages core prices a key at are 20 and 10. The fact filters are a stable range on f1’s ts beside an OR with an arm per kind (269,000 rows; the partitions leave out the arms their bounds refute and the kinds they imply), and a tsquery on f2 (100,000 rows). “Node” is the node the model builds; where it is not chosen, the plan with nested loops and merge joins disabled, which is the node over the hash join child the chosen plans have. “Nested loop” is the pushdown off with hash and merge joins disabled, “hash join” with nested loops and merge joins disabled. Bold is the plan chosen with nothing disabled; “core” marks core’s own choice with the pushdown off.

query, serial node nested loop hash join
(A) 986 (201,536) 10,721 (215,240) core; 6,068 (280,473) 1,026 (287,575)
(B) 1,029 (201,535) 15,454 (355,797) 988 (288,022) core
(A), NOT EXISTS on f1 767 (253,200) 5,336 (243,309) core 1,079 (287,593)
(A), GROUP BY d.region 981 (201,961) 10,311 (215,503) core 1,002 (287,833)
f2 JOIN (d ⋉ f1), GROUP BY d.region 963 (203,356) 15,911 (308,025) core 1,066 (433,004)
(A), 3,831 dimension rows (grade < 5) 655 (176,910); probed 738 (183,117) 601-660 (121,671) core 914 (284,603)
(B), 3,831 dimension rows 637 (176,917) 1,335 (209,420) core; 1,737 (280,004) 862 (285,122)
(A), 14,947 dimension rows (grade < 20) 715 (181,619); probed 993 (208,275) 1,349 (137,687) core 956 (285,840)
(B), 14,947 dimension rows 737 (181,619); probed 1,133 (208,274) 3,337 (267,891) core; 3,772 (337,255) 889 (286,343)
two workers
(A) 398 (162,699) 1,935 (158,588) core 749 (264,593)
(B) 405 (162,699) 501 (261,361) core
(A), NOT EXISTS on f1 344 (183,495) 1,834 (169,628) core 544 (260,563)

Every plan the model chose for the node counts f2 against the dimension semi-joined to f1, with core’s hash semi join of the dimension’s rows against f1’s filtered rows as the child, and collects f2’s filter. The node is ten to sixteen times faster than the nested loops core picks for (A) and the inner join, which core prices at the fact’s average rows a key, and about as fast as the hash joins core prices at 1.4 to 2.1 times the node. EXPLAIN ANALYZE of (A) says why no faster: of its 931 ms, Join Child Time is 710 - the child’s hash semi join, most of it the bitmap heap scans of f1’s partitions - and the node’s own lookups and counts 180 ms for 70,758 keys. The same key walk over one table’s child ran in 0.2 to 0.4 s in the benchmark’s two-table queries; over a joined dimension the child is the other half of the query, and costs what core’s plan of it costs.

Where the node loses, it loses on cost to a nested loop that core underprices: the anti join (253,200 against 243,309; 767 ms against 5,336), (A) in parallel (162,699 against 158,588; 398 ms against 1,935) and (A) over 14,947 dimension rows (181,619 against 137,687; 715 ms against 1,349). The node’s own price there is its child’s plus the lookups, as for one table, and nothing is taken off it; the nested loop’s estimate is core’s. For (A) over 3,831 dimension rows the nested loop is priced about right and is about as fast as the node (0.6 s against 0.66 s), and is chosen.

f1 is never the node’s fact here. Forced with f2 inside the dimension (a filter f1.id > 0 that no posting set answers keeps f1 out of the other orientation), the node over f1 is priced above core’s hash join and runs in about the same time; the two-table half alone, d ⋉ f1 through the node, ran in 792 ms at 426,754 against the hash semi join’s 698 ms at 176,583. Some 450 ms of the 792 is collecting the ts range in each partition it keeps while the filters are located (Range Sources Collected: 5), and the rest is 143,331 lookups, 75,000 keys in up to four partitions. When this was measured the collection was outside the node’s timers, priced at a fraction of its time, and every partition was priced for every key. Since 2026-09-29 it is timed (Fact Filter Locate Time) and priced as the union it builds, and each partition for the keys it looks up (§32, “What collecting a range costs”). Re-measured on data of the same kind, the two-table half then costs 316,865 against 512,027 before, and still loses to the hash semi join at 276,208 (441 ms against 667): its lookups over the partitions are priced at two to three times their time. (Refitted 2026-09-29, “The per-key terms, refitted”: a walked key is now priced at its lookup on the leaf and its place in the batch, not a page visit as well, and the semi joins over a partitioned fact of that refit’s measurements came down from a median of 792 units a millisecond collected and 1,279 probed to 613 and 786.)

Probe or collect for a few thousand dimension rows. The model collects f2’s filter in every plan above, and probing it is dearer in the model and slower in fact. Over 14,947 dimension rows (grade < 20; 3,527 estimated in the child) the node’s own work is 6,568 collected against 33,224 probed, 715 ms against 993 measured with the copy priced out (pg_lion.fkjoin_copy_container_cost = 1e9); over 3,831 (884 estimated in the child), 3,059 against 9,265, 655 ms against 738. Both are small beside the child’s 173,851 to 175,051. For (A) that puts the node above core’s nested loop: at 14,947 dimension rows the loop is priced at 137,687 and runs in 1,349 ms, against the node’s 181,619 and 715 ms - it probes f1 for the 3,211 rows core expects the dimension semi-joined to f2 to have, at the fact’s average rows a key again; at 3,831 it is priced at 121,671 and runs in 601 to 660 ms, about as fast as the node, and is chosen rightly. For (B) the node is chosen at both sizes, the probed plan never.

A dimension that is any join (2026-10-30)

A count of a fact filtered by lion and joined on its key to a list of keys fell back to a bitmap heap scan of the fact whenever the list was spelled as a join rather than an IN: the fact’s count(*), and count(DISTINCT key), over

fact f JOIN (d JOIN list l ON l.part = $1 AND l.k = d.pk
               LEFT JOIN excluded x ON x.k = d.pk WHERE x.k IS NULL) c ON c.pk = f.fk

was answered by reading the heap of every fact row the filter kept, for its key alone. Three rules of the recognition declined it, none of them about the count: the dimension’s tables were one table and semi or anti joins only, so d JOIN l declined; the dimension’s table could not be a partitioned parent, so l alone, named through its parent, declined; and an anti join made of LEFT JOIN ... IS NULL declined anywhere in the query.

Why any join is exact. An inner join’s count is the sum, over the dimension’s rows, of each row’s fact rows (“What is computed, and why it is exact”), whatever the rows are and however many share a key: the join pairs a fact row with each dimension row of its key, and the node counts it for each. count(DISTINCT x) is the core Agg’s over the node’s rows, one a dimension row that joins, and is blind to their multiplicity. So the dimension may be any join core plans of the tables that are not the fact, provided the fact joins it by one key equality and nothing of the fact is wanted above but what the node emits. Its rows are core’s: the node runs the join rel’s cheapest path as its child, as for a dimension of one table. What does rest on a key that never repeats - a GROUP BY of the key, the dimension-side LEFT JOIN to the fact, a batch’s keys united under a GROUP BY of a fact column - is taken only where an index proves it of the one visible table (“Uniqueness of the dimension key”).

Recognition (lion_fkjoin_try_fact(), lion_fkjoin_pick_dim()):

  • the dimension’s VISIBLE tables are those in no inner side of its own semi and anti joins: the tables whose rows its rows are (dimvisible). There may be several, inner-joined;
  • the key is taken from one of them, d: a table joined to the fact by exactly one clause, a strict operator between a plain column of each, and the table on which the semi or anti join with the fact is made, where there is one. With the key’s equivalence class holding a column of several of them (f.fk = d.pk = l.k) any of them is the key, the class holding them equal in every row; one whose column an index proves unique is taken first, so that a dimension of one table keeps its proof. d may be a partitioned parent: core plans it, and the lookups take the key from its rows;
  • an anti join made of LEFT JOIN ... IS NULL is the dimension’s own like a NOT EXISTS: core runs it inside the child, and its rows are its outer side’s. Its inner side’s columns are NULL in every row and are not among the visible tables, so a query that names one declines. With the fact its inner side - the dimension rows without a fact row - it declines, as before;
  • the node’s rows carry columns of any visible table (lion_fkjoin_child_has() checks the child emits them), for a GROUP BY, count(DISTINCT) or the aggregates of “Every aggregate over the node’s rows”.

Uniqueness (keyunique): the dimension’s rows hold each key at most once when there is one visible table and an index proves its key unique. Only a GROUP BY of the key (“Grouped by the join key”: each of the node’s rows a group of its own) and the keys united under a GROUP BY of a fact column (“Grouped by a fact column, the keys united”: a fact row counted once by the union) rest on it, and ask it; the dimension LEFT JOIN the fact keeps it as its scope guard; and the forward semi join over a joined dimension is the inner join only with it. Without it a GROUP BY of the key is grouped by the Agg above the node, as any other grouping column is.

Cost. Unchanged: the child’s price is core’s, and the lookups are priced per row of it. With a key that repeats the lookups repeat, priced so.

Tests: fkjoin_anydim.sql - an inner join inside the dimension, its other table’s columns counted and grouped by, three tables over one key class; the list through a subquery with LEFT JOIN ... IS NULL, with NOT EXISTS, and through its partitioned parent; a key that repeats, with GROUP BYs of the key; parallel plans; and the fact inside LEFT JOIN ... IS NULL and a column of an excluded table, declined. Every answer against core’s plan with the pushdown off. fkjoin_dimjoin.sql’s inner join inside the dimension is taken now.

Probed, then collected (2026-09-29)

Whether the node probes the fact filters at every count or collects them once is the planner’s choice, made from its estimate of the dimension rows (“Cost, revisited”). A benchmark counted a fact of a tsvector under a lion index against tens of thousands of dimension keys, the fact filter a tsquery, an AND of ORs of dense terms; core estimated an order of magnitude fewer dimension rows - the product of the dimension’s filters, which core takes for independent - so the plan probed. The keys the dimension kept were the heavy ones, hundreds of containers each where the model prices a key at the fact’s average, and the count phase ran for tens of seconds over three participants, building about two unions for every key container it read. The same fact counted against several times as many keys with the filters collected took under a second. Core underestimates semi joins over skewed keys by one to two orders of magnitude on that benchmark, and no price the planner makes from its own estimate can be right about a run it estimated that wrongly; the executor can see what it is paying.

The account (lion_join_count_key()). A run whose plan probes keeps an account of what probing has cost it beyond what the same counts against a copy would have, in the planner’s units, from the counters each count leaves: the containers read from the fact filters' sets (the count’s Containers Visited less the key’s own), each a seek of a posting tree and the container found there, at LION_PROBE_COST (LION_MEMORY_PROBE_COST for a set already in memory); the posting pages those seeks read, at LION_DESCENT_COST a page as a directory level is priced; the unions built at a key, at LION_UNION_KEY_COST; less the key’s own containers at LION_FKJOIN_COPY_PROBE_COST, the copy’s lookup each would have made instead. What both ways share - the key’s set read and counted, a count set up - is in neither. Deterministic: the same data and plan make the same account, which the tests rely on. (Since “The per-key terms, refitted” the account is kept at the planner’s own prices for the same operations - the probing count’s set-up and its union sets, LION_FKJOIN_PROBE_COST a filter container sought and LION_FKJOIN_PROBE_PAGE_COST a posting page, less the count against the copy’s set-up as well as its lookups - and never goes below nothing; the measurements below were made before.) A fact filter decided from stored positions (§17) is decided by each count that probes it and not by the counts against a copy, which was decided as it was made, so all that deciding it takes is probing’s: each member a lexeme’s cursor sought, at LION_POS_CHECK_COST, and each it decoded getting there, at LION_POS_MEMBER_COST (pos_seeks and pos_decoded of the count’s statistics, lion_posfilter_work()).

The price (lion_join_copy_price()), taken once, at the first count that probes, from the sets as they were located, which carry their rows and containers (ntids, ncontainers): every container of every set of every positive filter read once (LION_CONTAINER_COST), the posting pages a set on pages of its own takes (LION_DESCENT_COST each, its containers' bytes over a page’s room), each union’s image at each of its keys and its members set in it (LION_UNION_KEY_COST, LION_UNION_MEMBER_COST), and the copy’s containers made (LION_FKJOIN_COPY_CONTAINER_COST). The copy’s size is the planner’s formula over the rows the filters' product leaves, taken as independent, in no more containers than the sparsest filter has. A clause the row filter decides from stored positions (LionClauseState.posfilter) is decided as the copy is made: each row the product leaves sought under each of the clause’s lexemes, and each lexeme’s members decoded - its set’s rows, at most LION_POS_SEEK_MEMBERS a row - at the planner’s prices (§17, “The price”). No price - no switch - where no copy can be made: a hot standby, filters that only subtract, a list located a batch at a time. A copy expected not to fit a hash join’s memory (get_hash_memory_limit(), what a plan that collects is given) is priced as it will be made, SPILLED: its pages written once - and every count after the switch reads the containers it looks up back from the file, which the account takes off what each probed count would have cost against the copy (LION_SPILL_READ_COST). A copy that turns out larger than expected spills all the same, as a planned one does. A range among the filters is a collected set, in memory or in a file, and is read into the copy as any filter is.

The switch (lion_join_maybe_switch()). After each count, between two keys, a run whose account has reached the price collects the filters (lion_join_collect_into(), as a planned collection does before the first key) and counts every key after that against the copy. That is the ski-rental rule: a run that switches has paid for probing at most the price of collecting, and then the collection, so it costs at most about twice what the better of the two ways, chosen in hindsight, would have - where the probing plan it replaces costs as many times that as the keys are heavier than the planner thought. A run that never reaches the price was right to probe, and does. The walk of the fk index lets go of its leaf while the copy is made, and reads it again by its block number for the next key; the WHERE sets let go of their pins, as after a planned collection. Should the copy come out empty - the filters select nothing - the keys after it are looked up no more: an inner or semi join has no row left, and an anti join’s are every row, as when a filter has no entry.

  • The empty copy does not change how the run reads its child (2026-09-30 review). Over a plain fact, whether the rows come a batch at a time, walked in key order, or a row at a time is decided once, at the run’s first row (join->batch.begun, join->batch.walked), and kept to its end. It used to be worked out again at every row from wheremissing, which a clause with no entry sets before the first row and the empty copy sets part way through - so the row after the switch was read a row at a time: the rest of the batch in hand was dropped and the child read on, or read again where a finished scan starts over, and an anti join lost rows or counted them twice (on PostgreSQL 16, 1001 a region for 1000 over one batch, 867 over several at 64 kB, one in parallel). Each way sees to the empty copy itself: walked or a row at a time, an anti join’s rows after it go up with no lookup and a semi or inner join’s run ends; a semi or anti join path’s batch tested whole (lion_join_count_batch()) tests no key after it - those join nothing - and hands up what it tested, and its later batches are tested no more. The copy is set up as the counts' filters (join->sources) even when empty, so that join->filtered always means what the counts read. A partitioned fact’s leaves already stopped at their turn’s empty copy (lion_join_count_parts()), and that leaf has no turn after it.

  • §9. “Why a stale copy is safe” asks of a copy that it be made after the query’s snapshot was taken and only ever be counted beside a located fk set, which carries the interlock. A copy made part way through is both: the filters it is made from were located in this run, under the snapshot, and the collection reads them as a planned one does, pinning nothing and asking the map nothing; every count after it locates its fk set under its own pin, as every count does.

  • A partitioned fact keeps an account and a price per leaf (LionJoinPart), since each leaf’s filters are its own sets. A probing plan locates a leaf’s filters again at each of its turns (each batch of keys), and that goes on the leaf’s account too, its directory pages at LION_DESCENT_COST. A leaf that switches makes its copy within what the copies of the leaves before it left of the hash join’s memory (lion_join_part_maybe_switch()), as a collecting plan’s leaves do, and its later turns read the copy - so the switch saves the locating of the filters at every later turn as well.
  • Parallel. Each participant keeps its own account and makes its own copy, as the leader of a plan run without its workers does. The shared copy of “One copy per query” is not joined: a probing plan sized no dynamic shared memory for it, and the participants reach the price at keys of their own, each over its own share of the dimension rows. A participant whose share holds few heavy keys may never switch, and needs not.
  • Rescans. Each run starts its account again (lion_join_switch_reset(), from lion_reset_run()); the copy of the run before is released with the run.
  • EXPLAIN ANALYZE prints Fact Filters: probed, then collected in place of nothing (a probing plan prints no Fact Filters line), and Fact Filter Switches - the copies made so, one a run, a leaf or a participant - and Fact Filter Keys Probed, the keys counted by probing before each, summed; and the copy’s own counters (Fact Filter Rows Collected, Copy Containers Read, Seeks, File Reads, Fact Filter Collect Time), as a collecting plan does. pg_lion.enable_filter_switch (on) is a testing knob: off, a probing run probes to its end, with the same answers.

The reverse - a plan that collected and meets only a few light keys - is not switched: its collection is made before the first key, and what it cost is spent by then.

Measured (2026-09-29, PostgreSQL 18 packaged build, a VM of four cores that other work kept at a load of two to six, warm cache, shared_buffers 6 GB, work_mem 64 MB; medians of three runs after one, a probing run without the switch once). A synthetic star of 30 million fact rows in 230,000 heap pages (3,600 container keys), a tsvector of five dense terms (each on 6% of the rows) and two filler words under one lion index, the fk under another: 12 million of the rows on 30,000 hot keys, 400 each in some 380 containers; 12 million on 200,000 warm ones, 60 each; 6 million on 6 million cold ones - 4.8 rows a key on average, which is what the model prices a key at. A 1.43-million-row dimension whose two filter flags are true on the hot keys and nowhere else, which core multiplies into 549 rows for 30,000. The fact filter is a tsquery of two ORs, of three and of two of the dense terms: 592,134 rows. The node is forced (the join methods disabled; probing by pricing the collection’s pass out, the switch off or on); with nothing disabled core’s nested loop into the fk index is chosen for the first three sets, priced at the average rows a key. Times in ms:

dimension keys collected probed, before probed, trees probed probed, then collected (trees built / probed) keys probed before the switch nothing disabled
1,047 hot 118 3,511 2,065 143 / 140 8 nested loop, 572
10,147 hot 312 32,225 18,607 360 / 331 8 nested loop, 5,540
30,000 hot 801 96,345 64,786 781 / 728 8 nested loop, 18,656
200,000 hot and warm 1,565 252,835 169,041 1,744 / 1,603 8 the node, collected, 1,721
two workers: 1,047 114 1,456 1,157 166 / 156 24 nested loop, 386
10,147 253 11,439 11,352 350 / 317 24 nested loop, 3,624
30,000 549 33,540 30,890 532 / 561 24 nested loop, 9,867
200,000 1,000 93,551 84,139 920 / 1,021 24 the node, collected, 1,009

The probing run over the 30,000 keys did what the benchmark’s did: 11.4 million key containers, 73.6 million posting pages read by the seeks of the filters' sets, and 34.3 million unions built (none with trees probed, where the tree was evaluated for the key’s row or two at each of the 11.4 million keys instead). Collected, the same keys took 32 ms to collect and 637 to count. A run that switches pays eight keys' probing - some 30 ms, what the collection’s price says - and then runs as the collected plan does: 0.9 to 1.2 times its time, where probing to the end took 30 to 160 times it. With two workers each of the three participants switched, after eight keys of its own (Fact Filter Switches: 3, Keys Probed: 24); they gained less from probing trees, whose seeks of the same few hundred posting pages by three processes at once cost more than the unions they save. Over keys of one row each - the dimension’s cold keys, the shape the model prices - the account grows a container a key: 1,202 of them took 135 ms probed to the end and 130 collected (the collection 36 ms of it), and the run that may switch did so after 526 keys, in 136; 40,542 took 730 ms probed and 238 collected, and switched after 522, in 178 (its copy made in 28 ms where the planned one took 56, on a machine other work was loading). The account and the price are both in the planner’s units, and the rule’s factor of two holds in those; in time it holds as far as the two are calibrated alike. Over the 1,202 keys the run switched after 14 ms of probing for a collection that took 32: the price is the cheaper of the two to underestimate, since the account keeps growing at the next key while a price too high is never reached.

test/sql/fkjoin_adaptive.sql forces probing plans (the collection’s pass priced out of the planner’s choice, pg_lion.fkjoin_collect_container_cost, which the run’s own price does not read) over a fact whose forty heavy keys core estimates at one dimension row, and checks against the pushdown off: counts, a grouped count, an IN list, semi and anti joins, row at a time and walked in key order, each switching after a stated number of keys; filters that never meet, whose copy made part way through is empty, over inner and anti joins, row at a time and walked; the knob off, probing to its end, and at 64 kB a copy expected not to fit, priced spilled and switched to all the same; two rescans below a correlated subquery, two switches; a partitioned fact, three leaves switching each on its own, grouped, semi and anti joins; parallel plans, plain and partitioned, whose participants switch; and a dirty heap - rows deleted, moved to other keys and to other terms - before and after VACUUM. Its empty copies are all counts summed in the node - one call, which reads the whole child - with the join paths turned off, so the way of reading the child that an empty copy changed at the next call went unseen. test/sql/fkjoin_switch.sql (2026-09-30) is about the empty copy alone: filters that never meet, the rows handed up one at a time, against the pushdown off.

The per-key terms, refitted (2026-09-29)

“Cost, revisited” fitted the FK-side join’s terms over 23 shapes of two facts, and every term added since - the copy looked up by key, the walk in key order, the row handed up, the counts' unions probed rather than built (§29.11) - was derived or measured on its own. Refitted here together, against what the executor now does.

Measured (2026-09-29, PostgreSQL 18 packaged build, the machine of “Probed, then collected”, which other work kept loaded; shared_buffers 6 GB, work_mem 64 MB, warm; medians of three runs after one, timed and untimed). Four facts of two million rows, 10,800 heap pages (169 container keys), whose fk has exactly 1, 10, 100 or 1,000 rows a key scattered over the heap, a filter column of ten values under a lion index of its own and the fk under another; a dimension per fact, its keys ranked at random so that a range of the rank selects any number of them; and the fact of ten rows a key again, list-partitioned into four leaves. Dimension sets of 100 to 500,000 keys (at most the fact’s), each under three shapes - a count whose fact filter is an IN list of three values (a union of three sets, 30% of the fact), a grouped count under one equality (10%), a semi join under the same equality - with the filters collected and with them probed (the join methods disabled, the other way priced out, the switch off), serially and in two workers, and with nothing disabled and with the pushdown off. 352 runs; and the skewed star of “Probed, then collected” again.

The fit. Each phase’s time from the node’s timers (Join Lookup Time, Join Count Time, the node’s own time less its phases) against the counters of the operations it made, by non-negative least squares weighted to relative error, with each fk container charged LION_CONTAINER_COST’s 40 ns as the model charges it on both ways. Per operation:

operation counter serial two workers
a count that probes the filters, set up keys found 271 ns 737
a set of a union filter, per such count keys found x the union’s sets 217 ns 708
a container of the filters sought Containers Visited less the key’s own 51 ns 38
a posting page those seeks read Join Posting Pages Read 270 ns 281
a count against the copy, set up keys found 659 ns 832
an fk container ANDed with the copy, besides LION_CONTAINER_COST Join Key Containers Read 12 ns -
a member of it past its first 14 ns 20
a directory page Directory Pages Read 430 to 571 ns 403 to 572
a key looked up on its leaf Join Keys Looked Up 377 ns 474
a walked key, besides the leaves it steps over: its lookup and its place in its batch the same, walked 740 ns 784
a row handed up Actual Rows 340 to 630 ns 200 to 570

Median residuals 16% for the probing counts, 13 to 20% for the counts against the copy, 22 to 27% for the lookups; the ranges are two fits of the same term that differ in what they are fitted beside. In two workers a count costs two to three times what it does alone: the participants seek the same posting pages and sets at once. The model prices the serial operations and divides by the participants, as core does.

What was wrong. The probing count of a union filter was priced as the union’s merge again at every count - lion_merge_ops() of the whole union prorated to the probed keys, and LION_UNION_SET_COST’s 100 cpu_tuple_cost a set - from when the counts built it; they probe it now, a seek of each set at the key’s containers and its cursor set up, 217 ns a set. The IN list’s probing plans were priced at 2,149 to 16,139 units a millisecond (median 3,684), five to thirty times what they take. The probe of a filter set too large to be copied into memory for the counts (LION_MATERIALIZE_MAX_*) re-reads its posting tree at every count: from the second row on it is a NOPIN set whose cursor starts at the root, 2.9 to 3.7 posting pages a probe where a key’s one to ten containers lie far apart, 0.34 to 0.68 where it has 100 or all 169 of the heap’s container keys and the seeks step right. LION_FKJOIN_PROBE_COST’s 80 had those pages in it, measured at one fanout. A set that is copied is probed in memory, at LION_MEMORY_PROBE_COST, and the partitioned fact’s leaves' filter sets all are. The walk in key order was charged a page visit a row, where its counters show 0.02 to 0.3 directory pages a key while the keys are dense, besides a batch row priced as a page on purpose: a partitioned fact’s lookups, a batch sorted once and walked in every leaf, were priced at two to three times their time (“A dimension that is a join”, §32 “What collecting a range costs”). And the two counts' set-ups, 25 cpu_tuple_cost each, are 271 and 659 ns: probing a key whose filters are in memory is cheaper than looking it up in a copy.

What moved. The model’s shape: a union filter probed is its sets' seeks - no more of them than the sets have containers - and LION_FKJOIN_SET_COST a set, a count; a probe is lion_fkjoin_probe_one(): LION_FKJOIN_PROBE_COST and the pages it reads, LION_FKJOIN_PROBE_PAGE_COST each, the posting tree’s height and two where the key’s containers are eight container keys apart or more and a share of that where they are closer, or LION_MEMORY_PROBE_COST where the set can be copied; a union collected is its sets' containers read and its image built at each of its keys (LION_UNION_KEY_COST, LION_UNION_MEMBER_COST), as the leapfrog builds a driving union, not a k-way merge; and a walked row is LION_FKJOIN_LOOKUP_COST and LION_FKJOIN_BATCH_ROW_COST, with no page but the leaves it steps over. The constants, at 5 ns a cpu_operator_cost and 20 a cpu_tuple_cost:

setting (pg_lion.) before now
fkjoin_count_cost 25 cpu_tuple_cost 14
fkjoin_set_cost (new; was LION_UNION_SET_COST and the prorated merge) 100 cpu_tuple_cost 11
fkjoin_probe_cost 80 cpu_operator_cost 10
fkjoin_probe_page_cost (new; was in fkjoin_probe_cost) - 54 cpu_operator_cost
fkjoin_copy_count_cost 25 cpu_tuple_cost 33
fkjoin_copy_probe_cost 5 cpu_operator_cost 2
fkjoin_copy_member_cost 3 cpu_operator_cost 3
fkjoin_lookup_cost (new; was a page visit, descent_cost’s 120) - 75 cpu_operator_cost
fkjoin_batch_row_cost 120 cpu_operator_cost 75
fkjoin_row_cost 10 cpu_tuple_cost 20
fkjoin_collect_container_cost, fkjoin_copy_container_cost 2, 20 cpu_operator_cost unchanged (below)

The run’s account of “Probed, then collected” is kept at the same prices: a count that probes adds its set-up and its union’s sets, its filter containers sought and its posting pages read, less the count against a copy it would have made - its set-up and its fk containers looked up - and the account does not go below nothing, so that keys cheap to probe early in a run do not put off the switch once dear ones come.

Units a millisecond, before and after - each forced plan’s total cost over its untimed run, the same 352 runs with the build before the refit and after it (the executor the same but for the account); median, and the least and the most:

forced plans serial, before serial, now two workers, before two workers, now
collected, plain (51 each) 677 (156 to 2,308) 605 (276 to 942) 554 (303 to 2,070) 648 (266 to 1,074)
collected, partitioned (10) 808 (465 to 1,906) 634 (446 to 1,282) 658 (552 to 1,651) 450 (363 to 1,290)
probed, plain (51) 739 (193 to 16,139) 419 (215 to 974) 788 (119 to 15,377) 410 (164 to 933)
… of them the IN list (17) 3,684 (2,149 to 16,139) 383 (215 to 765) 2,310 (703 to 15,377) 373 (164 to 708)
probed, partitioned (10) 3,966 (1,023 to 22,785) 752 (631 to 1,412) 1,635 (808 to 18,327) 435 (362 to 1,401)
core’s plans, pushdown off (27) 223 (81 to 14,469) 257 (117 to 15,686) 289 (101 to 4,477) 303 (163 to 5,925)

The probing plans, priced at up to 45 times the 500 units a millisecond the rest of the model runs at, run at 164 to 1,412; what is left low is the IN list probed over keys of a hundred or a thousand rows (215 to 250 serially, below) and the two workers' probing counts, which take two to three times a serial one’s time each (164 to 183 over the ten-row fact’s largest sets). Core’s own plans are its prices, unchanged: its hash joins run at 120 to 300 units a millisecond, its nested loops over a hundred keys at 1,700 to 15,000 (they are priced for pages read at random that the warm cache holds).

The choices. With nothing disabled, the IN list over each fact and dimension set and the partitioned fact’s semi join, against the fastest of the node collected, the node probed and core’s plan: serially the chosen plan took 1.05 times the fastest at the median where it took 1.07, 1.81 at the 90th percentile where 2.17, and 2,251 ms in all where 2,794; in two workers 1.06, 1.83 and 1,386 ms where 1.09, 1.62 and 1,776. The mispicks the refit ends are the node’s collected plan refused for a hash join (serially, 10,000 keys of one row, 115 ms against 40, which two workers still refuse; the partitioned fact’s 10,000 keys, 131 against 45) and the probed plan chosen over the collected one (1,000 keys of a hundred rows, 15.6 against 6.6; of a thousand rows, 32.5 against 20.9). The mispicks left are hash joins that core prices at 120 to 270 units a millisecond, which the node at 600 loses to while it is up to two and a half times faster (20,000 keys of a hundred rows, 160 ms against the node’s 103; 2,000 of a thousand, 128 against 43), and a hundred keys of one row, where core’s nested loop, priced at 797 and run in 0.3 to 0.5 ms, now loses to the node’s 585 and 0.8 to 0.9 ms. Over the suite’s own small tables fkjoin_distinct.sql’s count(DISTINCT d.attr) over a semi join of 300 dimension rows is now pushed down, in 0.6 to 0.7 ms against the nested loop’s 1.0 to 1.1 it was left to.

Walks over small fk indexes. The refit takes the walk wherever the batch’s keys lie closer than a descent reads, which it did not over the suite’s small fk indexes. Over 400,000 rows of 4,000 and of 400 keys, whose descents read three directory pages a key, the walk and the descent were timed side by side (the walk forced by pricing its rows at nothing, the descent by pricing the walk’s lookups out): the whole of 4,000 keys took 9.0 ms walked against 9.2 descended, their lookups 2.2 ms against 4.3; 400 keys 1.1 against 1.1; 400 keys of a thousand rows 1.7 against 1.6 and 20.2 against 21.8 grouped. The walk is not slower where the refit now takes it, and saves half the lookups' time.

The skewed star of “Probed, then collected”, again (the same machine and runs as there, three of each). With nothing disabled the node, collected, is now chosen for 10,147 and 30,000 hot keys, in 315 and 770 ms, where core’s nested loop into the fk index was, in 5,540 and 18,656: the collected price came down with a union collected priced as its image rather than a k-way merge, below the nested loop’s price at the average rows a key. For 1,047 keys the nested loop stays, in 605 ms against 120 collected. The probing plans switch after 19 keys where they switched after 8 (57 over three participants where 24), at the account’s lower prices, and take 1.0 to 1.3 times the collected plan’s time serially (160 ms against 120 for 1,047 keys, 819 against 769 for 30,000, 1,604 against 1,554 for 200,000) and 1.04 to 1.8 in two workers, where each participant collects for itself: 125 to 152 ms of collection over the three against 50 to 65 for the planned shared copy. Over the cold keys of one row the 1,202 keys no longer switch and probe to the end, in 104 ms against 99 collected; the 40,542 switch after 1,144 keys, in 181 against 155.

Not refitted, and found. The collection’s own terms, once a run (fkjoin_collect_container_cost, fkjoin_copy_container_cost), are left as they were: one filter of 201,111 rows in 169 containers took 0.45 to 0.6 ms to collect, where it is priced at some 20 us (the three of the IN list, a union, took 1.1 to 1.5 ms, priced at 0.8 through the union’s members), and the skewed star’s union of 592,134 rows took 27 to 36 ms. The data holds two sizes of collection only, which cannot tell a fixed cost from one a row; the price is low for small dimension sets, where it makes the planner collect for a few hundred microseconds more than probing would take, and it makes the run’s switch come early, which costs at most the collection. A count’s containers that hold visible rows cost some 0.8 us each more, fitted on Visibility Map Checks, in both ways; the model has no such term. The AND of a key’s container of a few members with a copy’s container that is an array walks the copy’s members: the grouped count over a thousand rows a key, whose copy of one filter holds 1,190 members a container, counts three to four times slower than the IN list’s, whose copy is of bitsets. A union filter probed over keys of a hundred or a thousand rows is priced at half its time (215 to 250 units a millisecond): each of its containers at a key is probed as a union of three sets, some 1.4 us, where the model has three seeks. And in two workers a count takes two to three times what it does alone.

The suite’s plans change in three ways, each checked: over its small fk indexes the lookups now walk the keys in index order where they descended (Join Key Lookups: in index order); some small dimension sets over filters held in memory probe them rather than collect them; and fkjoin_distinct.sql’s count(DISTINCT d.attr) over a semi join of 300 dimension rows is pushed down, in 0.6 to 0.7 ms against the nested loop’s 1.0 to 1.1 it was left to. costgucs.sql lists the three new settings. Where a test counts the copy’s own counters (fkjoin_perkey.sql, fkjoin_semi.sql, fkjoin_walk.sql) it now forces the collection with pg_lion.fkjoin_count_cost, and fkjoin_adaptive.sql’s switches come a few keys later (13 where 10), its rescan case running two dimension sets of twenty heavy keys so that each run still reaches the price.

The semi and anti join as a join path (2026-09-29)

The node answered the FK-side semi and anti joins only as an aggregate over the whole query’s join, at the upper rel. Where the rows of d ⋉ f - the dimension rows that have a fact row under the fact filters - were the INPUT of something else, core computed that semi join itself: a nested loop that probes a btree on f.fk for each dimension row and tests the fact filters on the heap, or a hash join over all of the fact’s filtered rows. A benchmark had such a semi join over a large share of a dimension’s rows, a partitioned fact (list partitions, one sub-partitioned by range on a timestamp) and fact filters that are a stable range on the timestamp beside an OR with an arm per value of the partition key: 10 to 30 s through core’s nested loop or hash semi join, where the same semi join counted by the node (the reverse semi join) took some 150 ms. Its rows fed the node itself, as its joined dimension (“A dimension that is a join”, whose child is core’s cheapest path of the join rel {d, f1}), and other joins and aggregates.

So the semi and anti join is offered as a JOIN path of its own (set_join_pathlist_hook, lion_set_join_pathlist(), chained to any hook installed before it): a CustomPath of the join rel that emits the outer side’s rows. Its child is a path of the outer side; for each of the child’s rows the node looks the key up in the fact’s fk index and tests it under the fact filters - the upper node’s machinery unchanged: the filters collected once or probed, the lookups in key order a batch at a time, a partitioned fact’s leaves each looked up with the batch and matched by any (or by none), the §9 interlock per leaf - and emits the row when the semi join has a match, or when the anti join has none. EXPLAIN names it Custom Scan (LionSemiJoin) or Custom Scan (LionAntiJoin), prints Join Rows: the outer rows with a match (without a match; , in their order when it claims the child’s order, below) and every line and counter the upper node prints, and runs with the upper node’s executor: the same LionCount scan state under another plan name (lion_semijoin_scan_methods), the rows mode of count(DISTINCT) (LION_JOINFLAG_ROWS) with LION_JOINFLAG_OUTER (shape 16).

Which calls it takes (lion_fkjoin_recognize_join()). Core calls the hook once for each pair of rels it joins into the join rel and each kind of join it tries for them. A semi join (EXISTS, IN) comes as JOIN_SEMI, the special join’s left side outer; as JOIN_RIGHT_SEMI the other way round (PostgreSQL 18 and later); and as the right side made unique and inner-joined - JOIN_UNIQUE_INNER and JOIN_UNIQUE_OUTER up to 18, from 19 JOIN_INNER with a rel of unique-ified paths. An anti join (NOT EXISTS) comes as JOIN_ANTI, and as JOIN_RIGHT_ANTI the other way round (16 and later). Only the plain JOIN_SEMI and JOIN_ANTI are taken, with the fact as innerrel: their rows are the outer side’s, each at most once, which is what the node emits; the right-hand forms have the fact outer, whose rows the node never emits, and the unique-ified ones only repeat what JOIN_SEMI answers. Then:

  • the special join is the call’s own (extra->sjinfo), of the same kind, a semi join or the anti join of a NOT EXISTS - the anti join core makes of LEFT JOIN ... WHERE f.fk IS NULL is left alone, as the upper node leaves it;
  • its minimal and its syntactic inner side are the fact alone, and innerrel is the fact: a plain or partitioned table with no LATERAL reference and no TABLESAMPLE (lion_fkjoin_rel_ok()), whose own restrictions are the fact filters - every one answered by posting sets, with a lion index on the fk and on each filter in every leaf that keeps it, exactly as the upper node asks (lion_try_count_path(), lion_fkjoin_setup()), RLS on the fact declining as there;
  • the join’s clauses (extra->restrictlist, everything the join evaluates) are exactly ONE: a strict operator between a plain column of the fact and a plain column of a table of the outer side, which the index must answer as a §10 equality (lion_match_index()). A second correlation (f.y = d.attr), an outer-side qual inside NOT EXISTS (which stays a join clause; inside EXISTS it is pushed to the outer side and is a filter of the child), or anything but a Var on either side declines;
  • the outer side needs nothing from outside it (no lateral_relids, on it or on the join rel: a LATERAL subquery on the outer side is taken at the join rel that also holds what it references), its rows carry the key, and no pseudoconstant qual is anywhere in the query; the paths offered are unparameterized, over every unparameterized path of the outer side;
  • the statement is a SELECT with no row marks: a row it locks or modifies would have its EvalPlanQual recheck run through a join the node stands in for, which it does not do.

The outer side may be anything else: a base relation of any kind, an inner or outer join, a join rel holding other semi and anti joins - another LionSemiJoin among them - whatever core plans for it.

What the rows carry. The join rel’s target, which for a semi or anti join names the outer side’s columns - of any kind, system columns and whole-row Vars included; the node reads them from its child’s row as it reads a dimension column - and nothing of the inner side but what an equivalence class needs above the join. Two EXISTS correlated to one key make one class of the three columns (f1.fk = d.pk, f2.fk = d.pk), core forces every member to be available up to the class’s relids, and the join rel {d, f1} then carries f1.fk for a join with f2 that may compare f2.fk = f1.fk. Core’s semi join hands up its first matching fact row’s value; the node emits the outer row’s key in its place, which is every matching fact row’s value when the fk and the key are of one type, the join operator is that type’s own equality, and it implies one representation under the join’s collation (lion_fkjoin_fk_is_key(), §10’s value rule - “Grouped by the join key”). A key of another type (int4 fk against an int8 key, joined by a cross-type class) or of a type whose equal values are spelled differently declines, and so does any other fact column, an anti join’s fact column (it has no row to come from), and a PlaceHolderVar. No uniqueness is asked of the key: each outer row is tested and emitted on its own, two outer rows with one key being two rows.

Why the rows are the join’s. A semi join’s rows are the outer rows for which some fact row satisfies the join clause and the fact filters, under the query’s snapshot; per outer row that is §26’s existence test of the key’s fk set ANDed with the filters, which the upper node already makes for a reverse semi join, and the anti join’s rows are those for which it finds nothing. A NULL key joins nothing under a strict operator: a semi join never emits its row and an anti join always does (NOT EXISTS). NOT IN is no anti join: x NOT IN (SELECT f.fk ...) is not true for a NULL x nor when the subquery yields a NULL, and core makes no anti join of it on 16 to 18. From 19 it does, when it proves neither the outer expression nor the subquery’s output can be NULL - where the two are the same, and the node’s anti join is right for it. Over a partitioned fact a row matches when any leaf has a visible match (the leaves' rows are disjoint), and the anti join’s when none has; a leaf whose filters select nothing has no turn. Every count is the upper node’s, so §9, “Why a stale copy is safe”, “One copy per query” and “A partitioned fact table” hold as they are: the node is the upper node with the rows of a join instead of partial counts.

Order. A plan that looks the keys up a row at a time emits the child’s rows in the child’s order. One that walks the fk index in key order, and every plan over a partitioned fact, reads the child a batch at a time and sorts the batch into the index’s key order. The join path tests such a batch WHOLE before any of its rows goes up (lion_join_count_batch() for a plain fact, lion_join_count_parts() for a partitioned one, each key’s answer in its entry) and then hands up the surviving rows in their own order: each row’s place in the child’s order is its seq, 0 to n - 1, so lion_join_batch_unsort() swaps each row into its place - n swaps at most and no comparison. So the path claims the child’s pathkeys, as a nested loop claims its outer side’s (build_join_pathkeys()), and a Sort or a merge join above can use them; LION_JOINFLAG_ORDERED says so, set when the join rel has a use for them. Testing a batch whole also keeps the walk’s place from one key to the next with no row going up in between, where the rows mode of the upper node lets the walk’s leaf go at each row it hands up. What putting the rows back costs was measured on the data below (d’s 200,000 kept rows semi-joined to the partitioned f1, serially, a build that timed the put-back and one that skipped it): 0.9 ms of some 750, under 5 ns a row. The query ran in 736 to 759 ms over an ordered child (LionOrdered on d, or a btree index scan) with no Sort, and in 748 ms with a Sort above the path over an unordered child - the differences noise, as were those with the put-back skipped (782 to 862 ms). The model charges nothing for it.

Projection. The path supports projection (CUSTOMPATH_SUPPORT_PROJECTION): over the topmost join the query’s target list is computed by the node from its row, with no Result above it. Its tuple (custom_scan_tlist) is then every column of the join rel’s rows, since core may hand it another target list, or none - when it puts one on the plan afterwards (create_projection_plan()).

Parallel. A partial path over each partial path of the outer side, under the upper node’s rules (“Parallel”, “One copy per query”): the join rel, the fact rel and the outer side consider parallelism and the clause values are parallel-safe; each participant tests the rows its share of the child returns; the fact filters' copy is collected once, by the participants together, in the Gather’s shared memory; and the workers are what a parallel scan of the fk index pages the lookups read would get, or the child’s, whichever is more - unless the key’s table has a parallel_workers setting, which decides alone. Core puts the Gather (or a Gather Merge, over the claimed order) above it, as over any partial path of a join rel; its rows are the join rel’s divided as core divides a partial path’s. The serial path is parallel-safe when its child is, and a worker may run it whole - under a parallel join’s inner side, say - with a copy of its own. A parallel upper node over a partial semi join path is two parallel-aware nodes, each with its shared state keyed by its own plan node; each collects its copy before it reads a row of its child, so no participant waits at one node’s barrier while another waits at the other’s.

Rescans and parameters. A rescan is the upper node’s: everything a run built let go of, the copy made again for new values of the fact filters, and the child rescanned, core’s changed parameters handed on to it. So the path is right as the inner side of a nested loop that rescans it for every outer row, and inside a correlated subplan whose parameters are in the child’s quals and in the fact filters, a NULL one among them; a generic plan’s parameters are evaluated once a run, in every participant. A generic plan’s kind = $1 proves nothing of a partition’s bounds (§16), and needs a lion index on kind in every partition.

Cost (lion_add_semijoin_path()): the child’s total cost, plus lion_cost_fkjoin_path() for the child’s rows - a lookup and an existence test per outer row, every row taken to find an fk entry as the upper node prices a reverse semi join’s (found = the rows, “Cost, revisited”), each leaf of a partitioned fact for the keys it looks up - plus cpu_tuple_cost and the target’s cost per row emitted, as core charges a join’s rows. The rows are the join rel’s, core’s estimate of the semi or anti join, and a partial path’s a participant’s share of them. Nothing is taken off: not for the outer rows core’s estimate says have no match, and not for the fact rows the cost model has priced at their average number a key where the kept keys hold more. What the fact side costs does not depend on which of the outer side’s paths the child is, and is computed once for all of them.

Startup (2026-09-29, the same day). No row goes up before the first batch is tested whole: one batch of 200,000 rows at a work_mem of 64 MB, below, gave its first row at 978 ms of 992. The path first started when its child did, which told a LIMIT above that its first rows came at once. Its startup cost is now the child’s startup, the child’s run for the first batch’s rows (their share of its rows), and what the fact side does for them - lion_cost_fkjoin_path() for that many rows under the choices the whole run made (force): the fact filters located, the copy collected, the batch’s lookups and existence tests. The total is as it was. The batch is what lion_join_fill_batch() puts in one: rows until its memory context has work_mem allocated - its first block, the array of entries, 1,024 at first and doubled when full, and each row’s copy (lion_join_batch_rows()) - never more than the child’s rows; a plan that looks the keys up a row at a time has a batch of one row, and its startup is the fact filters' collection and one lookup. So over one batch the path’s startup is nearly its total, and a Sort above it gives the rows of an ORDER BY ... LIMIT as early and for less than the child’s order does (the ordered path is shown with sorts off in the test); at a work_mem of 64 kB, ten batches of a 4,000-row outer side, the startup is some 14% of the path’s cost and the ordered path under a LIMIT is taken with no Sort.

The upper node prices a row it hands up at LION_FKJOIN_ROW_COST (10 cpu_tuple_cost, “The cost of a key, after the per-key path”), and the join path at one cpu_tuple_cost, core’s price of a join’s row. For the rows of a reverse semi or anti join - the rows of a count(DISTINCT) of a dimension column, a GROUP BY of one - the two plans do the same lookups and hand up the same rows, and the model now prefers core’s Agg over the join path to the upper node’s rows by the difference. Measured, neither price is the time. Through an anti join whose fact filter selects nothing - 2,000,000 outer rows, every one handed up and none looked up - the path took some 65 ns a row over its child’s own, about 3 cpu_tuple_cost at §10’s units, and a count(DISTINCT) of a column of those rows ran as fast through the upper node’s rows as through core’s Agg over the path (310 and 320 ms). The two plans run alike, and the prices are left as they are. A count alone keeps the upper node, which hands up one row a participant.

Composition. The upper node over a dimension that is a join takes the join rel’s cheapest path as its child (“A dimension that is a join”); where that is the semi join path, the node counts one fact against the rows the path emits for another:

Finalize Aggregate
  ->  Custom Scan (LionCount)
        Lion Indexes: f2_df.fk (fk = d.pk), f2_df.doc (doc @@ '...'::tsquery)
        Fact Filters: collected once
        ->  Custom Scan (LionSemiJoin)
              Partitions: f1_a_old, f1_a_new, f1_b, f1_cd
              Lion Indexes: (fk = d.pk), (((kind = 'a') AND ...) OR ...), (ts >= (now() - ...))
              Implied by Partition Bounds: ...
              Refuted by Partition Bounds: ...
              Join Key Lookups: in index order
              Join Rows: the outer rows with a match
              Fact Filters: collected once
              ->  Bitmap Heap Scan on dim d
                    ...

Setting. pg_lion.enable_semijoin (on) offers the paths; pg_lion.enable_count_pushdown off turns them off with the rest of the pushdown. No enable_*join setting disables them: they are join methods of their own. The FK-side join tests that pin the upper node’s plans with core’s join methods disabled (fkjoin*.sql but this one, and countmultikey.sql) turn the path off: with it, their reverse semi and anti joins went to core’s Agg over the join path, by the row prices above, with the same answers - the whole suite was run so once, and every answer agreed.

Measured (2026-09-29; PostgreSQL 18, release build, shared_buffers 256 MB, work_mem 64 MB; medians of five warm runs; the machine shared with other work at a load of 4 to 9, so that a difference under some 20% is noise). Synthetic data: a dimension d of 500,000 rows, a unique key, a lion index over its filter columns and a filter that keeps 40% of it; f1, 20,000,000 rows LIST-partitioned by kind into four lists, one sub-partitioned by range on ts, with a partitioned lion index over its filter columns and the fk and a btree leading with the fk; f2, 10,000,000 rows with a lion-indexed filter column, a lion index over the fk, and a btree on the fk; 95% of either fact’s rows on the kept keys. f1’s filters, a range on ts relative to now() beside an OR with an arm per kind, keep 821,814 rows on 195,969 of the 200,000 kept keys; f2’s keep 400,000 rows (two words ORed) or, marked narrow, 10,000 (two ANDed). The queries: (1) the count of d’s kept rows with a row in f1 and a row in f2; (2) the count of f2’s rows whose fk is a kept d row with a row in f1; (3) the rows of d ⋉ f1 feeding a Sort on another column, and, joined to a 100-row table, a GROUP BY of that table’s label - the semi join below the join to it or, kept apart in a subquery by OFFSET 0, above it (a hash join, where core’s joins are enabled); (4) the anti join feeding a Sort; (5) the rows of d ⋉ f1, and their count. Each cell is the plan, its cost and its median time; “the path” has core’s three join methods disabled, “without it” the same and pg_lion.enable_semijoin off, and “core’s nested loop” the pushdown off and the hash and merge joins disabled.

Serial:

query nothing disabled enable_nestloop = off the path without it core’s nested loop
(1) hash semi joins, 292,095: 1,603 ms the same: 1,324 LionCount over LionSemiJoin, 732,064: 1,514 LionCount over d nested-loop-joined to f1 made unique, 477,689: 2,673 428,816: 17,765
(1) narrow nested loop semi join into f1’s btree over a hash semi join of d and f2, 53,691: 367 LionCount over the hash semi join, 102,375: 177 LionCount over LionSemiJoin, 429,796: 732 LionCount over d nested-loop-joined to f2 made unique, 116,121: 203 67,438: 358
(2) hash semi joins, 288,590: 2,395 the same: 1,488 LionCount over LionSemiJoin, 732,068: 1,638 477,687: 2,376 530,789: 33,232
(2) narrow nested loop semi joins, 72,510: 352 hash semi joins, 237,608: 1,561 730,097: 1,958 475,677: 3,253 72,510: 566
(3) Sort Sort over a hash right semi join, 218,126: 1,426 the same: 1,508 Sort over LionSemiJoin, 577,190: 846 core’s, 322,802: 1,879 322,802: 1,797
(3) grouped hash right semi join over the hash join of d and the table, 207,461: 1,059 the same: 1,157 LionSemiJoin over their nested loop, 255,519: 428 core’s, 319,994: 1,764 319,994: 1,728
(3) grouped, kept apart a hash join over a hash right semi join, 211,817: 1,015 the same: 969 a nested loop over LionSemiJoin, 575,322: 801 core’s, 320,876: 1,553 320,876: 1,547
(4) Sort over a hash right anti join, 219,900: 1,268 the same: 1,142 Sort over LionAntiJoin, 579,158: 990 a nested loop anti join into f1’s btree, 1,018,932: 4,662 1,018,933: 5,519
(5) rows hash right semi join, 210,450: 2,056 the same: 1,796 LionSemiJoin, 569,520: 1,009 core’s, 315,126: 2,283 315,126: 2,405
(5) count the same under an Agg, 210,664: 1,571 the same: 1,452 the upper node alone, 568,633: 833 the same: 765 315,339: 1,728

Two workers:

query nothing disabled enable_nestloop = off the path without it core’s nested loop
(1) a nested loop semi join into f2’s btree over a parallel hash semi join of d and f1, 212,485: 9,203 ms Parallel LionCount over the parallel hash semi join, 233,299: 1,238 Parallel LionCount over Parallel LionSemiJoin, 350,092: 1,114 Parallel LionCount over a nested loop semi join into f2’s btree, 433,370: 9,538 428,810: 15,761
(1) narrow a nested loop semi join into f1’s btree over a parallel hash semi join of d and f2, 40,984: 300 Parallel LionCount over the parallel hash semi join, 86,170: 328 Parallel LionCount over Parallel LionSemiJoin, 207,959: 546 116,114: 243 67,431: 382
(2) Parallel LionCount over the parallel hash semi join of d and f1, 233,298: 1,303 the same: 1,185 Parallel LionCount over Parallel LionSemiJoin, 350,094: 1,154 477,678: 2,624 530,780: 38,420
(2) narrow nested loop semi joins, 46,270: 385 parallel hash semi joins, 191,484: 969 349,206: 1,090 475,672: 2,729 46,270: 436
(3) Sort Sort over a parallel hash semi join, 178,515: 949 the same: 894 Sort over Parallel LionSemiJoin, 295,314: 725 core’s, 322,793: 1,810 322,793: 1,802
(3) grouped Parallel LionSemiJoin over the hash join of d and the table, 149,157: 399 the same: 373 over their nested loop, 151,015: 388 a nested loop semi join into f1’s btree, 151,812: 908 151,812: 950
(3) grouped, kept apart a hash join over a Gather of a parallel hash semi join, 175,537: 487 the same: 491 a nested loop over a Gather of Parallel LionSemiJoin, 296,737: 435 core’s, 320,876: 1,490 320,876: 1,537
(4) Sort over a parallel hash right anti join, 179,426: 753 the same: 845 Sort over Parallel LionAntiJoin, 298,291: 578 440,503: 1,865 440,503: 1,852
(5) rows parallel hash semi join, 174,159: 821 the same: 720 Parallel LionSemiJoin, 290,960: 574 core’s, 315,118: 1,775 315,118: 1,738
(5) count the same under an Agg, 165,202: 761 the same: 824 the upper node alone, 281,538: 643 the same: 651 315,338: 1,952

What they say:

  • The path is the fastest way to the semi join’s rows over f1’s wide filters: 1.3 to 2.5 times core’s hash semi join serially ((3), (4), (5)), 1.1 to 2.2 times with two workers, and 2 to 33 times the plans core makes with its hash joins disabled - a nested loop into f1’s btree, or over f1’s rows made unique. For the counts (1) and (2) it is about as fast as core’s hash semi joins serially, and as the upper node over core’s parallel hash semi join - the node’s child where the path is not taken - with two workers; with a narrow filter on f2 a plan that starts from f2’s 10,000 rows is faster (177 to 367 ms against 732 to 1,958), and the path is priced above it by as much.
  • With nothing disabled it was chosen once, rightly: (3)’s GROUP BY with two workers, 399 ms against the 868 ms of the plan core takes with the path off. enable_nestloop = off changes no choice of it; it moves (1) with two workers off core’s nested loop into f2’s btree (9.2 s) to the upper node over core’s parallel hash semi join (1.2 s), whose partial path is cheaper than the path’s (233,299 against 350,092) and as fast.
  • The path’s price is its time: 370 to 680 units a millisecond serially and 310 to 520 with two workers, around §10’s calibration. Core’s hash semi joins over the partitioned fact run at 100 to 240 (some 90,000 heap pages of fact rows read past shared_buffers, two million rows filtered), and its nested loop into f2’s btree at 23. So the path loses on price where it wins on time: serially it costs 1.2 to 2.7 times core’s hash semi join and runs faster. Nothing was taken off its price for that; a hash join priced at its time would lose to it.
  • Core’s estimate of d ⋉ f1 was 90,469 rows for 195,969 (ANALYZE’s n_distinct of f1.fk, 226,000 for 500,000): the join path’s rows are the join rel’s, and everything above them is priced for half the rows it gets, whichever path the join rel takes.

Grouped by a fact column (2026-09-29)

SELECT f.kind, count(*) FROM fact f JOIN dim d ON f.fk = d.pk WHERE ... GROUP BY f.kind used to be declined whatever indexes there were: the node’s row for a dimension row is one count, the rows of fk = d.pk ANDed with the fact filters, and nothing in that number says how the rows it counts divide among f.kind’s values. A lion index on kind did not help, because the join never asked for one. So the query was core’s - over a partitioned fact table with an OR of a filter per kind, a nested loop into every partition, at every dimension row. Written as a UNION of one ungrouped count per kind, each branch was the node’s, which is the whole of what this adds: the node counts each dimension row once per GROUP of the fact column instead.

What is counted. For a dimension row d and a value v of the fact column g, the rows the Finalize Agg’s group v wants from d are F(d) ∩ {g = v}: the key’s set, the fact filters, and the posting set of g = v - one more source of the same count. The node counts that for each group, and hands up a partial row (d's columns, v, count) for each count that is not 0; the Finalize Agg adds up the rows of each (v, d's grouped columns) as it adds up a dimension column’s. A dimension row’s rows in different groups are disjoint (a fact row has one value of g), so the counts are exact per group and add up to the row’s count. Grouping by g beside dimension columns, or beside the fact’s join key (“Grouped by the join key”), is the same rows with more columns.

The groups are a relation’s own. In the plain table, and in a partition whose bounds leave g several values, they are the entries of a lion index on g there: an entry is one group exactly when the index’s strategy 1 is the grouping equality under the grouping collation, and its stored key is what the row prints, so the index has to pass §10’s value gate - the same rules as a GROUP BY’s own driving index (lion_collect_targets(), given g as a driving column), which is why a numeric g declines. A partition whose bounds give g one value needs no index at all: every row of it is of that group, and a key’s count there is the group’s count, made as any partition’s is (lion_leaf_bound_value()). The proof is §16’s for an implied clause - the constraint strongly implies g = c under the grouping’s own equality and collation - taken for the few constants the constraint compares g with; and since c is what the rows print, the grouping equality has to be the type’s and imply an identical representation under that collation (lion_type_equalimage()), which is §10’s rule for a stored key. A partition of the NULLs alone gives the NULL group. So a fact table LIST-partitioned by kind, one partition per kind, needs no index on kind for this; a partition of several kinds, or a default one, needs one.

Executing it (lion_next_join_group()). The keys are read a batch at a time - the batches of “Lookups in key order”, taken for a plain table too - and each batch is taken to each relation in turn, the plain table or every leaf partition. In a partition whose bounds give g its value, the turn is the partition’s ordinary turn (lion_join_lookup_ent(), with its copy of the filters and its account of what probing them costs), each count put by with that value. Otherwise the entries of g’s index - open, with the relation’s others, for the life of the node (§16, “Why every leaf is open for the whole run”) - are walked a chunk of 64 at a time: the chunk’s sets located - and pinned, as an IN list’s are - each key of the batch looked up once and counted against each of them (lion_join_count_groups(): the key’s set, the filters or their copy, and the group’s set, ANDed by the one count function every count goes through), and the chunk let go of before the next. The cost model takes this for columns of few values; a column of more is correct, only slower, and a chunk after the first walks the fk index from its start again. The rows a turn counted go up once the turn is over: after the partition’s turn has ended, as a partitioned fact’s rows always go (§16), or, over the plain table, once its sets have let go of their pins (lion_pause_run()). They wait in memory until then - at most one per key and group with rows in the relation, no more than the batch’s join rows there. A turn whose filters select nothing has none. Fact Group Key names the column in EXPLAIN, Fact Groups From Partition Bounds how many partitions' bounds give it, and EXPLAIN ANALYZE’s Fact Group Counts how many counts were made per group - none in a partition whose bounds give the value.

What is taken. An inner join, and the forward semi join over its distinct keys, whose rows stand for the fact’s; a semi or anti join’s rows are the dimension’s, and the query above them names no fact column. One fact column other than the join key, itself a column of the GROUP BY (an expression of it is the Agg’s, above the node), hashable, with counts alone beside it: the rows carry no count per group for “Every aggregate over the node’s rows” to rewrite, and a count(DISTINCT) or a sum beside a fact column declines. Each row is a dimension row’s in one group - in one partition, over a partitioned fact - so “Grouped by the join key”’s rows-as-groups Agg is never taken beside it. Parallel plans divide the dimension rows as for any FK-side join, and each participant counts its share in every group.

Cost. A relation counts each key once per group, so lion_cost_fkjoin_path() multiplies its per-key counts - against the filters or their copy - by its groups: one where the bounds give the value, and otherwise the column’s distinct values in the relation (estimate_num_groups() over its rows, since every entry is counted whether or not the filters leave it rows). The key’s lookup is made once. The node’s rows are its keys' times the groups, as many as the join’s rows at most.

test/sql/fkjoin_factgroup.sql: every query against the pushdown off and serially, over a fact table LIST-partitioned by kind (a sub-partitioned list, a list of one kind, one of two and a default, NULL kind included) and a plain copy of it - grouped by the partition key, alone, beside dimension columns and the join key, under an OR of a filter per kind with a range beside it and an IN list of a dimension’s keys, and by other columns, one of a hundred values and NULL, more groups than one chunk; the forward semi join; HAVING, ORDER BY, LIMIT; filters that select nothing in some partitions and in all; batches at a small work_mem; rescans and a dirty heap; the counters; and the declines (an expression, two fact columns, a count(DISTINCT) beside, a numeric column, and a partition of several kinds without an index on kind - which declines only where a partition needs it).

Grouped by a fact column, the keys united (2026-10-11)

SELECT f.c20, count(*) FROM fact f WHERE f.c200 IN (SELECT id FROM dim200 WHERE region = 'region3') GROUP BY f.c20 over 5M rows counted each of the 29 keys in each of the 20 groups: 580 counts, every one of them reading its key’s set again and seeking the group’s set at each of its containers, 871,740 key containers in all, 393 to 600 ms. The same keys given as an array, = ANY ((SELECT array_agg(id) ...)), are an IN list - collected once, every group counted against it - in 12 ms. Nothing in the rows needed the keys apart: no dimension column is in them, and the Finalize Agg adds up a group’s rows whatever key they came from.

What is counted. When the target and the HAVING read no column of a dimension row - none of its own, nor the fact’s join column standing for its key (“Grouped by the join key”) - and the aggregates are counts alone, a batch’s keys are united: their sets located once, each letting go of its pin as soon as it is located, made one union source of a collection with the fact filters - or their copy - ANDed in (lion_sources_collect()), and each group of the fact column counted once against that copy (lion_join_unite_keys(), lion_join_count_united()). A group’s row then carries the count of all the batch’s keys together, and the Finalize Agg adds up the rows of each group as before - one per batch and relation now, where it was one per key. EXPLAIN says Fact Groups Counted Against: united join keys, and Fact Group Counts is the groups (times the batches) rather than the keys times the groups.

Why it is exact. The union’s count in a group is the sum of the keys' counts there because the keys' fk sets are disjoint: a fact row has one fk value, and the keys are distinct under the join’s equality - an inner join’s by the dimension’s unique key, the forward semi join’s because they are its distinct keys - which is the fk index’s, so no two keys locate one entry. And the copy is safe on exactly the terms the fact filters' copy is (“Why a stale copy is safe”): it is read after the query’s snapshot was taken, so it misses no row the snapshot sees, and it is only ever counted beside a group’s set, located from the index and pinned as an IN list’s is, which carries the §9 interlock for any dead TID the copy still lists; the dimension rows are read by the child under the same snapshot as before. A key the same bytes as the one before it in the batch - which that proof says does not come - is left out of the union and counted key by key, so the rows would still add up to the join’s. An inner join whose key is not proven unique (“A dimension that is any join”) is never united: two dimension rows of one key count its fact rows twice and the union once, and that bytes check does not catch the repeat under a nondeterministic collation, where 'a' and 'A' are one key in other bytes.

Bounds and fall-backs. The keys located at once are what work_mem holds of them (lion_array_batch_size() sets at most, and no more than half of work_mem’s worth of LionPostingSets, so the array of sets leaves room for the sets' copies); past that they are united a share at a time, each share a copy of its own counted beside every group, which adds up the same. A share takes one key at least whatever work_mem is left - lion_array_batch_size() alone never falls below 1000 sets, whose array outgrows a work_mem of 96 kB before a key is located - and the turn checks for interrupts before each. The copies take a hash table’s memory (get_hash_memory_limit()) and spill past it as the filters' copy does. A turn that cannot be united - a hot standby, which makes no copy; a range the counts walk (§32), which is not a set to collect; a copy that fails - is counted key by key, every key. A partition whose bounds give the column its value has one group, and its turn counts each key as before. With dimension columns in the rows, nothing changes: each key is counted in each group.

Cost. lion_cost_fkjoin_path() prices both ways and takes the cheaper (*unite). Per key, each key is counted once per group, and every count after the first reads the key’s set again; every count also seeks the group’s set at the key’s containers (lion_cost_fkjoin_group_counts()). Both of those used to be left out - only the count’s set-up and its filter probes were multiplied by the groups - which priced the 580 counts above at 77, against core’s 125,094 for 345 ms; they were chosen, rightly against core, at a twentieth of the time they took. United, the found keys' sets are read once into the collection (a set’s cursor, its containers, the union’s image at each container key, the filters sought at the keys of the sparser side, the survivors copied), and each group is a count against the copy (lion_cost_fkjoin_united_counts()), once a batch. A plan that unites makes no copy of the fact filters of its own: the union is ANDed with the filters as it is made.

Measured (2026-10-11, 5M rows, max_parallel_workers_per_gather 2, work_mem 64MB, best of 3):

query before after core
c20, c200 IN (29 keys of a region) 438 ms, 580 counts 14 ms, 20 counts 345 ms
the same as an inner join - 22 ms 311 ms
… and f.c2 = 1 - 20 ms 264 ms
c20, every key (200) - 43 ms 592 ms
c2, d.flag (half the keys) - 15 ms 503 ms
c20k, 29 keys (20,000 groups) - 209 ms 412 ms
c20, d.name, 29 keys (per key, unchanged) - 391 ms 351 ms

The last row is a per-key plan, chosen and slower than core’s: priced at 44,649 against core’s 126,053 even with the key sets read again counted, the per-key counts remain underpriced against core’s aggregate on this table, a calibration question this change leaves alone.

test/sql/fkjoin_unite.sql: every answer against the pushdown off with sequential scans only - an inner join grouped by a fact column under a dimension filter, a fact filter, both, HAVING, ORDER BY and LIMIT; NULL dimension keys alone and among others; keys without fact rows; an empty dimension result and a fact filter that selects nothing, each a value the planner does not see; a dimension that selects every key, joined and as an IN; the forward semi join over a non-unique key - duplicated and NULL keys, one key three times, an EXISTS; a dimension column and the fact’s join key in the rows, counted key by key; the counters (Fact Group Counts the groups, against the keys times the groups per key); batches at a work_mem of 64 kB; a correlated subquery that rescans the node; and a dirty heap before and after VACUUM. test/sql/fkjoin_factgroup.sql’s queries with no dimension column in the rows - partitioned, parallel, an OR of filters per kind with a range - now unite, with their answers unchanged.

The dimension LEFT JOIN the fact (2026-10-11)

SELECT d.name, count(f.id) FROM dim d LEFT JOIN fact f ON f.fk = d.pk GROUP BY d.name - the count of each dimension row’s fact rows, a dimension row with none counted 0 - was declined: an outer join was refused whatever its direction, and the plan was a hash join over the whole fact table (1,144 ms on the 5M-row synthetic table below, against 18 ms for the inner join through the node).

What is computed. The join’s rows are an inner join’s pairs, and each dimension row with no pair once more, null-extended. A dimension row d with n = |F(d)| fact rows under the fact filters is therefore max(n, 1) of the join’s rows: n holding a fact row, or one holding NULL in every fact column. The node looks the keys up exactly as for an inner join and hands up every dimension row with its n - a NULL key, a key with no entry, a count of 0 and a fact filter that selects nothing included, each a row with n = 0 (LION_JOINFLAG_LEFT, which goes with the counted rows of “Every aggregate over the node’s rows” and nothing else). The Agg above aggregates them as it does counted rows, over m = GREATEST(n, 1) where an aggregate counts the join’s rows:

the query’s over the dimension rows why it is the join’s
count(*), count(1) lion_join_count(m) a dimension row is m of the join’s rows
count(f.x), x the fact’s key or a column declared NOT NULL lion_join_count(n) x is NULL in the null-extended row, and not NULL in a joined one
count(x), x an expression of dimension columns (the dimension’s key among them) lion_join_count(m) FILTER (WHERE x IS NOT NULL) each of the row’s m join rows holds its x
sum(x), x such an int2 or int4 expression lion_join_sum(x * m) as for an inner join, over m
count(DISTINCT x), min, max and the others of “Every aggregate over the node’s rows”, of dimension columns as they stand each dimension row is in the join at least once

A group whose dimension rows have no fact row gets count(f.x) 0 from lion_join_count’s 0 over rows of n = 0, as the ordinary plan’s count of NULLs is; an empty dimension gives no group, or 0 and NULL without a GROUP BY. A NOT NULL fact column is the planner’s notnullattnums, which leaves out a constraint that is not valid. An inner join counts such a column as count(*) now too (lion_fkjoin_agg_is_count()), which WHERE f.x = 3 over a LEFT JOIN - strict, so core makes it an inner join - needs.

What is taken. Recognising it (lion_fkjoin_recognize()) takes one LEFT JOIN as the only special join of the query, its outer side one dimension table and its nullable side the fact, both minimal and syntactic, its own relid the join rel’s third member; the RIGHT JOIN spelling is the same join. The dimension’s key is proved unique as for an inner join, the same scope guard. The join clause is the one clause between the two, and it has to be every join clause either table has:

  • a condition of the ON clause on the fact alone is pushed down to the fact’s scan by core, where it is a fact filter like any other - it decides which fact rows match, which is what the count is of;
  • a condition of the ON clause on the dimension alone decides which dimension rows are null-extended, and stays a join clause, which declines;
  • a WHERE condition on the fact that is strict makes the join an inner join before the hook runs; one that is not (f.y IS NULL, coalesce(f.x, 0) = 0) is evaluated above the join over the null-extended rows too, and is a join clause of the fact that needs the outer join’s relid - which a pair’s clauses do not include, so the join clauses are checked whole, and it declines.

The aggregates over a fact column are the counts above; count of a nullable fact column, sum or min of one, a DISTINCT of the fact’s key, a GROUP BY of a fact column and a key pinned to a constant (core then joins the two on no clause at all) decline. The partitioned fact, parallel plans, batches and the walk in key order are the inner join’s.

Cost. The inner join’s counted rows (lion_add_fkjoin_paths()), every child row a row: the same lookups, and LION_FKJOIN_ROW_COST per dimension row rather than per one that joins.

Measured (2026-10-11, 5M rows, a 220-row dimension of which 20 keys have no fact row, work_mem 64MB, best of 5): d.name, count(f.id) ... GROUP BY d.name 1,144 ms to 16 ms; d.region, count(*) 888 ms to 18 ms; with AND f.c20 = 3 in the ON clause 168 ms to 106 ms.

test/sql/fkjoin_left.sql: every answer against the pushdown off with sequential scans only - counts of the join’s rows, of the fact’s key and its NOT NULL id, of dimension columns, sums, min, max, bool_or and a DISTINCT over a 400-row dimension with NULL keys and 100 keys past the fact’s range; grouped by a dimension column, by the key (the NULL keys one group), and not at all, folded to constants, empty; fact filters in the ON clause, one that selects nothing at run time, a subquery’s; a dimension filter; HAVING count(f.id) = 0, ORDER BY and LIMIT; the RIGHT JOIN spelling; the declines; the counters; batches at 64 kB; a rescan; a parallel plan; a partitioned fact; and a dirty heap on both sides before and after VACUUM.

A second dimension that only filters (2026-10-11)

SELECT d.region, count(*) FROM fact f JOIN dim d ON f.fk = d.pk JOIN e ON f.e_id = e.id WHERE e.flag GROUP BY d.region - a fact counted per dimension row, a second table joined to it only to filter its rows - was declined: three tables with no special join are an inner join inside the dimension (“A dimension that is a join”), and the plan was core’s hash joins over the whole fact table.

What is computed. A table X inner-joined to the fact F by F.x = X.k alone, X.k unique under the join operator, and nothing of X wanted above the join. Each fact row has one x, and each value of it at most one row of X, so the join with X keeps the fact rows whose x is the key of one of X’s rows that pass X’s own filters, each once, and adds nothing the rest of the query reads. It is a fact filter:

F.x = ANY (K),    K = { X.k of the rows of X under its filters, X.k NOT NULL }

the operator strict, so a NULL key joins no fact row and a NULL F.x none of the keys. The query left is the join of F with the rest of its tables, with one more fact filter, which the node answers as it answers any: the keys of the remaining dimension looked up under the fact filters, the IN list among them.

Recognition (lion_fkjoin_filter_of(), for each table of the join tried as the fact):

  • X is a plain table or a materialized view (not a partitioned or inheritance parent), in no special join - no semi or anti join’s side and no outer join’s - so that it is inner-joined;
  • no column of X is in the join rel’s target, which holds every column the query reads above the join (target list, GROUP BY, HAVING, ORDER BY): X’s columns are read by its own filters and the join clause alone;
  • exactly one clause joins X to the rest of the join - the equivalence classes' clauses between X and every other table, and X’s joininfo clauses within the join - and it is the clause the pair (F, X) gives (same class, or the same joininfo clause). A table also joined to the dimension, by a second clause, or by a condition over it and the fact together (e.flag OR f.x = 1), is not only a filter;
  • that clause is a strict operator between a plain column of each, and X.k is unique under it (lion_column_is_unique(), as a dimension’s key is proved);
  • X’s cheapest path is unparameterized and carries the key, the key’s type has an array type, and the rows the path is expected to return are no more than a count locates of one IN list at once (lion_array_batch_size(): a work_mem of 256-byte located sets, never fewer than a thousand), since the node holds the list’s sets whole where the ordinary plan would hash X’s keys within its memory.

lion_fkjoin_recognize() folds every table that qualifies; the tables left are the dimension, taken apart as ever (lion_fkjoin_try_fact()). When every table but the fact qualifies, each is tried as the dimension with the others folded, and the cost model chooses - an ungrouped count with a filter on each of two dimensions is either one’s count against the other folded. Folded tables go only with an inner join of the fact and the dimension (a LEFT JOIN, a semi or an anti join with the dimension declines), at most LION_MAX_JOIN_FILTERS (8) of them.

Execution. Each folded table is a child of the node’s, custom_plans[1 + i], core’s cheapest path of X. Its clause is an ARRAY clause of the fact filters like any IN list - the lion index on F.x matched for it by the join operator, a cross-type one (int4 against int8) included - whose value the node makes at the start of each run (lion_eval_clause_values(), lion_join_filter_values()): the child is run whole, in the query’s memory as every node is (its scan descriptor outlives the values' context), the key read from its column of each row, NULLs left out, the keys accumulated into an array in the values' context (past the cap, a tuplestore: “Past the cap” below). From there it is an IN list among the fact filters: located, collected into the copy or probed, counted. A rescan rescans the children and collects the keys again, so a filter of X’s on an outer query’s column is applied per outer row. The clause’s expression in custom_exprs is a NULL constant of the key’s type that nothing evaluates: a Param there would need an initplan to set it (SS_finalize_plan()). EXPLAIN prints it among the Lion Indexes as x = X.k, deparsed from the child’s column, and EXPLAIN ANALYZE counts the keys collected (Join Filter Keys). The join operator’s function is checked at executor startup as the key’s is (lion_replaced_functions()).

custom_private’s join member grows (shape 23) a (clause, column) pair a folded table, the column its attnum at the PATH stage and its position in the child’s target list at the PLAN stage (lion_plan_fkjoin_path(), lion_child_resno()); the decoder checks the clause is in range, not the key’s own, and an ARRAY clause, and the plan check that there is a child each and that the join is an inner one with none of the distinct-key, outer, LEFT or parallel flags.

Cost. The children run whole before the first key: their total cost is added to the node’s startup and total. The IN list is priced through a cost clause (lion_fkjoin_filter_costclause()): F.x = ANY (<n NULLs>), n the child’s estimated rows, which estimate_array_length() counts, in a RestrictInfo of the fact alone whose cached selectivity - clause_selectivity() returns it without reading the array - is n / max(nd(F.x), X's rows): each key one value of F.x with as many fact rows as any other, and where F.x has values no key has, the keys' share of those. The same share multiplies the fact filters' selectivity in the fact-side model (lion_fkjoin_filters_sel()), which the fact rel’s own estimate does not include. The uniform fk is an assumption: keys chosen by X’s filters that hold most of the fact’s rows make the plan cheaper than priced, never wrong.

Not taken, as yet: parallel plans (a participant would run each folded table whole; with one the node is offered serial only).

Past the cap. The cap is the planner’s estimate, which can be wrong by any factor. A folded table that returns more keys than lion_array_batch_size() puts them in a tuplestore instead of the array (LionClauseState.valstore), which keeps work_mem of them and spills the rest to a temporary file, and the IN list is collected from it a batch of keys at a time into one set, as a parameter’s long list is (§15, “A list too long to locate at once”; lion_locate_list()): in each leaf of a partitioned fact, and again at each rescan, the store ended before the values' context is reset (lion_clause_values_free()). A table of a million flagged keys whose statistics said none were, at work_mem 4MB on the assert build: 17 MB of the backend’s own memory at the peak, in 2.3 s, where the whole list in one array held some 287 MB in about the same time.

Measured (2026-10-11, 5M fact rows, a 220-row dimension of 10 groups, a 1,000-row filter table of which a third is flagged, best of 5, PostgreSQL’s own plan with its parallel workers): the flagged rows' GROUP BY 776 ms to 54 ms; a filter leaving 19 keys 343 ms to 51 ms; one leaving 899 keys 1,294 ms to 131 ms.

test/sql/fkjoin_star.sql: every answer against the pushdown off with sequential scans only, over a fact with NULL fks and a 400-row dimension with NULL keys and keys past the fact’s range, and a folded table with NULL keys and fact values it has no row for - grouped by dimension columns and by its key, the folded table’s own filters applied and none; two folded, one compared across types (int4 against an int8 primary key); the fact’s own filters beside them; no GROUP BY, with every other table only filtering; the IN spelling; a folded table no row of which passes its filter; HAVING, ORDER BY and LIMIT; the declines (a column of it grouped by or counted, a non-unique key, a clause to the dimension or a second one to the fact, a condition over it and the fact, an outer join of it and beside it, more keys than one list locates at 64 kB); the Join Filter Keys counter; batches at 64 kB; a correlated subquery that rescans the node with a new filter of the folded table’s; parallel settings; a partitioned fact; and a dirty heap without VACUUM - deletes and updates in the fact, the dimension and the folded table, whose rows move in and out of its filter, and new rows with NULL and new keys. node_contracts.sql and exec_hook.sql check the folded join’s operator, int48eq, refused as the ordinary plan refuses it.

The anti join over a small inner side (2026-10-20)

SELECT count(*) FROM t WHERE <clauses the count answers on t>
  AND NOT EXISTS (SELECT 1 FROM s WHERE s.a = t.k AND <filters on s>)

counts t’s rows WITHOUT a qualifying inner row: a complement of the forward semi join’s union, which no count per key gives, and core’s plan is an anti join over the whole of t. Where the inner side is a few rows, the complement is cheap the other way round: count t’s WHERE as the count pushdown counts any, and take away the rows of it that some inner row matches, which a B-tree on t.k finds a key at a time.

What is computed.

count = C - |M|
C = { rows of t visible under the snapshot that pass W }          (the count of W, §9)
M = { r in C : some row of the inner side passes every join clause with r }

W is t’s own restrictions (and, for count(col), the col IS NOT NULL the count adds to them). The node runs the inner side as its child, once. Each child row’s scan keys - the values of the inner expressions of the join clauses a B-tree on t looks up by - are evaluated; a row with a NULL key matches nothing (every key is a strict operator) and is skipped; the rest go into a sort by their keys in the B-tree’s order (“One lookup a key” below; “The anti join within work_mem”), with - where a join clause other than the keys' needs them - the inner columns the clauses read. Each key is then looked up once in the B-tree under the query’s snapshot, and every row found is tested on W, and if it passes, on the join clauses with each inner row of its key in turn - the scan keys' own hold of every row their lookup finds, and the others are tested; a row that passes is in M. A key’s inner rows alike in every column (by their bytes, datum_image_eq()) that the sort puts one after another are kept once. With two clauses on the same column (s.a = t.k AND s.b = t.k) both are scan keys of it: the B-tree’s preprocessing makes an inner row whose a and b differ an empty scan, and the clauses are tested on what is found anyway. Clauses that are not keys - s.b <> t.g, a clause of t alone inside the NOT EXISTS, which belongs to the join - are tested the same way.

The order the clauses are tested in. Core’s plan tests t’s restrictions in its scan of t, and the join clauses only on the rows the scan gives the join, the hash or index clauses first and the others after them; each list cheapest first (order_qual_clauses(): by security level, then by the cost of one test, in their written order where those agree). So does the node (lion_anti_order_quals(), lion_anti_join_clauses()): W, in that order, on each row found, and only on a row that passes it the join clauses, the scan keys' own first - true of every row their lookup finds - and the others in that order. A clause core runs only on rows a cheaper one has kept is not run here on a row that one rejects: count(*) FROM t WHERE y <> 0 AND NOT EXISTS (SELECT 1 FROM s WHERE s.a = t.k AND s.b / t.y > 1) divides by no zero y, nor does a WHERE of 10 / y > 1 AND x = 3 on a row whose x is not 3, where core’s plan answers. Beside a count by cells the predicates' truths are evaluated first (below): tests of a column against constants, which a lion index answers and which fail on no row.

Forms. count(*), count(1), count(col) where the count answers it (a column declared NOT NULL, or the IS NOT NULL clause it adds, which the subtraction tests too: a row with a NULL col is in neither C nor M), a count(DISTINCT col) of a column proved unique (§26, “A unique column”, which is that count(col)), and a GROUP BY of one or two of t’s columns that LionCount groups by (§10, §20), with a HAVING, or folded by the planner to one group; and a count by cells (§10, “Counts by cells”), a GROUP BY of expressions over predicates or a WHERE over them, whose cells lose the rows found (“Counts by cells beside the anti join” below). Not: the decoded walk of a GROUP BY’s columns (§34), a count(DISTINCT) walked, filtered counts, the aggregates over keys (§37), the top k (§36), a coalesce group, expression columns (§41), a partitioned or inheritance parent - each counts in a way the subtraction would have to follow, and none is needed by the query this is for.

Recognition (lion_fkjoin_recognize_small_anti(), from the join rel create_upper_paths_hook is given at UPPERREL_GROUP_AGG, where §27’s joins are recognised):

  • a sublink anti join - NOT EXISTS pulled up by core, its RTE_JOIN not in the FROM list - whose outer side is exactly one table t and whose two sides make the whole join rel; any other special join lies inside the inner side; no PlaceHolderVars, pseudoconstant quals or LATERAL references;
  • t a plain table or a materialized view as the fact of §27 is asked to be (lion_fkjoin_rel_ok()), not a parent; the inner tables base relations with no TABLESAMPLE (lion_fkjoin_member_ok()); the inner rel - one table, or the join rel of several - has a cheapest path that is not parameterized;
  • the join clauses: the equivalence-class clauses between the two sides and t’s joininfo clauses within the join rel, each non-volatile, with no SubPlan, aggregate, window function or set-returning function, over plain columns - t’s, and the inner side’s that its rel’s target carries;
  • at least one of them a strict binary operator between a plain column of t and an expression of the inner side alone: the keys.

LEFT JOIN s ON s.a = t.k WHERE s.a IS NULL, which core makes an anti join of, keeps its IS NULL quals above the join, and is not a sublink anti join: it is left alone, as not free and asked for by nothing. t.k NOT IN (SELECT s.a ...) is a hashed SubPlan on 16 to 18, which the node never sees. 19 makes an anti join of it where it proves that neither t.k nor s.a can be NULL (a NOT NULL column, or a strict qual on it) and its operator is a B-tree’s or a hash index’s, which gives no NULL for values that are not: then NOT IN and NOT EXISTS are the same join, a sublink anti join with its own join entry, and the node takes it as it takes the NOT EXISTS - its answer is core’s. A NOT IN over a side that may be NULL stays a SubPlan there too.

The B-tree (lion_anti_choose(), lion_plan_anti.c): a B-tree of t’s own that has amgettuple, not hypothetical, whole or partial with a predicate t’s restrictions imply (predOK: a row outside it fails W, and is in neither C nor M), with an equality scan key on its first column at least. A key is a scan key of index column c when the column is the key’s t column, the operator - commuted when t’s column is its right operand - is the equality of c’s operator family with c’s opclass type on the left, and its collation is the index’s (or the index has none). The scan keys cover a prefix of the columns, each with every key that is one of its; the B-tree that covers the most columns is taken, and of those the smallest. None, and the query is core’s. A key is the scan key of the first column it is one of, and of no other: of a B-tree that names t’s column twice, the second’s would be the same equality on the same value and narrow nothing. So each scan key is a join clause of its own, and the join clauses the node tests lead with each once (lion_anti_join_clauses()); the prefix searched by ends before the second, unless another key is one of its, and a key of a column after it is a join clause tested on the rows found.

One lookup a key. The keys are sorted by the < of their B-tree column’s operator family for two values of the key’s type (strategy 1 of the family for the operator’s right type on both sides; a family without one leaves the query to core), under the key’s collation: they come in the index’s order, and keys equal under the family - numeric’s 1.0 and 1.00, strings a nondeterministic collation makes equal - one after another, looked up once. Keys the sort keeps apart are unequal under the family, whose operators are transitive across its types, as a B-tree family’s must be: a row found by the lookups of two of them would make them equal. So the lookups find disjoint rows, and no row of M is found twice.

Why the subtraction is exact. The count and the lookups are made under one snapshot S, es_snapshot:

  • C is the count of W under S (§9: the interlock makes every count exact under the snapshot the query runs with, the visibility of each candidate row decided by S).
  • An MVCC index scan returns, for one scan key, each row version visible under S whose key equals it once: a heap-only tuple through its chain’s root entry, a version an index entry points at directly through that entry, and nothing for the entries of versions S does not see. Every version has an entry under its own key (or, heap-only, its root has one under the same key), so a row visible under S with t.k = v is found by the lookup of v.
  • Every row found is tested on W on the version S sees: a row in M passes W and is visible under S, so M ⊆ C, and no row is subtracted that the count did not count. Every row of C with a matching inner row has a key equal under the join’s operator to that row’s, which is not NULL (the operators are strict), and the lookup of that value finds it: M is complete.
  • Distinct lookups find disjoint rows: the keys the sort keeps apart are unequal under the B-tree’s family (“One lookup a key”), so each row of M is found, and subtracted, once.
  • The inner side is core’s plan under S, as the ordinary anti join’s is: the same inner rows.

So the count is |{ r ∈ C : no inner row matches r }|, which is what core’s plan counts. §11’s rule - lion’s VACUUM leaves a posting set’s TIDs alone while a scan may need them - is about lion’s own sets, which C reads under it; the B-tree is core’s, and core’s VACUUM removes an entry only once its version is dead to every snapshot, S among them. Each structure is exact under S on its own, so nothing ties the two together. Rows the query’s own transaction changed are seen as S’s command id says, by both.

Execution (lion_exec_anti.c). The run is made once, at the start of the first row’s call, after the clause values are evaluated and before the count locates anything, so that no lion pin is held while the child runs. Reading the child, looking the keys up (the B-tree opened then, under AccessShareLock, and checked to be a B-tree of t whose key columns the scan keys name), testing each row found in a slot of the node’s scan tuple - t’s columns read from the heap tuple, W tested over them, and on a row that passes each inner row’s beside them in turn, the join clauses tested over the two - and adding it to the subtraction: one total for an ungrouped count, or, for a GROUP BY, each row’s group (its grouping columns read from the heap tuple, hashed and compared with the grouping’s own equality and hash function, under its collation), or, beside a count by cells, the row’s cell (below); none of it held in memory past work_mem or hash_mem (“The anti join within work_mem”). lion_emit_keys() takes each row’s subtraction off its count: a group left with none is no row, as a group with no rows to start with is not, and so is the one group of a GROUP BY the planner folded; a plain count’s one row stays a row, of 0. A subtraction larger than the count is an error in a serial plan, which the argument above rules out. A rescan makes the run again: the child is rescanned with the changed parameters core does not hand to custom_ps, and a parameter of t’s WHERE or of the inner side’s filters is the new one’s.

Parallel. The ungrouped count, offered as §10’s parallel count: the participants divide the count, and the participant that comes to the run first (an atomic flag in the node’s shared memory, antiowner) makes it - runs the child, looks the keys up, and takes the whole subtraction off its partial count; the others take nothing off theirs, and the Finalize Aggregate adds them up. A partial count may go below zero. The child is the inner side’s cheapest path a worker may run, unparameterized and not partial (the serial node’s cheapest may be a Gather). Without a shared memory (none could be made) the participant subtracts. A rescan of the Gather clears the flag. EXPLAIN ANALYZE’s counters are every participant’s, summed. A parallel count by cells subtracts in every participant (below). A GROUP BY of columns stays serial.

Cost.

the count's price (converted, §39)
  + lion_units_price(the lookups)          (converted, as the node's own work)
  + the child's total cost                 (core's plan, in core's units, as a dimension's)

all of it before the first row. The lookups are priced as core prices an index scan repeated once per outer row of a nested loop (cost_index(), btcostestimate()): lookups = the inner key expressions' distinct values (estimate_num_groups()) over the child’s rows, no more than the rows; each finds t’s rows over the distinct values of its key columns in the whole table (lion_anti_key_values()) - a lookup finds every row of its key, and the restrictions are tested after, where estimate_num_groups() would scale the values down to the rows the WHERE leaves: an n-distinct extended statistics object’s count of as many of the key columns together as one has (the most, and of those the object of the fewest columns, as estimate_num_groups() picks one), times each other column’s n_distinct as lion_fkjoin_fk_ndistinct() takes it. The product of the columns' own takes them as independent: over 3M rows whose two key columns make 1,000 pairs, it makes a lookup of a pair a share of a row where it finds 3,000, and takes the node over core’s hash anti join at a third of its speed (334 ms against 122); with CREATE STATISTICS (ndistinct) on the pair, core’s plan is taken. A descent costs ceil(log2(index tuples)) * cpu_operator_cost and (height + 1) * 50 * cpu_operator_cost; each index tuple read cpu_index_tuple_cost and a cpu_operator_cost per scan key; the index pages over every lookup by index_pages_fetched() at the random page cost; the heap a read for each row found (no order) or for each page of a lookup’s share of the table’s pages (the index’s order), as far between the two as the correlation squared puts them, each read at the price of a page of the pages the reads come to, as many as index_pages_fetched() makes them (lion_anti_heap_page_cost(): from random_page_cost towards seq_page_cost as the table’s share of effective_cache_size holds them, not by density, since the lookups read the pages in the key’s order). A read is a buffer looked up, pinned and locked whether or not another lookup read the page before, and that, on a table the cache holds, is the lookups' time; core’s index scan, whose model this was, charges the distinct pages at the random page cost and none of the reads again, which over the table below priced 10,000 lookups at 4.2 times their time and 100,000 at about once, index_pages_fetched() by then at every page of the table. Each row found a cpu_tuple_cost and the clauses it is tested on; a cpu_operator_cost a row for each grouping column; and each child row’s keys evaluated, and the child’s rows sorted by them as cost_sort() prices a sort (lion_anti_sort_price(), after costsize.c’s cost_tuplesort()): of the keys' width, and the child row’s where a join clause other than the keys' reads it, with the pages written and read back past work_mem. A large inner side is many descents and many rows, which the ordinary plan’s one pass over t undercuts: the cost model, nothing else, leaves it to core. On PostgreSQL 18 the path carries the child’s disabled nodes.

The plan. custom_private grows (shape 29) an ANTI member, 24, after the cells' (§10): the B-tree, its scan keys' columns, operators, collations and the < they are sorted by, the number of keys, join clauses and restrictions, and the grouping columns' equalities, collations and a < of each equality’s family; at the PATH stage also the expressions - the keys' inner values, the join clauses and the restrictions - which lion_plan_custom_path() appends to custom_exprs after the residual clauses, where setrefs.c points their columns at custom_scan_tlist: t’s as a residual clause’s are (LION_TL_RESID), the inner side’s as LION_TL_ANTI entries naming their place in the child’s target list. The plan check requires its one child and refuses it beside the FK-side join, partitions, the decoded walk of a GROUP BY’s columns, a parallel GROUP BY other than a count by cells' (below) and the other forms above. EXPLAIN prints, under the node’s indexes:

Join Type: Anti
Join Rows: the rows counted, less those with a match
Join Lookup Index: t_k_idx
Join Cond: (s.a = k)

the join clauses deparsed as core prints a join’s (t’s columns bare, as in a Residual Filter), and under ANALYZE Join Child Rows, Join Keys Looked Up (distinct keys, NULL ones not), Join Rows Fetched and Join Rows Subtracted, with Join Child Time and Join Lookup Time under TIMING, and Join Spilled where some of the run went past its memory (“The anti join within work_mem”). The child is printed under the node. pg_lion.enable_count_anti turns it off.

Beside the FK-side anti join. A reverse anti join - SELECT count(*) FROM dim d WHERE NOT EXISTS (SELECT 1 FROM fact f WHERE f.fk = d.pk ...) - is both §27’s anti join, the dimension’s rows tested against the fact’s posting sets, and this one, with the fact’s rows as the inner side and the dimension’s primary key as the B-tree; the cost model chooses. The suites of the FK-side join that pin its plans turn this one off.

Measured (2026-10-20, PostgreSQL 18, a synthetic table of 3M rows of some 240 bytes - 710 MB - whose key has 1M values of three rows each scattered over the heap, a B-tree on the key and a lion index on the WHERE’s column; warm, best of 7 runs - of 5 for the larger inner sides - on a machine other work kept busy; core’s plan a parallel hash anti join with up to two workers):

WHERE inner keys LionCount core’s plan the model’s choice
a half 2 15 ms 181 ms LionCount, parallel
a half 2,000 26 ms 172 ms LionCount, parallel
a twentieth 2 2.4 ms 137 ms LionCount
a twentieth 2,000 12 ms 139 ms LionCount
a half 20,000 149 ms 240 ms core
a twentieth 20,000 128 ms 152 ms core
a half 200,000 1,463 ms 391 ms core
a twentieth 200,000 1,215 ms 184 ms core

LionCount’s times in the last four rows are with core’s joins turned off. The model left the inner side to core somewhere between 2,000 and 20,000 keys, where the node was still faster on a warm cache, its heap fetches priced at the random page cost as core prices its own index scans' pages: at a third of the keys where the two plans' times cross. Priced by the read (2026-10-23, a release build, PostgreSQL 20devel, the same shape at 3M rows in 40,000 pages, shared_buffers 512 MB, serial plans, medians of five warm runs):

WHERE inner keys LionCount core’s plan the model’s choice, before now
a tenth 2,000 10 ms 115 ms LionCount LionCount
a tenth 5,000 30 ms 117 ms LionCount LionCount
a tenth 10,000 54-56 ms 111-139 ms core LionCount
a tenth 20,000 101-108 ms 128-150 ms core LionCount
a tenth 50,000 259 ms 145-166 ms core core
a tenth 100,000 495 ms 171-191 ms core core
a half 10,000 49-72 ms 399-414 ms LionCount LionCount
a half 20,000 107-122 ms 413-453 ms LionCount LionCount
a half 50,000 277-291 ms 495-718 ms core core

The node’s price is now 1.4 to 1.8 times its warm time at every size, against core’s hash anti join, whose price grows with the table where the lookups' does not: the conservative side of the crossing, which a table of 50M rows moves further out. The last row is a hash join of a half of the table priced at the hash join’s rate (§39), below its time.

Counts by cells beside the anti join (2026-10-21)

SELECT CASE WHEN <score over predicates on t> >= 5 THEN 'high' ... END AS tier, count(*)
  FROM t WHERE <clauses> AND NOT EXISTS (SELECT 1 FROM s WHERE s.a = t.k ...)
 GROUP BY tier

is a count by cells (§10, “Counts by cells”) of t’s WHERE less the rows of it some row of s matches - and so is the WHERE form, count(*) of the rows whose tier is one. The rows of a group are the rows of the cells the group’s expression maps to, so the anti join takes each row it finds from its CELL, and core’s Finalize Aggregate adds up what is left.

What is computed. With C and M as above, and cell(r) the cell of row r - the outcome of each dimension its value or keys are in:

count(c) = |{ r ∈ C : cell(r) = c }| - |{ r ∈ M : cell(r) = c }|

The count by cells makes the first term, as ever; the lookups find M, as for any count, and each row of M is counted under its cell.

A row’s cell (lion_cells_code_of(), lion_exec_cells.c). Each predicate - the expression of its LION_TL_CELLPRED column of custom_scan_tlist - is evaluated on the heap tuple of a row of M (lion_anti_cells_truths()), NULL a truth of its own: the truths that, in the count, the labels give its cell. Of each dimension the row’s outcome is the one in which every predicate of the dimension has the row’s truth (lion_cells_truth_in(), the truth the node emits for a cell). There is one at most: a scalar dimension’s outcomes are the classes of its values by their signatures, so two outcomes differ in some predicate’s truth, but for a NOT NULL column’s NULL and a boolean’s other value, which no row has; a multi-key dimension’s outcome is the set of leaves true of the row, each leaf a predicate, or NULL, which every leaf is of. And there is one at least, the row’s own, by the argument of §10 that the cells' truths are the predicates' over the row’s values: a row in no outcome is an ERROR. The cell’s code is the outcomes in the tally’s mixed radix.

The outcome is not searched for outcome by outcome, which would try each of a dimension’s outcomes - up to 1,025 of a multi-key one and 1,026 of a scalar one - on its predicates, where the price below has a cpu_operator_cost a predicate. Once a run, for the first row taken away, each dimension’s outcomes are put in a table by their truths (lion_cells_bytruth()): every outcome’s tuple of its predicates' truths, a byte each, sorted, and of the outcomes that share a tuple the lowest, which the search in order would find first. A row’s outcome is a binary search of the table for its own tuple, of no more tuples than the outcomes or 3 to the power of the predicates: at most two comparisons a predicate. The table is the plan’s alone; it is made in the labels' memory, which the end of a run and a rescan reset (lion_cells_reset()), and made again by the next run that takes a row away. Measured (2026-10-21, PostgreSQL 18.6, medians of eleven alternated runs) on a synthetic table of 400,000 rows, 20,000 of them taken away, counted by ten leaves of an array column: the lookups took 54 ms searching outcome by outcome and 33 ms with the table, of a query of 77 and 56 ms; with one leaf, 15 ms either way.

Where the predicates are read. Every clause the subtraction tests is in custom_exprs, and setrefs.c replaces any subexpression of it equal() to a custom_scan_tlist entry with a reference to it: a WHERE clause over predicates (the CASE of the WHERE form, which is one of t’s restrictions) comes to read the predicates' LION_TL_CELLPRED columns, as the node’s qual does. Those columns are no column of the table, so the node puts each predicate’s truth, as a boolean or NULL, at its place in the slot it tests the row in, before the clauses run - when a clause reads one (predsread). When none does, the predicates are evaluated on the rows of M alone, to tell their cells, and not on every row a lookup fetches: the restrictions reject most of those where the WHERE is narrower than the key. On a million-row table with a WHERE of a tenth, 40,000 inner keys fetching 419,594 rows of which 41,977 are taken away, and five predicates, the lookups took a median 490 ms against 600 when every row fetched was evaluated (an assert-enabled build).

Why it is exact. M’s rows are rows of C (above), and each is in the count under the cell of its values and keys as the snapshot sees them (§10, “Why it is exact”: a row the snapshot sees has labels that are exactly its values' and keys'); the lookup fetches that same version - a heap-only tuple through its chain - and its truths are of those values, by the expressions whose truths the cells stand for. So each row taken away is taken from the cell that counted it, and every cell loses exactly its rows of M.

Execution. The run is the anti join’s own, before the count’s first row; each row of M is counted under its code as it is found - in parallel, kept with the container key its TID is in until its range is claimed (below). The count then goes up cell by cell, and a cell may go up more than once where the tally spilled (§34): so a cell that loses rows is held back (lion_anti_cells_hold()) - every count the tally gives of it added up - until the tally has given everything, and goes up last with what is left of it (lion_anti_cells_next()); a cell left with none is no row, and one left with fewer than none is an ERROR, which the argument above rules out. The other cells go up as the tally gives them.

In parallel (§10, “Counts by cells”). The participants count ranges of container keys, each into its own tally, so every participant makes the run - reads the child and looks the keys up - and takes away the rows of M whose TIDs are in the ranges it counts (lion_anti_cells_range(), at each range it claims; a heap-only tuple’s TID is in its chain’s block, the root’s, which is what the index holds). Its rows of M are sorted by container key, and the ranges a participant claims come in increasing order - the next range is an atomic counter’s - so each claim reads the rows of its range off the front of the sort and passes over those of the ranges other participants took. Every row of M is in one range, counted by one participant, so each participant’s partial count of a cell holds every row it takes from it, and the check above holds of a partial count too. The child is the inner side’s cheapest path a worker may run, as for the parallel count. EXPLAIN ANALYZE’s Join Child Rows, Join Keys Looked Up and Join Rows Fetched are every participant’s, summed; Join Rows Subtracted is the serial node’s.

Cost. The count by cells' price, and beside it the anti join’s: the lookups, with the predicates evaluated on each row found and a cpu_operator_cost a predicate to find its cell (lion_anti_price()’s rowexprs), converted as the node’s own work, and the child’s total cost, both before the first row. In parallel each participant pays both, undivided.

The plan. Both members: CELLS (23) and ANTI (24), the ANTI member with no grouping equalities - the walked column groups nothing - and custom_scan_tlist’s columns the predicates, the counts and the columns the anti join’s clauses read (LION_TL_RESID, LION_TL_ANTI). The plan check takes the anti join beside the decoded walk and a parallel GROUP BY only as a count by cells', with no grouping equality, and a count by cells' columns of the table or the inner side only beside an anti join. EXPLAIN prints both, the anti join’s lines first:

Finalize HashAggregate
  Group Key: CASE WHEN ((((CASE WHEN ((x = 1)) THEN 3 ...
  ->  Custom Scan (LionCount)
        Lion Indexes: t_gy.g (g)
        Join Type: Anti
        Join Rows: the rows counted, less those with a match
        Join Lookup Index: t_k
        Join Cond: (s.a = k)
        Group Strategy: Cells
        Cell Columns: t_x (x), t_gy.g (g), t_z (z)
        Cell Predicates: (x = 1), (x = ANY ('{2,3}'::integer[])), (g = ANY ('{2,3}'::integer[])), (g IS NULL), (z = 99)
        Cells: 18
        ->  Seq Scan on s

Correlated by two columns. s.a = t.k AND s.b = t.k is recognised as for any count (above): both clauses are scan keys of the B-tree’s column, an inner row whose a and b differ is an empty scan, and every row found is tested on both. A further inner column equal to the same outer key is, in effect, the inner filter s.x = s.a, and is tested so.

The decoded walk of a GROUP BY’s columns (§34) stays declined beside the anti join: it is the plan of a GROUP BY of two or more bare columns, not of the expressions over predicates the count by cells answers, and such a GROUP BY beside the anti join is the nested loop of §20, which the anti join takes already. Its groups would need the subtraction keyed by the combination of the columns' values the walk decodes - a third way of finding a row’s group - for a shape nothing here asks for.

Measured (2026-10-21, PostgreSQL 18.6 release build, a VM of four cores shared with other work, shared_buffers 1 GB, warm; medians of seven, alternated with core’s plan). A synthetic table of 3,000,000 rows of some 240 bytes - 756 MB, 96,775 pages - whose key has 950,555 values of about three rows each scattered over the heap, a B-tree on the key and lion indexes on the WHERE’s column (10 values) and the three columns of the tier’s predicates (50 values; 20 and NULLs; 20 and NULLs): the tier is a CASE over a score of three CASEs over seven predicates, 36 cells. The GROUP BY form counts a WHERE of half the table (1,500,824 rows), the WHERE form tier = 'high' within a tenth of it (441 of 299,650 rows); core’s plan is a bitmap heap scan under a hash anti join, with up to two workers in the parallel rows. Planned with nothing disabled:

form inner keys serial: LionCount core 2 workers: LionCount core
GROUP BY tier 2 26 ms 509 ms 15 ms 278 ms
GROUP BY tier 200 28 504 18 275
GROUP BY tier 2,000 40 532 44 281
GROUP BY tier 20,000 131 528 191 294
WHERE tier = ‘high’ 2 19 146 12 127
WHERE tier = ‘high’ 200 25 150 17 127
WHERE tier = ‘high’ 2,000 35 147 38 126
WHERE tier = ‘high’ 20,000 core’s plan, 147 147 core’s plan, 121 122

The count is the node’s whole time with a few keys; the lookups grow with the keys, 5 to 8 us a key and its three rows, and in parallel every participant makes them all, so from about 2,000 keys two workers gain nothing on one, and at 20,000 the GROUP BY form’s lose 60 ms to it - three participants' lookups on four busy cores, which the price, each participant’s lookups whole as its own, does not see. At 20,000 keys the WHERE form is left to core, where the node, forced, takes 108 to 148 ms against its 145: its heap fetches are priced at the random page cost, as core prices its own index scans. The rows a lookup finds are priced from the key’s n_distinct over the whole table (above): estimate_num_groups(), scaled down to the 1,479 rows the WHERE form’s restrictions leave, would make them 2,030 a lookup where there are three, and leave the WHERE form to core from 200 keys on.

The anti join within work_mem (2026-10-22)

What the run keeps grows with the inner side and with the rows its keys find, and the plan bounds neither: the planner may take the inner side for ten rows where the child gives millions, and the cost model’s choice is only as good as that estimate. So nothing the run keeps is held in memory past work_mem or hash_mem (work_mem times hash_mem_multiplier, get_hash_memory_limit()); past them it goes to disk, and the answer is the same.

  • The child’s rows go into a tuplesort within work_mem, as the run reads them: each row’s scan-key values and, where a join clause other than the keys' reads the inner row, the inner columns the clauses read after them. It is sorted by the keys, each by its < (“One lookup a key”), then by each inner column whose type has a default < (under its collation, where it takes one), so that equal keys come together and so do a key’s inner rows alike. The lookups read it in order: a key’s values copied, in memory reset at the next key, and its inner rows - each unlike the one before it - in a tuplestore within work_mem, on disk past it, which each row the key finds is tested against.
  • A GROUP BY’s losses go into a hash table of the groups, in a memory context of its own, while that is within hash_mem. Past it, each group’s loss so far goes into a sort by the GROUP BY’s columns, the table is let go of, and every later loss goes into the sort as a row of its own; the count’s rows are then held back into the same sort as they come, each with its count. When the count’s groups run out, the sort gives each group once, its counts and its losses added up, and it goes up through lion_emit_keys() as any group does (lion_anti_groups_next()): one left with nothing is no row, and the HAVING sees what is left. Each column is sorted by a < of its grouping equality’s family, NULLs last, under the grouping’s collation (lion_anti_group_order(): the type’s default < where its default equality is the grouping’s, else one of a family the equality is in; with none the plan is core’s), so that rows equal under it are one group. A GROUP BY whose path claims its groups' order (§21: one NOT NULL column, by its type’s own <) claims it beside the anti join only where that < is the sort’s, so the groups held back go up in the order claimed.
  • A count by cells' losses go into an array of codes and rows, which grows to hash_mem and is then sorted by code, each code once; if that leaves it more than half full, its losses go into a sort by code, as a GROUP BY’s do, and the tally’s counts are held back into it.
  • The rows taken away beside a parallel count by cells go into a sort by container key within work_mem, which the participant reads as it claims its ranges (“Counts by cells beside the anti join”).

The ungrouped count keeps one total. EXPLAIN ANALYZE prints what went past its memory in any participant, nothing when nothing did:

Join Spilled: keys, losses, held rows

keys the child’s rows, inner rows a key’s and ranges the rows taken away by container key, each on disk past work_mem; losses the groups' or the cells' losses past hash_mem, gone to a sort with the count’s rows held back into it, and held rows that sort on disk past work_mem. losses alone is a sort that stayed in memory. A participant that claims no range still made its sort of the rows taken away, and reports it gone to disk when it did.

Measured (2026-10-22, PostgreSQL 18, warm): a table of 4,000,000 rows, a B-tree on a key of one row each and a lion index on a column of 10 values, count(*) ... WHERE x < 5 AND NOT EXISTS (SELECT 1 FROM f(3000000) s(a) WHERE s.a = t.k), where f is a plpgsql function declared ROWS 10 that returns 3,000,000 keys, at a work_mem of 4 MB. The backend’s resident memory at its peak, above its own at rest: 9.8 MB and 3.0 s through the node (Join Spilled: keys, 3,000,000 keys looked up, 1,500,000 rows subtracted), against 13.2 MB and 1.4 s for core’s hash anti join; the same answer.

The choice, by WHERE and inner side (2026-10-22, PostgreSQL 18, serial, best of two). A synthetic table of 40,000,000 rows (2.0 GB, more than the 1 GB of shared_buffers, warm in the operating system’s cache), a key of 4,000,000 values of about ten rows each scattered at random over the heap with a B-tree on it, and a lion index on a column of 1,000 values that the WHERE takes a fraction of; the inner side a table of distinct keys. LionCount’s times with core’s joins turned off, core’s with the pushdown off; the model’s choice with nothing disabled:

WHERE inner keys LionCount core’s plan the model’s choice
a thousandth 1,000 23 ms 71 ms LionCount
a thousandth 10,000 181 ms 53 ms core
a hundredth 10,000 153 ms 839 ms LionCount
a hundredth 100,000 2,880 ms 849 ms core
a tenth 10,000 199 ms 1,516 ms LionCount
a tenth 100,000 2,830 ms 1,847 ms core
a half 100,000 3,115 ms 5,468 ms LionCount
a half 1,000,000 27,129 ms 7,820 ms core

Of sixteen pairs of a WHERE (a thousandth to a half) and an inner side (1,000 to 1,000,000 keys) the model chose the faster plan in all sixteen. A lookup here is some 28 us a key and its ten rows, the heap’s pages read from the operating system’s cache, and the choice moves with the table: a selective WHERE leaves core few rows to read and the node wins only with few keys, a wide one leaves it many and the node wins with many. On a table of 4,000,000 rows held whole in shared_buffers a lookup is some 2.5 us, below the random page cost its heap fetches are priced at: near the boundary the model left to core inner sides of 10,000 to 100,000 keys under a WHERE of a half or more that the node counts 1.3 to 1.6 times faster, and gave the node one of 10,000 keys in the heap’s order under a WHERE of a hundredth that it counts 1.4 times slower. With no WHERE at all, keys in the heap’s order, it left to core inner sides of up to 10,000 keys that the node counts twice as fast - by the price of counting the whole table against a merge join, the count’s own, not the lookups'.

The dimension from a B-tree (2026-10-20)

The node’s child is core’s: the dimension rel’s cheapest total path, or its cheapest partial path below a Gather (“Parallel”), priced by whoever built it. A B-tree that holds the columns the join needs of the dimension - its key, and a column grouped by - walked through lion’s set of the dimension’s filter, index-only, is such a path (§40.2: a LionBtreeScan, or a LionOrdered where core found the key’s order useful), and its partial form a partial path (§40.9), so the node takes it wherever it is the cheapest, with no code of its own: the dimension rows come from the B-tree’s index tuples, and its heap is read only for pages the visibility map does not call all-visible. Where the dimension’s walk must be a whole path below a Gather - the forward semi join over a non-unique key unique-ifies the whole plan (“Forward semi joins over a non-unique key”) - the serial walk is parallel_safe with the relation (§40.2).

Why it is chosen, and not because of cold reads. The walk is priced in pages it touches and entries it probes (§40.3): the B-tree’s pages and a probe per entry, against a bitmap heap scan’s heap pages and per-row work. When the dimension’s filter keeps a large share of its rows the bitmap scan reads nearly every heap page, which cost_bitmap_heap_scan() prices toward seq_page_cost, not random_page_cost; a heap many times the B-tree then outweighs the B-tree whatever the I/O constants say. On the synthetic table of §40.9 (3,000,000 rows, a 142,858-page heap and an 8,228-page primary key) as the dimension of a 6,000,000-row fact, filtered to a third of its rows, the walk of the primary key is priced at 133,974 against the bitmap heap scan’s 161,392 at random_page_cost 4, and at 108,765 against 160,867 at 1.0, a random page priced as a sequential one: chosen at both. test/sql/btreescan_parallel.sql pins the choice at 4 and at 1 on a dimension whose heap is twenty times its primary key.

Measured (2026-10-20, PostgreSQL 18.6 release build, four cores shared with other work, warm; medians of five): count(*) of the fact (a filter keeping a tenth of its rows) joined to that dimension’s third, 1,000,110 rows, the node forced with core’s joins off, the dimension read by a bitmap heap scan (the walks off) against the walk:

join serial, heap serial, walk 2 workers, heap 2 workers, walk
inner 1,639 ms 1,018 968 788
forward semi (f WHERE EXISTS (d ...)) 1,574 1,022 1,004 621
semi (d WHERE EXISTS (f ...)) 1,919 1,443 1,305 780
anti (d WHERE NOT EXISTS (f ...)) 1,913 1,297 1,173 758

With nothing disabled the inner join’s plan is core’s hash join, over the same walk: 824 ms serially against 1,777 over the bitmap heap scan, and 689 against 1,194 on two workers.

Declined in v1, and why

  • A non-unique dimension key in an inner join (above: a scope and costing decision, not a correctness one). The forward semi join over one is counted over its distinct keys (“Forward semi joins over a non-unique key”).
  • A partitioned fact table with a leaf that has no lion index on fk, or none for a fact filter its bounds do not imply; an old-style inheritance parent as the fact; partitionwise aggregation turned on, as for a count (§16); and run-time pruning of the leaves (“A partitioned fact table”, which counts every other one). A partitioned, inheritance or subquery DIMENSION is declined for want of an index list to prove uniqueness from; innerrel_is_unique() would prove some of those and is the natural v2.
  • An inner or outer join inside the dimension, composite keys, snowflake chains: a dimension that is a join is taken only as one table with others semi- or anti-joined to it (“A dimension that is a join”), whose rows are that table’s, each once; an inner join to a unique key would keep that too (core’s innerrel_is_unique() proves it) and is the natural next step. A table inner-joined to the FACT on a unique key, nothing of it wanted above, is folded into a fact filter (“A second dimension that only filters”); one joined to the dimension is not.
  • Fact columns in the output but for one fact column grouped by (“Grouped by a fact column”) and the fact’s join column, which stands for the dimension’s key (“Grouped by the join key”): two fact columns, an expression of one as a grouping column, a fact column beside aggregates other than counts, and a column whose index cannot print its keys. The join key’s exception does not cover an fk of another type than the key, a key whose equal values may be spelled differently (numeric, citext, bpchar, the floats, a nondeterministic collation), the key under another collation than the join’s, or an expression of it.
  • count(f.col) of a nullable fact column, and the aggregates “Every aggregate over the node’s rows” does not answer: of a fact column other than the key, ones that see multiplicity but counts and int2 or int4 sums (avg, sum of int8 or numeric, string_agg), and any with FILTER or ORDER BY.
  • A DISTINCT over an expression or over a fact column other than the key, an fk of another type than the key, a DISTINCT under another collation than the join’s (“count(DISTINCT)” above). A count of rows beside a count(DISTINCT) used to be declined here (“Every aggregate over the node’s rows”).
  • An anti join from LEFT JOIN ... IS NULL, and a NOT IN that is no anti join (above): every one on 16 to 18, and on 19 one over a side that may be NULL.
  • Outer joins but the dimension LEFT JOIN the fact: the fact LEFT JOIN its dimension and a FULL JOIN keep fact rows with no dimension row, which no lookup per dimension row finds; a LEFT JOIN beside other special joins, or with another join clause, is not taken apart (“The dimension LEFT JOIN the fact”).
  • Driving the count from the collected copy of the fact filters when it is small against the keys: that is core’s plan (“The cost of a key, after the per-key path”).
  • Partial aggregation of a GROUP BY’s rows inside the node. A summed join adds its counts up because a count alone needs no grouping (“The per-key path, end to end”); folding the rows of a GROUP BY d.attr before they go up would need d.attr’s grouping equality and hashing, which are core’s (§10’s finding 3), in a hash table of the node’s own. Core’s Finalize HashAggregate does it above the node.

Tests

test/sql/fkjoin.sql: every query against the same query with the pushdown off, as a multiset both ways round, on a clean heap and on a dirty one (deletes and updates of fact and of dimension rows before VACUUM) - GROUP BY one and two dimension columns and an expression, the ungrouped count with a dimension filter, fact filters (=, IN, IS NULL on another column, OR), NULL fks, NULL dimension keys and a NULL group, HAVING, ORDER BY and LIMIT on top, cross-type keys (int4 fk against int8 pk and int8 against int4), text keys, count(1) and count of either join column, generic plans with Param fact and dimension filters (a NULL one included) and, since 2026-09-27, IN lists of unknown length (= ANY ($1) with two values, none, NULL and 1,500 of them, ANY (ARRAY[$1, $2 + 1]), and a stable expression’s array), and a correlated subquery whose dimension filter is an exec Param, so that the node and its child are rescanned per outer row; the declines - a non-unique dimension key, a key unique only under another collation than the join’s (and accepted once a unique index under the join’s own exists), a second join clause, an outer join, three relations whose third is grouped by (a third that only filters is folded, “A second dimension that only filters”), a fact filter the posting sets cannot answer, a dimension key pinned to a constant, fact RLS, a fact column in the output, a volatile grouping expression, another aggregate, count of a nullable fact column, a partitioned fact one of whose partitions has no lion index on fk (the same table with one on every partition is pushed down) and a partitioned dimension; the cost model’s choice with nothing disabled, including a 50,000-row dimension it must refuse unless its own quals leave few rows; dimension RLS applied (a policy hiding rows changes the answer exactly as it does the ordinary plan’s) and column privileges on d.attr enforced; the Join Keys Looked Up / Join Keys Without Entry counters (NULL keys not looked up; keys whose entries VACUUM deleted); EXPLAIN through lion_explain_norm(). No new concurrency argument is introduced - the fact side is §9 per dimension row and the dimension side is a core scan under the same snapshot - so no isolation spec is added.

test/sql/fkjoin_semi.sql (2026-09-27), the same way against the pushdown off: the forward EXISTS, IN and JOIN spellings over a unique key with fact filters of every kind (=, IN, a tsquery, an OR, IS NULL, IS NOT NULL), none, and ones that select nothing, a dimension filter that passes nothing, cross-type keys and a grouped join; a non-unique key, whose forward semi join is counted over its distinct keys, and - once the dimension has lion indexes - also offered as the reverse shape with the roles exchanged, which the model does not take; the reverse EXISTS and IN forms, grouped, with HAVING, ORDER BY and LIMIT, count(1) and count of the key, duplicated and NULL dimension keys, keys with no fact rows (lion_sj_val() pins the answers themselves) and fact rows whose key has no dimension row, and a sum and a count of a nullable dimension column (declined before “Every aggregate over the node’s rows”); the anti joins, NULL keys among them, count of the key there (a count of a column, over the join’s rows, since then), and the LEFT JOIN and NOT IN forms left alone; the declines (a second correlation, an outer-side qual inside NOT EXISTS, a fact filter the posting sets cannot answer, another aggregate - avg - an inner join inside the dimension, a subquery that cannot be pulled up; the dimension semi-joined to a second table was declined as three relations until “A dimension that is a join”, and is pushed down now); generic plans with Param filters on both sides, NULL ones included; correlated subqueries that rescan the node with a new dimension filter and with a new FACT filter, whose copy is made again; EXPLAIN of the collected and the probed plans and of a dimension filtered through its own lion index; the Fact Filter Rows Collected, Join Keys and Containers Visited counters (an existence test reads fewer containers than the counts); a copy that outgrows the memory limit at run time (work_mem and hash_mem_multiplier both set) and falls back with the same answer; the model’s choice, including a 60,000-row dimension it refuses forward and reverse unless its own filter leaves few rows; and a dirty heap before and after VACUUM. Its two EXPLAINs of a dimension filtered by its own index and of two dimension rows use plans of the child that PostgreSQL 16 to 19 all make (two equalities in the Index Cond; a sequential scan with index scans off): 16 keeps a second column’s IN list out of the bitmap’s Index Cond and costs pk IN (5, 6) as a sequential scan where 18 takes the primary key. In test/sql/fkjoin.sql a 60-value IN list, one union for the scan instead of one per count, is now collected and chosen. test/isolation/fkjoin_vacuum_race.spec is the race of the copy against VACUUM, above.

test/sql/fkjoin_distinct.sql (2026-09-27), against the pushdown off: count(DISTINCT f.fk) over the inner join and its EXISTS and IN spellings, with fact filters of every kind, none, and one that selects nothing, grouped by one and two dimension columns and by an expression, with HAVING, ORDER BY and LIMIT, a GROUP BY folded to a constant over something and over nothing, and an ungrouped count over nothing (lion_dj_val() pins 0); a text key; count(DISTINCT) of dimension columns (several in one query, a nullable one) over inner, semi and anti joins; duplicated and NULL dimension keys; a count of rows beside a DISTINCT count, over an inner and a semi join (declined before “Every aggregate over the node’s rows”); the declines (a non-unique key, forward and inner; a fact column other than the key; an int4 fk against the int8 key, while a dimension column over that join is accepted; a DISTINCT under "C"; an expression; FILTER; string_agg(DISTINCT ... ORDER BY); sum(DISTINCT)); EXPLAIN of the Sort and the Agg over the rows, forward and anti; the existence tests of an inner join’s rows reading fewer containers than its counts; generic plans with Params, a NULL one included; correlated subqueries that rescan the node; and a dirty heap before and after VACUUM.

test/sql/fkjoin_parallel.sql (2026-09-27): a 3,000-row dimension with parallel_workers = 2, parallel_setup_cost, parallel_tuple_cost and min_parallel_*_scan_size at 0, and parallel_leader_participation off so that the workers do the counting whenever they start (the suite’s own max_parallel_workers_per_gather = 0 is SET per session). lion_pj() runs every query three ways - as planned, in parallel; with max_parallel_workers_per_gather = 0; and with the pushdown off - and compares both of the first two with the third: reverse, anti, forward and inner joins, grouped, with HAVING, ORDER BY and LIMIT, folded GROUP BYs over something and nothing, count(DISTINCT) of the key and of dimension columns, duplicated and NULL dimension keys. EXPLAIN shows the Gather over the parallel node for partial counts and for the rows of a count(DISTINCT); the workers max_parallel_workers_per_gather and the dimension’s parallel_workers allow (one worker without the leader is not chosen, with it it is; none, and none on a dimension set to 0); and the serial node under debug_parallel_query’s single-copy Gather, whose answers are checked too. The summed counters (Join Keys Looked Up is the dimension’s 3,000 rows however they were shared, and 18,000 over six rescans); rescans of the Gather as the inner side of a nested loop; a generic plan’s Params evaluated in every participant, a NULL one included; the fact filters' copy under a 64 kB memory limit with the answer unchanged; and a dirty heap before and after VACUUM. Since 2026-09-28 (“One copy per query”) it checks that the copy is shared: collected once a run over the six rescans of the Gather (Fact Filter Copies Shared 6 and one chunk each); a copy of some 90 kB that a hash join’s 64 kB would spill but the participants' shared memory - 64 kB for each of the three the plan was made for - holds, collected once in one chunk and not spilled; the same plan with max_parallel_workers = 0, whose leader runs it alone with a copy of its own that spills, and shares nothing; and a copy of some 240 kB over three chunks, past the shared memory, spilled once for the query, its containers read back from the chunks' files (Fact Filter Copy File Reads), with the answers of a semi join and of a grouped join the same as with the pushdown off. Its plans are ones every supported release makes alike (a parallel sequential scan of the dimension, the worker count from its parallel_workers); the expected output was checked on PostgreSQL 16 and 18. Section 9 of test/sql/countmultikey.sql (2026-09-27) does the same for the values every participant evaluates for itself: multi-key fact filters known only at run time (§17), exact, widened and every row, and IN lists of unknown length - a parameter, a NULL one, 1,500 values, IN ($1, $2) and a stable expression’s array - and checks serially that the 1,500-value list, too wide for the open budget at a work_mem of 64 kB, is collected once.

test/sql/fkjoin_nonunique.sql (2026-09-27), the forward semi join over a non-unique key, against the pushdown off: EXISTS and IN over a 900-row dimension of 400 keys, most of them two or three times, with fact filters of every kind (=, IN, a tsquery, an OR, IS NULL, IS NOT NULL) and none, dimension filters and none, a fact filter with no entry, a dimension filtered to nothing and one that leaves only NULL keys (lion_nj_val() pins 0); heavy duplication (3,000 rows over six keys, one without fact rows) and every row the same key (the answer pinned against that key’s own count); cross-type keys both ways and a text key; count(1), count of the fact’s key, and count(DISTINCT) of it, over nothing too, and count(*) beside it (declined before “Every aggregate over the node’s rows”); HAVING, ORDER BY and LIMIT; the counters (Join Keys Sorted without the NULL keys, Join Keys Looked Up the distinct keys, Join Keys Without Entry); the declines (a fact column in the GROUP BY, another aggregate, a second correlation, the anti join the other way round, a fact filter the posting sets cannot answer); generic plans with Params on both sides, NULL ones included; correlated subqueries that rescan the node, its child and its sort with a new dimension filter and with a new fact filter; the model’s choice, including a 60,000-row dimension it refuses unless its filter leaves few keys that are fk values; and a dirty heap before and after VACUUM. Its parallel section runs every query as planned, serially and with the pushdown off (lion_nj_par(), as lion_pj() does): EXPLAIN of the Gather over the parallel node, the leader taking part and not, Join Keys Looked Up summed over the participants equal to the distinct keys and Join Keys Sorted to one participant’s, both over four rescans of the Gather as well, a generic plan, a volatile dimension filter that keeps the node serial, parallel_workers = 0 and debug_parallel_query. EXPLAIN prints the node and what is above it, and the dimension’s own plan as one line (lion_nj_explain()), because that plan is whatever each major makes of a small scan.

test/sql/fkjoin_walk.sql (2026-09-28), lookups in key order, against the pushdown off: a fact of 24,000 rows whose fk runs over 12,000 keys, every key twice in an order unrelated to the key’s, with NULL fks and keys without rows, under an fk index built at fillfactor 20 so that its directory has height 2 with some five hundred leaves; a 4,000-row dimension over every third key, some of them absent from the fact and ten past its range, and a 6,000-row dimension over 3,000 keys, duplicated, with NULL keys. Each answer is checked as lion_wj() checks it, which also says whether the plan walked: inner joins grouped by one and two dimension columns and ungrouped, with fact filters (=, IN, one with no entry) and dimension filters, HAVING and ORDER BY; the forward EXISTS over the unique key; count(DISTINCT) of the key and of a dimension column, and beside counts of join pairs (the counted rows of “Every aggregate over the node’s rows”, walked too); reverse semi and anti joins over both dimensions, duplicated and NULL keys among them, and an anti join whose fact filter selects nothing; the forward semi join over the non-unique key, counted over its distinct keys, and its count(DISTINCT), alone and beside count(*); and a dimension filtered to a few dozen keys, which the model looks up a row at a time. EXPLAIN shows Join Key Lookups: in index order for the walked plans and not for that one. The counters: Join Keys Looked Up and Without Entry (every key looked up, a duplicated one too), the distinct keys of the forward semi join, a walk that reads fewer directory pages than it has keys (a fact filter few keys have rows under, so the walk stays on each leaf from key to key) against the row-at-a-time plan’s three pages a key, and more than one batch at a work_mem of 256 kB, with the answers unchanged. Then correlated subqueries that rescan the node - its batches and its walk - with a new fact filter per outer row, generic plans with a Param fact filter (a NULL one included), parallel plans of each shape (lion_wj_par(), as lion_pj() does), and a dirty heap - deletes and updates on both sides, and fks moved to other keys - before and after VACUUM. The walk’s choices do not depend on work_mem above a few hundred kilobytes, and at the default the batches are one.

test/sql/fkjoin_perkey.sql (2026-09-28), where a key’s time goes, against the pushdown off: a fact of 120,000 rows whose fk runs over 60,000 keys, every key twice and half the heap apart, and a 20,000-row dimension whose lion-indexed region selects 5,000 keys, 1,250 of them without an entry; inner joins grouped and not, a semi and an anti join and a grouped join under another filter, each answer checked. The counters are read from EXPLAIN’s JSON (lion_pk_node()) and checked against the data and each other: the child’s rows and the keys looked up, 5,000 each, 1,250 without entry; two fk containers read a key at most; a copy seek a key container at most; the copy’s containers read at most the key containers plus the keys; the visibility map asked at most once a key container, and once a key for a semi join; no posting page read for INLINE sets. A count alone goes up as one row, and is estimated as one, where the same keys grouped hand up the 3,000 that join; either run pins the visibility map’s page once. The timings are there with TIMING and not without. A copy that spills at a work_mem of 64 kB, from a generic plan made at the default: spilled once, read back, its file reads at most the keys' containers and the copy containers read, and the temporary blocks the node read at most those reads and one - the count’s reads of a spilled copy are bounded by the key, not by the copy. And a parallel plan, whose counters are the participants' sums. Since “The copy, looked up by key”, a copy whose container keys are too far apart for the direct index - the first and the last thousand rows of a 60,000-row fact - against keys whose rows lie anywhere, so that most key containers find nothing of the copy at their key and the key is sought on: inner, grouped, semi and anti joins against the pushdown off, and the copy’s containers read fewer than the key’s. test/unit/container_test.c checks the AND of an ARRAY of one to three rows against every kind of container, and damaged ones.

test/sql/fkjoin_mixed.sql (2026-09-28), every aggregate over the node’s rows, against the pushdown off: a 3,000-row dimension with nullable int, int2 and text columns, a 60,000-row fact whose fk runs over 3,600 keys (with NULL fks, keys past the dimension’s and dimension keys without fact rows) and an int4 copy of it, and a 3,000-row dimension over 1,500 keys, duplicated, with NULL keys. lion_mj() reports the plan’s form - counted rows below a LionJoinAgg, rows as they stand, partial counts, or none. count(DISTINCT) beside count(*), count(1) and counts of either side of the key, the dimension rows through an inlined CTE, a cross-type key; grouped by columns and by an expression, HAVING on a count, on a DISTINCT count and on a count the target does not name, ORDER BY an aggregate and LIMIT, expressions of aggregates and one aggregate twice, GROUP BYs folded to a constant, over something and over nothing, and answers over nothing pinned (lion_mj_val(): 0 for the counts, NULL for a sum and a max); min and max of text, int8, int and int2 columns and of expressions, bool_and, bool_or, every, bit_and and bit_or, beside counts and alone (the rows then need no counts), over rows whose values are all NULL; count(col) of nullable columns and expressions, 0 where every value is NULL; sum of int4 and int2 columns and expressions, NULL where every value is NULL; semi and anti joins over unique and duplicated keys with NULL keys, their aggregates as they stand; the forward semi join over the non-unique key, its distinct keys counted. The declines: fact columns but the key, min of the fact’s key, FILTER, ORDER BY and DISTINCT in other aggregates, avg, sum of int8, bit_xor, array_agg, a volatile expression. EXPLAIN (VERBOSE) of the rewritten aggregates, grouped with HAVING under an ORDER BY, the fact’s key counted distinct and the forward semi join’s keys; a hashed Agg with no Sort where nothing is DISTINCT; a semi and an anti join’s aggregates as they stand; counts alone still partial. Containers Visited of counted rows equal to the partial counts', and above the rows alone’s existence tests. The cost model with nothing disabled: counted rows where the counts alone are pushed down, and not where they are not. Generic plans, NULL Params included; correlated subqueries that rescan the LionJoinAgg, the Agg and the node; parallel plans (lion_mj_par(), as lion_pj()), the Gather below the plain Agg, hashed and sorted, semi, anti and the forward semi join, Join Keys Looked Up summed over the participants, and the serial plan under debug_parallel_query. The aggregates themselves: 0 and NULL over no rows and over NULLs, “bigint out of range” past int8. A role without EXECUTE on lion_join_count() gets the ordinary plan with the same answer, while the rows alone, which need none, are still pushed down; one without USAGE on the extension’s schema (moved there for the test) gets the counted rows. And a dirty heap before and after VACUUM. Beside it, fkjoin_distinct, fkjoin_semi and fkjoin_nonunique answer the queries they used to decline, and fkjoin_walk walks counted rows in key order.

test/sql/fkjoin_bykey.sql (2026-09-28), grouped by the join key, against the pushdown off. The data is fkjoin_mixed.sql’s, with a unique text key beside the dimension’s int8 one and its fact side, plus a numeric dimension and fact whose equal keys are spelled 1.00 and 1.0. lion_bk() reports the plan’s form, and how the Agg nearest above the node groups: hashed, sorted, each row a group, or no groups. The cases are:

  • the CTE form as written (GROUP BY f.fk) and with GROUP BY a.pk, under other filters, with other aggregates at the outer level, and over nothing;
  • GROUP BY f.fk with every count, fact filters of each kind, expressions the target computes from the grouped column, ORDER BY and LIMIT, the text key, and a dimension column beside the key in either order;
  • GROUP BY d.pk, both keys in either order, and the dimension’s key over an int4 fk;
  • HAVING on a count, and on the key alone, which becomes a range of the fk index;
  • the aggregates of “Every aggregate over the node’s rows” grouped by either key (count(*) with max(d.attr), counts of columns, sums, min, bool_or, bit_and, count(DISTINCT) of a dimension column and of the key), and one CTE level above them;
  • the forward semi join’s distinct keys grouped, and a reverse semi join whose key an index proves unique; an anti join’s key, and a non-unique key, are hashed;
  • the declines: an int4 fk against the int8 key, alone and in the CTE form; the numeric key, whose answer (lion_bk_val()) keeps the fact’s spelling where GROUP BY d.pk prints the dimension’s (and where count(DISTINCT) of it is still taken); the key under "C"; GROUP BY f.fk + 1; other fact columns. Taken but hashed: another unique dimension column, the dimension’s key under "C", and d.pk + 0.

From EXPLAIN’s JSON (lion_bk_plan()): the per-row Agg starts when the node does, while a hashed Agg waits for the node’s last row; the per-row Agg’s cost is the same at 64 kB and at 64 MB of work_mem, where a real grouping of as many groups costs more at 64 kB; and counted rows with a DISTINCT have no Sort below them when grouped by the key, while grouped by a dimension column they have one. The estimate is the node’s rows in the CTE form, fewer under narrow fact filters, and within a factor of 1.5 of the actual rows either way. The cost model with nothing disabled takes the node for a few dimension rows grouped by either key and in the CTE form, and the hash join for all of them. Also covered: generic plans with NULL Params included, correlated subqueries, parallel plans (the Finalize GroupAggregate over a Gather, forward semi join included), and a dirty heap before and after VACUUM. fkjoin_walk groups walked rows by either key, and the forward semi join’s walked distinct keys by the fact’s.

test/sql/fkjoin_partition.sql (2026-09-28), a partitioned fact table: 60,000 rows LIST-partitioned by kind, which no lion index covers - one list sub-partitioned by year on ts, one alone, two kinds together in the one partition with a lion index on kind, and a default partition that takes the rest and NULL - with a partitioned lion index on (tags, ts, fk) and one on another column; fk skewed (a few keys with many rows), scattered, NULL on some rows and past the dimension’s keys on others; a 3,000-row dimension and one of 3,000 rows over 1,500 keys, duplicated, with NULL keys. lion_xj() runs every query as planned (in parallel wherever the plan is), serially and with the pushdown off, and compares both of the first two with the third: reverse semi and anti joins over one list partition, a sub-partitioned one, several and all of them, duplicated and NULL keys among them; inner joins, ungrouped, grouped by one and two columns and an expression, with HAVING, ORDER BY and LIMIT; count(DISTINCT) of dimension columns and of the key, and beside count(*) under a LionJoinAgg; the forward semi join over the non-unique key and its count(DISTINCT); no fact filter left in any partition (the key clause implied everywhere), a filter that selects nothing in some partitions or all; a filter per kind ORed, which each partition narrows to the arm its bounds leave it (§16, “OR arms the partition bounds refute”) - semi, anti and grouped inner joins, the mixed aggregates, a range beside it, and the default partition declining the arm it does not decide; and a partition only run-time pruning would remove. EXPLAIN of each shape (the dimension’s plan as one line) shows Partitions, Lion Indexes and Implied by Partition Bounds - a list one partition answers from its own index on kind, a range on ts one sub-partition’s bound implies. The counters: Join Keys Looked Up a key per partition that has a turn for an inner join, fewer for a semi join, and none in a partition whose filters select nothing; more than one batch at a work_mem of 64 kB with the answers unchanged; Fact Filter Copies Shared one per partition, and one per partition and run over three rescans of the Gather. A generic plan with a Param fact filter (a NULL one too), one whose OR has an arm on an IN list over a stable expression - planned under one value, run under another, and no arm refuted or implied by the first - and a correlated subquery that rescans the node with a new fact filter; the declines (a partition without a lion index on fk, a key clause the default partition’s bound does not imply and no index of it answers, alone and among others, an old-style inheritance parent, partitionwise aggregation turned on); the model’s choice with nothing disabled (the node for a few dimension rows against filters that leave many fact rows, and not for a filter that leaves few); and a dirty heap - deletes and updates in three partitions, an fk moved - before and after VACUUM.

test/sql/fkjoin_dimjoin.sql (2026-09-28), a dimension that is a join, against the pushdown off (lion_gj(), as lion_xj() does: as planned - in parallel where the plan says so - serially, and with the pushdown off and sequential scans only, and where asked which table was the fact or whether the plan was parallel): a 3,000-row dimension with a primary key and a lion index on its filter columns, a 3,000-row dimension over 1,500 keys, duplicated, with NULL keys; f1, 40,000 rows LIST-partitioned by kind with one list sub-partitioned by range on ts, whose ts lies relative to now() so that ts >= now() - interval '120 days' - a stable range - keeps the same rows on any day; f2, 30,000 rows of a plain table with a lion index over a tsvector and fk; and f3, a third fact - every column drawn from a hash of its own, so that no filter lines up with a key by the arithmetic of the row numbers and every answer has rows to count (the answers of (A) and (B) are pinned). EXPLAIN (with the dimension’s plan as one line saying it is a join) of (A) with each fact the only one there can be, with the filter per kind ORed and the stable range when f1 is the fact; (B) over the unique key, as the inner join it is, and over the non-unique dimension, its keys made distinct; an anti join inside the dimension and one as the node’s; an inner join grouped by a dimension column and by the key; the counted rows below a LionJoinAgg. The answers of each shape, and of (A) with both facts, three facts, grouped, with HAVING, ORDER BY and LIMIT, count(DISTINCT) and the aggregates of the node’s rows, GROUP BY either key, (B) grouped by the fact’s key and counted distinct, anti joins both ways, inner joins with counted rows, a partitioned fact against a dimension anti-joined to f2, an inner join and a nested EXISTS inside an EXISTS of the dimension; (A) and (B) pinned by value. NULL and duplicated keys: reverse semi and anti joins over the non-unique dimension, joined, grouped by its key (hashed, not a group a row); an inner join over it declined; the forward semi join over it, its keys made distinct; a dimension filtered to nothing, fact filters that select nothing. The counters: Join Child Rows and Join Keys Looked Up equal to the dimension rows the dimension’s own semi join leaves, and the forward semi join’s keys looked up equal to its distinct keys. Parallel plans over the join rel’s partial path - reverse semi, inner, forward and anti joins, and rescans of the Gather - with hash joins allowed and nested loops and merge joins not: with every join method disabled, PostgreSQL 16 and 17 add disable_cost to the serial node’s price and to the parallel one’s, through their children, and add_path() takes the two for one price and keeps the serial path, which is parallel-safe. The forward semi join over the non-unique dimension run whole in every participant is therefore parallel from 18 and serial before, its answer checked either way and its keys looked up once each; a volatile (PARALLEL SAFE) qual of a table semi-joined inside the dimension keeps it serial on every major. Generic plans with Params in the fact’s, the dimension’s and the other fact’s filters, NULL ones included; correlated subqueries that rescan the node with a new filter of each. The declines: an inner join inside the dimension, an outer join and a PlaceHolderVar, a LATERAL reference, a second join clause, another fact’s EXISTS correlated to the fact, the fact’s EXISTS nested inside another table’s, a pseudoconstant qual. The cost model with nothing disabled: (B) over a few dimension rows is the node’s, and the whole dimension against both facts unfiltered two hashed semi joins. Which of two facts that qualify the node takes, and whether it runs in parallel, follows core’s price of the joined dimension’s plan, which differs between majors (right semi joins from 18, for one); the test prints either only where the query leaves no choice or a setting makes it. And a dirty heap - deletes, updates, fks moved, on every table - before and after VACUUM. fkjoin_semi.sql’s “three relations”, a dimension semi-joined to a second table, is pushed down now; beside it an inner join inside the dimension still declines.

test/sql/fkjoin_adaptive.sql (2026-09-29), probing plans that collect part way through: what “Probed, then collected” lists - the switch’s key in EXPLAIN ANALYZE, row at a time and walked, a copy that comes out empty, the knob off and a copy too large for its memory, rescans, a partitioned fact’s leaves, parallel plans and a dirty heap before and after VACUUM - each answer against the pushdown off.

test/sql/fkjoin_switch.sql (2026-09-30), a switch whose copy comes out empty, made part way through a batch: the upper node handing up a row per dimension row - anti, semi and inner joins grouped by a dimension column, the rows of an anti join below core’s Agg and the counted rows below a LionJoinAgg - and the count summed in the node; the semi and anti join paths over an outer side whose keys repeat and are sometimes NULL; each walked over one batch and over several at a work_mem of 64 kB and a row at a time; rescans (the upper node below a correlated subquery, the paths below a nested loop’s LATERAL), each run switching again; parallel plans of both; a partitioned fact; and the switch off. EXPLAIN (COSTS OFF) of the plans, and each answer against the pushdown off as multisets. Before join->batch.begun every anti join walked in key order - the upper node’s and the join path’s, over one batch and several, rescanned and in parallel - lost or repeated rows, ten answers of the file; the others passed then too, and hold the ways the fix did not mean to change.

test/sql/fkjoin_semipath.sql (2026-09-29), the semi and anti join as a join path, against the pushdown off (lion_sp(), as lion_gj() does: with core’s join methods disabled - so that the path is taken wherever it is offered - as planned, in parallel where the plan is, then serially, and with the pushdown off and sequential scans only, saying which of the two paths the plan took and, where asked, whether it was parallel and below the upper node). The data is fkjoin_dimjoin.sql’s kind: a 3,000-row dimension; an outer side of 3,000 rows over 1,500 keys, duplicated, with NULL keys; f1, 40,000 rows LIST-partitioned by kind with one list sub-partitioned by range on ts and ts relative to now(); f2, a plain fact with a tsvector, its fk and an int4 copy of it; and a fact whose fk index is built at fillfactor 20, a directory of height 2, with a dimension over every third key read through an index on another column. EXPLAIN (COSTS OFF) of each shape: semi and anti joins over the plain fact and the partitioned one (a stable range beside a filter per kind ORed), the IN spelling with a filter that only subtracts, an outer side that is an inner join (its plan as one line) and one that is itself a semi join path, the outer side’s order kept with no Sort, the rows feeding a Sort and - nothing disabled - a hash join, the upper node over the path for (A) and (B), and the parallel path below a Gather. The answers: semi and anti joins over both facts, fact filters of every kind, one that selects nothing (no row of the semi join, every row of the anti join), the int8 key against the int4 fk; outer sides that are an inner join, the path’s own rows, an anti join over a semi join and an outer join whose NULL-extended rows have a NULL key; a system column, a whole row and expressions, computed by the path’s projection; two facts on one key, whose class makes the join rel carry the first fact’s column, an int8 fk and an int4 one; NULL keys (the semi join’s count pinned without them, the anti join’s with them) and duplicated keys, each outer row emitted once per its own occurrence (how many keys are emitted how many times, against the outer side’s own counts); the rows feeding a sort, core’s Agg (avg, string_agg) and a hash join; the composition, (A) and (B) pinned by value. Order (lion_sp_seq(), the rows in the order they come against the pushdown off, and whether a Sort was needed): descending through a backward index scan and by another column first through an index on it, the batch sorted by key and put back, over many batches at a work_mem of 64 kB, and over the partitioned fact - each checked, by a build that skipped the put-back, to fail without it. The counters: every outer row a child row and every non-NULL key looked up, none looked up when a fact filter selects nothing, and a parallel plan’s sums. Rescans: the path as the inner side of a nested loop with no Materialize, serially and as a Gather; a correlated subplan with a new outer-side filter and a new fact filter per outer row, a NULL one among them; generic plans with Params on both sides, NULL ones included, and a generic plan’s kind = $1, which declines. The declines: a second correlation, an outer-side qual inside NOT EXISTS (and inside EXISTS, where it is the outer side’s and taken), a fact filter no posting set answers, NOT IN over a nullable fk (and over two sides never NULL, which is the path’s on 19 and core’s SubPlan before it, core’s answer either way), the anti join of LEFT JOIN ... IS NULL, a PlaceHolderVar among the join rel’s rows, the fact joined outer; a LATERAL outer side taken above what it references; pg_lion.enable_semijoin and pg_lion.enable_count_pushdown off. The cost model with nothing disabled: the path for a few outer rows against a filter that leaves most fact rows, core’s join for all of them against one that leaves few, and the upper node for a count of the first. The startup (lion_sp_start(), from EXPLAIN’s JSON costs): nearly the whole of the path’s cost past its child’s startup for one batch, in key order and over the partitioned fact; about one batch’s share of it at a work_mem of 64 kB, against the batches the executor made; next to nothing a row at a time; and an ORDER BY ... LIMIT 5 over the path, a Sort above it over one batch and the outer side’s order with no Sort over several. And a dirty heap - deletes, updates, fks moved, NULL keys made, on every table - before and after VACUUM.

test/sql/countanti.sql (2026-10-20), the anti join over a small inner side, every answer against the pushdown off with sequential scans only, as a multiset both ways round (lca()), over a 20,000-row table whose key has 400 values of some 50 rows each and NULLs: a WHERE of a half and of a tenth; NULL keys on both sides, duplicated inner keys, inner keys of no row and of rows the WHERE leaves out, an empty inner side, a WHERE with nothing left after the subtraction; the two-clause form over one column with inner rows whose two values differ; a clause of both sides beside the key, a clause of the counted table alone inside the NOT EXISTS, a key of an inner expression, a cross-type key, numeric keys equal in other spellings (one key, looked up once); an inner side that is a join; residual clauses beside the count’s; count(1), count of a key declared NOT NULL and of a nullable one, count(DISTINCT) of the primary key; GROUP BY of one and two columns, a group that drops to nothing and one the planner folded, HAVING; a two-column B-tree with the key first, by one scan key and by two, and one whose first column is no key’s (declined); B-trees that name the key’s column twice, alone and before a column of another key, with nothing disabled and with the pushdown off; a dirty heap - deletes, updates of the key, of the WHERE’s column and of a column no index has, new rows, the inner side updated - before and after VACUUM and inside the transaction that made it; generic plans with Params in the WHERE, in the inner side’s filter and in a key’s inner expression; a correlated subquery that rescans the node and its child; parallel - EXPLAIN of the Gather over the parallel node, the counters of the one participant that subtracts equal to the serial node’s, a correlated rescan, the leader not taking part, and the GROUP BY left serial; the counters (lca_counters()), and EXPLAIN of the node, a GROUP BY, an inner join as the child, and of ANALYZE. Beside a count by cells: a GROUP BY of a CASE over predicates and of views' tier and score over them, a WHERE over them, and both; predicates with NULL truths, ten IN lists of 1,025 outcomes, the correlation by two of the inner side’s columns, inner keys of no row, NULL and duplicated, an empty inner side, a clause of both sides, a key of an inner expression, numeric keys, an inner side that is a join, HAVING, a tally spilled at a work_mem of 64 kB, generic plans and a correlated rescan, a dirty heap, and parallel, the rows subtracted every participant’s together the serial node’s, with the leader and without it. The order the clauses are tested in: a join clause that divides by a column of t beside a WHERE that keeps that column’s zero out - with nothing disabled, by the key of a row and of some 50, grouped, beside residual clauses, count(col), a count by cells and in parallel - and a WHERE written to divide by a column before the clause that tests it, ungrouped and grouped; and EXPLAIN of the join clauses in core’s order. Past work_mem and hash_mem, both 64 kB, over a 60,000-row table and an inner side of every one of its 12,000 keys and 5,000 more rows of one key: the child’s rows sorted on disk, and an inner side a plpgsql function declared ROWS 10 returns, which the model takes the node for with nothing disabled; a key’s 5,000 inner rows on disk, each of its rows tested against them and one passing none; the losses of 6,000 groups held with the count’s rows and merged - groups that come to nothing among them - with a WHERE, by two columns, by a column with NULLs and by a text, with a HAVING, and in the order the plan claims for an ORDER BY, no Sort above the node (lca_ordered(), the rows in the order they come against core’s); a correlated rescan, ungrouped and grouped; the losses of 2,500 cells of a count by cells, with a HAVING; and in parallel the ungrouped count, the inner rows and the count by cells, whose rows taken away go to disk by range, the rows subtracted every participant’s together the serial node’s - each with Join Spilled (lca_spilled()), and nothing spilled at the default work_mem. The declines: no B-tree on the key, LEFT JOIN ... IS NULL, a NOT IN over a nullable column (and over two sides never NULL, which is the node’s on 19 and core’s SubPlan before it, core’s answer either way), a WHERE clause nothing answers, a NOT EXISTS beside a second table in the FROM, a correlation by no equality, RLS on the counted table, a partitioned table, pg_lion.enable_count_anti off; and the cost model with nothing disabled: the node for a few keys of a row or so each, core’s plan for an inner side of 10,000 keys. fkjoin_semi.sql and fkjoin_parallel.sql turn it off, so that their reverse anti joins stay the FK-side join’s plans they pin. priv_decode.sql (test/modules/lion_hooktest) refuses an ANTI member without its child, one of no key, one of fewer join clauses than keys, ones whose count of join clauses, of restrictions or of the three together is more than a List holds, a scan key without its < and a grouping equality without its <, an inner column without an ANTI member, and one past the attribute numbers there are.

28. Range predicates over the sorted directory (v1, implemented)

Formerly §12 item 5. k < v, k <= v, k >= v, k > v and BETWEEN on a scalar lion column, answered by the bitmap scan and by the count pushdown. What §21 made possible is the order: within one key column and one kind the directory is sorted by proc 4 first ((attno, kind, proc 4, hash, bytes)), so the entries a range selects are one contiguous run of VALUE entries, and a range is a descent to where that run begins and a leaf walk to where it ends.

Strategies and opclasses

Four new strategy numbers, in btree’s order and after the five §17 uses: 6 <, 7 <=, 8 >=, 9 > (LION_STRAT_LT .. LION_STRAT_GT; amstrategies 5 -> 9). They are added in place - 0.1 is unreleased - to every SCALAR class of the install script that has proc 4, and to citext_ops:

  • integer_ops and float_ops get the four operators for every type pair they already have an equality and a cross-type proc 4 for (<(int4, int8) beside btint48cmp, and so on), so int4col < 5000000000::int8 descends exactly as int4col = 5::int8 does;
  • every other ordered class gets its own type’s four: oid, bool, char, name, text (varchar through binary coercion, as for =), bpchar, bytea, uuid, date, time, timetz, timestamp, timestamptz, interval, numeric, macaddr, macaddr8, inet, jsonb, pg_lsn, xid8, tid, enum; and citext. enum_ops is declared FOR TYPE anyenum, so a bound of the column’s own enum type is the class’s own type, not a cross-type search (lion_range_add(), lion_match_index()); the count pushdown of an enum EQUALITY still does not make that step and is declined, as it was before this section;
  • xid_ops and cid_ops get none: they have no proc 4, their directory is in hash order and a range has no run to walk. array_ops and tsvector_ops get none: their entries are extracted keys, not column values, and tags < '{a}' compares whole ARRAYS.

The script rules of 7c6e439 hold: the operators are core’s, which an unqualified operator name in an extension script finds in pg_catalog first; citext’s are named @extschema:citext@.< and so on, never through the target schema (test/sql/hardening.sql plants a < there and shows it is not the one chosen). Cross-type date/timestamp comparisons (datecol < now()) came later, as a family of their own (below, “Cross-type date and timestamp”): a family gains a type pair whole or not at all, equality included.

lionvalidate() accepts strategies 6..9 in a scalar family only for a type pair that ALSO has proc 4 - the walk needs the comparison, and an operator the index could only answer by testing every entry is a promise the AM should not make - and never in a multi-key family. Equality stays required for every type pair the family knows, so a range-only pair is still refused.

What k op v means to the index is what it means to btree: the rows whose key compares so under the family’s proc 4 for (key type, v’s type), under the INDEX’s collation. For these families that comparison IS the operator’s (both are core’s btree support), which is the promise §21 already asks of proc 4 about strategy 1; an opclass author whose proc 4 disagrees with its < gets wrong answers from btree as well. Collation is the planner’s rule for equality, unchanged: the bitmap scan is only offered where IndexCollMatchesExprColl() holds, and the pushdown makes the same test in lion_match_index(). NULL keys never satisfy a range: the walk visits VALUE entries only, so the reserved NULL (and EMPTY) entries are never in one.

Cross-type date and timestamp (2026-10-28)

date_ops, timestamp_ops and timestamptz_ops are one family, datetime_ops, as in btree, with five of btree’s six cross-type pairs: =, the four comparisons and <>, and core’s own proc 4 for each (date_cmp_timestamp, timestamptz_cmp_timestamp and the rest). So the commonest dashboard filters reach the column’s index where they used to go to the heap: created_at >= current_date on a timestamptz column (a date bound), tz >= localtimestamp - interval '7 days' (a timestamp bound), ts >= current_date - 7 on a timestamp one, d = timestamp '...'.

The pair left out is a timestamp KEY against a timestamptz value (ts >= now()), because its comparison is not monotonic in the key. timestamp_cmp_timestamptz turns the key into an instant at the session’s TimeZone, and a local time a DST change skips is taken at the offset before the change: in New York 2026-03-08 02:30 is 07:30 UTC while 03:00 is 07:00 UTC, so the key that sorts first compares later. A descent by that comparison lands in the wrong place, and a walk that ends at the first key past a bound stops before keys that still match; with the pair in the family, an equality probe at 03:00-04 found one of the two matching rows. btree’s datetime_ops has the pair and the same fault (PostgreSQL’s B-tree gives wrong answers on the same data), but lion’s rule is the sequential scan’s answer, so the clause stays a filter. The other direction is safe: a timestamptz key against a timestamp value converts the VALUE once, and the keys are instants in order already; a date converts to the instant of its first moment, which never goes backwards from one day to the next. Keeping the pair would need a lookup that tests every key within the largest UTC offset (about 16 hours) of the value with the operator itself, and trusts the order only outside that window; that is not built. The index compares a key with the value exactly as the operator does, since proc 4 IS the operator’s comparison; where one side is timestamptz that comparison reads the session’s TimeZone, which is why those operators are STABLE, and a scan evaluates them at run time as btree’s does (test/sql/datetime_crosstype.sql shows one query answering differently under two zones, the same three ways: pushdown, bitmap scan, no index).

What the integer and float families have and this one does not is a compatible HASH: a date and the timestamp at its midnight are equal and hash apart (hashdate of a day number, timestamp_hash of microseconds), and a timestamp and a timestamptz are equal only at a time zone the hash cannot know. §21’s directory orders VALUE entries by (proc 4, hash, bytes), and a lookup used to place a cross-type value by both terms, the hash being the search type’s own on the promise that a family’s types hash compatibly. That promise is now asked only of an UNORDERED column (§21, “Cross-type searches”): a cross-type search of an ORDERED column leaves the hash term out (LionSearchKey.nohash, set by lion_probe_search_key() whenever the probe has the family’s cross-type proc 4), so the key stands before the whole run of entries proc 4 ties with it, and the run scan’s equality decides among them - one entry, for these types, whose comparison never ties two different values. Every path that locates a cross-type value goes through that one search key (the single lookup, the batched list walk, the bitmap and plain scans, the joins' key lookups, the SQL counts), so none of them can find a hash where another finds none. A date, timestamp or timestamptz column is always ordered - its proc 4 is the type’s default btree comparison, which a build can always reproduce (§21’s rule 1) - so the unordered route never meets this family.

Bit slices (§51, §53) still answer only a bound of the column’s own type (lion_slice_range_type_ok()): a cross-type bound on a sliced column is a range of entries or summaries, as on any other column, and the slices still answer the aggregates over it.

The bounded walk (LionRange, lion_range.c)

A column’s range clauses become one LionRange, resolved once per scan (per partition, §16):

  • Each bound’s comparison comes from §21’s probe resolution, lion_probe_init() for the bound’s type, so the walk and the lookups cannot disagree about the order: the column’s own proc 4 for its own type or a binary coercion to it, the family’s cross-type proc 4 otherwise. A bound for which that resolution finds no comparison - an UNORDERED column, or a family with a cross-type < and no cross-type proc 4, which the validator refuses but a hand-edited catalogue could hold - is tested with the OPERATOR itself instead, entry by entry over the column’s whole run: correct and linear, and never what a shipped class does - but it is what a user class whose proc 4 has a SQL-standard body gets, since a build lays such a column out in hash order (§21). Whether the column is ordered is the RECORDED order of §21 (“The order is the index’s”), never the catalog’s of the moment; §21 also says why that record is a best-effort guard against a comparison changed after the build, as it is for btree, and not a guarantee. Like every comparison of a descent, these calls run with a leaf share-locked; §21 says why a SQL-language operator that reads the same index can hang its own backend there, and why that is a superuser’s concern.
  • Positioning. When every bound has a comparison and the column is ordered, the walk descends to the first entry of the column that does not sort below the TIGHTEST lower bound: a search key of kind VALUE with that bound as its key, hash 0 and no stored form, which compares as the smallest member of its run (§21), so the descent lands on the first entry whose proc 4 is at or above the bound. With more than one lower bound it descends once per bound and keeps the landing furthest right (lion_range_side_leaf(), since §32; it used to descend to the first one and skip the entries below the others). Without a lower bound it starts where the column starts. Either way the scan’s resume position is the column’s (attno, MINF), so the entries of an earlier column on the landing leaf are passed by the ordinary resume comparison and the bound needs no special case there.
  • The test per entry. A reserved entry is skipped. An entry failing a LOWER bound is skipped - the order leads with proc 4, so those are a prefix of what is walked: the landing leaf’s entries below the bound, and those below any lower bound other than the one descended with. The first entry failing an UPPER bound ends the walk. With operator tests nothing ends it early.
  • Several bounds on one column are ANDed into one walk: BETWEEN a AND b is k >= a AND k <= b, one descent to a and a walk that stops past b. An empty range (k > 5 AND k < 5), an inverted one (BETWEEN 9 AND 3) and bounds outside the key domain need no special case: the walk starts past its end, or runs off the column, or the cross-type comparison says every key is below (btint48cmp compares an int4 key with an int8 bound exactly, so int4col < 5000000000 is every row and int4col > 5000000000 none, as with btree). NaN sorts above every float in btfloat8cmp, as float8lt orders it, and -0 ties with 0, whose entry is one (hashfloat8 agrees). A NULL bound - a Param that came out NULL - selects nothing, every operator involved being strict.

Resuming, splits, deletes: the walk IS §21’s entry scan (lion_entry_scan_begin_range()), which resumes at “the first key above the last one returned” and so already survives splits and VACUUM’s entry deletion; what differs is only the leaf it reads first, which a descent chose and which may have split since - its entries then moved right, where the walk goes next, and a directory leaf is never unlinked (§21). The bitmap walk is lion_emit_all_keys()’s copy-a-leaf walk with a start and a stop. So there is no new concurrency argument. An isolation spec is added all the same (count_range_vacuum_race.spec, below), because the per-entry §9 argument of the count is the thing a reviewer will want to see exercised under a range.

One read per leaf (2026-09-27)

A benchmark (2026-09-26) showed a range reading the directory once or twice PER KEY: 4,012 directory pages for 2,000 keys and 1.93 million for 1.91 million, and the same ratio on a very large table. There was no second descent; the entry scan re-read the leaf. lion_entry_scan_next() gave its lock up after every entry it returned and came back for the next one by reading the same leaf again and binary-searching it for “the first key above the last one returned” (§21). Once the leaf ran out it read it one more time, found nothing after the last key, and only then followed the right link. A walk over n entries on m leaves therefore read n + m leaves. That is two reads per key when an entry fills a leaf (2,000 keys of a thousand-row column whose INLINE entries are 3.8 kB each) and one when a leaf holds a hundred and fifty of them (a unique timestamp).

The walk now reads a leaf ONCE (lion_entry_scan_fill()). Under one share lock it:

  • finds where to resume on the leaf, which is still the first key above the last one examined;
  • copies every entry the walk selects into a private batch, which is at most a block’s bytes, since that is where they came from;
  • remembers the last entry it examined as the next resume key;
  • and remembers the leaf’s right link as the next leaf to read, unless the walk ended on this leaf.

It then gives the lock up and hands the copies out one by one. The next leaf read goes to that right link without reading this leaf again. A walk over n entries on m leaves reads m leaves: 1,000 reads for 1,000 such entries (the thousand-row column, one per leaf) and 15,820 for 1,913,601 (the unique timestamp).

Why the copies may outlive the lock is §21’s argument, applied per leaf instead of per entry:

  • A split moves entries only rightwards, onto a new page linked immediately right of the one it splits. The entries the leaf held after the resume key when it was read are in the batch. A split after that moves them, and entries inserted since, to a page between this leaf and the right sibling the batch recorded, and the walk never reads that page. The inserted entries hold only rows no older snapshot can see (§29.5 case 3), which is the freedom the walk has always had behind its position. If the recorded right sibling splits before the walk reads it, the high key and the resume key route the walk right as before.
  • VACUUM deletes an entry only once its set is empty. A CHAIN copy whose set has been freed reads as empty through the owner check of §18.
  • An INLINE copy is counted against the visibility map, so §9 wants the leaf it came from pinned from the copy until the count. The scan keeps that pin (batchbuf) while the batch holds an INLINE copy still to be handed out. Every INLINE set handed out takes a pin of its own, and the last one takes the scan’s - except in the sums over a range (lion_sum_walk() and the range taken as a source), which take a whole batch, count it and release every set of it before the next leaf is read: there the scan’s one pin covers all of the batch’s INLINE sets and is dropped at that read (lion_entry_scan_share_pin()), because a few hundred pins of one buffer cost the resource owner a search each when they are released. VACUUM’s cleanup lock on that leaf therefore waits as it waited for the one set’s pin before, only for a batch’s length. A batch of CHAIN copies pins nothing, which vacuum_entry_delete.spec relies on: it parks a walk and needs VACUUM to get through.

A walk that returns rows pauses. A GROUP BY hands a row to the executor per group, and a cursor can hold that row for ever, so the node calls lion_entry_scan_pause() whenever it returns a row. A batch that still holds INLINE copies drops them with their pin, and the next call reads the leaf again from the last entry handed out. A GROUP BY over INLINE entries therefore still reads a leaf per group, as it always did; it emits a row per group anyway. A batch of CHAIN copies survives the pause. The node’s located sets give their pins up at the same point (§15, “Paused and finished counts”). The sums never pause: the sum over a range, the sum-over-all of §14, the count(DISTINCT) walk of §26 and lion_index_count_group_stats() each walk to the end within one call. §28’s pinned-GROUP-BY test (range.sql §14, at most one leaf pinned) holds as before.

Two injection points park a walk with no lock held: lion-entry-scan-resumed between the first and second entry handed out of a leaf, as before, and lion-entry-scan-leaf before every leaf read but the first, in a counting walk as in a fetching one. test/isolation/count_range_split_race.spec parks a range sum at each point, splits every leaf of the column under it with 2,000 inserted keys that it then rolls back, and gets the exact count.

Bitmap scans (liongetbitmap())

A scan key with strategy 6..9 on a scalar column is a RANGE key, and every range key of a column is consumed by ONE walk, so BETWEEN needs no recheck. A column with an equality, a list or IS NULL as well answers those (§5 SCAN step 5) and leaves its range keys to the heap recheck: they can only shrink the sets, one bound per row, where the walk would read every entry of the range.

The walk emits entry by entry: an INLINE payload from the leaf’s private copy, a posting tree through lion_emit_chain() one page at a time, with nothing pinned between entries. A range of any width therefore holds at most one index pin at a time and draws nothing from §15’s pin budget - the bitmap scan needs no §9 interlock at all, since the executor visits every TID it emits - and its memory is the TIDBitmap’s own work_mem budget, which goes lossy rather than growing.

On a column with SUMMARY posting sets (§32) the walk is the summed walk of a count (lion_entry_scan_begin_summed(), from lion_walk_begin()): the keys of the buckets at the range’s two ends one by one, and every bucket the range covers whole as its summary - pairwise disjoint sets that together are exactly the range’s rows, by §32’s argument (“Readers”). Whoever takes an entry’s posting set cannot tell one from the other, so the bitmap scan emits the same TIDs from a few thousand entries where a near-unique column’s range has a million keys. The entries are the entry scan’s copies, handed out with no pin (lion_entry_scan_next_copy()); the plain scan’s walks (§29.3) take them the same way, except one that must keep its pins (a non-MVCC snapshot, §29.5), which walks the keys. (Deviation from §32 as first written, which left summaries to the counts and walked the keys here: a wide range of a timestamp was a walk of every key it holds, summaries or not.)

k < ANY (array) arrives as an array key with a range strategy (amsearcharray is on for the whole family); only ANY is ever an index qual. The union of k < e over the elements is k < max(e), so it is ONE walk to the widest element - the largest for < and <=, the smallest for >= and >, found with the probe’s comparison of the array’s type (§21) - and only a column with no such comparison walks once per element. NULL elements are skipped (a strict comparison with NULL is never true, and neither is ANY of NULLs and falses); an empty or all-NULL array selects nothing. (Deviation from the first version, which walked every element into the same bitmap: 200 near-equal bounds over a million-row unique column emitted ten million TIDs, 1.4 s against 20 ms (2026-09-24 review). One walk now emits the 432,200 matching rows once, at the cost of the single widest bound, 108 ms on the assert build.) lioncostestimate() charges such a column the entries of that one walk, as it does a plain range. The array’s length is a parameter’s, without a cap: the search for the widest element answers a cancel every thousand elements, and beside other columns (lion_scan_col_ranges(), below and §29.3) the scan makes the one range it walks. (Until 2026-10-18 it made room for a range per element and used the first, 48 bytes each: “invalid memory alloc request size 1104000048” at 23 million elements, where the ordinary plan answers. A column without the comparison still holds a range per element, in a huge allocation.)

On a MULTICOLUMN index (§24) a range column cannot be a node of the set tree - its answer is a union of an unbounded number of entries - so its WALK is intersected with the other columns' answer a container at a time (lion_emit_intersect()), exactly:

  • The other columns' set trees are ANDed into ONE stream of containers in ascending container key, planned against work_mem as lion_sets_iterate() plans it (§15, “Bounded cursors”), holding no pin between two containers (lion_stream_begin_bounded()).
  • The stream is taken a WINDOW at a time: its next containers, copied, until they take max(pg_lion.scan_window_floor, work_mem), the plain scan’s window memory (§29.3). Each range column is then walked once for the window, every entry sought to the window’s first container key and read up to its last - an INLINE entry from the walk’s copy of the leaf, item by item, a posting tree through a stream - and the window’s position found by binary search from where the entry has got to, so a window of any size costs a walk and not a walk per container.
  • The LAST range column’s walk ANDs each of its containers with the window’s container at that key and adds what is left to the caller’s bitmap at once - a bitmap is a set, so it does not matter which entry a TID came from, and the entries of one scalar column are disjoint anyway. A range column before it ORs what its walk finds into a bitset image per window container instead, and the window’s containers are cut down to that before the next column’s walk; those windows count the image a container may need against the memory as well.
  • With nothing beside the ranges that the sets can express, the first range column is COLLECTED into a set of its own (lion_range_collect(), summaries and all, into a temporary file past the window’s memory, §32) and stands for the other columns; a range left on its own is walked straight into the bitmap, rechecked for the qual that was dropped. A column that cannot order the elements of col < ANY (array) has a range per element, and their union is collected as the one set (lion_ranges_collect()): every window of the collection walks each range into the one union, so the ranges share the window’s memory and a spill’s one file. (Until 2026-10-18 each range was a set of its own, collected in all of the window’s memory and ORed in the tree: as many windows' memory as the array had elements, and a temporary file for each that spilled.)
  • A long IN list among the other columns' quals is located a batch at a time (§15, “A list too long to locate at once”): each batch’s union, ANDed with the other trees, is a stream of its own, taken window by window through the range walks into the same bitmap. The walks are made again for every batch.

A selective answer of the other columns is one window, and each range one walk - with summaries a walk of the range’s buckets. The result goes into the caller’s bitmap, which a BitmapOr above may share with its other arms, and the scan reports the TIDs it added: the intersection’s own count. k < ANY (array) is such a column too - its one walk to the widest element - although it arrives as an array key, as a list does. (Deviation from the first version, which sorted columns into set trees and walks by that rank alone: the array range went to the set trees, which cannot express it, and was dropped for the heap recheck while lioncostestimate() and the plain scan both counted it as answered - a < ANY ('{-40,-45}') AND b = 3 handed the heap every row of b = 3 to throw away (2026-09-25 review). test/sql/range.sql shows no row removed by the recheck now.)

(Deviation from the second version, which walked each range column into a TIDBitmap of its own, of work_mem, and intersected the bitmaps with tbm_intersect() - core’s BitmapAnd inside one scan. A range whose rows lie on more heap pages than a bitmap of work_mem holds exactly went LOSSY, and tbm_intersect() keeps every exact page of the other side that falls on a lossy page, only marked for recheck. A time window over rows stored in no time order has a row on nearly every page, so the range removed almost nothing: in a benchmark, an equality, a multi-key overlap and a time window on three columns of one index handed the heap about every row the first two selected, most of which the recheck then threw away, with many lossy pages - after walking the range key by key into its bitmap, its summaries unused. The scan reported the smaller input’s count as its own, which made it look as though the range had not been answered at all. Nothing else dropped it: a bound that is a STABLE expression arrives as a runtime key and is read at every rescan like a constant; neither the multi-key clause beside it nor the summaries changed the choice; and range keys are left to the recheck only on a column that has an equality or a list of its own. test/sql/rangebitmap.sql has the shape - a synthetic table, the bound a literal, a stable expression and a generic plan’s parameter, with and without summaries - at a work_mem that holds the answer exactly and the range alone lossily: no row removed by the recheck and no lossy page.)

amcostestimate

genericcostestimate() already charges what the walk reads in proportion to the selectivity - that share of the index’s pages at random_page_cost and of its tuples at cpu_index_tuple_cost - which is about the leaves and the posting pages in the range. On top of that the walk is charged per ENTRY: n_distinct(k) x selectivity of them, each a decode and one comparison per bound. On a low- or mid-cardinality column that term is small and the index is a fraction of a btree’s, so the lion bitmap scan wins; on a near-unique column every row is an entry, the index is larger than the btree (an entry header per row) and the entry term adds to the per-row charge, so btree - which also has a correlation to exploit, and a lion scan never has one - wins. test/sql/range.sql pins both choices. A bound past the histogram’s end is estimated with the column’s actual end, read from the directory (The endpoint probe, below).

A column with SUMMARIES walks fewer entries, and is charged them (lion_cost_walk_entries(), which lion_plain_walk_entries() shares): the keys of the buckets at the range’s two ends and one entry per bucket it covers whole, by the count’s model of a summed side (§32, “Costs”: a bucket holds max(summary_tids, rows per key) rows, or what the column’s first summaries hold on average when that is more than twice as many, and a run of n keys covers about n / keys per bucket - 1 buckets whole). The index is opened at plan time for that, as the count’s costing opens it; a hypothetical index is priced as walked. A plain scan that keeps its pins walks the keys all the same (§29.5) and is charged the summaries' walk: an index-only scan, which lion builds only for a query that needs no column (§29.9), and a non-MVCC one.

On a MULTICOLUMN path genericcostestimate() prorates by every column’s selectivity together, while the range column is walked whole whatever the others select; the postings of the range that the prorating leaves out are charged on top, and the entries once per window of the plain scan’s WINDOW shape (§29.11). The bitmap scan walks once per window of the same memory, its windows holding the other columns' containers at their own size, so it makes at most as many walks as it is charged. (Added 2026-09-25: before it, a BETWEEN 1 AND 900000 AND b = 5 over a million unique a was priced at two thirds of the sequential scan and ran 1.4 to 1.6 times as long.)

Count pushdown: a range BOUNDS the driver, it is never a source

The design question was whether a range should be a count SOURCE - the union of its entries, merged k-way like an IN list (§15) - or a restriction on the entry walk that DRIVES the count. It is the second, and the reason is §15’s own argument: the entries of one scalar column are disjoint, so the rows whose k is in a range are the disjoint union of the range’s entries, and

count(*) WHERE k IN range AND F  =  sum over the entries e in the range of |e ∩ F|

which is §14’s sum-over-all with the walk bounded, while GROUP BY k WHERE k IN range AND F is §10’s group walk bounded the same way. The union has no plan-time size (an IN list’s is capped at 1000 at plan time; a range has whatever entries the data holds), needs every entry located - a pin each under §15’s budget, NOPIN re-locates and all - and pays a k-way heap per count. The driver form holds ONE entry’s pin at a time plus the WHERE sources', which is §10’s GROUP BY and needs no new §9 argument: each entry is counted as a single-key count is, intersected with the WHERE sources, and the pin on the page its containers came from is kept until they have been through the visibility map. The WHERE sets are materialized on their second use as for any GROUP BY (§9), so a selective filter is probed in memory, entry after entry.

(2026-10-27, §53: on a column a lion index slices (§51) a range has a third form, whose work does not grow with it: the rows in range folded from the column’s bit slices at each container key of its existence slice E, collected into a copy that is a source, ANDed with E located with its pin. It is offered by a build of its own, priced, beside the walk and the summaries; a range from slices never drives the sum, and a range that bounds a GROUP BY’s or count(DISTINCT)’s walk of its own column is still a bound.)

Accepted (lion_try_count_path()), every other rule of §10, §14 and §16 unchanged:

  1. count(*), count(k) or count(col) under §14’s rules, WHERE <range on k> [AND ...] without a GROUP BY: the sum-over-all driver (LION_FLAG_SUMALL) with k as the driving column and the walk bounded (LION_FLAG_RANGE). count(k) is answerable because a range proves k non-NULL (the operators are strict). Clauses on other columns are the ordinary sources, an IS NOT NULL on another column included, and a k IS NOT NULL beside the range is dropped as §14 drops the driver’s own.
  2. GROUP BY k WHERE <range on k> [AND ...]: the group walk bounded. The value-representation contract of §10 applies to emitting k as it does without a range, and so do the pathkeys of §21
    • the walk is still in key order.
  3. count(DISTINCT k) WHERE <range on k> [AND ...] (§26 shape 1, bounded), and g, count(DISTINCT k) ... WHERE <range on g> GROUP BY g (§26 shape 2 with its OUTER walk bounded). A folded single group does not ask k’s NULL entry whether the group exists when the range is on k, because the range excludes exactly those rows.
  4. Partitioned tables (§16): shapes 1 and 2, with the range resolved against each partition’s own index, key column and heap numbering.
  5. Params as bounds (generic plans, nested-loop exec Params): evaluated at scan start with the other clause values (§10); a NULL one makes the count 0, and a GROUP BY empty.
  6. Any number of range clauses on the SAME column, combined into one walk. They are exempt from the one-positive-clause-per-column rule among themselves, and from nothing else.

Declined at first, and since §32 answered: a range on a column that does not drive (g, count(*) ... WHERE <range on k> GROUP BY g, count(DISTINCT g) WHERE <range on k>, a range beside a two-column GROUP BY, two range columns), a range under an OR (§19), a range beside =, IN or IS NULL on the same column, and a range as an FK-join fact filter (§27). Each needs the range as an intersected SOURCE; §32 (“A range as a source”) collects it into one private set once per relation - cheap with summaries - or, when it does not fit in a hash table’s memory, sums over its walk at every count. Without a GROUP BY the widest range drives the sum and the others are sources; a range on a column with a positive clause of its own is always a source.

Still declined:

  • min(k)/max(k) (§23). Core’s own min/max optimisation (planagg.c) needs an ORDERED index scan - amcanorder and amgettuple - which lion does not have. Doing it here means a new node shape that PRINTS a value (the value-representation contract), walks from either end to the first entry with a visible row (an existence test per entry, §26), and for max walks backwards, which §21’s leftlink allows on paper and nothing reads yet. It fits, but not cleanly enough to ride along with this.

Cost (lion_cost_count_rel()). (What follows is the model of the GROUP BY and count(DISTINCT) walks over a range. The SUM over a range is priced as it now runs, by lion_cost_range_sum(): see “The cost of a summed range” below.) The range clauses are not sources: no lookup, no chain and no container term of their own. They scale the DRIVER: the entry scan’s pages by the fraction of the column’s entries in range - the combined selectivity of the range clauses alone (clauselist_selectivity()), which is the right fraction for a column whose rows are spread evenly over its values and an overestimate of the walk for a skewed one - and the entries walked (numgroups of the sum-over-all and distinct walks) by the same fraction of n_distinct(k). Everything per entry is then priced as the unbounded walk prices it, plus a fixed cost per entry walked, LION_RANGE_ENTRY_COST (40 cpu_tuple_cost): the walk resumes at a key (a leaf read and a binary search), copies the entry out and sets up and tears down a merge for it. Measured on the assert build at 100k rows over a unique timestamp, 3601 entries took 6.7 ms summed and 5.2 ms grouped - 1.4 to 1.9 us each - against the btree index-only scan’s 0.64 ms for the same rows at 171 cost units; without the term both were chosen at about a tenth of that estimate, with it the near-unique column goes to btree and a range over a 200-value column (21 entries, 0.18 ms against btree’s 0.9) stays with the node. A count(DISTINCT) walk already pays §26’s per-test cost for the same work and does not pay this one too. (Deviation from this section’s first draft, which expected the unbounded walk’s per-entry terms to be enough; the unbounded group walk of §10 keeps its own calibration, which its own tests pin.)

The dirty heap is priced per entry too. §10 charges each dirty page ONCE per query, because the visibility cache answers every later visit - but the cache resolves a page on its SECOND visit, and a range walk counts entry by entry, each count flushing its own recheck batch, so a page holding rows of several entries is fetched for each of them until the cache has it. At five million rows with every heap page dirty, a 30-day range over randomly placed days rechecked 68,384 block visits for 37,133 pages (166 ms, against the bitmap heap scan’s 46 ms) and was chosen; the same range over days stored in order rechecked 581 (3.8 ms, against the btree’s 11.5) and was refused. So for a ranged walk the recheck pages are entries x the pages one entry's candidates lie on x dirtyfrac, the per-entry pages interpolated between one per row (values scattered) and the rows' share of the heap (values in heap order) by the squared correlation, as cost_index() does, and never more than twice §10’s figure, since the cache stops a page being fetched a third time while it has room. Both choices above are now the faster plan. The correlation’s square is harsh on NEARLY ordered data: a 100k-row column whose correlation updates had brought down to 0.96 was priced at 960 page visits against the 154 measured, and btree’s 1.7 ms was chosen over the node’s 0.55. That is the safe side of the error, and the §23 fix below - carrying the recheck batch across a sum’s entries, as §15’s disjoint sum does across a list - would make the question moot.

EXPLAIN prints the driver with its range in place of (k) / (all keys): Lion Indexes: ix_d (d >= '2024-01-01'::date AND d < '2024-02-01'::date), ix_s (s = 3) - a Param as $1 - and Group Key and Distinct Key as before. Directory Pages Read (§21) shows the walk is bounded.

Privileges (§9). The range operators' functions (int4lt, date_ge, …) are WHERE-clause functions, so lion_replaced_functions() already lists them in LION_PRIV_EXECUTE, and the node checks EXECUTE on them at startup as the ordinary plan’s scan does. test/sql/range.sql revokes int4lt from PUBLIC and gets core’s error from both plans.

Counting a walk (2026-09-27)

The sum over a range counted each entry on its own, with one lion_count_sources_cached() per entry. Each of those builds a memory context, the source trees, a plan and a cursor per set, and the visibility-map state, and tears them down again. On the assert build that was about 1.5 us per entry whatever the entry held, which is the whole cost of an entry of one or two rows: 1.91 million one-row timestamps took 2.9 s beside country = 'c7', and 2.1 s alone.

lion_sum_walk() now counts the entries of one directory LEAF together. They go into the driver’s slot as ONE source whose sets are DISJOINT: they are distinct entries of one scalar column, the argument §15 makes for an IN list and §14’s sum-over-all has always rested on. So the count of the batch is the count of their union ANDed with the WHERE sources, and the sum over the leaves is the sum over the entries. lion_count_sources_cached() then treats the batch as it treats an IN list: summed set by set in one shared context when it is the only positive source and summing is cheaper (lion_sum_is_cheaper()), and merged as a k-way union beside the WHERE sources otherwise. A dirty page with rows of several entries of the leaf is then rechecked once for the leaf instead of once per entry, which is the §23 item (“carry the recheck batch across a sum’s entries”) done a leaf at a time. §9 needs no new argument. Each INLINE set of the batch holds the pin of its leaf until the batch has been counted, so the batch is a positive source that holds a pin at every container key. That lets the WHERE sources be materialized on their second use, exactly as a GROUP BY’s are. A walk of any width holds one directory leaf pinned.

The union pays only for SMALL entries. LION_SUM_UNION_MAX_ITEMS (4) containers per entry on average is the cut. Above it each entry is counted on its own, as before, because the merge folds fewer than LION_OR_BITSET_MIN containers at a key pairwise (a lion_container_or() and an optimize per pair). For entries of 230 containers each, that fold cost more than the per-count set-up it saved. Measured with 1,000 such entries ANDed with country = 'c7': 39 ms one by one against 165 ms as unions of eight leaves' worth. For one-row entries the union is what wins: 86,399 of them took 40 ms a leaf at a time, against 148 ms one by one on the base build.

The complement (2026-09-27)

An ordered range selects one contiguous run of its column’s VALUE entries. The entries it does not select are the run BELOW it, whose entries fail a lower bound and no upper one, and the run ABOVE it, whose entries fail an upper bound. Every row with a value of k is under exactly one entry of k, and the WHERE sources F pick rows, so

|INSIDE ∩ F|  =  |F − NULL(k)|  −  |BELOW ∩ F|  −  |ABOVE ∩ F|

|F − NULL(k)| is one merge: F with k’s reserved NULL entry subtracted, which is the negated source k IS NOT NULL has always been (§14). So a range can be answered from either side, and lion_sumall_relation() takes the smaller:

  • full domain: BELOW and ABOVE are empty. The range covers every key the column has, it is exactly k IS NOT NULL, and nothing is walked at all. country = 'c7' AND pc >= 0 over a column whose 2,000 values are all at least 0 took 69 ms and now takes 0.5 ms.
  • complement: BELOW and ABOVE together have fewer entries than INSIDE. Their entries are summed, as INSIDE’s would have been (lion_sum_walk() with LION_WALK_BELOW and LION_WALK_ABOVE), and subtracted.
  • inside: otherwise, as before.

Which side is smaller is decided exactly, at execution time (lion_range_choose()). The INSIDE walk and the BELOW-then-ABOVE walk are stepped a leaf at a time in turn. They only COUNT what they would return (lion_entry_scan_skip_leaf()) and locate nothing. The race stops as soon as one side runs out, so it costs at most twice the leaves of the side that is then walked, which is small next to counting that side. Deciding by leaves weighs a side by its bytes: many small entries or few large ones.

The decision needs no statistics, which is the point: the planner’s n_distinct times histogram fraction can be off by orders of magnitude (a benchmark’s very large table was), and the first and last entries of the column are what says exactly whether a range covers everything. The walks BELOW and ABOVE ran to their ends, so “no entry outside” is exact:

  • an entry with a row visible to the count’s snapshot exists from before that snapshot until the count ends, because VACUUM deletes only empty entries (§18);
  • a key-resumed walk sees every entry that exists for the whole of the walk (One read per leaf, above).

The same holds for the walks that then count BELOW and ABOVE. Each of the three counts is exact under the one snapshot (§9), so their difference is exact too. lion_index_verify() after the tests and count_range_split_race.spec, which parks the race between two leaves and splits every leaf of the column, check both.

Where the walks START: BELOW starts at the column’s first entry, (attno, MINF), exactly as an unbounded walk does. ABOVE descends to its upper bound’s leaf, as INSIDE descends to its lower bound’s. With more than one upper bound (k < 10 AND k <= $1) each is descended to and the leftmost landing - the tightest bound - is where ABOVE starts (§32, “Two bounds on one side”; it used to start where INSIDE does, which was correct but made that side as long as the inside, so the race never took the complement). On a multicolumn index (§24) each walk stops at the first entry of another column, as every walk does.

Not taken, and the inside walked as before:

  • an unordered range, where there is no run to take apart;
  • no positive WHERE source, or none but ranges collected as sources (§32), which hold no pin (lion_source_drives()). |F − NULL(k)| needs one to drive the merge (§14), and the count of every row of the table is not something an index can give. count(*) WHERE k >= 0 alone is the sum over every entry, as it always was. Summing another column’s entries, NULL included, and subtracting k’s NULL entry from each would give it, but that is a walk of another column, and only worth it when that column has far fewer entries. Not done.
  • a partial driving index. Its entries hold only the rows its predicate admits, and F − NULL(k) would count the others too. The planner never picks one (lion_find_roaring_index()), and the executor checks anyway.
  • the GROUP BY and count(DISTINCT) walks, where every entry is a group or a test of its own.

EXPLAIN ANALYZE prints Range Evaluation: inside | complement | full domain. A partitioned table takes its way per partition, and a rescan takes it per scan, so with more than one way taken it prints how often each was taken: complement 1, full domain 2. Under time partitioning every partition the range covers whole is a full-domain one. Posting Sets Summed counts the entries summed on whichever side was walked, and Directory Pages Read includes the race’s leaves.

The cost of a summed range (2026-09-27)

lion_cost_range_sum() prices count(*) WHERE <range on k> [AND F] as lion_sumall_relation() runs it, and lion_cost_count_rel() adds F’s own terms (located once, and a walk of each of its sets) and nothing per entry. The earlier model charged the walk LION_RANGE_ENTRY_COST per entry and ONE container per entry: the rows of the whole count, spread over the entries. That priced 2,000 keys of a thousand rows each beside a selective equality at 1,344 for 920,000 containers (69 ms), and it could not see that the keys outside a range might be a handful. Now:

  • One side is walked. INSIDE has nin = n_distinct(k) × sel entries. When F has a positive source that can drive a count and k’s directory is ordered, the complement has n_distinct(k) − nin entries plus one count of F minus the NULL entry: F’s containers again, and a descent. The cheaper side is charged, plus the race: two leaf reads for every leaf of that side. A range taken as a source (§32) is no such source: it is collected into a copy that holds no pin, and the executor never lets one drive (lion_source_drives()), so beside nothing but such ranges the inside is charged, as it is walked. (Until 2026-09-30 any positive source would do here, and count(*) WHERE mid > 10 AND hi > 10 over 2M rows - hi > 10 summed, mid > 10 collected - was priced at a tenth of the inside walk it made: 38,488 for 2.5 s, where the sequential scan it rejected, at 63,522, takes 0.33. It is 316,199 now, and the sequential scan is chosen. A range too large to collect is collected in windows and cannot drive either; one the plan walks at every count can, and the model takes it as collected all the same, which prices such a count at its inside, what it costs at most.)
  • Each entry walked costs a fixed amount, LION_RANGE_UNION_ENTRY_COST (12 cpu_tuple_cost) for a small entry counted with its leaf and LION_RANGE_ENTRY_COST (40) for one counted on its own. On top of that come its OWN containers: the rows of one key of k (tuples × sel / nin), not the rows that survive F. They lie in as many containers as the column’s correlation says, between one per row and their share of the heap, as cost_index() interpolates pages. The key steps are the smaller of the entry’s and F’s containers, once for every source, since the smaller one drives the merge and the other is probed at its keys.
  • The leaves: the column’s share of the index pages, divided among its n_distinct entries, once per entry walked.
  • The heap the map cannot vouch for, entry by entry as before (below). For the complement that is F’s candidates in the count of F and the outside entries' in theirs.

With this, on the repro table below, each summed range costs 350 to 400 units per millisecond on the assert build, against about 140 for the sequential scan. The one exception is the thousand-row keys, whose containers are charged the two cpu_operator_cost every lion count charged per container then. The planner now takes the node where it is the faster plan: the 1.9-million-key range beside country = 'c7' is chosen at 14,675 and runs in 41 ms, against the sequential scan’s 316 ms. It leaves the node where it is slower: a 7-day window of 605k keys is costed at 97,424 and goes to a plain index scan, 72 ms against 280.

On the release build (2026-09-27; §31) - PROVISIONAL, because the range work under way will rework these terms. The entry constants hold: an entry of one row counted with its leaf takes 290 to 300 ns there (86,400 to 2.6M one-row timestamps, 450 to 550 units a millisecond as priced), which is LION_RANGE_UNION_ENTRY_COST’s 12 cpu_tuple_cost at §10’s 500 units a millisecond. The per-container charge was not: an entry’s containers, and each of F’s sources probed at them, are now §10’s merge units (LION_CONTAINER_COST and LION_MEMBER_COST a member for the entry’s, LION_MEMORY_PROBE_COST and LION_AND_MEMBER_COST for each probe into F’s sets, which a walk copies into memory on their second use), and the complement’s count of F the same. Over the 35 range shapes of §31’s micro-benchmark that moved the node from 14 to 8,049 units a millisecond (median 225) to 47 to 2,294 (median 500): scattered days of 2,500 rows went from 150 to about 400, and the same beside a 20-value status from 80 to about 450. What is left is the executor more than the model: days stored in value order (2,500 rows each in one or two containers, fifty entries to a leaf) run at 62 to 66 units a millisecond, because §15’s density test sends a leaf of them to the k-way union, which builds a bitset image and optimizes it back into a run container at every container key (6 us a key, measured with a stack sampler), where summing them would have been a few hundred nanoseconds each.

What this cannot fix is the estimate itself. n_distinct × sel is what the planner believes, and a stale histogram or an n_distinct that ANALYZE’s sample got wrong makes a range of millions of keys look like a range of thousands. That is what happened on a very large table: tens of millions of keys priced as a few hundred thousand. Execution is robust to it now: the race picks the smaller side whatever the plan thought. The plan choice is not. A bound past the histogram’s end is now read from the directory, as core reads it from a btree (The endpoint probe, below); a histogram that is stale inside, or an n_distinct that is wrong, is not. InnoDB’s records_in_range, which descends to both bounds and estimates the leaves between from the internal levels, would correct both, and is still not done.

The endpoint probe (2026-09-27)

ANALYZE’s histogram ends at the largest value its sample saw, and a column that grows at one end - a timestamp, a serial id - puts every row added since past that end. Core estimates an inequality past it at a hundredth of one bin (ineq_histogram_selectivity() clamps to that when it has no actual end), so ts >= now() - interval '30 days' on a table grown by a tenth since its last ANALYZE came out at 330 rows for 148,760. Core corrects this with get_actual_variable_range(): whenever the histogram’s binary search is about to compare the constant with the first or last entry, it reads the column’s actual minimum or maximum from an index and puts it in place of that entry, and the bound then falls inside the last bin and is interpolated there. It reads only ordered indexes that can return their first column - a btree - so a column indexed by lion alone never got the correction, in lion’s own estimates or anyone’s.

Lion’s cost model now makes the same correction itself, from lion’s directory (lion_selfuncs.c):

  • Where. Exactly where core would read an end: lion runs core’s binary search over the histogram with the clause’s own operator and constant (lion_probe_ends_wanted()), under core’s own conditions for using the histogram (its collation, comparison_ops_are_compatible(), statistic_proc_security_check()). The constant is what get_restriction_variable() makes of the other side, so now() - interval is one. A Param is not, and is not probed. The search calls the clause’s operator on the histogram’s values, which the user may not be allowed to read - a column without SELECT, a table behind a security barrier view or a row-level policy - so, as core’s search does, it runs only when the operator is leakproof or the user can read every row of the column; otherwise core uses no histogram for the clause and the probe reads nothing (2026-09-28 review: the search ran regardless, and a leaky operator was handed the histogram).
  • What is read. The column’s first or last VALUE entry whose posting set has a live row. The directory is in the key’s order within each key column (§21), so a descent to (attno, VALUE) with no key lands on the first VALUE entry (lion_dir_value_start()), and one to (attno, the first kind after VALUE) just past the last (lion_dir_value_end()), whichever key column of the index it is. The walk goes on from there, a leaf at a time, to the right or through the left links (lion_dir_step_left(), nbtree’s _bt_walk_left()), copying each leaf’s entries under its share lock and testing them with none held. A row is live when the visibility map vouches for its page or SnapshotNonVacuumable accepts it, core’s test: a key whose rows are all deleted is not an end, and one whose rows are recently dead or uncommitted is. Only an ordered scalar column is probed, and only when its order is the histogram’s (the key type’s default btree comparison, the index’s collation the statistics').
  • Bounded, as core’s is. LION_PROBE_HEAP_PAGES (100, core’s VISITED_PAGES_LIMIT) heap pages without a live row, LION_PROBE_HEAP_TIDS (10,000) TIDs without one, or LION_PROBE_LEAVES (100) directory leaves, and the probe gives up and the histogram’s own end stands. The leaves bound matters where core’s does not: until VACUUM, the entries of a column’s deleted newest keys stay, each with its dead rows, where a btree marks its entries dead as the probe meets them. An index that grew by inserts holds about 34 one-row keys of a column to a leaf, so the 10,000 newest keys deleted are past the bound and 1,000 are not; VACUUM then takes the entries and their leaves out (§21, “Page deletion”), and the probe reads the live end again (test/sql/rangeprobe.sql §3 shows all three). The TIDs bound is there because core’s probe asks about a dead TID once - it marks the btree entry dead (kill_prior_tuple) and the next plan skips it - where a posting set has nothing to mark, so every plan until VACUUM asks about the same dead rows again, and 100 pages of small rows hold up to 29,100 of them; 10,000 is those pages at 100 rows a page. Each TID used to be asked about through table_index_fetch_tuple_check(), which on 16-19 builds and drops a slot and a fetch state per TID; the probe keeps one of each (lion_tid_fetch_begin(), lion_compat.h), and the pin on a page while its TIDs are asked about (2026-09-28 review).
  • Once per planner run. The ends are cached for the PlannerGlobal of the query, in its memory, and forgotten when that memory is reset. A planner run nested inside another - a function an estimate evaluates that plans a query of its own - keeps a cache of its own beside the outer run’s, which it leaves as it found it (lion_run_cache_for()). Neither a hypothetical nor a partial index is read, nor the parent of an inheritance tree. A column core reads itself is left to core: one that leads an index core reads ends from - ordered, whole, able to return the column, under the histogram’s collation and in the order of its sort operator (lion_core_probes()). A btree on the column under another collation, or a text_pattern_ops one under text’s own <, is one core passes over, and the probe reads the column’s ends from lion.
  • An end found empty is not read again at every plan (2026-09-28 review). Where the probe found no live key - it reached a bound, or read the column’s whole run - the backend remembers so, per index, key column and end (lion_probe_missed(), sixteen at most), in place of the dead entries core’s probe would have marked; before, the end whose newest rows were deleted in bulk was walked to the bounds by every plan until VACUUM. A dead row never becomes live again, so only new rows, VACUUM or a new relation under the same OID can change the answer, and the memory is forgotten when the pg_class row of the table or the index changes - VACUUM and ANALYZE change it whenever they update the statistics there, and so does DDL that rewrites or drops either (a syscache callback on RELOID; a relcache callback would take one of the ten slots a backend has for all of them, and running out of those is FATAL) - or once the table or the index has more pages than when it was made, which rows added past the end soon give it. Rows that grow neither and come with no VACUUM or ANALYZE are not seen until one of those; the estimate meanwhile is the histogram’s own end, which is what core’s probe leaves when it gives up.
  • Lion’s estimates only. The probed ends go into a copy of the column’s pg_statistic row, which get_relation_stats_hook hands to core’s own clauselist_selectivity() while lion prices one of its accesses (lion_probe_begin() .. lion_probe_end()): lioncostestimate() for every lion index path - the selectivity genericcostestimate() prorates the index by, the entries lion_range_walk_cost() prices, the heap side priced from them, and so LionOrdered’s lion side - and the count pushdown’s paths (lion_try_count_path(): the range’s share of its entries, and the rows lion_cost_count_rel() and lion_cost_range_sum() recheck, lion_probe_rel_rows(), which since 2026-09-28 also carries the intersection probe’s correction, §29.11). The relation’s row count and every other path’s estimates stay core’s. Core caches a clause’s selectivity in its RestrictInfo the first time it is asked (clause_selectivity_ext()), so the scope clears that cache on each clause it corrects and puts it back when it ends.

The correction is core’s, and so is its reach: the bound lands in the last bin, which holds a hundredth of the rows ANALYZE saw at the default statistics target, so an estimate past the end rises to at most that share however many rows were added. On a very large table it would lift a wide range’s estimate many times over and still leave it several times short - and whether that alone moves the plan depends on the range’s price per key.

Measured on the assert build (PostgreSQL 18.6), best of three warm runs; “before” is the build without the probe:

table, query actual rows lion’s estimate before / now lion’s price before / now chosen, ms
3.3M rows, the newest 300k past the histogram and in heap order: sum(id) WHERE ts >= now() - interval '30 days' 148,760 330 / 14,769 plain 605 / 24,249, bitmap 1,261 / 32,181 plain scan 83-87 (bitmap 38-48, seq 663)
the same, count(*) 148,760 330 / 14,780 LionCount 75 / 3,344 LionCount 63
the 300k newest rows in the space a DELETE and VACUUM of a tenth freed, on 42,773 of 42,858 pages: sum(id), 30 days 148,760 300 / 13,363 plain 552 / 22,027, bitmap 1,148 / 29,246 plain scan 100 (bitmap 84)
rangeprobe.sql’s table, 1,500 rows analyzed and 28,500 added: count(*) WHERE ts >= '2026-01-02 02:00' 28,441 60 / 5,926 LionCount 69 / 1,274 LionCount 12.6-16 before, seq scan 4.1 now

The first three plans are the same before and after: a 30-day range of a large table is a small share of it, which lion reads quickly whatever it was priced at, far ahead of the sequential scan. Two of them are the fastest plan. The first is not - the plain scan streams its 148,760 one-row entries a stream each (§29.11) and takes twice the bitmap scan’s time - and is still chosen because the estimate it is priced by, 14,769 entries, is a tenth of them: the probe’s reach, above. The last row is a range over most of a table whose statistics saw a twentieth of it, with a histogram of five bins so that its last one holds a fifth of the rows: there the count pushdown walked 28,441 entries it had priced as 60, and a sequential scan does the count in a third of the time. The probe costs 7 shared-buffer reads and about 0.05 ms of planning per planner run (0.12 ms against 0.07 for the 30-day count), and none when no bound is past an end.

Dropping the range for a heap recheck (considered, not done)

When the range is wide and F is selective, the node could count F and test the range on each candidate’s heap tuple instead of walking the range’s entries. That is a Bitmap Heap Scan of F with the range as its filter, which the planner already builds when F’s column has an index of its own, at the price of a heap visit per candidate and never the visibility map. The node’s recheck asks only visibility. Teaching it to evaluate a qual would give LionCount a second, bitmap-shaped execution with a cost model of its own, for a case where a plan the planner already has is at best equal. With the walk priced as it runs, the planner already declines the pushdown for it: the 7-day window below goes to the index scan. When F and the range are columns of ONE multicolumn index, core has no F-only index path, and the lion bitmap scan answers the range by a walk ANDed with F’s containers (above) - with summaries, a walk of the range’s buckets. Leaving that range to the heap recheck is a question for lioncostestimate() and the bitmap scan, not for this node, and the bitmap scan does not: the walk it would save is short when the range is summarized, and bounded by F’s windows when it is not.

Measured (2026-09-24, the distinct slot’s PostgreSQL 20devel, assert-enabled: ratios, not absolute numbers)

Five million rows over 37,133 heap pages, all-visible after VACUUM: d - about 2000 days stored in order (2500 rows a day), dr - the same days in random order, status - 20 values, ts - a timestamp unique per row and in heap order, each with a lion and a btree index (lion d 320 kB against btree’s 33 MB, dr 47 MB against 33, status 10 MB against 33, ts 325 MB against 107). Best of five EXPLAIN ANALYZE runs, milliseconds; “chosen” is the plan with nothing disabled, the other columns force one path each (the btree column is its best plan, an index-only scan for a count):

query chosen LionCount lion bitmap btree seq
count(*) WHERE d 1 day node 0.19 0.19 0.47 0.26 237
count(*) WHERE d 30 days node 0.24 0.23 7.14 5.78 236
count(*) WHERE d 365 days node 0.93 0.90 85.6 69.9 263
count(*) WHERE d 2000 days node 3.08 3.00 351 288 354
count(*) WHERE dr 1 day node 0.25 0.24 8.48 0.26 252
count(*) WHERE dr 30 days node 2.31 2.43 57.3 5.84 253
count(*) WHERE dr 365 days node 25.9 26.0 187 70.7 286
count(*) WHERE dr 2000 days node 106 105 480 285 372
... dr 30 days AND status = 3 node 4.42 4.41 30.4 32.5 261
dr, count(*) ... GROUP BY dr, 30 days node 2.21 2.14 68.6 7.55 266
count(DISTINCT dr), 365 days node 1.33 1.29 267 73.3 357
sum(x) WHERE d 7 days btree index scan 2.66 - 2.07 2.59 237
sum(x) WHERE dr 7 days btree bitmap 27.1 - 27.2 26.8 250
count(*) WHERE ts 1 hour btree IOS 0.10 0.41 0.32 0.09 258
count(*) WHERE ts 1 day btree IOS 0.48 3.88 0.98 0.47 256
count(*) WHERE ts 30 days btree IOS 11.2 109 20.4 11.1 261

The count pushdown is chosen wherever it wins - by one to three orders of magnitude over the btree for a clustered column and three to thirty times for a scattered one - and a near-unique column goes to the btree, where the node would be four to ten times slower. The lion BITMAP scan is at parity with the btree’s where the heap dominates (a sum(x)), and the planner picks between them on cost. With 1% of the rows updated so that every heap page is dirty: d 30 days, the node 3.5 ms (chosen) against the btree’s 11.5; dr 30 days, the node 157 ms against the bitmap heap scan’s 58 (chosen); dr 30 days AND status = 3, the node 18 ms (chosen) against 30; count(DISTINCT dr) over 365 days, the node 22 ms (chosen) against 263.

Measured again (2026-09-27, PostgreSQL 18.6 assert build, pg_lion preloaded, rmgr WAL mode)

These are section 4 of the benchmark’s repro script. The table is 2M rows over 14,720 heap pages, all-visible, with one lion index on (status, flag, pc, ts, country):

  • pc has 2,000 values, all at least 0, a thousand rows each, one 3.8 kB INLINE entry per leaf;
  • ts is a timestamp unique per row, in heap order, about 150 entries per leaf;
  • country has 50 values.

The base build is ad9c155 and “now” is this section’s three changes. Times are medians of five warm EXPLAIN ANALYZE runs (TIMING OFF) in ms. The cost is the node’s own estimate, forced when it is not the chosen plan, and “pages” is Directory Pages Read:

query base: cost / ms / pages / summed now: cost / ms / pages / summed, evaluation chosen now
country = 'c7' 166 / 1.0 / 4 / 0 166 / 1.0 / 4 / 0 node
... AND pc >= 0 (all 2,000 keys) 1,344 / 69.5 / 4,012 / 2,000 173 / 0.49 / 18 / 0, full domain node
... AND ts >= '2025-01-02' (1.91M keys in, 86k out) 812,380 / 3,713 / 1,929,429 / 1,913,601 14,675 / 39 / 2,165 / 86,399, complement node 41 (base: seq scan 365)
... AND ts < '2025-01-02' (86k keys) 38,091 / 148 / 87,126 / 86,399 14,370 / 40 / 2,165 / 86,399, inside index scan
... AND ts 7-day window (605k keys) 258,312 / 1,128 / 609,813 / 604,801 97,424 / 280 / 15,020 / 604,801, inside index scan 72
ts >= '2025-01-02' alone 812,572 / 2,069 / 1,929,425 / 1,913,601 258,544 / 648 / 15,820 / 1,913,601, inside seq scan
country = 'c7' AND pc >= 1000 (1,000 keys) 705 / 36 / 2,013 / 1,000 2,903 / 40 / 3,019 / 1,000, inside node
sequential scan of the 1.9M-key count 44,807 / 316 the same

Directory reads went from about one per key to about one per leaf. A range whose outside is small costs what its outside does, and one covering every key costs one count of F. The one row that got no faster is the thousand-row keys: they are counted one by one as before, plus the leaves of the race (3,019 pages against 2,013). Their estimate rose from a fifth of the time’s worth to about half of it, measured against the other ranges' 350 to 400 units per ms.

Tests

test/sql/range.sql, written before the code and shown failing first: every strategy and combination (<, <=, >, >=, BETWEEN, open-ended, empty, inverted, equal bounds, bounds outside the key domain), commuted forms (5 < k), cross-type int2/int4/int8 and float4/float8 with -0, NaN and Infinity, text under the default collation and under "C" with a mismatched collation declined, citext, numeric, date/timestamp/timestamptz, uuid, bool, enum; NULL keys excluded; a multicolumn index (a range on one column and an equality on another, and two ranges); a partial index; after deletes and updates (a dirty heap) and after VACUUM (a clean one); every answer equal to a sequential scan’s. The count pushdown for each accepted shape with its plan and exact answers, each decline, Params under force_generic_plan (a NULL one included), a partitioned table, the index pins of a GROUP BY parked mid-walk (pg_buffercache: one entry’s, however wide the range), EXECUTE revoked on a range operator, and plan choice with nothing disabled: a lion bitmap scan over btree for a range on a low-cardinality column, and btree or a sequential scan over lion on a near-unique one. EXPLAIN goes through lion_explain_norm(). test/isolation/count_range_vacuum_race.spec parks a range count with an entry’s container pinned while the VACUUM of that entry’s dead rows waits, as count_vacuum_race.spec does for an equality.

test/sql/rangesum.sql (2026-09-27) covers the sum over a range. Each answer is checked against a sequential scan with the pushdown off and printed with its Range Evaluation and Posting Sets Summed. The cases:

  • full-domain ranges on int and text keys, including cross-type bounds outside the int4 domain, with NULLs in the column;
  • near-full ranges taken as their complement, on either side and on both;
  • ranges whose inside is smaller;
  • empty and inverted ranges;
  • two upper bounds;
  • ranges beside an IN list, an OR, IS NULL and IS NOT NULL on other columns, and k IS NOT NULL on their own;
  • the GROUP BY and count(DISTINCT) walks;
  • Params under a generic plan (a NULL one included) and a rescan per outer row;
  • a dirty heap with a key inserted above every other one and one below, before and after VACUUM;
  • a range on each column of a multicolumn index;
  • a partitioned table whose partitions each take a different way.

It also checks that a sum over 20,000 one-row entries reads each leaf once, and that a full-domain range reads a few descents' worth of pages.

test/sql/rangeprobe.sql (2026-09-27) covers the endpoint probe on a table analyzed at 1,500 rows, sampled whole, and grown to 30,000 past both ends of two of its columns, the first and the third of the index. Written before the code, it fails on the build before it: lion’s estimate of a range past either end is core’s, the count and the sum over it go to the count pushdown and to the plain scan, where a sequential scan is three times faster. It checks lion’s estimate against core’s; the plans, and the answers against a sequential scan’s; a bound inside the histogram, which is not probed; the newest 1,000 rows deleted, before VACUUM and after, where lion’s estimate is the number core makes with a btree on the column; 9,000 more, past LION_PROBE_LEAVES, where the probe gives up and the histogram’s end stands; a partial index, and a Param. test/sql/rangeprobe_leak.sql (2026-09-28) counts the calls of a non-leakproof >= of a lion operator class made while a query is planned: the table’s owner has the histogram searched with it, a role under a row-level policy does not, and the same role does once the operator is leakproof. test/sql/rangeprobe_memo.sql deletes the newest 25,000 of 40,000 keys of a one-column index and vacuums - keys that arrive in order fill that index’s leaves, about 130 one-row keys to a leaf against the 34 of rangeprobe.sql’s three-column index, so it takes some 13,000 of them to empty LION_PROBE_LEAVES leaves: the first plan reads a hundred leaves and gives up, the next reads none (their planning buffers differ by 100 or more), and rows added past the end, more than the deletion left room for, grow the table and the index and have the end read again. test/isolation/count_range_split_race.spec parks a range sum between two entries of a leaf (lion-entry-scan-resumed) and the race of the complement between two leaves (lion-entry-scan-leaf), splits every leaf of the column under them, and checks both counts against the heap.

test/sql/rangebitmap.sql (2026-09-28) covers “Bitmap scans” on a multicolumn index of a text column, a text array, a timestamp and an integer, over two copies of one synthetic table - one with summaries = auto, which summarizes the timestamp and the integer, one without - laid out nine rows a page, the timestamp a permutation of the heap order, so that a window of it has rows on nearly every page. At a work_mem of 64kB, which holds the answer’s pages exactly but not the range’s, an equality, an overlap and a range with the bound a literal, a stable expression (a runtime key) and a generic plan’s parameter (a NULL one included), a lower bound, an upper one, both and one past every key: each answer against a sequential scan’s, and from EXPLAIN ANALYZE only that the range is an Index Cond, that the recheck removed no row and that no page was lossy - the version before this one removed rows and read lossy pages there. Then that the summarized walk reads under half the index blocks of the other; two ranges across the windows of the least pg_lion.scan_window_floor; ranges alone, the first one collected; a qual dropped for the recheck beside them; and the plain scan’s walks over the summaries.

29. Index scans: amgettuple, ordered scans, index-only scans

Until this section lion answered only bitmap scans. Measured on release PG18.6 at 98ecedd (bench/results/2026-09-24-98ecedd-*), row fetches that visit the heap anyway were at parity with btree except in two shapes that only a plain Index Scan serves: a very selective lookup (WHERE c1m = ..., one row: btree 0.023 ms against lion’s bitmap 0.027 ms at 1M rows, the difference being the bitmap’s fixed setup) and a low work_mem (64 kB: WHERE c200 = 17, 0.5% of 5M rows, btree Index Scan 13 ms against a LOSSY lion bitmap 92 ms, which rechecks every row of every lossy page). A lion index could not do better because it had no amgettuple. This section adds it (step 1, implemented), and records what ordered scans (step 2, deferred to the backlog) and index-only scans (step 3) would need from it, so that neither has to redesign the scan.

29.1 Handler settings (step 1)

amgettuple = liongettuple (every supported major, 16 .. 20)
amcanbackward = false       ammarkpos/amrestrpos = NULL     amcanparallel = false
amcanorder = false          amcanreturn = NULL              amsearcharray = true (unchanged)
amsearchnulls = true (unchanged)                            amoptionalkey = true (unchanged)
  • No backward scans. amcanbackward = false is what ExecSupportsBackwardScan() reads, so a SCROLL cursor over a lion Index Scan gets a Material node from the planner, and a NO SCROLL one refuses FETCH BACKWARD in the executor before the AM is asked. The only other source of a backward index scan is an ORDERED index (indxpath.c builds a backward path for every index with a sort order, whatever amcanbackward says), which lion is not; liongettuple() therefore raises an internal error for any direction but forward, and nothing can reach it.
  • No mark/restore. A merge join over a lion Index Scan would need one only if lion claimed an order, and it claims none; with ammarkpos NULL the planner puts a Material node under a merge join that needs to restore anyway.
  • kill_prior_tuple is ignored in this version. btree uses it to set LP_DEAD on index tuples whose heap tuples are dead to everyone; lion has no per-TID flag to set (a TID is a bit in a container), and removing the bit would be a write under a share lock on a page other scans are reading. VACUUM removes them. Known cost: a plain scan over a hot key with many dead versions keeps visiting their heap slots until the next VACUUM, as a bitmap scan does today.
  • No parallel scan. amcanparallel stays false; the parallel callbacks stay NULL.
  • No returnable column (amcanreturn NULL, step 3) - but an index-only scan of a query that needs no column at all is possible the moment amgettuple exists, and it is served (§29.9).

29.2 The scan plan, shared with the bitmap path

Both entry points begin with the same choice of the quals to answer (§5 SCAN step 5, §24, §28), lion_scan_choose(): EVERY set qual of every key column - an equality, a list, IS NULL, a multi-key query - is a set tree, all of them ANDed, several of one column as well as those of several columns; a column without one is WALKED, for its range (all its range keys at once, or op ANY (array) with a range strategy) or else its IS NOT NULL - which beside anything that answers is no walk but its column’s NULL entry taken away from the answer, or its column walked where that reads less (§14, “IS NOT NULL in the scans”). A range beside its column’s sets is left to the heap recheck, IS NOT NULL beside a strict qual of its column is implied by it, and IS NULL beside either selects nothing. The set trees (lion_scan_col_tree(): an equality is a leaf, an IN list an OR over its located sets, IS NULL the reserved NULL entry, a multi-key query its extracted tree, §17) are built by the same function for both paths. What differs is only how the answer is delivered: into a TIDBitmap, or as a stream the executor pulls one TID at a time. (Deviation from this section’s first version, which answered the most selective-looking qual of a column - equality, list, range, null test - and left the others to the recheck, §5 SCAN step 5.)

29.3 The TID source (LionSource, lion_scan.c)

A plain scan is a source of TIDs, opened on the scan keys at the first amgettuple call after amrescan. It has one of five shapes:

  • NONE: an answered qual selects nothing (col = NULL, an empty or all-NULL list, a key absent from the index, a multi-key query in mode NONE, IS NULL beside a strict qual of its column). amgettuple returns false at once.
  • SETS: every answered qual is a set tree. The trees are ANDed (§24) and the whole expression is ONE stream of containers in ascending container key, produced by the very evaluator the count and the bitmap path already share (§9 cursors, §15 k-way OR, §22 leapfrog AND: LionExprCursor) - since 2026-09-28 the count’s own leapfrog, the tree of fewest rows driving whatever the order of the index’s columns (§29.11, “The AND, as the count makes it”). An IN list is the OR of its sets - a union by container key - so the TIDs come out in HEAP order, each exactly once, and the heap is visited in physical order, one pass. Equality, IN, IS NULL, multicolumn ANDs, several quals of one column and multi-key queries (arrays, tsvector) in mode KEYS all take this shape, and IS NOT NULL beside them: the NULL entries of its columns taken away from the AND by an AND-NOT node over it (§14).
  • WALK: one scalar column is walked for its range (§28, including op ANY (array)), an IS NOT NULL that nothing else answers, or the scan has no key at all (a partial index whose predicate the query implies, §24, which walks column 1 with its NULL entry). The column’s entries are walked in directory order exactly as the bitmap walk does (lion_emit_all_keys_ext(): copy a leaf, release it, test each entry with lion_range_test(), stop at the first entry past an upper bound or of the next column), and each selected entry is streamed as AND(entry, rest), where rest is the AND of the other columns' set trees (empty for a one-column scan). On a column with summaries (§32) the “entries” of one range are the keys of the buckets at its ends and the summaries of the buckets between, as for the bitmap walk (§28, “Bitmap scans”) - disjoint, so the WALK still returns no TID twice - unless the scan keeps its pins (§29.5), whose walk is the keys'. Within one entry the TIDs come out in heap order; across entries in key order. A second walk column is dropped and rechecked: a range column cannot be a set-tree node (its answer is a union of an unbounded number of entries), and the bitmap path’s answer for two range columns - the first walk ORed into an image per container of a window of the others, the second ANDed with that - is not one stream. An op ANY (array) range walks to the widest element (§28) when the column orders the elements; an UNORDERED column walks ONCE, testing each entry against every element’s range (the bitmap path walks once per element and lets the bitmap absorb the overlap, which a stream cannot). IS NOT NULL walks only alone, the NULL entries of any other IS NOT NULL columns taken away from each walked entry; next to anything that answers - a set tree, a range, a long list - its NULL entry is taken away from that answer instead, or, beside sets alone, its column is walked as a range is when that reads less (§14). (Until 2026-10-19 it was dropped there and rechecked, as the bitmap path dropped it.) (Deviation from this section’s first version, where IS NOT NULL always drove a walk: every entry of its column ANDed with the other columns' stream - a IS NOT NULL AND b = 5 over a million unique a took 4.37M buffer hits and 1.8 s against the bitmap scan’s 19 ms, while the cost model priced it by b alone, 2026-09-25 review.)
  • WINDOW: a WALK beside the other columns' set trees that is LONG - whose restarts would read more than those sets have posting pages, and more than 16 (lion_source_walk_is_long(), which counts them from the directory leaves alone and stops there). Restarting AND(entry, rest) for an entry re-descends each of the rest’s posting trees and then reads, of each, the posting pages at the entry’s container keys: at most one per TID of the entry (its header’s ntids), at most the set’s own pages (lion_source_restart_pages()). An entry of a few rows costs a descent; an entry spread over the heap - every entry of a low-cardinality column - costs reading the rest whole. A WINDOW reads the rest once. The rest is ONE stream, and a window is its next lion_walk_window() containers, each ORed into a bitset image; the range is then walked once for the window (its buckets' summaries and edge keys, on a summarized column), each entry sought to the window’s first container key and read up to its last - an INLINE entry straight from the walk’s copy of the leaf, item by item, a posting tree through a stream - and ORed into a second image wherever the rest has a container. The two images' AND is the answer for the window, handed out in container key order, so the whole scan is one ascending stream and the heap is visited in physical order, as by a bitmap scan. Each window walks the range again, so the window is wide: max(pg_lion.scan_window_floor, work_mem) / 8 kB containers, 512 at the default floor of 4 MB (32768 heap blocks if the rest has a container at every key), up to 65536. Built only for a scan that drops its pins (§29.5); one that keeps them keeps the entry-by-entry WALK, whose pins are the ones it needs. A short walk stays a WALK too: its restarts cost no more than reading the rest once. (Deviation from this section’s first version, which had only the WALK: a BETWEEN 1 AND 400000 AND b = 5 over a million rows, a unique, took 1.06M buffer hits and 818 ms where the bitmap scan read 5481 buffers in 72 ms, and the planner chose that plain scan for the 200000-row range, 409 ms against 38 (2026-09-25 review). As a WINDOW it reads 5485 buffers in 36 ms, and the 200000-row range 2751 in 17 ms; an INLINE entry read through a stream of its own, as the first cut of the WINDOW did, took 113 ms for those 200000 entries.) (Deviation from the WINDOW’s first version, which counted ENTRIES against the rest’s posting pages, a restart being taken for a descent. That holds for a column of a few rows per key, the unique a above, and not for a low-cardinality one: each of pc < 100’s 100 keys (4000 rows each, 8M rows, 190k heap pages) has rows at nearly every container key, so with four dense sets beside it - 4700 posting pages, more than 100 entries - the scan stayed a WALK that streamed those sets once per key: 10.8 s against the bitmap scan’s 0.77, 2.5 s against 0.27 for 20 keys, 36 s against 1.7 for 400, and the always-true pc >= 0 (2000 keys) past two minutes against 2 s. As WINDOWs: 0.14, 0.44, 0.97 and 2.2 s. A plain Index Scan of this shape on a much larger table did not finish in 60 s (2026-09-27): an always-true range over about two thousand keys beside four sets. test/sql/plaincost.sql §3.)
  • LIST: an IN list on a scalar column longer than a batch (§29.4) is located and streamed a batch at a time, each batch the SETS shape with the other columns' trees ANDed in and the NULL entries of IS NOT NULL columns taken away. It outranks a walk, which is then left to the recheck, and a second long list is left to the recheck too.
  • UNION: a MULTI-KEY column would have to be read whole - IS NOT NULL, a query in mode ALL (§17) with nothing else expressible, or a no-key scan whose first column is multi-key - or a scalar column’s walk, a range or a lone IS NOT NULL, is long beside nothing but the NULL entries of IS NOT NULL columns, which it would restart for each entry (2026-10-19): the union of the entries the walk selects, its ranges applied. A row is under several entries of a multi-key column, so walking them would return it several times. The scan streams the exact UNION of the column’s entries instead, a WINDOW of container keys at a time: every entry is sought to the window’s first key and read to its end, and ORed into one bitset image per container key of the window; the next window starts at the smallest key any entry had past this one. Each window walks the column’s entries once, so the work is windows x entries, and the window is therefore WIDE whatever work_mem says: lion_union_window() = max(pg_lion.scan_window_floor, work_mem) / 4 kB container keys, 1024 at the default floor of 4 MB (65536 heap blocks, 512 MB of heap), up to 65536, and a key’s 4 kB image is made only when an entry has a container there, so a window costs what it holds - at most 4 MB at the floor, which with the WINDOW’s is where this scan may exceed a tiny work_mem. The floor is a setting so that the regression suite can cross window boundaries on a table of a few megabytes (test/sql/indexscan.sql §10 and §11 lower it to 64 kB: 16 keys, or 8 containers); it is a testing knob more than a tuning one. The NULL entries of IS NOT NULL columns beside it are one stream, sought forward to each container key the union hands out and taken away there (§14). Rechecked for a query in mode ALL, which needs it, and for a scan that keeps its pins (§29.5), which reads copies; exact otherwise. (Always rechecked until 2026-10-19, when IS NOT NULL on a multi-key column, which the UNION answers exactly, became an answered qual.) Correct and expensive, which is what the cost model already says about these quals; lioncostestimate() also charges each window past the first another read of the index (the IndexPath is shared with the bitmap scan, which is overcharged by that, only on a heap past 512 MB). (Deviation from the version before it, whose window was max(16, work_mem / 4 kB) keys: at 64 kB that is 1024 heap blocks, and a 2M-row table with 200k distinct keys took 17 windows and 1.40 s for an index-only count() against 0.48 s at 64 MB - and the planner picked that scan at 64 kB; 100M rows would have meant some 1600 walks of every entry (re-review, 2026-09-24). Keeping every entry’s cursor across windows instead would have made the memory grow with the number of distinct keys, and identifying an entry across windows, while inserts and VACUUM change the directory between two calls, needs its key - both worse than a wider window. test/sql/indexscan.sql §10 counts the index blocks one count reads at 64 kB and at 64 MB: the same, where they were five times as many at 64 kB.) (Deviation from this section’s first version, which ran liongetbitmap() into a private TIDBitmap and, on a LOSSY page, returned every offset up to MaxHeapTuplesPerPage “with recheck set”. That is right for a bitmap heap scan, which re-applies the index PREDICATE to a lossy page (bitmapqualorig), and wrong for a plain Index Scan, whose recheck quals are the index quals minus what the predicate implies, and for an index-only scan, which rechecks nothing: on a partial (tags) WHERE flag index at 64 kB of work_mem, count(*) WHERE flag answered 175,076 for 100,000 and a plain scan returned 140,216 rows with flag false - the 2026-09-24 review. The rule that fixes it is §29.6’s first one: no shape returns a TID the index does not hold.)*

Scalar entries of one column are disjoint (one entry per equality class, §21), so a WALK never returns a TID twice either, and neither does SETS, whose OR node merges by container key, nor a WINDOW, whose windows are disjoint runs of the rest’s container keys and whose images are sets.

Why an IN list is a union and not a walk of its entries. With no order to keep (§29.8), the union is strictly better for a plain scan: it visits each heap page once, in physical order, while an entry-by-entry list restarts at the beginning of the heap for every value; and it is already the shape the bitmap path evaluates. An ordered scan would need the other one (§29.8).

The stream (LionSetStream, lion_expr.c: lion_stream_begin(), lion_stream_next(), lion_stream_end()) is the LionExprCursor behind a pull interface - lion_sets_iterate(), the push form the bitmap path uses, is now a loop over it - with one new switch, keeppins, described in §29.5. A walked entry becomes a LionPostingSet without a lookup: an INLINE entry’s payload points into the walk’s private copy of the leaf, a CHAIN entry’s head is its posting tree’s root.

29.4 Batches and memory

Every located set costs its cursor, and an IN list is located whole - which the first version of this section missed: a located set with the cursor that reads it takes ~20 kB (the cursor’s 4 kB staging container, its share of the OR node, the set), so a 20,000-value list held 220 MB and c1m = any(array(select ... 300000 ...)) LIMIT 10, planned as a plain Index Scan, peaked at 1.37 GB to return ten rows (2026-09-24 review). The bitmap path never had it: lion_emit_array() looks its values up one at a time. So a list longer than a BATCH - max(32, work_mem / 32 kB) values, a batch’s cursors near 60% of work_mem (below) - is the LIST shape: its values are sorted once into lookup order (lion_probe_sort(): the probe’s comparison, then the hash), its byte-for-byte repeats dropped (lion_probe_sort_unique()), and located and streamed a batch at a time, the previous batch’s sets and pins gone before the next is located (the §11 order). A batch is cut only where the HASH changes, and every value of one equality class compares equal and hashes alike, so the class’s values sit next to each other and never straddle two batches: no entry is located twice, and since distinct entries of a scalar column are disjoint, no TID is returned twice - citext’s 'Alice' and 'alice' included. (The repeats were kept until 2026-10-18: one value repeated a million times is one class, and ran its batch on to a million sets' room, as the count’s batches did before they dropped them. The cutting is now one piece of code, lion_scan_list_locate(), which the bitmap scan’s batches use as well, §15.) What stays is the list itself, sorted: 12 bytes per value next to the executor’s own array. The heap is visited in physical order within a batch and restarts per batch, which is what a btree scan of the same list does per value. No new cost term: with batches the plain scan’s work is linear in the list, like the bitmap’s, and a list that is a Param - the reviewed case - has no length the planner could price anyway. Not covered: a multi-key column’s op ANY (array of queries) still extracts every query up front (per query, not per row; arrays of queries are rare). The BITMAP path’s intersections (lion_emit_columns(), lion_emit_intersect()) located a list whole, as they always had, until 2026-10-18; they now take a list longer than the count’s batch a batch at a time too (§15, “A list too long to locate at once”). The same section sized a cursor’s staging buffer by its entry’s payload, so the ~20 kB per set above is what a CHAIN set’s cursor (a page image) still comes near and an INLINE one no longer does.

What a batch takes, measured. A located set and its cursor take up to ~19 kB in the source’s entry context: two 8 kB allocator chunks - the cursor’s 4104-byte staging container and, for a set with a sparse segment (§13), the segment’s buffer of the same size, each rounded up to a power of two, or a posting-tree cursor’s 8 kB page image - and the set, its share of the OR node and the context’s slack. At 32 kB of work_mem a set, a batch is ~60% of work_mem, and the scan as a whole - the sorted list’s 12 bytes a value included - stays inside work_mem for a list of up to one value per 36 bytes of it, 29,000 values at 1 MB: the 20,000-value list of the tests takes 905 kB at 1 MB, 2.7 MB at 4 MB and 10.1 MB at 16 MB. The entry context has blocks of one size, 64 kB, so what it takes is what it holds give or take one block (and a walk’s per-entry stream, which fits one block, is reset from entry to entry with no allocation between); the floor of 32 sets, ~620 kB, is the bound below 1 MB. (Deviation from the first version, whose comment said ~19 kB a set but whose entry context grew by doubling blocks, the last one half empty: a batch took twice what it used, 1073 kB at 1 MB of work_mem, 4180 kB at 4 MB and 16.9 MB at 16 MB, and the test that was meant to show it inside work_mem compared with the server’s own work_mem, 64 MB on the development servers, so it only failed on a server whose default was 16 MB or less, PostgreSQL’s 4 MB included (2026-09-25 review). test/sql/indexscan.sql §9 now sets 4 MB and 16 MB itself.)

amgettuple returns TIDs out of a BATCH: the members of the container the stream is standing on, expanded into an array of at most LION_CONTAINER_RANGE lo values (lion_container_to_array(), turned into a TID one at a time, in the per-heap-block order lion_container_to_tbm() emits). The next container is pulled only when the batch is used up. What the scan holds between two calls is therefore bounded by work_mem (save the windows' floor) and the query, never by the data: one container’s members; one 8 KB image per posting-set cursor of the tree (the page its current container came from), for at most a batch of an IN list’s sets (above); the INLINE payload copies of those sets (at most an entry each, LION_MAX_ENTRY_SIZE); for a WALK one directory leaf image; for a UNION the bitset images of its window (made on first use; up to 4 MB at the floor, §29.3); for a WINDOW two images per container of its window, the rest’s and the walk’s (up to 4 MB at the floor, work_mem above it), and the rest’s one stream; and a long list’s values, sorted. A posting set of any size is STREAMED container by container - a 5M-row entry is never materialized - and a walk of any width holds one entry’s stream at a time. Every amgettuple call runs in a per-scan context that amrescan resets, and every amgetbitmap call in one of its own, reset as the call returns - not in the context the executor calls them in, its per-query one, which lasts as long as the query - so that what a call allocates as it goes, such as the buffer a posting-set cursor takes for the first sparse segment it meets, goes with the scan or with the call. A walk’s leaf image and range ends go with the walk, and its per-entry cursors are in a per-entry context reset before the next entry. So a nested-loop inner scan rescanned a million times, plain or bitmap, and a range over a million entries both run in constant memory (test/sql/indexscan.sql §12).

29.5 Pins, and what a scan paused between calls may hold

The executor returns a row to its caller between two amgettuple calls, so a scan can be paused for any length of time - a cursor held open by a client, a nested loop’s outer side, a LIMIT that never comes back. The rules: no LWLock is held across calls, ever; a PIN may be, and whether it is depends on the snapshot, as in btree.

MVCC snapshots drop every pin (dropPin, mirroring nbtree’s so->dropPin, set in btrescan, and _bt_drop_lock_and_maybe_pin(); nbtree/README “Making concurrent TID recycling safe”): the scan copies what it needs and holds no pin at all while the executor has the row. With keeppins off every cursor releases a posting leaf the moment it has copied it, and a located or walked INLINE set is unpinned as soon as its payload is copied (lion_posting_set_unpin(), as the bitmap path already does). The condition is btree’s: IsMVCCLikeSnapshot() (IsMVCCSnapshot() before 19, which already included historic snapshots), !xs_want_itup (always true in step 1: there is no amcanreturn) and a heap relation. This matters beyond speed: a cursor that pinned a posting leaf would make every VACUUM of the table wait for it (§11 takes a cleanup lock on every page), for as long as the client leaves the cursor open.

Why an MVCC scan needs no pin, case by case. What it holds across a pause is private memory only: copies of posting leaves (each with the right link it had when copied), INLINE payload copies, the posting-tree root of each CHAIN set, a directory leaf image with its right link, and a batch.

  1. TIDs VACUUM removes after the scan copied them - the heap fetch applies the snapshot: the tuple is dead to it, or its slot is LP_UNUSED, or it was reused by a tuple inserted after the old one became dead to EVERY snapshot, ours included, which our snapshot cannot see either. That is nbtree’s argument verbatim (README, “Making concurrent TID recycling safe”: “An MVCC snapshot is only sufficient to avoid problems during plain index scans because they must access granular visibility information from the heap proper”); a plain lion scan fetches every TID it returns, and xs_recheck is irrelevant to it.
  2. TIDs VACUUM removes before the scan copies them were dead to every snapshot, so missing them is right.
  3. TIDs inserted after the scan copied a page belong to transactions our snapshot cannot see (an index entry is written before its inserting transaction commits, so every row visible to the snapshot was in the index before the scan read it). Missing them is right, and so is seeing them.
  4. A posting leaf splits under a paused cursor (it holds an image of P with right link R): the split moves P’s upper items to a new page N between P and R. The cursor already has them in its image of P, follows R, and never visits N - nothing missed, nothing twice; what N gains later is case 3. R itself splitting is found through R’s own right link when the cursor reads R. This is §4/§22’s rule that items only ever move right, onto a page linked immediately right of their source, and it holds whether the new page was extended or recycled (§18).
  5. A root push-down (§22) moves a head’s items to a brand new child: a cursor that copied the old head as a leaf has its items; a cursor that has not started (or has a seek pending) descends from the root, which lion_posting_search() handles at any level.
  6. An INLINE entry spills to a CHAIN (§4): the scan’s payload copy is complete as of the copy; case 3 covers the rest.
  7. VACUUM deletes an entry and frees its posting tree (§18), and the pages are reused by another key’s set: a cursor that follows its stale right link or descends from its stale root lands on a page that is DELETED or claims another owner, and ends the set there - the owner check every cursor already makes (lion_page_owns(), §18). Ending there is right because VACUUM deletes an entry only once its set holds no TID at all. (While an MVCC snapshot is registered, the freed pages cannot even be handed out again: lion_alloc_page() takes a DELETED page only once its safexid is older than every snapshot, §18. The owner check is what the argument rests on; the horizon is a second fence.) A LIVE set loses only the leaves VACUUM emptied (§22, “Empty leaves”): such a leaf is taken out of the set but stays an empty leaf of it, with the right link it had, so a cursor that follows a stale link to it walks through it to the leaf that took its range over. Here the horizon is what the argument rests on: the leaf is not handed out again while the scan’s snapshot lasts (test/isolation/posting_reclaim_cursor.spec).
  8. The directory leaf under a paused WALK splits, or loses entries to VACUUM: the walk holds an image of the leaf and its right link, exactly as the bitmap walk does between leaves (§21 “Ordered iteration”, §28): entries moved right are in the image and are not visited again on the new page; a deleted entry had an empty set; an entry inserted after the image was taken is case 3. A leaf VACUUM unlinks meanwhile (§21, “Page deletion”) is still a leaf, with a MINUS INFINITY high key and the right link it had, and it is not reused while the scan’s snapshot lasts: the right link always names a leaf, and from a deleted one the walk moves right.
  9. The scan’s own INLINE copies taken at locate time (SETS: every set is located at the first call) are cases 1 to 3 again.

Non-MVCC snapshots keep the pin on the page the current batch came from, as btree’s !so->dropPin scans do. keeppins on: every cursor keeps the pin on the posting leaf its current container came from until the stream moves past that container (§9’s cursor rule, unchanged), a located INLINE set keeps its leaf pin for the life of the scan, and a WALK keeps its current directory leaf pinned while that leaf’s entries are streamed. The batch is the current container, and the stream moves past it only inside the NEXT amgettuple call, so every TID is returned and fetched from the heap while the page it was read from is pinned. VACUUM cannot remove a TID from a page without a cleanup lock on it, nor from any page after it in chain order before it has had that page’s cleanup lock (§11’s guarantee, which is what makes a split or a spill that moves the TID harmless), and the heap’s second pass - the only thing that makes a line pointer reusable - runs only after ambulkdelete has finished on every index. So no TID the scan holds can be recycled under it: the same property btree gets from its leaf pin, obtained from the protocol §9 already proves for the count. Why it matters: a non-MVCC snapshot (SnapshotDirty, SnapshotSelf, SnapshotAny) can see the tuple a recycled slot now holds, so without the pin a scan could return a row under a key it does not have, or return one row twice (once under the old TID’s entry, once under its own). For an AND of several sets one pin suffices (the TID is in all of them, and ambulkdelete has to pass every page of the index), for an OR every child standing at the current key keeps its own - the same rules §9 states for the count (lion_source_pinned()).

  • An IN-list set located past the §15 pin budget is NOPIN and carries no interlock. A non-MVCC scan that has one sets xs_recheck, which filters a recycled slot whose new tuple does not match; the one residual effect - the same matching row twice through two stale entries - needs a thousand-leaf IN list under a dirty snapshot, which no caller in core builds (below).
  • The UNION shape (§29.3) holds no pins in either mode (its windows copy containers out of many pages) and is always rechecked; a union cannot return a TID twice. The WINDOW shape copies out of many pages too, and is simply never built with keeppins: a scan that must hold the page of each batch walks a long range beside other columns entry by entry, as a WALK, whose pins are the ones this section needs. Paused between two calls, a WINDOW holds its images, the rest’s stream (copied leaves with their right links, cases 4 to 7) and nothing else; the next window walks the directory afresh, as a UNION’s does, so what changed in between is cases 1 to 3, 7 and 8. A multi-key query’s sets are located unpinned in either mode too (the tree builder the bitmap path shares drops their pins), and a multi-key scan is always rechecked (§29.6); EXCLUDE USING lion (tags WITH &&) is such a scan, and a conflict it reports through a recycled slot is a row that really overlaps.
  • The §11 deadlock rule for readers holds in both modes: every set is located - every directory lock taken - before the first container page is pinned, and a WALK takes the next directory leaf only after the previous entry’s stream has been closed and its container pins dropped. A WINDOW’s walks, and the count that decides whether a walk is long, lock directory leaves while the rest’s stream is paused, which holds no pin (it is never opened with keeppins). Holding directory-leaf PINS (INLINE sets, the walk’s own leaf) while locking directory pages is what the count’s GROUP BY walk has always done; pins do not block share locks, and VACUUM never waits for a cleanup lock while holding any lock (§11).

Who scans a lion index with a non-MVCC snapshot, and the decision for each:

  • Exclusion constraints (check_exclusion_or_unique_constraint(), SnapshotDirty). DefineIndex refuses EXCLUDE USING <am> for an AM without amgettuple, so EXCLUDE USING lion (k WITH =) becomes possible with this section - an equality-only exclusion constraint, i.e. a unique constraint served by lion’s single-key lookups. It is allowed: its scan is exactly the pinned mode above (one equality key per column, no arrays), the constraint code rescans after waiting for an in-progress conflict, and it re-applies the operator whenever xs_recheck is set. test/sql/indexscan.sql tests it, and test/isolation/gettuple_dirty_pin.spec shows that the pin is held (VACUUM waits for it) where an MVCC scan at the same point holds none.
  • Logical replication, REPLICA IDENTITY FULL (RelationFindReplTupleByIndex(), SnapshotDirty). On 16 and 17 IsIndexUsableForReplicaIdentityFull() accepts only btree and hash, by AM Oid. From 18 it asks IndexAmTranslateCompareType(COMPARE_EQ, ...) for every key column and requires amgettuple; lion has no amtranslatecmptype, so today it is never chosen on any version. If §29.8 ever adds one, lion becomes eligible, and that is fine: the scan keys are one equality (or IS NULL, SK_SEARCHNULL) per column, which is the pinned SETS shape, and the apply worker compares the whole tuple (tuples_equal()) for any index that is not the replica identity index. Nothing to refuse; a note for whoever adds the translation.
  • SnapshotAny: CLUSTER needs amclusterable (false); nothing else in core scans a user index with it. get_actual_variable_range() (SnapshotNonVacuumable) needs an ordered index with amcanreturn; lion has neither.

29.6 xs_recheck

First, the rule every shape keeps: no TID the index does not hold is ever returned - only TIDs that some entry’s posting set held when the scan read it, each at most once. xs_recheck is a statement about the QUALS the index was asked to answer, and it is not a license to return a superset of the index: a plain Index Scan of a partial index has the quals its predicate implies removed from its recheck, and an index-only scan evaluates no recheck qual at all (the planner only builds one for a query that references no column, so there is none to evaluate). A row that the index does not hold therefore comes back as a row. A TIDBitmap’s lossy page is the one superset source lion ever had, and the UNION shape (§29.3) replaced it; every other source is a stream of posting-set containers.

Then xs_recheck is set, for every TID of the scan, when any of these holds; otherwise the TIDs are exact:

  • a qual was not answered: a range beside its column’s sets (§29.2), a second WALK column or long list, or a multi-key column in mode ALL (dropped, as in the bitmap path). IS NOT NULL is always answered (§14); until 2026-10-19 one next to another column that answered was dropped here and rechecked;
  • a MULTI-KEY column’s query in mode ALL (strategies 2 .. 5, §17: a prefix, a NOT, weights, a NULL key, INCLUDE_EMPTY, <@, too many keys): every row, rechecked. A query in mode KEYS is exact on the plain path as it is on the bitmap path, which has always passed it through unrechecked: both rest on lion_extract_query()’s classification - mode KEYS is exactly the rows an AND/OR of whole key sets selects, and a key’s set is its own, a hash tie being resolved by the opclass equality in the directory descent (§21) - and on the one lion_scan_col_tree() they share, which fails the tree (*ok = false) unless every element of the query is in mode KEYS. (Until 2026-10-06 the plain path rechecked every multi-key answer, on the ground that one operator call per fetched row costs nothing next to the fetch. §40’s index-only mode broke that ground: a set reported inexact made the node fetch every member’s heap tuple for the recheck, for a filter the index had answered exactly.)
  • the UNION shape (§29.3) for a query in mode ALL, or in a scan that keeps its pins;
  • a non-MVCC scan with a NOPIN set (§29.5), or with a NULL entry to take away (§14).

A range is exact (the walk applies every bound of the column, as a WALK or a WINDOW), an IN list is exact (a union of whole entries), IS NULL is exact, IS NOT NULL walked or its NULL entry taken away is exact, and so is a cross-type or binary-coerced equality (§21’s probe). A collation the index cannot answer never reaches the AM (the planner matches IndexCollMatchesExprColl() first).

29.7 Rescans, keys that change, and cleanup

  • amrescan copies the new keys, closes the stream, releases every located set (their pins and their pin-budget share, §15), drops the walk’s leaf pin, and resets the per-scan context; the source is opened again, from the new keys, by the next amgettuple. A nested loop’s inner index scan with a Param (t.k = outer.x) is exactly that: one rescan per outer row, one descent per rescan. The per-column probe cache (LionScanCol, keyed by the key’s subtype) survives rescans, as it did for the bitmap path.
  • ScalarArrayOpExpr arrives as ONE key with SK_SEARCHARRAY, as for the bitmap path: amsearcharray stays true because amgettuple answers every array key the bitmap path answers (equality lists as a union, range arrays as one walk, multi-key query arrays as an OR of their trees) - the executor never has to iterate the elements with rescans.
  • pgstat_count_index_scan() and, on 18+, instrument->nsearches are counted once per source opened, i.e. once per rescan, as btree counts one per primitive scan.
  • Statistics. The custom nodes and the SQL functions keep pg_stat_user_indexes and pg_stat_user_tables as the scans above do, so that an index lion reads never shows as unused. A scan of an index (pgstat_count_index_scan(), which sets last_idx_scan too) is counted once per scan and per rescan of each index read: LionCount (lion_count_report_reads(), as a run’s relation closes or the node rescans: each of the relation’s indexes whose directory was read since, which lion_dir_read_since() answers from the mark every directory page read leaves (lion_dir_count_read(); no count asks about a read before its own transaction, so a transaction that leaves more than 64 marks drops them at its end), with the rows counted since, statrows - so an index opened and never read, the group column of a GROUP BY whose WHERE selects nothing or the fk index of a join no dimension row reaches, counts no scan), the FK-side joins, LionOrdered and LionBtreeScan for each lion index whose set they build (the B-tree they walk is counted by core’s own scan under them), LionBm25 and the lionvec scan once per scan, and the SQL functions once per call - lion_bm25_score() once a query, when it reads the index’s statistics into its cache. idx_tup_read counts what the index gave: a count’s rows counted (the distinct values of a count(DISTINCT), the rows of the keys lion_index_facets() counted), credited to each index of the count, since every one of them answered for each; a semi or anti join’s rows returned; the members of LionOrdered’s sets and the entries of its walks of a lion column; the entries of BM25’s walks. idx_tup_fetch (pgstat_count_heap_fetch()) counts the rows a node fetched by the index’s order itself - a walk of a lion column, the rows BM25 returns - and the rows fetched by a set’s TIDs or rechecked in the heap by a count go to the table alone, as a bitmap heap scan’s do (tuples_fetched of the heap relation’s pgstat_info). No node counts a sequential scan. test/sql/index_usage.sql reads the deltas of every path after pg_stat_force_next_flush().
  • amendscan does everything amrescan does and frees the contexts. An ERROR in the middle of a scan leaves pins to the resource owner, which releases them, and the list pin budget is zeroed at the end of the transaction (§15).

29.8 Ordered scans (step 2): deferred

Deferred to the backlog (§23) on 2026-09-24, in favour of an ORDER BY node built on a btree (below). What it would take, so that the step is not re-derived: amcanorder on 18+ only, where the planner maps an index’s < through amtranslatecmptype (strategies 6..9 to COMPARE_LT..GT, 1 to COMPARE_EQ; scalar families only) - on 16 and 17 core looks for btree strategy 1, which is lion’s =, and so could never see an order; a get_relation_info_hook that trims sortopfamily for a column whose RECORDED order is hash order (§21, “The order is the index’s”), for multi-key columns, for multicolumn indexes (lion orders one column at a time) and for DESC/NULLS FIRST indoptions, which amcanorder makes CREATE INDEX accept; NULLs returned last (the reserved NULL entry sorts first, §21); IN lists walked entry by entry in directory order instead of the union of §29.3, because the AM is never told whether the plan needs the order (the executor passes only a direction), so an ordered AM has to produce it always; backward paths, which indxpath.c builds for every ordered index regardless of amcanbackward, either served by a descending walk (§21’s leftlink) or pruned; and ordered-scan costing with a startup cost that lets LIMIT stop early. Everything else - the walk, the per-entry stream, the batch, the pin rules - would carry over unchanged, since a WALK is already in key order.

What the btree-ordered node planned next can use. That node walks a btree on the ORDER BY column and tests each heap TID against lion’s exact answer to the WHERE clause. The source above is that answer: LionSource opened on the WHERE clause’s scan keys in the SETS shape is one ascending, duplicate-free stream of containers (container keys strictly increasing), so “build once, then contains(tid)” is a copy of the containers the stream produces into an array - memory equal to the answer’s container bytes - searched by container key and tested with lion_container_contains(); or, if that is too large, a TIDBitmap filled from the same stream. A WALK source (a range) streams entry by entry, so its union has to be accumulated (a TIDBitmap, or container keys ORed as §15 does) before it can be probed. lion_source_exact() says whether the set is exact (§29.6). The entry points are lion_source_open(), lion_source_next() and lion_source_close() in lion_scan.c, declared in lion_count.h; they take an index, scan keys and the keeppins switch of §29.5, and no IndexScanDesc. §30 is that node (LionOrdered): it copies the containers of one source per lion leaf of core’s bitmap qual tree into a sorted array, sorts and merges a WALK’s or a LIST’s once they are complete, and ANDs and ORs the leaves' sets.

29.9 Index-only scans (step 3), for later - and the one kind that exists already

An index-only scan of a query that needs no column exists already (found while implementing step 1: count.sql failed with core’s no data returned for index-only scan). check_index_only() allows an index-only scan when every column the query needs can be returned, and a query that needs none - SELECT count(*) FROM t, count(*) WHERE <a partial index's predicate> - needs none; amoptionalkey lets the planner scan a lion index with no key at all, and once amgettuple exists it builds that path and sometimes picks it. It is served, not refused:

  • the executor reads the tuple from xs_itup (and its shape from xs_itupdesc, which ambeginscan now sets), so the scan hands it a tuple of NULLs of the index’s own shape, formed once; nothing reads it, because there is no column in the target list or the quals to read (the recheck qual is empty too: a key would put its column among the needed ones);
  • xs_want_itup turns dropPin off, so a WALK keeps the directory leaf of each INLINE entry and the posting leaf of each container pinned until the next batch - the §9 interlock, which is what makes the executor’s visibility-map test of each returned TID safe, exactly as for the count - except on a hot standby over a generic-mode index, whose replay takes no cleanup lock: there every shape’s TIDs are looked up in the heap first, as the UNION’s are below (§9, “Hot standby”; 2026-09-27 review);
  • the UNION shape (a multi-key first column) returns only TIDs the index holds (§29.6) but pins none of the pages they came from, so each of them is looked up in the heap under the scan’s snapshot first (lion_table_fetch_tid()) and only a visible one is handed on - which stays visible to that snapshot, so nothing can take it away before the executor’s own test. A visible tuple at a TID the index holds is a row of the index: a HOT chain cannot change a column the predicate reads. Slow and exact; the count pushdown answers the common shapes instead anyway.

What a real amcanreturn would need from this scan:

  • A value to return. A lion column’s stored key is the value only under the §10 value-representation contract (BTEQUALIMAGE_PROC, the bpchar allowlist): citext’s two spellings share an entry, so the stored key is not every row’s value. amcanreturn is per column and has to apply that contract; multi-key columns can never return (their keys are extracted, not the column). The value comes from the ENTRY, not from a TID, so the scan sets xs_itup (or xs_hitup) once per entry and returns it with every TID of that entry - which needs the WALK shape or a per-entry list, since a SETS union by container key does not know which entry a TID came from. A single-key equality knows it trivially.
  • The §9 interlock. An index-only scan skips the heap for all-visible pages, which is exactly the count’s situation: the pin on the page a container came from has to be held until the executor has checked the visibility map for that container’s TIDs. xs_want_itup therefore turns dropPin off (btree does the same, nbtree README above), and the batch rule of §29.5 already holds the pin until the next batch is pulled - i.e. until every TID of the current container has been returned and its VM bit tested. Nothing else changes; a NOPIN set then has to be relocated under a pin (lion_posting_set_relocate(), as the count does) rather than rechecked.
  • Costing: cost_index() applies the all-visible fraction itself once canreturn is set.

29.10 Backward scans and more (step 4, later)

A descending walk (directory leaves by leftlink, which §21 keeps and verify() checks; posting leaves have none, so a descending container order needs the tree’s pivots or a reversed batch), (a, b) order on multicolumn indexes, and - if ever wanted on 16/17 - an ordered CustomScan.

29.11 Cost (lioncostestimate())

Plain and bitmap index paths are the same IndexPath and share amcostestimate; the planner prices the heap side of each (cost_index() for a plain scan, cost_bitmap_heap_scan() for a bitmap one). The only input that differs is indexCorrelation, which only cost_index() reads, and lion used to return 0 for it. It then returned what btcostestimate() returns: the ANALYZE correlation of the column whose qual the scan answers (STATISTIC_KIND_CORRELATION for the type’s default <, the operator ANALYZE computes it with), times 0.75 for a multicolumn index as btree does; 0 for a multi-key column and whenever no statistic exists - on the argument that within one key a lion scan returns TIDs in heap order (as btree does since its heap-TID tiebreaker), so how the matching rows are spread over the heap is what the column’s correlation describes. The index side is unchanged. The count pushdown competes at the upper rel with its own cost (§10); the regression suite pins its choices. A benchmark (2026-09-27) showed that number to be wrong twice over, and it is now two things (lion_plain_heap_correlation()):

The correlation, less ANALYZE’s tie-break (lion_index_correlation()). ANALYZE sorts its sample by value and equal values by their place in the heap, and correlates the sorted order with the heap order. For a column of few values placed at random that comes out at exactly the sum of its values' squared frequencies S, not 0 - 1/k for k equally common values, 0.82 for a 90/10 boolean; on 8M rows placed at random: 0.4406 for three values at 60/20/20% (S = 0.44), 0.8171 for 90/10 (S = 0.82), 0.6852 for 80/20 (S = 0.68). cost_index() read that as the rows of one value packed on fewer pages than they touch, and lion is made for exactly such columns. The correlation is now taken beyond that baseline, (corr - S) / (1 - S) clamped to [-1, 1], S from the MCV list and an even share of the rest: 0 for values placed at random, still 1 for a column stored in value order (-1 for one stored in reverse, whose values are packed as well), and the raw number for a column of many values. This applies to every plain lion scan, and is the number handed on for those that are not in heap order.

A scan in heap order is priced as the bitmap heap scan of the same rows, and a little more. The SETS, UNION and WINDOW shapes (§29.3, lion_source_sorted(); lion_plain_scan_shape() mirrors the choice at plan time) hand out their TIDs in heap order whatever the column’s correlation: they read every page that holds a match once, in block order - exactly the bitmap heap scan’s pages, in its order. cost_index() interpolates its I/O by the correlation squared between rows packed on sel x heap pages read in order and rows scattered over Mackert and Lohman’s count of pages, each a random read; a scan in heap order has the first end but not the second, where it reads what compute_bitmap_pages() counts (Cardenas’s formula) at cost_bitmap_heap_scan()’s price a page (random_page_cost falling to seq_page_cost as the pages approach the whole heap). So the I/O is that, interpolated towards the packed end by the correlation above, and the correlation returned is the one that makes cost_index() arrive at it. Per row cost_index() charges cpu_tuple_cost and the quals left to the heap; the plain scan is charged what cost_bitmap_heap_scan() charges a row besides - the bitmap entry (a tenth of a cpu_operator_cost, LION_BITMAP_ROW_COST, core’s own number), and every index qual only when the scan RECHECKS them (§29.6: it dropped a qual, it reads a multi-key column, or it is a UNION; lion_plain_scan_shape() answers that too) - and LION_PLAIN_FETCH_ROW_COST, one cpu_tuple_cost, for every row past the first on its page: the bitmap heap scan locks a page and walks its matches once, the plain scan makes an amgettuple call and takes a buffer lock and a HOT search per row. Measured on the release build (backend CPU time, heap in shared buffers, the scans interleaved; §31): 11.5 ns a row more than the bitmap heap scan for 40,000 of 60,000 numeric rows stored in value order, 21 ns for 75,000 rows of 30 days stored in order, 20 ns for 43% of the 200,000-row table of plaincost.sql on every page - about the 20 ns a cpu_tuple_cost stands for at the 500 units a millisecond of §10’s “The units” (it was half that, from 20 to 50 ns on an assert build against 55 ns for a sequential scan’s row). Where the rows are scattered the plain scan spends 0.5 us a page LESS than the bitmap heap scan - 23 ms against 33 for 25,000 rows at 0.4 a page, 71 against 104 for 100,000 at 1.5 - which is not credited: the plain-scan costing under way may. (Until 2026-09-27 every index qual was charged per row, as cost_bitmap_heap_scan() charges it. An exact scan evaluates none - and neither does a bitmap heap scan on an exact page, though core charges it - and the charge lost pushdown.sql’s lion_pdn count, 40,000 of 60,000 numeric rows stored in value order, to a sequential scan that takes twice as long: its one numeric = a row is priced as an int’s.)

No operator per index tuple (2026-09-27). genericcostestimate() charges every index tuple it expects to visit cpu_index_tuple_cost and a cpu_operator_cost per index qual, as a btree evaluates its quals on its tuples. A lion scan evaluates none: its index tuples are the members of the posting sets its quals located, turned into TIDs a container at a time. lioncostestimate() takes the operator back out and leaves cpu_index_tuple_cost a TID, which is about what a TID costs there (5 to 15 ns into a bitmap, against the 10 ns cpu_index_tuple_cost stands for at 500 units a millisecond). With it, lion_pdn’s bitmap scan (2.5 ms of CPU) is chosen at 1,085 over the plain scan (3.0 ms, 1,099) and the sequential scan (5.3 ms, 1,116); the test’s plan with every scan disabled prints it.

The AND of sets (2026-09-27, lion_set_merge_cost()). genericcostestimate() prorates the index by the selectivity of all the quals together, which prices an AND by what it RETURNS. What the scan reads is the sets: every set qual it answers (§29.2), of every key column and as many as one column has, is a posting set (an equality, IS NULL, a multi-key query) or an IN list’s union (a multi-key query that needs every row is dropped and rechecked instead, and is none of them), and it ANDs them as a count does - the smallest drives, the others are sought at its container keys - so lioncostestimate() now charges §10’s merge CPU for them (lion_merge_cpu_cost(), the count pushdown’s own price) and the pages the driver walks and the probes touch, beyond the prorated share, at seq_page_cost. One set alone, and a range or IS NOT NULL (a walk, priced by lion_range_walk_cost()), add nothing. Four dense sets whose AND was 1,730 rows of 8M ran 52 ms on lion against 1.2 ms on a btree over the four columns at about the same cost (6.5k against 6.2k). On §31’s 5M-row w the same shape, four sets ANDed to 48,000 rows, raised the lion plain scan from 80,300 to 83,900, and the planner now takes the btree’s bitmap heap scan (79,800; 52 ms of CPU) over it (82 ms). Plain and bitmap paths share the estimate, and both make the same AND (§29.2). (Superseded by “One price for the AND of sets”, below: the merge is still the count’s, and now so is everything else about the sets.)

One price for the AND of sets (2026-09-28, lion_cost_set_and()). The count pushdown and a lion index scan make the same AND of the same posting sets, and were priced for it by two models. A benchmark’s count over several correlated filters answered by one multicolumn index ran faster as LionCount than as the plain lion scan the planner chose, and was priced well above it, for two reasons; this is the first (“Correlated sets”, below, is the second). For the same clauses the two prices differed in:

  • the lookups. The count charged each clause’s directory leaf at random_page_cost and its descent, LION_DESCENT_COST a level (§21), and an IN list’s values their leaves, read in key order at lion_heap_page_cost(), and a search each. The scan charged none of them: genericcostestimate() prorated the index by the selectivity of the quals together - the pages of the RESULT’s postings at random_page_cost, one at least, spread over an IN list’s num_sa_scans - so clauses whose product is a few rows paid for about one page however many there were.
  • the unions. The count read every container of an IN list’s sets (LION_CONTAINER_COST, and LION_MEMBER_COST a member up to LION_MEMBER_CAP) and merged them (lion_merge_ops(), in cpu_operator_cost); the scan took the list for one set and charged neither.
  • the containers. The count counted a key’s containers by its column’s correlation with the heap (lion_key_containers()), the scan as scattered (lion_containers_for()); both merge them with lion_merge_cpu_cost().
  • the posting pages. The count charged the driver’s walk at seq_page_cost and the pages the others' probes touch at lion_heap_page_cost(), a clause’s pages being its column’s share of the index’s container pages - the index less its meta page and directory - by the clause’s selectivity, one at least (an IN list: one a value). The scan charged the whole index’s share by the selectivity, and only what exceeded the prorated pages, all at seq_page_cost.
  • the clauses. The scan answered one qual per key column - an equality before a list before a range before a null test, the rest left to the recheck - until 2026-09-28, and every set qual of every column since (§29.2, samecolumn_scan.sql), as the count ANDs every positive clause; lion_set_merge_cost() then counted every set qual a source, priced as above. A scan and a count over the same clauses now AND the same sets. What is no set is each path’s own: a range, which the scan walks on a column without sets (lion_range_walk_cost()) and the count may take for a source (§32), and a multi-key query that needs every row, which both leave to a recheck of the rows the other sources leave, and neither price counts a source of its AND: the scan’s by lion_cost_qual_is_full(), the count’s since 2026-09-28 by lion_multikey_cost_mode(), the count’s own extraction of the same query (§17, “A query known only at run time”).

One function prices it now. lion_cost_set_and() (lion_plan_cost.c) prices each clause as lion_cost_count_rel() prices a WHERE source - lion_cost_set_clause(): its lookups, what a walk of its sets reads, its containers and, for a list, the union - intersects them with lion_merge_cpu_cost(), and charges the driver’s walk and the sought sources' pages (lion_cost_set_pages()), and the lookups' directory leaves (lion_cost_leaf_pages()); lion_cost_count_rel() takes the same terms from the same functions for its sources, clause by clause. lioncostestimate() feeds it every set qual the scan answers, of every key column and several of one - item for item the classification lion_scan_choose() makes, which lion_cost_col_quals() mirrors, less a multi-key query that needs every row (lion_cost_qual_is_full()) - and where there are two or more, charges it in place of the pages genericcostestimate() prorated (lion_generic_page_cost(), that function’s own formula) and of lion_set_merge_cost(), which is gone; it keeps what is the scan’s: a TID a result row at cpu_index_tuple_cost, the quals' arguments, and the heap, which core prices. A nested loop’s inner scan repeats the AND, and its pages are spread over the repetitions as genericcostestimate() spreads its own (index_pages_fetched()), as are a correlated subquery’s over its calls (§40.11). What is the count’s own is the heap recheck of the pages the visibility map cannot vouch for, and its row; its visibility-map checks are inside the merge’s constants, which were fitted on counts (LION_MEMBER_COST is the mask built from a container’s members), so the scan is charged them too - 0.74 ns a member beside the 10 ns a TID it pays anyway. Over the same sets an ungrouped count therefore costs the scan’s index side, its recheck and a row, and no more than the scan with its heap. One set alone is priced as before: what the scan reads of it is what it returns, which is what the prorating prices.

Every lion path of two sets or more is priced anew. A small result over several clauses costs more: a directory leaf a lookup at random_page_cost where the prorating charged one page for all of them, and their descents; but no more leaves of one index than its directory has (lion_cost_leaf_pages()): lookups into a directory of one leaf read it once, and the second is a buffer hit. The count had charged every lookup its own leaf, which only ever differed for several clauses on one multicolumn index with a small directory, and it now charges the same. Two equalities on a larger directory cost about 5 more at the default random_page_cost; on a directory of one leaf, about 1.5. A large result costs less there, its sets' pages walked and sought at seq_page_cost and the cache-aware price where they were prorated at random_page_cost: plaincost.sql’s 43% query, three dense sets, by roughly a third on the index side, and by a few percent at a random_page_cost of 1.1 (derived from the model, not measured). On PostgreSQL 16, in both WAL modes, no plan the regression suite pins moves but one. ordered.sql’s LionOrdered over a = 3 AND (b = 5 OR c = 7) ORDER BY k LIMIT 50 took for its lion side the BitmapOr of a = 3 AND b = 5 and c = 7, and now takes a = 3 alone: the arm’s AND rose by its lookups, from 11.3 to 17.1, and the node is priced 86.6 with a = 3 against 86.9 with the BitmapOr and 88.4 for the ordered btree scan. The new side is the faster: 0.12 to 0.14 ms for the 50 rows against 0.15 to 0.17, every page in shared buffers - 308 heap fetches against 192, but one set located where the BitmapOr locates three and ANDs two. Two units from the btree scan is too close to pin, and the test is about the BitmapOr (a restriction clause an arm reuses stays a filter), so it now shows the node’s plan with plain index scans off, which leaves core’s bitmap path, the BitmapOr, for its lion side; its expected plan is the one it printed before. Disabling every core scan instead would not do: core prices a disabled bitmap path out of the lion-only relation the node takes its lion side from, and the node then shows a = 3 whatever the price. The §31 comparison above of four sets against a btree’s bitmap heap scan was made with the model this replaces and has not been made again.

Correlated sets (2026-09-28, the intersection probe, lion_isect_factor() in lion_selfuncs.c; pg_lion.enable_intersection_probe, bool, default on, PGC_USERSET). The second reason: core estimates a conjunction as the product of its clauses' selectivities, and filters on columns derived from one hidden attribute are far from independent. The benchmark’s came out a small fraction of their rows, the heap paths were priced for that, and the count, whose price barely depends on the rows, lost to them on paper. Each such clause is a posting set with an exact row count, and the index can AND the sets' containers at any container key, so lion measures the AND for its own estimates.

  • What. The probe locates each clause’s sets as a scan locates them (lion_scankey_sets(), §29.2’s per-column trees), takes the source of the fewest rows as the driver, and reads it a container at a time. Since 2026-10-19 it samples the driver’s containers by their rank among its own (“A sample of the driver’s own rows”, below): a driver of a few posting leaves is walked, the containers some other source has no container at are ruled out, and the rest are counted - every one ANDed - or sampled every n/32-th; a single larger set is sampled at its posting tree’s separators. A driver neither ranks - a list or a multi-key query of many sets - is read as every driver was before: the heap’s container keys in strata, and the driver’s first container in each (lion_stream_at()), so that the sample spreads over the heap - every container of a heap of no more container keys than strata. At each container sampled it ANDs the other sources' containers at that key, the sources of fewer rows first, and stops at the first that has none. A sample counts for its stratum’s width, and the strata still ahead widen when the samples so far, at what they have cost, would overrun the probe’s budget, so that the sample covers the heap whatever the seeks cost (lion_isect_pass()); a driver found in fewer than LION_ISECT_MIN_KEYS strata - rows in one stretch of the heap, as a hidden attribute that follows the order the rows came in lays them out - is sampled again, in strata of that stretch. The share of the driver’s rows that survive, times the driver’s rows, is the conjunction’s rows: the sets' exact count for one set or an IN list’s disjoint entries, core’s estimate for a multi-key query of several keys, whose sets overlap - or, for a driver walked, the walk’s own count of the rows its sets hold.
  • Where it can matter. Two or more set clauses of one lion index with a value to look up at plan time - an equality, an IN list of at most LION_ISECT_MAX_SETS values, IS NULL, a multi-key query, each with a constant (a Param has none); since 2026-09-28 in their canonical form (lion_canonical_clause()), an OR of equalities on one column as the IN list it spells and a boolean column tested by itself (NOT flag) as its equality (“An OR of equalities is its IN list”, below) - on an index that is neither partial nor hypothetical, of a relation that is not an inheritance parent; and core’s estimate of their AND at most a third (LION_ISECT_ERROR) of the smallest clause’s own, which is the most the AND can hold. Independent clauses that are both selective pass that test as well; the probe then finds what core estimated, and changes nothing. And only where the user may read every row of those columns - SELECT on the table or on each column, no security barrier view’s or row-level policy’s quals, an inheritance child judged by the table the query names - which is what core asks before a column’s statistics reach a user’s function (all_rows_selectable()): the probe counts the rows themselves, the ones the user may not see included, and EXPLAIN shows the estimate.
  • What it corrects. A factor on core’s estimate of the same clauses, applied when the probe finds it LION_ISECT_ERROR (3) times off or more, either way: less is within what ANALYZE’s per-column estimates and the sample leave of an independent conjunction, and would move plans on noise. A position filter’s candidates take the probe’s measure however near core’s (lion_isect_measured_factor(), §17 “The price”): they are not an estimate from which a plan is sized but the rows the filter decides, each at a price, and a phrase’s words go together. A sample lowers an estimate no further than what it can rule out, and one that met the AND in fewer than LION_ISECT_MIN_HITS containers raises it no further than the rows it met, or than the longest part of the AND that enough of them met says (“What a sample can rule out”, “Wide lists”, below); a count - every container of the driver ANDed - stands as it is. It applies to lion’s OWN estimates: the selectivity lioncostestimate() returns for a plain or bitmap path - the Bitmap Index Scan’s rows, the heap side of both paths, LionOrdered’s lion side (§30.3) - and the rows the count pushdown’s recheck is priced for (lion_probe_rel_rows(), the relation’s restriction clauses on the lion index that answers most of them), which since 2026-09-28 are also LionOrdered’s rows (§30.3), and the merge’s guess of how often the AND’s later sources are sought (“Wide filters”, below). Whatever else a conjunction holds - a range, a clause without a constant - is taken as independent of them, as core takes it.
  • What it leaves. Every other path’s estimate stays core’s. rel->rows did too until 2026-10-07, and is now corrected (“The relation’s rows”, below). Core’s own answer to correlated columns applies to every path: CREATE STATISTICS ... (dependencies, mcv) on them corrects core’s estimate, and the probe then agrees with it and changes nothing. (Until 2026-09-28 the probe did not correct the merge’s guess of how often the later sources are sought either - lion_merge_cpu_cost()’s 1 - exp(-lambda), §22’s open item - which took the sets as independent; it does now, for an ungrouped count and a scan, “Wide filters”, below. A grouped count’s merges still take them as independent.)
  • Visibility. The TIDs are counted whatever their visibility: the probe reads no heap page and holds no lock or pin from one container to the next (the sets are located and their pins let go of before the first container is read, and the streams copy each posting leaf and release it, §29.5), so it has none of the endpoint probe’s heap bounds to keep (§28). A dead row not yet vacuumed is a row of its sets, as in their own counts, and dilutes the driver and the survivors alike; a heap path fetches it too. What that misjudges is an AND whose rows were deleted in bulk and not yet vacuumed, which is measured as still there until VACUUM - where the endpoint probe, which stakes an estimate on one key at one end, must and does test liveness.

  • Bounded. LION_ISECT_KEYS (32) driver containers, however many sets the clauses locate, and a probe that cannot afford LION_ISECT_MIN_KEYS (8) of them gives up; at most LION_ISECT_MAX_SETS (32) sets sought at each, a conjunction of more measured without its widest clauses (“Wide lists”, below); and a budget of buffer accesses a planner run (pgBufferUsage, lion_isect_run_budget()): LION_ISECT_RUN_BUFFERS (1,024), and beside it what 32 samples of the widest conjunction the run measures cost at LION_ISECT_SEEK_BUFFERS (2) a seek of each of its sets - 2,048 more for 32 sets. A probe may take what the run has left, the relation’s own conjunction first; a conjunction that what is left cannot afford eight samples of is measured without its widest clauses, and past that the probe gives up and what the run measured of a part of the conjunction stands (lion_isect_part_factor()), or core’s estimate where it measured none, as it does when the heap has outgrown rel->pages twice over. At most LION_ISECT_RUN_PROBES (8) probes a planner run, and each conjunction measured once in it (cached beside the endpoint probe’s ends, per PlannerGlobal) - once for its table: the cache matches a conjunction’s clauses by the table’s columns, and every lion index of the table that answers the same clauses takes the one measurement, whichever asked first (LionIsect). The share of the rows an AND keeps is the heap’s, and each index holds the same TIDs. Measured once an index, a table of three lion indexes would spend three probes a conjunction, the run’s eight would run out at the third conjunction’s third index, and that index’s path would keep core’s product - one row for 1,280 on isectwide.sql’s table - under a relation of the probe’s rows (lion_isect_rel_factor() takes them from the index with the most set clauses), and be taken for the price that gives it. The budget is checked before each clause is located and each sample taken, so a probe overruns it by one IN list’s lookups or one sample’s seeks at most: a planner run accesses 3,072 buffers and that much more at worst, the scale of a dozen endpoint probes (100 heap pages and 100 leaves an end, §28). The sample by rank reads inside the same budget: a driver’s walk at most LION_ISECT_WALK_PAGES (32) buffers, the other sources' presence walks 32 each, a posting tree’s separators at most LION_ISECT_SEPARATOR_PAGES (33) pages (“A sample of the driver’s own rows”, below).

    Measured on PostgreSQL 16 (2026-09-28: EXPLAIN (BUFFERS)’s planning buffers with the probe on and off, and each probe’s own count). The five-clause count of isectprobe.sql plans with 79 buffer accesses against 20 without the probe - two conjunctions, of 22 and 37 - and the same shape on 2M rows (782 container keys) with 530 against 20: 198 and 237, the second’s 21 strata widened to 19 to fit its budget. Other conjunctions of two to five clauses on 2M rows took 13 to 155 a probe, three clauses one of them an IN list of 20 values 126. The first version - 64 containers, 2,048 buffers a probe and 8,192 a run - took 547 for the five clauses on 2M rows and 889 for the planning. Every one of those conjunctions came out within 6% of its true count, uniform, clustered in one stretch of the heap, or sparse, where the decision needs a factor of three: the containers are a small multiple of what it needs, a probe’s budget is the dearest probe measured, and a run’s four of them, where a query measures one or two conjunctions a relation. Planning rose from 0.12 to 0.74 ms for the 2M-row count and from about 0.17 to 0.3 to 0.5 ms for the test’s, every page in shared buffers; on a cold cache the accesses are reads, as many as the run’s budget at most.

test/sql/isectprobe.sql pins it: five filters derived from one hidden group, whose product is two rows for 2,000, estimated at 2,000 with the probe and at 2 without it; a pair of independent columns whose estimate the probe leaves exactly as it was; and the ungrouped and the GROUP BY count going to LionCount only with the probe. Without it the lion scan that leaves the IN list to its heap filter and ANDs the other four sets is priced for two rows, a third of the ungrouped count and a ninth of the grouped one; with it, at fourteen and four times theirs. The ungrouped count ran in 0.2 ms against that scan’s 1.2 to 1.7.

The relation’s rows (2026-10-07, lion_correct_rel_rows() in lion_selfuncs.c; pg_lion.enable_rows_correction, bool, default on, PGC_USERSET). Correcting lion’s own estimates left the joins above the relation sized from core’s product: on a private benchmark, a join of a dimension filtered by correlated columns was estimated at a small fraction of the rows the filters left, and the nested loops chosen for that ran far past the alternatives. Core sizes a relation (set_baserel_size_estimates()) before it builds any of its paths and has no hook between the two, so the correction is made in set_rel_pathlist_hook, first in lion’s chain, once the paths are built - and it is made to the paths too:

  • rel->rows becomes lion_probe_rel_rows() - core’s estimate of the restriction clauses with the probed AND in place of core’s product - where the probe found that AND LION_ISECT_ERROR times off (lion_isect_rel_factor() other than 1), and wherever lion’s estimates take the restriction clauses in another form than core’s (lion_rel_canonical_changes()): an OR of equalities as its IN list (“An OR of equalities is its IN list”, below), a run of <> on one column as one exclusion (“A run of <> is one exclusion”, below), the tests of whether a lion-indexed array is empty as the one whole-array clause they come to (§17, “Non-empty arrays”) - with the probe’s factor or without it, the probe on or off, of a relation one of whose indexes is lion’s. Elsewhere nothing changes.
  • Every path already built takes the same ratio on its rows. An unparameterized scan’s rows are rel->rows (a partial path’s divided among its workers) and its cost does not depend on them - a sequential scan’s on the tuples it reads, an index or bitmap scan’s on its own clauses' selectivity - so the paths built before the hook and after it, the Gathers and the joins, all see one relation. A parameterized path’s rows are its ParamPathInfo’s (a partial path’s divided among its workers): the same restriction clauses with its join clauses ANDed in, taken as independent of them. Each ParamPathInfo’s rows are computed again as get_parameterized_baserel_size() computes them, with the restriction clauses as lion_probe_rel_rows() takes them and the factor applied before the rows are clamped to one, and no more than the corrected rel->rows (lion_param_rows()); each path takes the ratio of its ParamPathInfo’s new rows to its old (lion_correct_path_rows()), so that it still agrees with it. The ratio of rel->rows would not do: a lookup on a unique key is a small fraction of a row in core’s product, which core clamps to one, and one row times the factor is as many rows as the factor - 125 for the nested loop’s inner scan of the primary key in rowscorrect.sql, which no unique key returns. A ParamPathInfo first made after the hook (a reparameterization) has core’s, which is no more than the corrected rel->rows.
  • lion_probe_rel_rows() then reads rel->rows as it is, rather than applying the factor twice; inside a probe scope it recomputes from the clauses, and applies it, as before. The factor multiplies core’s rows before they are rounded and clamped to one row (“Wide lists”, below), for rel->rows and a ParamPathInfo’s alike.
  • A plain relation of its own only: an inheritance parent’s rows are its children’s sum, added up before any child’s paths are built, which a child’s correction would not reach.
  • It is made, too, where the relation tests whether a lion array_ops column is empty (§17, “Non-empty arrays”): lion_rel_canonical_clauses() puts in place of every such clause of a column the one clause core estimates from the column’s statistics for the rows they select together (lion_nonempty_rel_clauses()), where core estimates each spelling - a function of the column - from none, and multiplies them. That correction needs no factor of the probe’s, so it is made where lion_isect_rel_factor() is 1 and with the probe off as well; a ParamPathInfo’s rows take the same clauses (lion_param_rows() is handed the canonical ones).

test/sql/rowscorrect.sql pins it: three filters derived from one hidden group, whose product is about 20 rows for 1,000, are estimated within a factor of two at the scan and at a join and semi join of 3,000 rows above it, and fifty times short without the correction; an independent pair is left exactly as it was, and with the probe off nothing changes. Inside a nested loop, the same filters beside a lookup on the primary key are estimated at one row with the correction as without it, and beside a lookup on a column of seven values independent of the group at a seventh of the filters' rows, where core’s product says one. Two tests of whether an array is empty beside a lookup on a column of 31 values are estimated within a tenth of the rows they return, with the probe on and off, both as a nested loop’s inner scan and with the value a constant, the two estimates agreeing; without the correction, at about a third.

(2026-10-19: the correction carries what the probe measures, mistakes included. On a private workload’s table loaded in batches, a sample that met none of a conjunction’s millions of rows measured it empty, and the table was sized at one row for every plan above it. The probe now samples the driver’s own containers and lowers no estimate further than its sample can rule out (“A sample of the driver’s own rows”, “What a sample can rule out”, below); the correction is as it was.)

An OR of equalities is its IN list (2026-09-28, lion_or_as_array(), lion_canonical_clause()). A benchmark wrote one wide filter two ways, each multi-value column as a = x OR a = y and as a IN (x, y), and the two planned differently, at different prices, and were estimated differently. They select the same rows, and where lion answered, priced or measured them they diverged in six places, found on a synthetic repro (6M rows, 200 hidden groups each driving several columns of one wide lion index, the filter of one group: equalities, short IN lists and a multi-key && - twenty posting sets, 3,835 rows):

  1. Core’s selectivity. Core estimates the OR as though its arms could overlap, 1 - (1 - s1)(1 - s2), and the list as the disjoint entries they are, s1 + s2: 15 rows against 22 for the same filter. lion’s own estimate of the relation’s rows (lion_probe_rel_rows()) is computed as set_baserel_size_estimates() computes it, from the list in the OR’s place (lion_rel_canonical_clauses()), and so is the probe’s estimate of the conjunction it measures; and rel->rows is corrected to the same estimate (“The relation’s rows”, above).
  2. The probe. lion_isect_scankey() took an equality, a list, IS NULL and a multi-key query, no OR - and no boolean column by itself, which is how flag = false reaches the restriction clauses (NOT flag) while the index path is handed flag = false. So the count’s rows (lion_isect_rel_factor(), over the restriction clauses) were measured on another conjunction than the index path’s - 8 clauses against 9, a second probe - and, for the OR form, on 4 of the 9, the ORs left out. Every clause is taken in its canonical form now (lion_canonical_clause()): the list for such an OR, flag = false for NOT flag; on 18 the list core derives for the index and the one lion derives from the restriction are one conjunction of the run’s cache (equal()), and the filter is measured once whichever form it has.
  3. The count’s price. An OR’s leaves were priced as clauses of their own - a directory leaf at random_page_cost and a descent each, their union members x log2(leaves) (§19) - and a list as §15 prices it - its leaves in key order at lion_heap_page_cost(), a search each, lion_merge_ops(), and no union at all where it is summed: 37,379 against 37,406 for the repro’s filter, and three to four times apart for a longer list alone (k of 200 values, 10 of them: 3,177 against 654; k of 5,000 values, 1,000 of them: 56,221 against 25,994).
  4. The count’s execution. An OR’s leaves are located one by one, a list’s in one walk of the directory: 40 directory pages against 22 for the repro’s filter. What the merge ANDs is the same union either way - 27,365 containers visited for both forms, 73 seeks avoided, the same buffers but those directory pages, and the same time (110 to 135 ms warm on PostgreSQL 18, which the shared machine’s noise does not separate). A list that is the only source is summed (§15) where the OR was merged; ANDed with anything, both merge.
  5. The index quals of the plain and bitmap scans. PostgreSQL 18 matches such an OR to an index column as the list (match_orclause_to_indexcol()): an IndexClause whose rinfo is the OR and whose index qual is the list. 16 and 17 match it to nothing but the arms of a BitmapOr, each an index path of one arm and the clauses beside the OR, so the OR form’s lion scan left the lists to the heap filter where the IN form’s ANDed them. lion_or_list_paths() builds the paths 18 builds, for the lion indexes only: create_index_paths() on a scratch copy of the relation that sees them, with the list in each such OR’s place among their restriction clauses and no join clause, as LionOrdered builds its lion side (§30.2); then hands them back, each list’s IndexClause pointing to the OR, so that the plan’s index condition is the list, its recheck the OR, and neither is left to the filter. The count’s own planner and LionOrdered’s lion side take the same lists. Not beside a security barrier’s or a policy’s quals.
  6. The BitmapOr’s arms. Every arm is an index path of its own, and each was a conjunction to measure: on 18 the OR form’s planning measured five - the arms of the ORs with the other lists - and spent the run’s budget (1,038 buffers), and the count’s own conjunction was refused its probe. An arm of such an OR is its whole list in the probe now (lion_or_arm_list()): the conjunction measured is the one with the list in the arm’s place, and its factor is applied to core’s estimate of the arm’s, which is the list’s times the arm’s share of it. On 18 that is the relation’s own conjunction - the OR form measures one conjunction, as the IN form does, 352 buffers - and on 16 one conjunction an OR.

Which form executes: the list’s, everywhere - its lookups are one walk, its batches bound a list past the budget (§15), and alone it is summed. Measured on PostgreSQL 18, LionCount forced, median of five, before (the OR’s union of leaves) and after (the list’s), the IN form’s own time beside it:

query IN OR before OR now
k200 IN (2) alone 0.29 ms 0.59 0.24
k200 IN (10) alone 0.96 4.97 0.87
k200 IN (100) alone 10.2 10.0 10.2
k5000 IN (10) alone 0.25 0.72 0.42
k5000 IN (300) alone 8.0 25.4 13.0
k5000 IN (1000) alone 21.7 88.3 27.9
k200 IN (10) AND x = 1 10.9 10.8 11.2
k5000 IN (300) AND x = 1 30.0 31.0 24.8
k5000 IN (1000) AND x = 1 101.6 107.9 110.1

The AND rows do the same work in both forms and differ by the machine’s noise (the same query’s runs spread by 20 to 30% there); the list alone is up to five times faster than the union. On 16 the same: k200 IN (10) alone 6.9 ms as the OR, 0.75 now.

What stays an OR: arms of another operator or constant type, a NULL arm (which eval_const_expressions leaves as an OR of a NULL), a Param (core builds an ARRAY[...] of them for the index; lion leaves the restriction to §19), two columns, a range arm. test/sql/orlist.sql pins it: the repro’s filter and two others in both forms, the same count plan at the same cost on every version, with the probe and without; LionOrdered’s rows (§30.3) the probe’s for both forms, within twice the 287 rows the filter holds, where without the probe they are core’s product, a hundred times short; and every answer - the count, the rows, a GROUP BY, a page - the sequential scan’s, for those and for five ORs that are no list. On 991c315 the plans and costs differ, the count is the OR’s union, and LionOrdered is priced for core’s product in both forms.

A run of <> is one exclusion (2026-10-20, lion_neq_runs(), lion_canonical_clause_list(), lion_cost_canonical_sel()). A filter of a column with a high null fraction by dozens of exclusions - col <> 'x' AND col <> 'y' AND ... - is sized by core’s estimate at a tiny fraction of the table’s rows, and a nested loop over another table’s B-tree chosen for that can run for minutes on a count. Core estimates k <> v as 1 - freq(v) - nullfrac (neqsel(): eqsel()’s estimate of k = v, less the NULLs, which no <> keeps) and a conjunction as the product of its clauses, so the NULLs are taken away once a clause. k <> ALL (array) - NOT IN - fares no better: scalararraysel() sums it as disjoint, 1 - sum(freq(v) + nullfrac), and once that is below zero - two values of a column half NULL - falls back to the same product. On a synthetic table (PostgreSQL 18, 1,000,000 rows, a text column 35% NULL over 150 values):

filter core lion actual
13 <> 3,452 594,741 593,523
80 <> 1 303,577 303,087
NOT IN of the same 80 values 1 303,577 303,087

The rows of two values are disjoint, and a NULL is neither, so the share of the rows a run keeps is 1 - nullfrac - sum of freq(v) over its distinct values, freq(v) being what eqsel() takes for k = v - its frequency among the most common values, or the average of the others - and nullfrac the one neqsel() reads. That is what lion’s estimates take:

  • Which clauses. In a list of clauses of one relation, every expr <> Const, either way round (commuted, so that expr is on the left), and every expr <> ALL (array Const) whose operator core estimates with neqsel(), over an expression that holds a column and calls nothing volatile; those with one expression, operator, input collation and constant type are a run, NOT IN and <> alike. A NULL constant and an array with a NULL element are not taken (no row passes them, which is core’s estimate), nor an empty array (every row), nor a Param: a generic plan’s run is core’s.
  • One clause. The run is ONE clause of the list, in its first clause’s place: expr <> ALL of its distinct values, a RestrictInfo whose cached selectivity (norm_selec, outer_selec) is the share above (lion_neq_run_clause(), lion_neq_run_sel(), which asks var_eq_const() what neqsel() asks it). clause_selectivity() returns a RestrictInfo’s cached selectivity before it looks at the clause, so every estimate that reads the list through clauselist_selectivity() takes the share, and multiplies it with the other clauses as core multiplies any clause it has nothing to combine with. Extended statistics on the column and another (CREATE STATISTICS ... (mcv)) still see the clause, and may estimate it with them.
  • Distinct values. A value written twice, or two that the type’s equality calls one ('Red' and 'red' under citext, or under a nondeterministic collation), are one most common value to eqsel(), and are counted once: the values are sorted with the comparison of their type’s default btree class under the clauses' collation and told apart by it, which requires that class’s family to hold the run’s equality, so that two values the comparison calls equal are equal to the operator. A type without one keeps core’s estimate, and so does a run of one value, which core estimates exactly.
  • Past every row. The share stands while the frequencies add up to no more than the rows that are not NULL. A value that is not a most common one is taken at the average of the others, as if it were in the column, and a run of more such values than the column has - values mostly in no row, k NOT IN a list of another table’s keys - adds up to more than every row: 2,400 values of a column half NULL whose 2,000 values have 25 rows each come to 60% of the rows, against the 50% not NULL. There the values are taken as independent instead, as scalararraysel() takes <> ALL once its sum goes past every row: the rows not NULL, times the product over the values of the share of them each leaves, 1 - freq(v) / (1 - nullfrac), so that the NULLs are still taken away once. That run keeps 50,000 rows and is estimated at 15,055; the share clamped to zero would be one row, and core’s product - of 1 - freq(v) - nullfrac a value - is one row too. A table never analyzed is no different: each value is taken at one over the default count of distinct values, 200, with no NULL, so a run of more than 200 is past every row, and the estimate is core’s own, below the 200 and past it. Up to the rows not NULL means to the rounding of the statistics' float4 frequencies: a run of every value a column has, all of them most common ones, can come out past those rows by parts in 108, and up to 10^-6 past them (LION_NEQ_RUN_SLACK) the share is zero - every row is left out.
  • Where. lion_canonical_clause_list() is the form lion’s estimates take a list of clauses in: each OR of equalities as its IN list (“An OR of equalities is its IN list”, above), then each run as its exclusion; lion_canonical_rel_list() then takes a relation’s tests of whether a lion-indexed array is empty as the one clause they come to (§17, “Non-empty arrays”). The relation’s restriction clauses in that form (lion_rel_canonical_clauses()) give the rows lion_probe_rel_rows() computes - the count pushdown’s recheck, LionOrdered’s rows - and each parameterized path’s (lion_param_rows()), and rel->rows is corrected to them wherever they differ from core’s (“The relation’s rows”, above). An index path’s quals in that form give its selectivity (lion_cost_canonical_sel(), in lioncostestimate()): genericcostestimate()’s selectivity is scaled by lion’s estimate of the quals over core’s, which leaves a partial index’s predicate’s share as it was, and the index tuples and the pages prorated by them are computed again from it as genericcostestimate() computes them, so that the heap side of the plain and the bitmap path, and LionOrdered’s lion side, follow. A NOT IN is no index qual (core matches an op ANY only, §35); the run of the index’s <> quals is.
  • What it leaves. The answers: the clause is an estimate’s, which no plan holds; the count pushdown counts a run exactly, as negated sources (§35). The intersection probe measures set clauses only, and a run beside them is taken as independent of them, as core takes any clause the probe has no value for.

Measuring a run in the intersection probe was considered and not done. The run’s own estimate is exact from the statistics; what the probe would add is how the run correlates with the set clauses beside it, which would need its AND to subtract the values' sets and the NULL entry’s at each container it samples - the count’s negated sources (§35) - where it ANDs positive streams from a driver; and a run with no set clause beside it has no driver to sample.

Wide filters (2026-09-28). The probe exists for the correlated filters core’s product underestimates, and the wider such a filter, the more it is off. On the repro the twenty sets estimate at 22 rows for 3,835 - and the probe, as it was, gave up on them:

  • What a sample cost. A seek past the posting leaf in hand stepped right twice before it descended (LION_POSTING_SEEK_STEPS), which pays for a target a leaf or two away; the probe’s strata are a hundred leaves apart. A sample of twenty sets read 60 to 76 buffers. Sampled streams now descend at once (lion_stream_far()): 41.
  • What a probe could spend. 256 buffers a probe (LION_ISECT_BUFFERS): the locate’s 22 (82 on a cold cache) and three or four samples, where LION_ISECT_MIN_KEYS is eight. The probe gave up after 204 to 213 buffers, or, where its strata widened to fit, took four samples and applied what they said. A probe may now take what the run has left of its budget (lion_isect_run_budget()), and gives up as soon as it knows it cannot take LION_ISECT_MIN_KEYS samples - after its first - leaving what it has not spent to the run’s other probes, where it used to read to its budget’s end first; and it no longer applies a sample of fewer (four containers of a BitmapOr arm, none of which held a row of the AND, had measured it empty).
  • Which probe goes first. Core prices the paths of subsets before the relation’s own conjunction is asked about - an index path of the clauses one column answers, the arms of a BitmapOr - and those spent the run’s budget first. Before a conjunction is measured, the relation’s own - every set clause of its restriction clauses the index answers, the AND the count makes and a scan of every clause makes - is, when it is another (lion_isect_factor_run()).
  • What a conjunction the run could not measure is. Core’s product, as before, unless the run measured others of the same table: then between the largest of them that is a part of it and the smallest it is a part of, geometrically in the number of clauses (lion_isect_part_factor()). On 16 the repro’s five clauses beside the four lists measured a factor of 3.1, all nine 186, and the five with one of the lists 9.1, where this says 8.6.

  • What the AND costs. The merge’s guess of how often the later sources are sought (lion_merge_cpu_cost(), 1 - exp(-lambda)) took the sets as independent: after five or six of the twenty, the product says the intersection is empty at most keys and the later sources are rarely sought. The repro’s merge sought every source at every key the driver produced - 27,365 containers for 1,374 keys, 73 seeks avoided. What the probe measured of the AND is spread over the sources after the driver, a share each (isect to the power 1 / (nsrc - 1)), for an ungrouped count and a scan (lion_cost_set_and()); a grouped count’s merges still take the sets as independent. The repro’s count rises from 37,394 to 39,010.

  • What a few containers can say. A sample that meets the AND’s rows in one or two of its containers says little of how many there are: on the same table loaded group by group, so that each group’s rows lie in a stretch of seven container keys, the twenty sets' eight containers met the two matching groups' stretches once, and that container stood for its stratum - 79,790 rows for 3,608. Fewer than LION_ISECT_MIN_HITS (3) containers that keep a row of the AND raise the estimate as far as the rows they kept, which are there whatever the rest is, or as far as the longest part of the AND that three of them or more kept a row of says, the rest taken as core takes it and no more of the part’s rows than the rows of the AND the sample met allow (“Wide lists”, below); one that lowers an estimate lowers it no further than the sample can rule out (“What a sample can rule out”, below), and a sample of every container of the heap is a count, not a sample. A driver whose containers the probe can rank is sampled by rank among its own, or counted, rather than in the heap’s strata (“A sample of the driver’s own rows”, below). The stretched table’s narrower conjunctions keep the error a stratified sample of such a heap has: five of the clauses at 66,233 rows for 11,385, four at 82,389 for 16,279 (93,000 on 991c315).

After: the twenty sets are measured in 352 buffers (412 on a cold cache), eight samples, at 4,091 rows for 3,835; the repro’s filter widened to thirty-two sets (lists of 8, 8, 6 and 4) in 542 (602 cold), at 10,570 rows for 10,183. Planning the twenty-set count takes 1.4 ms against 0.2 without the probe, every page in shared buffers. The worst case is the run’s budget: a probe checks it before it locates each clause and before each sample, so a run accesses at most its budget (lion_isect_run_budget(), up to 3,072 buffers, “Wide lists” below) and one sample’s seeks (65 for thirty-two sets) or one list’s lookups more, and its eight probes at most. test/sql/orlist.sql’s wide filter, twenty sets over 400,000 rows, needs 301 to 326 - past the old budget of one probe.

Measured on the repro (PostgreSQL 18 and 16, release builds, shared buffers of 256 MB, the heap of 687 MB in the OS cache, max_parallel_workers_per_gather 0; warm: median of 7 to 11 runs, the two plans alternating in one session where both can be had by a setting; cold: the cluster restarted and the relation’s files dropped from the OS cache before each run, three runs each). Every plan that changed:

version, query before now warm, before / now cold, before / now
18, count, OR form LionCount, OR’s leaves LionCount, the lists 124 / 124-134 ms (the same work)
16, count, IN form LionCount, or the plain scan of five clauses and a filter (the probe’s widened sample decided which) the plain scan 102 or 36-44 / 45-57 ms
16, count, OR form BitmapOr of the m4 arms, the other lists filtered the plain scan, as the IN form 85 / 44-45 ms 460-569 / 276-487 ms
16, rows, OR form BitmapOr the plain scan 66 / 42-45 ms
16, page (ORDER BY ts LIMIT 100), IN form lion scan and Sort LionOrdered 44.3 / 50.3 ms 591-621 / 108-136 ms
16, page, OR form BitmapOr and Sort LionOrdered 79 / 41-55 ms
18, page, both forms lion scan and Sort LionOrdered 114-117 / 124-139 ms 461-639 / 449-1000 ms (medians 471 / 495)
18, page, eight clauses (5,512 rows) lion scan and Sort LionOrdered 131 / 133 ms 534 / 201 ms
18, page, seven clauses (12,376 rows) lion scan and Sort LionOrdered 142 / 128 ms 870 / 201 ms

The one that is not better is 18’s page of the nine clauses, a draw cold and 6 to 20% slower warm: LionOrdered ANDs the twenty sets as the lion scan does, walks the btree, and its executor switches to fetch and sort (§30.4) after 120,160 entries with 80 of the 100 rows met - 32 entries a member not yet met, 3,755 of them - so it fetches what the lion scan fetches, after the walk. The switch comes before the LIMIT wherever the members are fewer than about sqrt(LIMIT x entries walked / 32), 4,300 here; a filter of more members than that for its table - the eight- and seven-clause pages above - is walked to its LIMIT, and fetches 100 rows where the lion scan fetches thousands. Pricing the switch where the LIMIT’s walk would pass it was tried and left out: it reads the members from the same estimate, and with the probe off, core’s product put the five-clause page’s 11,700 rows at 3,509, predicted a switch that does not happen, and sent the page to the btree walk with a heap filter - 202 ms against 40 for LionOrdered. The switch bounds what a wrong bet costs, and the model does not second-guess it.

A sample of the driver’s own rows (2026-10-19, lion_isect_by_rank(), lion_isect_walk(), lion_isect_presence(), lion_posting_separators()). On a table loaded in batches, each batch’s rows in the order of one column, a conjunction of millions of rows was estimated at one row with the probe on (near its true rows with it off), and so was every conjunction of those columns and one more equality; the rows correction made the one row the table’s (“The relation’s rows”, above), and every plan above it was built for it.

Reproduced on 1.5M rows loaded in 300 batches of 5,000 - a source column of 12 values by a wave column of 25, source-major - each batch’s rows in the order of a kind of four values (45, 25, 11 and 19% of them), so that a batch is two or three of the heap’s 734 container keys; two tag columns hold values for some sources only, NULL for the rest. wave = 5 AND kind = <the last kind> holds 11,400 rows, and the probe measured none of them and estimated one (core: 11,556). The driver, the wave’s 60,000 rows, lies in twelve stretches of two or three containers, sixty keys apart, and the probe sampled the driver’s FIRST container at or past the first key of each of 32 strata of the heap’s keys: a stratum that begins between two stretches - nearly every one - samples the first container of the next stretch, which is a batch’s start, whose rows are the first kind’s alone. The strata after it began past the stretch, the second pass “in strata of that stretch” began at the same first container, and no container sampled held a row of the last kind: the AND was measured empty. It is the inspection paradox of a systematic sample whose units are not the population’s - strata of the heap’s keys, where the population is the driver’s containers - and the more the driver’s rows cluster, the worse it is: the same pairs on a heap whose hidden groups follow the load came out within 26%, two or three containers holding a row of the AND, and on one where the groups take turns row by row, exact.

So the probe samples the driver’s containers by their rank among the driver’s own, whatever the heap between them:

  • A driver of a few posting leaves is walked (lion_isect_walk()): its own stream, read once, its non-empty containers' keys and members kept - at most LION_ISECT_WALK_PAGES (32) buffers and LION_ISECT_WALK_KEYS (8,192) containers, about what its samples' seeks would read anyway. The other sources are then walked along those keys as the merge seeks them, the fewest rows first (lion_isect_presence(), at most 32 buffers a source): a key some source has no container at holds no row of the AND and is ruled out, not sampled. What is left - on a batched heap, the stretches where every source’s batch lies, which is what a column present for some sources only comes to - is counted when it is no more containers than the other sources' sets can be sought at within the probe’s seeks (LION_ISECT_KEYS, or LION_ISECT_SEEKS over the other sources' sets where that is more): every one ANDed, the AND’s exact count. More are sampled every n/32-th, from the middle of the first n/32, each standing for its share of the containers left and those for their share of the driver’s rows. The driver’s rows are then the walk’s count of them, where they were core’s estimate for a multi-key query.
  • A single set of more is ranked by its posting tree (lion_posting_separators()): the separators of its root and, while they are fewer than the samples and within LION_ISECT_SEPARATOR_PAGES (33) pages, of the level below. A posting tree splits its pages by bytes, so the separators of a level are about equal-byte quantiles of the set; the sample is the driver’s container at 32 of them, spread evenly, and since a page’s first container is met in proportion to its size, as a boundary between equal parts of a sequence falls in an item in proportion to the item’s length, each is weighted by the inverse of its bytes (and a line pointer’s). No presence walk here: the budget would not cover one.
  • What neither ranks - an IN list or a multi-key query of more sets than a walk reads - is sampled in strata of the heap’s keys, as before.

Measured on PostgreSQL 18 (lion’s own estimate, the Bitmap Index Scan’s rows, with the probe’s own count beside it where that differs; the buffers each probe accessed, from its DEBUG2 line) on the batched heap above, on 1.2M rows in 100 hidden groups that take turns row by row (uniform, 138 container keys), and on the same rows loaded group by group (clustered), 21, 13 and 13 conjunctions:

heap, conjunctions before now buffers a probe, before / now
batched: a wave and a kind, with a third clause or not 1 for 11,400 (twice), 950 and 2,856 core’s 11,115 to 11,639, 925 and 1,489: the probe counts 11,400, 950 and 2,856, within LION_ISECT_ERROR of core, which stands 40 to 65 / 46 to 120
batched: a source, a wave, a tag and a list core’s 202 for 3,334 (one container held the AND) 3,327 (counted) 48 / 68
batched: the same with a tag present for some sources only 1,192 for 1,192 (core 374), 715 for 714 (core 194) 67 to 78
batched: a source and a tag it never has 1 1 (counted) 40 / 41 to 49
batched: a column of four values in the odd waves only, beside a kind (a driver of over 100,000 rows: its separators) the probe’s measure within 32% within 5% (core’s, within LION_ISECT_ERROR, stands) 57 to 146 / 57 to 162
uniform the probe’s measure within 0.2%; an empty AND, 1 within 0.1%; an empty AND, 13 (“What a sample can rule out”) 33 to 212 / 51 to 211
clustered the probe’s measure within 26% (1 to 12 containers held a row of the AND) exact: every one counted 19 to 66 / 19 to 143

A probe that runs out of its budget gives up, and the walks are inside it (lion_isect_spent() before each container): a run’s probes access at most the run’s budget and one sample’s seeks. Planning a four-clause count of the batched heap takes 0.12 ms against 0.05 without the probe, every page in shared buffers.

What a sample can rule out (2026-10-19, lion_isect_take(), lion_isect_upper(), lion_isect_factor_run()). A sample that met no row of the AND measured it empty, and an empty AND is estimated at one row: the last step of the batched heap’s 11,400 rows becoming one. Thirty-two containers can rule out only so much, and the rows of one container are no independent draws: on a heap loaded in batches a container holds one stretch of rows, and a source absent from it is absent whatever the AND holds elsewhere. So the sample counts its evidence in units:

  • a container some source has no container at is one unit - it says the AND skips one stretch, no more;
  • a container where every source has members is as many units as the driver has members on the heap blocks every source has members on, and one at least: rows side by side on one page, where the AND’s absence is a fact about rows, not about which stretch the sample met.

The AND’s share of the driver’s rows is at most the Poisson’s 95% upper limit of the rows of the AND met over the units - Wilson and Hilferty’s form, (h + 1)(1 - 1/9(h + 1) + 1.645/3 sqrt(h + 1))^3, 2.97 for none (the rule of three), 4.73 for one, within 1% of the exact limit from there on - and an estimate is lowered no further than that limit times the driver’s rows. A count - every container of the driver ANDed, or every container of the heap - is no sample, and stands exactly as it is: an AND it finds empty is still one row. Which keeps the empty and the tiny ANDs:

  • on the batched heap a source and a tag only other sources have is counted - the presence walk rules out every container - and stays at one row, as the clustered heap’s empty pairs do;
  • on the uniform heap, whose groups share every page, an empty pair met no row in thousands of units: 13 rows of 1.2M, not 1, where core said 4,728 to 11,451;
  • and a sample that meets the AND in fewer than LION_ISECT_MIN_HITS (3) containers (“What a few containers can say”, above) raises the estimate as far as the rows it met - they are there, whatever the rest is - or as the longest part of the AND that enough containers met says (“Wide lists”, below), and no further than the sample can rule out: before the presence walk counted it, the batched heap’s four clauses went from core’s 187 to 1,319 to 1,968 for 3,334.

test/sql/isectstretch.sql pins both on 192,000 rows of the batched shape, four sources by twelve waves in 93 container keys, sampled whole by ANALYZE: three conjunctions the sample of old estimated at 1, 1 and 161 rows for 760, 507 and 759 now within a factor of three, the relation’s rows with them; a tag only some sources have counted with the probe’s presence walk; a source and a tag it never has, and one more such, still one row; and the count of four clauses chosen with the probe and without it (DESIGN.md §39, “The AM’s own scan of the same sets”).

Wide lists (2026-10-20, lion_isect_run_budget(), lion_isect_measured_part(), lion_isect_probe(), lion_isect_take()). A filter of several multi-value IN lists on correlated columns is the widest conjunction the probe meets: every value of every list is a posting set the probe seeks at each container it samples, and the more columns such a filter names, the further core’s product falls - to a fraction of a row while the filter holds thousands. What the probe does with one:

  • The same sample, however many sets. The probe takes LION_ISECT_KEYS (32) of the driver’s containers whatever the sets. The wider a filter of correlated columns, the smaller the share of its driver’s rows the AND keeps, and the fewer containers hold a row of it: a conjunction whose rows narrow into a quarter of the heap as columns are added is met in about two of LION_ISECT_MIN_KEYS (8) containers, too few to measure how many rows it holds, and in about eight of 32. A probe that cannot afford eight samples gives up.
  • A budget that grows with the sets (lion_isect_run_budget()). A seek of a set at a sampled container costs about two buffer accesses - the strata are many leaves apart, so each is a descent of a two-level posting tree; 1.9 measured over a run, the locate and the driver’s own seeks included - so a run’s budget is LION_ISECT_RUN_BUFFERS (1,024) beside 32 samples of the widest conjunction the run measures at LION_ISECT_SEEK_BUFFERS (2) a seek of each of its sets: 1,216 for an equality and a list of two values, 3,072 for a conjunction of LION_ISECT_MAX_SETS (32). The relation’s own conjunction, measured first, then leaves the run’s other probes about 1,024 whatever its width. A probe checks what the run has left before each clause it locates and each sample, so a run overruns the budget by one sample’s seeks or one list’s lookups at most; the strata ahead widen (lion_isect_pass()) and the ranks ahead thin (lion_isect_sample_at(), every k-th of them, evenly) to keep a sample’s worth of it to spare.
  • What is too wide for a probe is measured without its widest clauses (lion_isect_measured_part()). A conjunction of more than 32 sets, or of more than what the run has left affords eight samples of, is measured on the clauses of the fewest sets while their sets fit, the more selective of two of as many first; the clauses left out are taken as independent of the rest, as core takes them. The factor the probe finds for the part - its measure over core’s estimate of the part - is the conjunction’s, and the part is remembered as measured as well, which is what a wider conjunction of the run interpolates from (lion_isect_part_factor()). A list of more than 32 values is measured by a sample of its values (“Long lists”, below). Only a conjunction of which not two clauses fit is not measured: what the run measured of its parts stands, or core’s estimate.
  • What a sample that met the AND in few containers says (lion_isect_take(), lion_isect_probe()). The AND is made source by source at each sampled container, the driver first and then the sources of fewer rows, and what each prefix of the sources kept is counted beside the whole’s, at no seek more. When fewer than LION_ISECT_MIN_HITS (3) containers kept a row of the whole AND, the longest prefix that three or more kept a row of is measured: its share of the relation’s rows, times the share of it that core’s product of the rest of the clauses keeps - and no more of the prefix’s rows than the rows of the whole AND the sample met can have come from, at 95% (lion_isect_upper_count()). The estimate is raised as far as that or the rows of the AND the sample met, whichever is more, never past what the sample can rule out (lion_isect_upper()) nor past the smallest clause’s own; less than LION_ISECT_ERROR times core’s is no measure. A rest whose rows lie away from the prefix’s - which core’s product cannot see either - raises nothing. An estimate is lowered no further than the sample can rule out (“What a sample can rule out”, above).
  • The relation’s rows (lion_probe_rel_rows()). The factor multiplies core’s rows before they are rounded and clamped to one: where core’s product is a fraction of a row, rel->rows, one, times the factor measured against that fraction would be as many times too many.

Measured on PostgreSQL 18 (release build; lion’s own estimate, the Bitmap Index Scan’s rows, with the probe on and off; the relation’s rows; planning’s buffer accesses with the probe on and off) on test/sql/isectwide.sql’s table, 160,000 rows in 119 container keys: a family of one row in sixteen, spread over the heap, shares the values of six columns' clauses, and the family’s rows in the first quarter of the heap the values of three more:

conjunction rows core probe relation buffers, on / off
six clauses, 20 sets: the family 9,961 19 10,120 10,120 160 / 8
eight clauses, 28 sets: the family’s rows in a quarter of the heap 1,280 1 1,288 1,288 205 / 10
nine clauses, 31 sets: the same 1,280 1 1,316 1,316 219 / 11
ten clauses, 41 sets: measured without a list of ten values 652 1 656 656 220 / 12
seven clauses, 21 sets: the family’s rows in under a twentieth of the heap, met in two containers 360 11 365 365 183 / 9

Twelve conjunctions of 32 sets in one statement plan in 1,827 accesses. On 12M rows in 8,000 hidden groups (5,514 container keys), nine columns each following its row’s group nine times in ten, conjunctions of 6 to 32 sets came out within 16% of their rows where the groups take turns row by row, and within 15% where the AND narrows into a quarter of the heap as the filter widens; core’s product was 1 to 3,482 rows for 5,540 to 69,370. Planning took 439 to 2,119 buffer accesses with the probe, 5 to 12 without it. Where the AND lies in fewer containers than the sample has strata - the same rows loaded group by group, a filter’s few groups in a handful of the 5,514 containers, or in batches that put them in one hundredth of the heap - no sample of 32 meets it, and core’s product stands: 1 to 2,735 rows for 5,407 to 11,327.

test/sql/isectwide.sql pins it: the conjunctions above within a factor of three of their rows, the relation’s rows with them, where core’s product is ten times short or more; the 41 sets measured without the widest list, the 21 by the prefix the sample met often; the planning of the 31 sets, and of the twelve conjunctions in one statement, within the run’s budget; and, with three lion indexes of the same columns, three conjunctions of 32 sets in one statement each estimated at its relation’s rows by the Bitmap Index Scan of whichever index the plan takes.

Long lists (2026-10-31, lion_isect_list_sample(), lion_isect_factor_run()). A list of more than LION_ISECT_MAX_SETS (32) values was never located for the probe and was taken as independent of the rest of the AND, as core takes it. A list of ids chosen because they pass the other filters is the opposite of independent: every listed value’s rows pass them, and core’s product put a count over about a thousand such ids at a twentieth of its rows, which priced lion’s plain scan for that many rows and chose it over the count. Such a list is now measured through LION_ISECT_LIST_SAMPLE (16) of its non-NULL values, evenly spaced over it: the sample is located and sought as a list of 16 values is, and what the probe measures of it is scaled by core’s estimate of the whole list over core’s estimate of the sample (at least one) - the clause’s scale, applied to the measured rows, to the bounds a sample can rule out and raise to, to a measured part’s rows, and to the driver’s rows where it is the sample. A sample none of whose values has an entry says nothing of the list’s other values, and its clause is left out of the measure as a clause too wide is, not taken as an AND that is empty. On test/sql/isectlist.sql’s table (100,000 rows, 200 hidden groups, a four-valued column that follows them), lion’s estimate of the Bitmap Index Scan with the probe on and off:

conjunction rows probe core
c = 1 and 40 values of g all with c = 1 20,000 20,000 5,000
c = 2 and the same 40 values 0 1 4,995
c = 1 and 20 such values, 20 absent 10,000 10,000 2,500
c = 1 and 40 such values, 40 with c = 3 20,000 10,000 10,000

The last is core’s estimate within LION_ISECT_ERROR, which the probe leaves as it is. On a synthetic table of 1M rows, that count is now estimated at 20,033 rows for its 20,033, where it had been 994.

The AND, as the count makes it (2026-09-28, lion_leapfrog()). “One price for the AND of sets” prices a scan’s AND as the count’s merge, because the two make the same AND of the same sets. They did not make it the same way. A benchmark’s count over a wide correlated filter, answered by a plain lion scan and an Aggregate, ran several times as long as the same count as LionCount, while the two were priced within a few units of each other - a coin toss that cost a multiple when it fell the wrong way - and a Bitmap Index Scan of the same kind of AND, building a parallel bitmap heap scan’s bitmap in one process, was as slow.

  • The two ANDs. The count’s merge (§22, §25) drives from the source of fewest members, seeks the others to its keys in ascending members, one at a time, each folded into the running intersection before the next is touched, and gives the key up at the first that empties it or has no container there. A scan’s AND was an AND node of the evaluator (§17), whose children are the quals' trees in the index’s column order: it stepped its FIRST child, sought every other child to the largest key any of them stood at, ANDed all of their containers, and at a key whose AND came out empty stepped every child. The answers were the same; the work was not.
  • The repro (synthetic, PostgreSQL 18 release build, random_page_cost 1.1, 8 GB of effective_cache_size, 2 GB of shared buffers holding the heap and the index, serial plans): 8M rows in 1,000 hidden groups that take turns over the heap, 117,198 heap pages, one wide lion index. The filter of one group - a few dense clauses on the index’s first columns, then selective ones: an equality, short IN lists and a multi-key && - is twenty posting sets and 2,184 rows; the product of the clauses says 2, the probe 2,028. LionCount took 49 ms and 2,377 buffers; the plain scan and its Aggregate 92 ms and 4,536 - the same 2,373 index pages and 2,163 heap pages; the Bitmap Index Scan alone 84 ms.
  • Where the time went (perf on the backend, each path forced and run thirty times): the scan’s AND spent 44% of it in lion_container_and(), the count’s merge 11%. The first children in the index’s order are the dense ones, so every key began by ANDing two of their RUN containers and optimizing the result, before a selective set was reached; the count’s first AND is its driver’s small ARRAY probed against the next set. The lists' unions - lion_container_or() and its optimize - took two thirds of the count’s time, and the same work of the scan’s. The buffers were the same here, because every set has a container at every key; where the sets leave keys empty, stepping every child read the dense sets' pages at keys the others had ruled out - 47 index buffers against the count’s 22 on the table of scanand.sql (§29.12), 21 now.
  • The change. One function, lion_leapfrog(), is the merge’s loop taken out of lion_run_merge(), which calls it for its positive sources; an AND node orders its children once, when it is built, by the merge’s own estimate (lion_node_members(): a set’s ntids, a union’s sum, an intersection’s least), steps only the first of them, seeks only it when it is sought itself, and builds its container with the same function. The §9 argument moved with the loop and is unchanged: a key given up asks the visibility map nothing, a cursor not sought keeps its pin, and the key handed up has every child standing on it with its container’s pin - for a trimmed AND (§15, “Bounded cursors”) the one child that keeps pins. EXPLAIN’s Probes Avoided counts the merge’s own sources, as it did. The plain scan’s shapes all AND through it - SETS, a WALK’s entry and the other columns' trees, a WINDOW’s rest, a LIST’s batches - and so do the bitmap scan’s lion_sets_iterate(), the intersection probe’s samples and the count’s multi-key trees.

Measured on the repro, medians of eleven runs, the old and new builds alternating (the machine was shared, the runs spread by 20 to 40%); no price changed, so no plan did:

query (the plan chosen) path before after buffers cost
count, twenty sets (LionCount) LionCount 48.8 ms 48.1 ms 2,377 17,892
plain + Aggregate 91.6 51.7 4,536 20,165
bitmap + Aggregate (index side) 88.4 (84.4) 52.9 (48.0) 4,536 (2,373) 20,159
page, ORDER BY ts DESC LIMIT 100 (plain + top-N) plain + top-N 91.6 58.6 4,536 20,188
LionOrdered 94.9 55.0 6,893 22,760
rows, six clauses, ten sets, 8,533 rows (plain) plain 73.6 44.0 9,970 23,069
bitmap (index side) 75.4 (59.0) 41.1 (24.4) 9,970 (1,745) 23,044
count, the same six clauses (LionCount) LionCount 22.7 25.4 1,749 14,121
plain + Aggregate 72.0 38.6 9,970 23,076
arr @> '{13,30}' beside two dense sets (LionCount) plain + Aggregate 25.3 19.2 8,169 13,275
bitmap + Aggregate (index side) 28.4 (14.1) 20.7 (6.6) 8,169 (751) 13,253
a range of 4 keys beside three sets (LionCount) plain + Aggregate, a WALK 32.8 23.3 826 5,485
a range of 3,900 keys beside three sets (plain) plain + Aggregate, a WINDOW 400 388 80,943 151,784
  • The price is true now. The model priced the scan’s AND as the count’s, and the scan now runs it in the count’s time: 370 to 390 units a millisecond for all three paths of the twenty-set count, where the plain scan ran at 220 before, and 540 to 600 for the six-clause one, where it ran at 320. What is left between the paths is what they differ in. The plain and the bitmap path are charged the heap for the rows the probe measured and not for core’s: lioncostestimate() returns the corrected selectivity, and cost_index() and cost_bitmap_heap_scan() take the rows they fetch, and the pages those lie on, from it - 2,273 units over the count for 2,163 heap pages and 2,028 rows, 3.6 ms measured. Without the probe all three come out within three units (15,570, 15,572 and 15,572), core’s product saying two rows: that is the coin toss, and it is what any path of a filter nothing measures is priced for; the executors now make it a toss between 48 and 52 ms rather than between 49 and 92. The count’s visibility-map checks stay in the merge’s constants, which were fitted on counts, and the scans are charged them as well - under 1% of the count’s time in its profile, against the 3.6 ms the scans spend on the heap.
  • The plans. The count stays LionCount, now ahead of the scans by their heap and nothing else; the page stays the plain scan and its top-N sort, a tie with LionOrdered, which makes the same AND before its walk (§30.4); the six-clause rows stay the plain scan, a tie with the bitmap scan. A parallel bitmap heap scan’s bitmap is still built by one process, as core builds it, and that process’s AND is the table’s bitmap index side.
  • The unions (this item said “not done” until 2026-09-28). The lists' unions were the larger part of both executors' time on such filters: each key’s union of a list’s containers was built whole, and optimized, before the running intersection, a few rows or none, was ANDed with it. The evaluator’s OR node now builds its union only when it is asked for, the intersection’s members are looked up in each set of the list where that is cheaper, and the merge’s constants were refitted on the unions as they are met: “Unions probed”, below.

Unions probed (2026-09-28, lion_leapfrog(), lion_or_probe_pays(), lion_container_and_union_raw(), lion_merge_cpu_cost_sets()). A filter of several IN lists - or a multi-key &&, or an OR of equalities, which is its IN list - beside other clauses is an AND some of whose sources are unions of posting sets. The leapfrog met every such union with the union built: seeking it to a key built it there in full - every member’s container ORed into a bitset image, or folded pairwise - and only then was the running intersection ANDed with it. On a synthetic table (8M rows in random order, 70 to a heap page, 1,786 container keys; one lion index over a few dense columns, the columns of short IN lists and an int[] column; the filter: a few dense clauses, several short IN lists and a multi-key &&, eighteen posting sets, 4,573 rows), LionCount took 28 ms and the plain and bitmap scans 89 and 92. The profiles (perf, the backend alone, each path forced and run for eight seconds): the plain scan spent 38% in lion_container_optimize() and 34% in array_union() - the pairwise fold of each list’s containers, optimized at each step - and the count 19% in container_or_bitset(), the images of the unions, and 36% ANDing what came out of them: an image of a hundred-odd members is a BITSET, and a BITSET ANDed with a dense RUN set fills the image and clears each gap between its runs (bits_clear_range() 26%). Where the intersection has a few rows at a key, all of that is spent to keep those few.

  • The choice, per key. An OR that an AND meets - a positive source of the count’s merge, a child of an AND node - is lazy (LionExprCursor.lazy): it seeks its children and stands at the key with the ones that have a container there, as before, and does not build their union until lion_ecursor_container() asks. The leapfrog asks for the driver’s, since the intersection starts from it, and for any other only where lion_or_probe_pays() finds building it the cheaper way; elsewhere lion_or_probe() looks the intersection’s members up in the children’s containers, a member once found not again, and keeps the ones found. Both give acc ∩ (c1 ∪ … ∪ ck), the same members. The estimate reads the containers' headers: probing costs acc’s members a child - a bit test in a BITSET (2 ns), a galloping search of an ARRAY of s members (1 + log2(1 + s/acc) steps of 3 ns), a search or a merge of a RUN’s runs - and acc’s extraction when it is not an ARRAY; building costs the image’s clear and count (150 ns) and each member (1 ns for an ARRAY’s, 170 ns for a BITSET, 2 ns a run), or for ARRAYs of 128 members together or fewer the fold, and acc’s AND with it. It is taken as though every member of acc were looked up in every child, which errs towards the union, and an intersection of more members than an ARRAY holds is never probed. The numbers are a harness’s beside the container library (the gallop’s 2 ns there is 3 in a count, whose containers are not all in the first-level cache); the estimate is integer arithmetic with a fixed-point logarithm, because in floating point it took 2 to 3% of a count that builds its unions anyway, and rounded to whole steps it probed 120 members against two ARRAYs of 90 - one step each, where they take 1.8 - and the count went from 2.8 ms to 4.2.
  • The primitive. lion_container_and_union_raw(a, b[], n, dest) extracts a’s members - an ARRAY’s in place - keeps the positions not found yet in a list that each pass over one b shortens, marks the ones found in a bitmap, and writes the found members, in a’s order, as an ARRAY. A position is written back and kept by adding a comparison rather than branching on it. Against an ARRAY it gallops (a merge of the two sorted sides is a chain of dependent loads and never won: 16 values against 450 members 0.19 us galloped against 1.6 merged, 256 against 97 0.84 against 1.6); against a RUN it searches the runs for fewer values than one in eight runs, else merges with them. Damaged containers are handled as the rest of the library handles them: a well-formed ARRAY comes out whatever a was, no index leaves a or b, and an a whose payload holds more than an ARRAY’s worth is taken as empty (lion_container.c, “untrusted containers”). container_test.c checks it against the reference over a’s of every representation and lists of every shape, disjoint, overlapping, holding all of a or none, empty, and damaged on either side.
  • An AND node’s intermediates are raw. A scan’s AND node used to make its intersection with the optimizing lion_container_and() - every step counted its runs and picked a representation - and its OR children built their unions the same way. The children of an AND node are raw now (their unions lion_or_hot_raw()’s, as a count’s), the leapfrog’s steps are lion_container_and_raw(), and the node optimizes what it hands out, once, where a stream’s caller keeps it (LionExprCursor.raw, lion_ecursor_init_ex()); the output is the same container, byte for byte, since the smallest representation depends only on the members. An OR at the top of a stream (one list alone) builds its union as before.
  • §9. The argument on lion_leapfrog() holds unchanged, and a probed union adds nothing to it: its children were sought to the key exactly as for a union built - lion_ecursor_seek() pops every child below the key off the OR’s heap and seeks it, under the same rule that a cursor sought past a key lets its page go only where nothing of that key is counted - the ones standing at the key keep the pins their containers came with until the OR moves past it, and what the probe keeps is a subset of the running intersection, every member of which lies in a container that the cursors before it stand on with their pins. Nothing is probed that is not a child standing at the key, and no child is left below it.
  • What shows it. LionCount’s EXPLAIN ANALYZE prints Unions Built and Unions Probed where there was a union to meet: the container keys at which a union of two containers or more was built, and those at which the intersection was looked up in the containers instead. pg_lion.enable_union_probe (on) is a testing knob: off, every union is built, as before, and the answers are the same.

Measured on that table and on the same rows clustered by the hidden group, PostgreSQL 18.6’s packaged release build, a cluster of 2.5 GB of shared buffers holding both tables and their indexes, random_page_cost 1.1, 8 GB of effective_cache_size, serial plans; the old and the new library alternated, installed and the server restarted each time, two rounds each; each path forced (LionCount with the scans off, the plain and the bitmap scan with the pushdown and the other scan off), and each figure the lower of the two rounds' medians of 3 to 9 warm runs (the machine was shared, and a query’s runs spread by 10 to 20% from one round to the next). The U rows are filters whose unions should still be built: dense sets before a list of dense values, or lists of dense values alone. The plan chosen with nothing disabled was LionCount for every one, before and after, and at a random_page_cost of 4 as well.

query (t_r: random order; t_c: clustered) rows LionCount: cost / ms, before -> after plain + Aggregate bitmap + Aggregate unions built / probed
W1 b1 AND b2 AND d = 0 AND k1 IN (3) AND k2 IN (2) AND k3 IN (4) AND k4 IN (3) AND arr && (3) 4,573 31,675 / 28.0 -> 9,285 / 16.8 34,765 / 89.0 -> 12,375 / 20.5 34,751 / 92.3 -> 12,360 / 19.9 5,371 / 3,558
W2 the same AND e = v 1,496 25,730 / 14.3 -> 2,359 / 9.9 26,624 / 75.7 -> 3,253 / 11.4 26,620 / 70.8 -> 3,249 / 12.2 477 / 7,864
S1 e = v AND k1 IN (3) AND k3 IN (3) 8,600 7,149 / 4.2 -> 1,570 / 3.3 15,621 / 22.1 -> 10,042 / 6.1 15,697 / 27.7 -> 10,118 / 9.4 477 / 3,088
S2 k3 = v AND k4 IN (4) AND b1 180,205 25,935 / 10.0 -> 6,948 / 11.3 128,657 / 134.5 -> 109,670 / 93.1 129,214 / 172.8 -> 110,227 / 136.9 1,786 / 0
S3 arr && (3) AND b1 AND b2 AND k4 IN (2) 157,719 12,773 / 18.0 -> 8,107 / 18.6 100,673 / 168.5 -> 96,007 / 140.7 99,562 / 156.3 -> 94,897 / 137.2 3,572 / 0
S4 k1 IN (3) AND k2 IN (2) 54,437 3,020 / 2.9 -> 1,452 / 3.0 49,067 / 45.6 -> 47,499 / 44.8 49,289 / 60.5 -> 47,721 / 57.8 3,572 / 0
S5 e = v AND arr && (5) 6,482 1,050 / 2.8 -> 1,892 / 2.8 3,478 / 28.4 -> 4,320 / 7.7 3,478 / 32.9 -> 4,320 / 8.6 210 / 1,576
S6 e = v AND b1 AND d = 0 (no union) 12,276 3,664 / 1.6 -> 3,664 / 1.5 16,313 / 6.8 -> 16,313 / 8.1 16,368 / 16.4 -> 16,368 / 12.6
U1 d = 0 AND b1 AND k4 IN (7) 5,356,870 65,136 / 22.4 -> 16,315 / 22.3 326,957 / 741 -> 278,136 / 615 334,799 / 748 -> 285,978 / 636 1,786 / 0
U2 k4 IN (2) AND k3 IN (6) 599,393 28,162 / 8.3 -> 7,072 / 8.2 158,044 / 248.0 -> 136,954 / 186.8 159,187 / 299.7 -> 138,097 / 230.6 3,572 / 0
U3 k3 IN (2) AND k4 IN (5) AND b1 393,201 39,522 / 15.7 -> 9,730 / 15.9 164,845 / 201.7 -> 135,053 / 159.4 165,444 / 254.3 -> 135,652 / 212.6 3,572 / 0
C1 = W1 over t_c 4,573 26,674 / 3.8 -> 3,946 / 3.7 26,676 / 11.8 -> 3,948 / 5.1 26,676 / 11.4 -> 3,948 / 4.5 3,642 / 621
C2 = S1 over t_c 8,600 6,955 / 1.5 -> 969 / 1.2 6,957 / 4.0 -> 971 / 2.0 6,957 / 4.2 -> 971 / 2.1 11 / 1,555
C3 = S3 over t_c 157,719 11,560 / 16.4 -> 6,977 / 17.6 97,942 / 101.1 -> 93,359 / 98.7 96,885 / 83.8 -> 92,302 / 81.6
C4 = U1 over t_c 5,356,870 64,108 / 20.2 -> 15,009 / 19.8 325,425 / 652 -> 276,326 / 560 333,252 / 728 -> 284,153 / 614
C5 = U2 over t_c 599,393 27,994 / 6.9 -> 6,799 / 6.9 158,058 / 163.4 -> 136,863 / 117.0 159,202 / 180.4 -> 138,006 / 138.4

(k IN (n) is a list of n values and arr && (n) an overlap of n keys; b1, b2 and d = 0 are the dense clauses, e = v the selective equality, and the lists' columns follow a hidden group, as the table’s rows do.) Where the intersection is small by the time a list is met, the count is faster by up to 40% (W1 40%, W2 31%, S1 and C2 20%; S5, whose five keys' images cost about what probing them does, and C1, which probes a key in seven, even) and the scans by two to six and a half times: they gained the probing and, where no list is probed, the raw intersection (S2, S3, U1 to U3: 12 to 31% off paths that fetch 150,000 to 5.4 million rows). Where every union is built the count is where it was: interleaved in one session, S2, S3, C3 and U2 ran within the machine’s noise of the old library (S2 9.6 to 11.3 ms against 10.1 to 12.3, S3 19.1 to 27.8 against 18.6 to 23.0, U2 7.8 to 8.4 against 7.7 to 8.2), and S4, whose work is two small unions a key and nothing else, 5 to 7% slower (3.0 ms against 2.8), which is what the laziness costs a key where there is nothing else to spend on. The profiles after: the plain scan of W1 spends 0.1% in lion_container_optimize(), 8% building the unions it builds and 23% probing (array_gallop() 13%, the primitive 6%, the extraction of an image’s members 4%), 31% seeking and 11% on the heap; the count 10% building, 23% probing and 45% in its seeks - the page copies of lion_cursor_take_page() 16%, buffer lookups 10% - which is where its time is now.

The price (the AND’s, one for a count and a scan: “One price for the AND of sets”, above). The model charged a union source of an AND every container of its sets read (LION_CONTAINER_COST and LION_MEMBER_COST each) and lion_merge_ops()’s k-way merge - a sift per container and members x log2(k) comparisons at a cpu_operator_cost, 5 ns, each - whether it drove the AND or was sought at a few of the driver’s keys; and it took a multi-key && for one set, though the executor meets it as the union of its keys. That part had never been fitted: §31’s fit was of single sets, pairs, triples and a quadruple. lion_merge_cpu_cost_sets() prices each union as the leapfrog meets it, from its sets and their containers (nsets, setcontainers, setpages: a list’s values, a &&’s keys where its extraction ORs them, lion_multikey_cost_mode_ex(), an OR’s leaves):

  • one that DRIVES reads its sets' containers (LION_CONTAINER_COST each) and builds its union at each of its keys, LION_UNION_KEY_COST a key and LION_UNION_MEMBER_COST a member;
  • one that is SOUGHT seeks its sets at the keys it is sought at, no more of them than they have containers - a probe each (LION_PROBE_COST) for a set on posting pages of its own, a container read for an INLINE one, whose seek steps over the items of a payload in memory - and pays at each key the cheaper of the union built and ANDed (LION_UNION_KEY_COST, and LION_UNION_MEMBER_COST each of its members and of the intersection’s) and the intersection’s members looked up in its sets' containers (LION_AND_MEMBER_COST a member a container), as the executor chooses.

lion_cost_set_and() (the scans) and an ungrouped count’s merge use it in place of the old terms; a grouped count and the FK-side join rebuild their unions per group or per key and price them as they did (LION_UNION_SET_COST, lion_merge_ops()). The two new constants are settings, pg_lion.union_key_cost (30 cpu_operator_cost, 150 ns) and pg_lion.union_member_cost (0.25, 1.25 ns). They and the seeks' prices come from series that vary one thing each over the 8M-row table - a list of 2 to 16 values driving an AND; one sought beside a set of a few rows a key (2 to 32 values: probed) and beside one of a few hundred (2 to 12: built); a && of 3 and 6 keys - where a set of a list costs 110 to 400 ns a key, about 130 of it the seek, a member of a union 1.2 to 1.5 ns and a member looked up 1.5 to 3.6. A seek priced at LION_PROBE_COST for every set made inlist.sql’s k = ANY (1000 values) AND lo = 1, whose sets are INLINE, cost 4,517 for 2.45 ms and lose to a bitmap heap scan of lo = 1 and a filter (3,688, 6.1 ms); priced as a container read for an INLINE set it costs 1,479 and is the plan again.

Over 77 counts of the three tables of these measurements (the two above and 3M narrower rows of a second shape) - sets alone, pairs to quadruples, lists driving and sought, &&, ORs across columns and the series - the model now prices the counts at a median of 711 units a millisecond, nine in ten between 399 and 1,237 (a standard deviation of 0.45 in the logarithm); it priced them at 1,289, 412 to 4,098 (0.81). The 55 with a union moved from a median of 1,785 (up to 7,349) to 688 (462 to 1,213); the 22 of sets alone did not move, at 773. What is above §10’s 500 is mostly their pages, priced as I/O by §10’s convention where these were warm; the plain and the bitmap scans of 54 of the same filters, whose heap core prices, ran at medians of 1,011 and 795 units a millisecond, and their AND is the count’s, priced alike - one price, now for the same work. What the model still misprices, and was left: the AND of two dense RUN containers of hundreds of runs each (335 to 375 units a millisecond: LION_MEMBER_CAP charges a thousand members where run_and_run() steps through every run of both); a wide correlated filter whose later sources the leapfrog seeks at nearly every key where the model, the intersection probe’s factor and all, has them sought at a few (W2, 239); and counts over the narrower rows whose ARRAYs of 1,500 members are ANDed through an image of the larger side (269 to 332). The AND’s existing constants were confirmed by the same data, LION_PROBE_COST and LION_AND_MEMBER_COST among them, and stand.

test/sql/unionprobe.sql checks the answers of both ways against a sequential scan, through the count, the plain and the bitmap scan with probing on and off, over fourteen filters that probe, build, or do both at different keys (a column dense on the first fifth of the heap and sparse on the rest), with a NULL in a list and an IS NULL, an empty list, lists naming no entry, a one-entry list, two lists on one column, &&, @> and an OR across columns; through a cursor fetching one row at a time with other scans of the index between, and parameterized inner scans rescanned per outer row, plain and bitmap; on a heap made dirty by updates and deletes, and after VACUUM. It shows the count’s counters making each choice, and every union built with the knob off. costgucs.out lists the two new settings.

Trees probed (2026-09-29, lion_lazy_and(), lion_and_align(), lion_and_build(), lion_tree_probe_pays(), lion_tree_probe()). Union probing applied to an OR met directly by an AND in the leapfrog. A source whose own tree is more than one union - a tsquery (a | b | c) & (d | e), an AND of two ORs (and each OR of three a binary OR of an OR, as a tsquery’s operators nest); an OR across columns of ANDed clauses, an OR of ANDs - was a cursor of its own and was evaluated whole at every key it was sought to: its AND node leapfrogged its children, and its driver OR’s union was always built, a bitset image of every member of its sets' containers at the key, however few rows the outer intersection had left there. A benchmark’s FK-side join that probed a tsquery, an AND of ORs of dense terms per key built about two unions for every key container it read, of a row or two each, all of them to keep those rows.

  • Lazy trees. Under a leapfrog an AND node is lazy as an OR is (LionExprCursor.lazy): sought to a key, it winds its children to the first key at or after it that every one of them has a container at (lion_and_align(), the leapfrog’s seeks without its ANDs) and stands there PENDING, its intersection not made; the children of a lazy OR are lazy too, so an OR of ANDs, or of ORs, is pending until it is asked. When it is asked - it drives the leapfrog, or building it is the cheaper way - its children are intersected at the key as the leapfrog does it (lion_and_build()), each of them again built or probed by the same rule, and the container may come out empty, which the leapfrog above takes as a key without a row. An AND node that drives a leapfrog - the merge’s, or its parent AND node’s - is not lazy (lion_leapfrog_lazy()): it is intersected at every key it stands at anyway, and its own leapfrog stops at the first child that empties a key, where winding would seek every child there first.
  • The choice, per key (lion_lazy_and()). A pending union whose children all have their containers is decided as before (lion_or_probe_pays()); anything else pending is a tree, and lion_tree_probe_pays() estimates both ways from the containers standing at the key, in the same quarters of a nanosecond and the same integer arithmetic: probing costs every member of the running intersection looked up in every leaf (the lookups lion_or_probe_pays() prices, now one function, lion_up_lookup()) and its extraction; building costs each union’s image and its members, each AND node’s ANDs by the sizes of what it ANDs, and the running intersection’s AND with the result (lion_tree_estimate()). A running intersection of more members than an ARRAY holds is never probed.
  • The probe (lion_tree_probe(), lion_tree_eval()). The intersection’s members are extracted once, and the tree is evaluated over their positions: a leaf’s (or a built node’s) container probed with the positions still in question, an AND node’s children each given the positions the one before it kept, in the leapfrog’s order, an OR node’s children that have a container probed one after the other with what none before found, then its pending children evaluated for what is left. The found positions come out as an ARRAY in the intersection’s order. The three steps are the container library’s (lion_container_extract_members(), lion_container_probe_members(), lion_container_array_from_marks()), which lion_container_and_union_raw() is now made of too.
  • §9. The argument on lion_leapfrog() holds unchanged. Winding a tree’s children to a key is seeking them there, and nothing of a key they pass over reaches the visibility map; a pending tree stands at its key with every child standing there with the pin its container came with - a trimmed AND with its kept child’s - until it moves past the key; and what the probe keeps is a subset of the running intersection, whose members lie in containers the cursors before it stand on with their pins. A tree that is built is the eager node’s container at that key, made the same way. What changes is where a key the tree has no row at is found out: by the leapfrog that meets it, at the cost of a probe, instead of by the tree running on to its next row - which the leapfrog would then have sought every other source to.
  • What shows it. LionCount’s EXPLAIN ANALYZE prints Trees Built and Trees Probed where the AND met a tree past its driver - the keys at which it was built there, and those at which it was evaluated for the intersection’s members; a tree that drives is built and counted in neither. pg_lion.enable_tree_probe (on) is a testing knob: off, no AND node is lazy and every tree is built where it is sought, as before, with the same answers. The same leapfrog is the plain and the bitmap scan’s AND (above), so both probe trees too.

Measured on the FK-side join of §27’s “Probed, then collected” (the probing runs there, trees probed against built: 30 to 45% off serially, where the unions were most of a key’s work beside its seeks, and 1 to 20% with two workers, whose seeks of the same posting pages contend), and on 4 million rows of 562 container keys under one lion index over a selective column (e, 0.1% of the rows, scattered), three dense ones of four values and a tsvector of five dense terms, each on three rows in ten; the same build and machine, medians of five, in ms, with tree probing on and off (the count forced as LionCount, the scans forced; the times are a few milliseconds on a loaded machine, so only the larger differences mean anything):

filter count, probed / built plain scan bitmap scan counters, probed (built)
e = 7 AND doc @@ '(t1 | t2 | t3) & (t4 | t5)' 8.2 / 16.6 3.4 / 6.1 4.2 / 15.3 562 trees probed (1,686 unions built)
e = 7 AND ((a = 1 AND b = 2) OR (a = 3 AND c = 1)) 2.0 / 11.7 - - 562 trees probed (562 unions probed over ANDs built)
e = 7 AND doc @@ '(t1 & t2) | (t3 & t4)' 2.1 / 2.2 3.4 / 11.1 3.0 / 7.3 562 trees probed (562 unions probed)
e = 7 AND b = 2 AND doc @@ '(t1 | t2) & (t4 | t5)' 2.4 / 8.8 7.1 / 9.8 2.9 / 4.3 560 trees probed (1,120 unions built)
a = 1 AND doc @@ '(t1 | t2 | t3) & (t4 | t5)' (dense: built) 7.2 / 9.6 1,124 trees built
doc @@ '(t1 | t2 | t3) & (t4 | t5)' (the tree drives) 7.3 / 9.4 562 trees built inside it

The OR across columns is no index qual of a plain or bitmap scan (core’s scans take it as a filter), so only the count meets it as a tree. The last two rows are where no probe pays, and the tree is built as before - an OR of an OR and a term, as a tsquery’s t1 | t2 | t3 nests, now built when it is asked for rather than wherever its inner OR stands - in no more time than before.

test/sql/treeprobe.sql checks the answers against a sequential scan through the count, the plain and the bitmap scan, with tree probing on and off, over fourteen filters: ANDs of ORs and ORs of ANDs in a tsquery, an OR of ANDs across columns, a deeper tree, both kinds beside each other, a three-term AND, trees met with a dense intersection (built), a tree built at the keys where a column is dense and probed where it is sparse, a tree that drives, a term with no entry in an OR and in an AND, a tree met after another clause, a term twice in one tree; the counters of each (and, for two of them, the numbers, probing on and off: the unions a probed tree never builds); a nested loop’s inner scans rescanned per outer row, plain and bitmap; a dirty heap before and after VACUUM. The container library’s three steps are unit-tested against the reference alone and combined over random trees of ANDs and ORs of every representation, and on damaged inputs (test/unit/container_test.c).

A WALK is priced as a heap pass per entry (2026-09-27). A range alone - one column’s entries in key order, each entry’s TIDs in heap order (§29.3) - got the column’s correlation, as a scan not in heap order does: btree’s price, Mackert and Lohman’s pages each a random read, whatever its entries held. What it does is a pass over the heap per entry, each reading the entry’s pages in block order: a bitmap heap scan of one entry’s rows, done once per entry. So it is priced as compute_bitmap_pages() prices a bitmap heap scan repeated that often, lion_plain_walk_entries() passes of rows / passes rows each: Mackert and Lohman’s pages for all the rows together (a page a later pass reads again is in the cache while the cache holds the heap), each at the price of ONE pass’s pages - random_page_cost for a pass of a few pages, falling to seq_page_cost for one over most of the heap - interpolated towards the packed end by the column’s correlation as before; the fetches past the first on a page counted per pass; and LION_WALK_PASS_COST, 5 cpu_tuple_cost, for each entry past the first, which the plain scan streams through a stream of its own where the bitmap scan adds it to its bitmap (0.30 to 0.35 us an entry more than the bitmap scan over 4,000 to 390,000 one-row entries). One entry is a scan in heap order, priced as k = v is; one-row entries are btree’s uncorrelated end and a stream each. The price a WALK gets against the bitmap scan’s, and the time it takes against the bitmap scan’s, before and after (assert build, median of five; 1M rows of the benchmark shape in shared buffers, 8M from the OS cache):

query entries plain / bitmap price, before now plain / bitmap time
1M pc BETWEEN 1 AND 1 1 1.12 1.00 (plain chosen) 0.61-0.70
1M pc BETWEEN 1 AND 2 2 1.17 1.05 0.58-0.79
1M pc BETWEEN 1 AND 20 20 1.74 1.56 0.81-0.98
1M pc BETWEEN 1 AND 100 100 3.77 3.38 1.21-2.07
1M status >= 'val2', 40% of the rows on every page 2 2.93 1.05 1.19-1.46
1M flag >= true, 80% on every page 1 2.40 1.08 (a sequential scan chosen) 1.14-1.41
8M pc BETWEEN 1 AND 20 20 1.79 1.60 0.56-0.74 warm, 2.83 cold
8M pc BETWEEN 1 AND 400 400 3.24 2.95 5.2-7.2

A benchmark’s pc BETWEEN 1 AND 20 (the plain scan 349 ms against the bitmap scan’s 600, and the bitmap scan chosen) is the 8M row: 20 passes read 79,000 page visits where the bitmap scan reads its 65,734 pages once. The plain scan’s lead is not the WALK’s: with the heap in the OS cache and PostgreSQL 18’s default io_method=worker, the bitmap heap scan hands each read to an I/O worker through its read stream, which for a page already in the OS cache costs more than the plain scan’s synchronous read (the plain scan 371-451 ms against 536-700). With io_method=sync the two tie (392 against 388), and cold the bitmap scan’s read-ahead makes it 2.8 times faster (4,454 against 1,573). The bitmap scan is still the choice, now at 1.6 times the plain scan’s price for what it does rather than 1.8 for a btree’s.

What the model still cannot see: a range over a column stored in heap order, ts BETWEEN two days of a timestamp, is priced at the packed end - rightly, it reads a few hundred pages in order - while cost_bitmap_heap_scan() prices the bitmap scan’s pages by Cardenas’s formula with no correlation at all, as scattered over the heap; the plain scan is chosen at about the bitmap scan’s price and takes 2.3 times as long (30,000 one-row entries: 17 ms against 7.2), the difference being its stream per entry. The bitmap scan’s price is core’s.

What cost_index() cannot charge is charged to the path (2026-09-27). cost_index() charges no more than its uncorrelated end, Mackert and Lohman’s pages at random_page_cost. At a random_page_cost near seq_page_cost (1.1, as on the benchmark’s server) a result of many rows on nearly every page costs the plain scan more than that end - by its fetches past the first on a page, which the bitmap scan does not pay - and the correlation cannot carry it: the two scans came out within 1% of each other, add_path() took that for a tie, the plain scan won it on its lower startup cost, and the bitmap path was discarded. Nothing amcostestimate returns tells the two apart: cost_index() stores the same index cost for the bitmap path the moment amcostestimate returns, and reads nothing else lion gives it. So lion_plain_heap_correlation() notes what is left over for an unparameterized plain path (lion_plain_note_remainder()), and lion’s set_rel_pathlist_hook adds it to the path once the relation’s paths are built and offers the bitmap heap scan of the same index path again beside it (lion_plain_pay_remainders(), in lion_selfuncs.c, run by lion_plain_set_rel_pathlist() before LionOrdered’s hook sees the paths); set_rel_pathlist() allows a hook to modify the core paths, and the two are then compared as any others. The plain path keeps its startup cost, which is what a LIMIT asks of it. The hook offers the sequential scan and the TID scans again as well, rebuilt as set_plain_rel_pathlist() builds them (2026-09-28 review): core adds the sequential scan before the index paths, and a plain path priced short by its remainder could dominate it and have add_path() free it, leaving a relation whose best path was the sequential scan without one. Where nothing had been displaced the rebuilt path is a copy of one still there, and add_path() throws it away. The other indexes' paths are not rebuilt - that would mean building every index path again, lion’s among them - and would have had to cost less than the plain path with its remainder and more than without it. plaincost.sql’s 43% query at 1.1 is one that had lost its sequential scan (about 7,550 by core’s formula, against the plain path’s 7,294 before the charge); the bitmap scan, at 7,273, still wins it. At 1.1, 43% of the rows, 18 to a page, on every page (plaincost.sql’s shape, status = 0 AND supp = 0 AND flag):

table plain / bitmap price before now plain bitmap
8M rows, heap from the OS cache 304,596 / 303,920 (plain chosen) 329,081 / 303,920 1,540-1,712 ms (1,558 under load) 1,240-1,430 ms (1,680-1,750 under load)
the same, cold 3.0-6.7 s 1.36 s
1M rows, heap in shared buffers 37,985 / 37,898 (plain chosen) 41,038 / 37,898 135-163 ms 114-131 ms
plaincost.sql’s 200k rows 7,294 / 7,273 (plain chosen) 7,896 / 7,273 26 ms 22 ms

A result of 2.5 rows to a page on 93% of the pages (status = 0 AND supp = 1) fits under the end and is unchanged; it is the plain scan’s in shared buffers (34 ms against 39-49) and the bitmap scan’s from the OS cache (815-1,087 against 975-1,105), which is a tie. With parallel plans the large ones go to the parallel sequential or bitmap scan, before and after.

There is no parallel plain lion scan (amcanparallel is false, §29.1): the planner compares the serial plain scan with a parallel bitmap heap scan, and it credits the parallel one with its CPU divided among the workers and nothing for its I/O, as it does for every AM - which is why, against a parallel bitmap heap scan, the plain scan’s pages are now priced as one process’s reads (“One process against several”, below). Index-only scans keep the column’s correlation: lion’s are of queries that need no column, and the one kind in heap order, a multi-key UNION, fetches every TID it hands out whatever the visibility map says, which cost_index()’s all-visible fraction would not describe.

On the benchmark shape (8M rows, 190k heap pages, a five-column index; assert build, warm OS cache and 256 MB of shared buffers, the paths interleaved in one backend, median of three; count pushdown off, parallel plans off unless marked; L13 is status = 0 AND supp = 0 AND flag AND tags && '{ga}', 13% of the rows on every page, L13r the same AND pc >= 0 (2000 keys), M4 country IN ('c1', 'c2'), 4% on 82% of the pages, M2 country = 'c7', 2% on 57%, M05 0.5% on 13%; cold is after evicting the relation from shared buffers and the OS cache):

query plain before plain after bitmap parallel bitmap (3) chosen before, rpc 4 / 1.1 chosen after
L13 1569 ms 1569 ms 1040 ms 653 ms bitmap / plain bitmap / bitmap
L13r > 138 s (WALK) 2162 ms (WINDOW) 1661 ms 1234 ms bitmap / plain bitmap / bitmap
M4 768 ms 768 ms 913 ms 486 ms bitmap / bitmap plain / plain
M2 516 ms (733 cold) same 983 ms (1834 cold) - bitmap / bitmap plain / plain
M05 52 ms (611 cold) same 59 ms (1453 cold) - bitmap / plain plain / plain

With parallel plans the large ones go to the parallel bitmap or sequential scan, before and after, at the default random_page_cost; at 1.1 L13 did too, by 2%. The prices behind the columns: L13 at 1.1 was 219k for the plain scan against 228k for the bitmap scan and is now 235k; M2 at 4 was 458k against 196k and is now level with it. At the default random_page_cost the old number overcharged every scattered result (the uncorrelated end is a random read a page, which a scan in heap order never pays), so results of a few percent went to a bitmap scan up to 1.9 times slower warm and 2.5 cold; at 1.1 the tie-break correlation gave large results the packed end’s discount, which is how a benchmark’s large count (costed just below the parallel bitmap scan’s) went to the plain scan: reconstructed from its plan, a correlation of about 0.38 took a large share off its heap I/O, and the WALK above did the rest. For comparison, a btree on (status, supp, flag, pc, country) reads L13’s rows in key order, the heap at random: 17 s against its own bitmap scan’s 2.5, priced 795k against 316k (2.5 times for 6.8). The column-correlation model is btree’s; applied to a lion plain scan, whose heap is read in order, it erred the other way - 3 times the bitmap scan’s price at the default random_page_cost for 1.5 times its time.

Measured again with the endpoint probe (§28), the WALK priced as passes and the remainder charged to the path (above) (2026-09-27, the same table and build, a loaded machine): every choice of the “chosen after” column holds at both random_page_costs. The plain and the bitmap scan took 1,631 and 1,215 ms for L13, 2,398 and 1,943 for L13r, 839 and 837 for M4, 546 and 927 for M2, and 53 and 60 for M05; the chosen plans 1,064, 2,294, 835, 592 and 50.

The count’s recheck is priced the same way (2026-09-24 review). With a correlation, a plain scan of a clustered value is cheap: 5000 rows of one value on 32 heap pages. The count pushdown charged its heap recheck one page per candidate TID up to the dirty part of the heap, so when the visibility map pg_class remembers had gone stale - relallvisible is only refreshed by VACUUM, while updates, and on 19+ on-access pruning, change the map - it priced the same count at 5036 against the scan’s 3252 and lost, running 2.4-2.8x slower (0.6 ms against 0.24). The recheck visits each page holding a candidate once, in block order, so for a count whose WHERE is one plain equality the pages are now what cost_index() would say they are: one per row when the values are scattered, the rows' share of the heap when they are stored in order, interpolated by the correlation’s square (lion_cost_count_rel(), lion_single_eq_var()). Only ever lower, and only for that shape; a range already had the rule (§28), and other WHERE shapes keep the old bound. (Since 2026-09-27 that correlation, the range’s, and the containers a key’s rows lie in (lion_key_containers(), §10) are all read through lion_var_heap_correlation(), the plain scan’s own number less ANALYZE’s tie-break, where they read the raw statistic before: a column of a few values placed at random looked packed to the count as it had to the plain scan - 0.44 for three values at 60/20/20% - and its recheck was priced on a fifth fewer pages than it visits.)

A range beside other columns' sets is priced as walked (2026-09-25 review). Because the two paths share the IndexPath, the index side has to be what both of them read, and a plain scan that read far more than the bitmap scan - a WALK restarting the other columns' stream for every entry, §29.3 - was priced as though it did not: the planner chose it for a BETWEEN 1 AND 200000 AND b = 5 over a million unique a at 409 ms against the bitmap scan’s 38. The executor fix (the WINDOW shape, and IS NOT NULL dropped beside anything that answers) makes both paths read the same, and the model now charges what that is:

  • the range’s entries once per WINDOW (lion_range_walk_cost()): the heap’s container keys over lion_walk_window(), 1 below 32768 heap blocks at the default floor - an upper bound, since the other columns may have containers at fewer keys; the bitmap scan walks once per window too, its windows holding those containers at their own size rather than as images (§28, “Bitmap scans”), so it makes at most as many walks. With summaries (§32) the entries are the walk’s edge keys and summaries (lion_cost_walk_entries());
  • and the postings of the range itself. genericcostestimate() prorates a multicolumn path’s index by every column’s selectivity together, which is about what an AND of set trees reads (they leapfrog, §22), but a range column is walked whole whatever the others select (§28): it reads the postings of every row in its range. The rows the prorating left out are charged a tuple cost each and their share of the index’s pages at seq_page_cost - the walk reads directory leaves in key order through their right links, laid out in that order by ambuild. Charged at random_page_cost, as genericcostestimate() charges its own pages, the 400,000-row range of the example went to the sequential scan at 133 ms against the plain scan’s 43; without the term the 900,000-row range went to the bitmap scan at 1.4 to 1.6 times the sequential scan’s time. A one-column path is prorated by the range already and pays nothing more.

On that million-row table with nothing disabled: the 2000-, 200000- and 400000-row ranges beside b = 5 choose the plain scan (0.2 ms), the plain scan (17-33 ms, was 409) and the bitmap scan (70-140 ms on a loaded machine, where the plain scan takes 36-70 and the sequential scan 108-200); the 900000-row range the sequential scan; a IS NOT NULL AND b = 5 the bitmap scan (15-19 ms, the plain scan now the same). test/sql/indexscan.sql §11 pins three such choices on a smaller table and shows the plain scan reading within twice the bitmap scan’s buffers, where it read 36 to 390 times as many.

IS NOT NULL beside quals that answer (2026-10-19, lion_cost_notnull(), lion_cost_notnull_quals()). Taken away rather than dropped (§14), it reads its column’s NULL entry, and genericcostestimate(), which prorates the index by the selectivity of every qual together, prices what the scan returns and none of that. Each IS NOT NULL beside quals that answer is now charged its lookup (a directory leaf and a descent, as IS NULL locates it) and the NULL entry sought to the container keys of the others' answer - rel->tuples times the path’s selectivity without the IS NOT NULL quals' - as an AND’s other sources are sought: the posting pages those seeks reach, of the entry in the form it takes (lion_null_entry_pages()), a probe each, and the answer’s members taken out at each key the entry has. Or, where the executor’s choice would walk the column (lion_scan_notnull_walks(): a scalar column, beside sets with no range or long list), the walk, made from the statistics as the executor makes it from the directory: the NULL entry from the null fraction, the walk from the column’s ndistinct entries of the other rows, an entry small enough to be INLINE (LION_DEFAULT_INLINE_LIMIT) its bytes of the leaves and any other its pages, once per window. A lone IS NOT NULL walks its column, which the full walk’s price already covers. The plain scan no longer counts as rechecking for it (lion_plain_scan_shape()), and lion_path_rechecked_quals() names what a plain scan’s source does leave to the recheck, for §40.3. indexscan.sql’s plan pins, a IS NOT NULL AND b = 5’s plain scan among them, are unchanged.

One process against several (2026-09-28). Pricing a scan in heap order as the bitmap heap scan of the same pages compares one process with one process. A benchmark found the planner taking the plain scan where the other side was a PARALLEL bitmap heap scan: large results from a large table, and counts the count pushdown did not answer, ran many times slower on the plain scan than on the parallel bitmap heap scan the planner used to choose. The plain scan is one process (amcanparallel is false, so cost_index() divides none of its price, and none of lion’s own terms divide either) that reads a heap page when it needs it and waits for the read. The bitmap heap scan reads ahead of itself (effective_io_concurrency; a read stream from 18 on), and in parallel several processes do, each over its share of the pages. core’s cost model prices neither: effective_io_concurrency appears nowhere in it, and a parallel path is credited with its CPU divided among the participants and nothing for its reads. The one term that stands for the bitmap heap scan’s way of reading is cost_bitmap_heap_scan()’s price of a page - random_page_cost falling to seq_page_cost with the square root of the share of the heap it reads - and that is the credit the plain scan took over above.

So where core would run the bitmap heap scan of the same pages in parallel - the relation may be scanned in parallel and compute_parallel_worker() gives compute_bitmap_pages()’s count of them a worker, which is what create_partial_bitmap_paths() asks (lion_plain_meets_workers()) - the plain scan’s pages are priced as one process’s reads: random_page_cost, but seq_page_cost for a page that follows the page read before it, which the kernel’s read-ahead of a consecutive run serves. For pages spread over the heap that is their share of it, d, where the bitmap heap scan’s price takes sqrt(d); a WALK’s passes are priced so pass by pass. The rest is as before: the fetches past the first on a page, each pass’s stream, the packed end the correlation interpolates towards (a run of consecutive pages, read ahead by the kernel for one process as for several), the remainder charged to the path. A synthetic example, the index side left out since both paths share it: 300,000 rows scattered over a 1,000,000-page heap lie on 260,870 pages, d = 0.26. The plain scan used to come out 0.13% above the parallel bitmap heap scan (two workers) at the default random_page_cost and 0.3% above it at 1.1 - within add_path()’s 1% fuzz, where the plain scan’s lower startup cost wins - and is now 30% and 2.7% above it.

Why not everywhere: a walk priced so at every size would take every scattered result of more than a page or two off the plain scan. 20 rows over a 1,112-page heap lie on 20 pages, d = 0.018: at the default random_page_cost 3.60 a page at the bitmap heap scan’s price and 3.95 at a single reader’s, 7 units more on 72, against the 0.05 the plain scan saves by not rechecking its qual. That is how core prices a btree’s scan of scattered rows against its bitmap scan; lion’s plain scan was measured as fast as the bitmap scan or faster there (above), one process against one, and a selective result stays on it. So below core’s parallel size, and wherever no parallel plan can be made (max_parallel_workers_per_gather at 0, a parallel-unsafe query, a temporary table, a cursor, a table whose parallel_workers is 0), the two are compared as before. An inheritance child is given a worker whatever its size, for a Parallel Append to combine; below min_parallel_table_scan_size a child is taken as a base relation is.

Not offering the plain path at all above some number of rows was considered and not done: the model has no such number. Past core’s parallel size the plain scan still wins on the model’s own terms where the bitmap heap scan goes lossy at a low work_mem (compute_bitmap_pages() charges it every row of its lossy pages), under a LIMIT (the plain scan’s startup cost is the index’s, the bitmap scan’s the whole bitmap), and over a column stored in heap order (the packed end); with the walk priced as a single reader’s the planner makes the comparison itself.

Plain scans switched off (2026-09-28, pg_lion.enable_plain_scan, bool, default on, PGC_USERSET). Off, the planner plans lion indexes as it plans GIN’s, for bitmap scans only: no plain and no index-only scan of a lion index is offered, parameterized or not, and every other access method’s index scans are left alone - enable_indexscan = off, the only switch before it, takes btree’s away as well. The mechanism is the one core has for an access method without amgettuple. get_index_paths() builds an index’s IndexPaths - costing each through amcostestimate - and then offers them as plain or index-only scans only if the IndexOptInfo’s amhasgettuple is set; the bitmap paths are built from the same IndexPaths. lioncostestimate() clears the flag, so no plain path of the index reaches add_path() in that planning run, and the next run reads it from the handler again (plancat.c). The two ways first considered both come too late or land wrong: a plain path removed from the set_rel_pathlist hook, or given a disable_cost / disabled_nodes penalty there, has already made add_path() discard the paths it dominated, of which the hook can offer back the bitmap scan of the same index and the sequential and TID scans (as it does for a remainder, above) but not another index’s scans or a BitmapAnd; and a penalty in amcostestimate lands in the index cost the bitmap path shares. LionOrdered, which runs create_index_paths() over a copy of the relation that sees only its lion indexes, takes its lion side from a bitmap path when no plain path is offered - the same lion access (§30.2). It is a planner setting: an exclusion constraint’s check calls amgettuple whatever it says, and a plan cached before it changed is not planned again for it, as with core’s enable_* settings.

29.12 Tests

test/sql/indexscan.sql, written before the code and failing on HEAD (no plan can show an Index Scan using a lion index there): every answer compared with a sequential scan’s as a multiset; plans with nothing disabled for a one-row lookup and for low work_mem, and every predicate kind forced through a plain Index Scan - equality, IN and = ANY with NULL elements, ranges, IS NULL and IS NOT NULL, multicolumn ANDs, arrays and tsvector with recheck, cross-type keys, citext, a collation mismatch declined; nested-loop inner scans with rescans; cursors (FETCH forward, SCROLL over a Material, NO SCROLL refusing backward); a dirty heap after updates and deletes, then VACUUM; a posting set far larger than one batch with its memory flat; an exclusion constraint; the count pushdown still chosen for its shapes, a clustered value under a stale visibility map included; a partial multi-key index read whole at 64 kB of work_mem, by a plain scan and by index-only scans with nothing disabled, clean and dirty (the review’s repros: rows the index does not hold came back); a 20,000-value Param list read through a cursor within a work_mem of 4 MB and of 16 MB, set by the test, and long lists with duplicates, NULLs, citext spellings, a range and another column beside them, at 32 values a batch; a multi-key union across windows at the least pg_lion.scan_window_floor; ranges and IS NOT NULL beside other columns' sets (§11: one-row INLINE entries, sparse segments and posting trees walked in WINDOWs, one window and several, < ANY, a short walk, empty sides, an IS NOT NULL whose NULL entry is taken away - until 2026-10-19 a dropped one with the rows its recheck removed - a dirty heap and after VACUUM), the plain scan’s buffers within twice the bitmap scan’s, the window’s memory, and three plan choices; index-only scans of no-column queries (a column’s walk, a partial index, a multi-key column’s bitmap), clean and dirty; and nested loops whose inner plain or bitmap scan is rescanned 5,000 times through a cursor - keys whose sets meet a sparse segment after their first container, a list of them, ranges over a column with summaries and without - with the executor’s memory of the cursor flat (§12). Existing tests whose helpers exist to exercise the BITMAP path, or to force the count pushdown by disabling every other scan, now disable plain index scans as well; the plan pins that changed are one-row multi-key lookups, which are now plain Index Scans. Isolation: gettuple_pause.spec (cursors - the pause the executor really makes, between two amgettuple calls - so it runs on every major) parks a scan after five rows while the posting leaves under it split, the INLINE set it copied spills, the directory leaf under a range walk splits, and VACUUM deletes the NEXT entry of the walk and frees its posting tree, whose pages an insert then takes back, or - a WINDOW parked in the first of two windows - the entries its next window walks, whose heap slots an insert takes back under the same keys; each drains to exactly the rows its snapshot sees, none twice, and pg_buffercache shows the paused scan pinning no index page, so the VACUUM completes under it. gettuple_dirty_pin.spec (injection point lion-gettuple-batch, 17+) parks an exclusion- constraint check with its batch loaded and shows VACUUM waiting for its pin, and a plain MVCC scan parked at the same point holding none while the same VACUUM completes.

test/sql/plaincost.sql (2026-09-27, §29.11) pins the plain-against-bitmap choice on 200k rows of the benchmark’s shape, sampled whole so the statistics are exact: counts of 13% (a multi-key column, rechecked) and 43% of the rows, on every page, go to the bitmap scan, and 0.5%, 0.1% and 0.01%, about a row to a page, to the plain scan, at the default random_page_cost and at 1.1 (the 43% one at 1.1 since the remainder is charged, above); a range alone is a heap pass per key - one key goes to the plain scan as pc = 5 does, twenty keys and one key over every page to the bitmap scan, with the sequential scan’s answers; a column stored in value order keeps its correlation; and a range beside dense sets - pc < 20, the always-true pc >= 0 - is a WINDOW whose plain scan reads within twice the bitmap scan’s buffers, with the sequential scan’s answers. Tests whose plans changed: those meant to show the bitmap path (basic.sql’s first plans, directory.sql’s bitmap_one) now disable plain scans; the others print a plain Index Scan for a key of about a row to a page, which the plain scan reads as the bitmap scan does without building a bitmap - measured on the tests' own tables, 2.2 ms against 2.75 (lis_f, k = 17), 1.5 against 1.7 (lis_w, a IS NOT NULL AND b = 5), 0.97 against 1.24 (ldbl, g = 5), a tie on lion_in’s a IN (1, 2, 3). pushdown.sql’s lion_pdn count, 40000 of 60000 narrow rows stored in value order, went from a plain scan to a sequential scan: the plain scan paid per row what the bitmap scan does and more (5.2 ms against its 3.55), and both came out above the sequential scan, which takes 10 ms - the price of a numeric =, one cpu_operator_cost like any other, is core’s. Since 2026-09-27 (above: no index qual charged where it is not evaluated, no operator per index tuple) it prints the bitmap scan, the fastest of the three: 2.5 ms of CPU on the release build against the plain scan’s 3.0 and the sequential scan’s 5.3.

test/sql/plainscan.sql (2026-09-28, §29.11 “One process against several” and “Plain scans switched off”), on 20,000 rows of 18 to a page placed at random: a sixtieth of the rows, on about a quarter of the pages, goes to the plain scan with parallel plans off and while its pages are fewer than min_parallel_table_scan_size, and to the bitmap scan once core would give the bitmap scan of those pages a worker (the plain scan some 30% dearer); 20 rows stay on the plain scan either way. With pg_lion.enable_plain_scan off the plain scan of the lion index gives way to its bitmap scan and the primary key’s btree keeps its index scan; with the other scans disabled, the index-only scan of a partial lion index and the parameterized lion scan inside a nested loop are made with the setting on and not with it off, the btree’s scan is made either way, and LionOrdered (every core scan disabled) is chosen either way - each with the sequential scan’s answers. The existing pinned plans do not move: every test that pins a plain lion scan sets max_parallel_workers_per_gather to 0, or pins a result of a few hundred pages at most under the default min_parallel_table_scan_size, and the tests that lower min_parallel_table_scan_size to 0 (fkjoin_parallel.sql, fkjoin_nonunique.sql, countmultikey.sql) print which count node is chosen, which a dearer plain scan does not change.

test/sql/samecolumn_scan.sql (2026-09-28, §29.2) puts several quals on one column - two overlaps of a text[], two matches of a tsvector, two lists or an equality and a list of an int - alone, beside the other columns of a three-column index and beside a range on one of them, on 12,000 rows seven to a page: through the bitmap scan each answer is the sequential scan’s, the recheck removes no row and the Bitmap Index Scan emits exactly the rows of the answer; through the plain scan, in its SETS, WALK, WINDOW (several windows at the least floor) and LIST shapes, the recheck removes none either. Contradictions (tags @> '{a1}' AND tags @> '{a2}', k = 3 AND k = ANY ('{4,5}'), k IS NULL AND k = ANY (...), && '{}') come out empty, and what the index still leaves to the recheck - a range beside its column’s list, a multi-key query in mode ALL beside one in mode KEYS, a second long list - is removed by it. The count pushdown ANDs the same clauses, and with nothing disabled a mode-ALL overlap beside an exact one goes to the lion scan whichever comes first, where the first used to be priced as a whole-index walk and a full recheck. Then a dirty heap (rows moved into the answer, rows deleted) and the same after VACUUM. indexscan.sql pins the plain scan’s recheck: none for two lists, the rows a range beside a list removes. No existing plan moves: the suite’s other queries with two set quals on one column - now charged their merge on a one-column index too (lion_set_merge_cost(); lion_cost_set_and() since, §29.11) - force their scan or are counts, which the dearer scan leaves to the pushdown, and the qual evaluation per row a plain scan saves on an implied IS NOT NULL is far below any margin a pinned plan has.

test/sql/scan_notnull.sql (2026-10-19, §14 “IS NOT NULL in the scans”) runs 26 queries, each through the bitmap scan, the plain scan and LionBtreeScan or LionOrdered over a covering btree (index-only), against a sequential scan’s answer, on 60,000 rows: IS NOT NULL beside an equality, an IN list, a range and a range beside an equality; two and three of them beside a set and alone; on a column without NULLs; implied by an IN list of its own column and contradicted by IS NULL; as <> ANY of two values (§35); on a column NULL on 29 rows in 30; on three multi-key columns (text[], tsvector, jsonb), each with NULL and empty values, beside a scalar equality and a multi-key query, two and three alone, and beside a mode-ALL <@, the one qual still rechecked. Every answer is the sequential scan’s, no recheck removes a row but the mode-ALL query’s, and every node’s set is exact. Then LionOrdered in heap mode and under LIMITs (the lazy set), lists longer than a batch at 64 kB of work_mem, windows at the least pg_lion.scan_window_floor, the walk-or-subtract choice at DEBUG1 - the 60,000-row table’s columns subtract, a heap of 72 container keys whose column is NULL on 399 rows in 400 walks, both through every scan - three plans with nothing disabled, a dirty heap (rows moved into and out of NULL entries and EMPTY entries, rows deleted), the same after VACUUM, and EXPLAIN ANALYZE of a heap-mode and an index-only node: exact sets, the index-only one fetching no heap row. indexscan.sql’s a IS NOT NULL AND b = 5 no longer reports rows removed by its recheck.

test/sql/isectprobe.sql (2026-09-28, §29.11 “One price for the AND of sets” and “Correlated sets”), on 100,000 rows in 50 hidden groups that take turns row by row, sampled whole so the statistics are exact, and alike in every container, so that the probe’s share is the table’s whichever containers it samples: the Bitmap Index Scan’s rows for five filters derived from the group - equalities, an IN list, a multi-key && - within twice the true count with the probe and fifty times short or more without it, and for two of them likewise; a pair of independent columns and a single clause estimated exactly as without the probe; the ungrouped and the GROUP BY count going to LionCount only with the probe, by margins of three to fourteen times either way; and every answer the sequential scan’s. Its expected output, and every other line this change moved, come from a run on PostgreSQL 16. The first version of this change expected the ungrouped count to be LionCount’s without the probe as well, on the argument that a count costs no more than a scan of the same sets and its heap; but the scan need not AND the same sets - it leaves the dense IN list to its heap filter, the count cannot - and priced for two rows it won by three times.

The tests whose plans pin a lion path of two sets or more - plaincost.sql (the 13%, 43%, 0.1% and 0.01% queries), rangeprobe.sql (a range beside two equalities), multicolumn.sql, countbudget.sql, pinbudget.sql, fkjoin_semi.sql, fkjoin_nonunique.sql (a dimension’s two equalities) and samecolumn_scan.sql - keep their plans, run on PostgreSQL 16 in both WAL modes. The correlated shapes among them (multicolumn.sql’s k = 7 AND t = 't47', whose t implies k, and the five-clause contradiction over lion_mcp; rangeprobe.sql’s c = 0 AND v = -700) are corrected by the probe, and their plans are the ones pinned already; ordered.sql’s LionOrdered, the one plan that moved, moved with the price (§29.11, “One price for the AND of sets”).

test/sql/orlist.sql (2026-09-28, §29.11 “An OR of equalities is its IN list” and “Wide filters”), on 400,000 rows in 100 hidden groups that take turns over the heap, nine columns following the group seven times in ten: a filter of twenty sets - four equalities with a boolean, four lists, a multi-key && - a list alone and two lists beside an equality, each as lists and as ORs; the count’s plan and cost the same for both forms with the probe and without it; LionOrdered’s rows for both forms the probe’s, within twice the filter’s 287 rows, and without the probe a hundred times short; and every answer - the count, the rows, a GROUP BY, a page - the sequential scan’s, for those and for five ORs that are no list. Its expected output is the same on 16 and 18, and on 991c315 every plan, cost and row check of it fails and every answer passes. No other expected output moved on 16, 17, 18 or 19, in either WAL mode: the suite’s ORs are across columns or not of constants, its probed conjunctions fit the old budget, and the merge’s correction moves no pinned plan.

test/sql/neqrun.sql (2026-10-20, §29.11 “A run of <> is one exclusion”), on 100,000 rows: a text column 35% NULL over 150 values, ten of them common, an integer column half NULL whose 2,000 values are each as common as the others, so that none is a most common value, and a citext column. Seven filters - thirteen <>, the same values as a NOT IN and as a mix of both with values repeated and a clause commuted, forty <>, a NOT IN of eighty values, a NOT IN of sixty values of the integer column, and the thirteen beside an equality on an independent column - are each estimated within twice their rows at the scan and at a join above it, and a hundred times short or more without the correction; the three spellings of one filter are one estimate, with the probe on and off. A nested loop’s inner scan of the run, lion’s bitmap index scan and LionOrdered are within twice their rows, LionOrdered’s with the correction off too. One value, one value twice, two columns of a value each, a NOT IN holding NULL, <> ALL of no value, a generic plan’s Params and a table without a lion index keep core’s estimate exactly; three spellings of one citext value are counted once; and every answer - the count, the rows, a GROUP BY, a page - is the sequential scan’s. No other expected output moved on 16, 17, 18 or 19. A run past every row: a NOT IN of 2,400 values of the integer column, its even numbers, which it has none of, the same with one in twelve of them in it, and the first as <>, are within four times their rows at the scan and at a join above it, where core’s estimate is one row; on a table never analyzed, a NOT IN of 301 values and the same as <> are estimated as core estimates them, in the thousands of rows.

test/sql/scanand.sql (2026-09-28, §29.11 “The AND, as the count makes it”), on 120,000 rows in 60 hidden groups that take turns over the heap, two dense columns first in the index and five columns following the group eight times in ten: nine filters - the group’s fourteen sets, an AND inside a multi-key @>, a range beside the sets as several WINDOWs at the least floor and as a short WALK, a list of 429 values beside them in batches of 32, a dense set beside one or two selective ones, and two contradictions - each answered through the plain scan and through the bitmap scan with the sequential scan’s rows; a cursor that pauses the plain scan between rows, a nested loop’s parameterized inner scan rescanned per outer row, an exclusion constraint’s pinned check of three equalities, a dirty heap and the same after VACUUM; the count of the filter planned as LionCount at a random_page_cost of 1.1, both scans priced above it by more than half its rows with the probe and by less without it (core’s product); and, on a table where two dense columns come first and the selective two meet at four container keys, the plain and the bitmap scan reading fewer index buffers than the count, whose one visibility map page is the difference. On bc2a017 that last check fails - the scans read 47 index buffers to the count’s 22 - and every other passes: the answers were right, and the plan was already the count’s. No other expected output moved on 16, 17, 18 or 19, in either WAL mode: no price changed, the suite’s counts print the same Containers Visited and Probes Avoided, which are the merge’s, and its scans' buffers are not printed where an AND of several sets would move them.

29.13 Measured (2026-09-24, the prune slot’s PostgreSQL 20devel, assert-enabled: ratios, not absolute numbers)

One million rows of the benchmark’s scalar table (bench/comprehensive/workloads.py, scalar_data), twice: one copy with lion indexes on c200, c20k and c1m, one with btree indexes on the same columns, both vacuumed and warm. The queries are the benchmark’s heap fetches (sum(id), sum(length(payload)), so neither index can skip the heap), count pushdown off; median of 21 runs (9 for fetch_medium) of EXPLAIN (ANALYZE, TIMING OFF)’s execution time, in ms. “default” is the plan with nothing disabled; the other columns force one path each.

query work_mem lion default lion Index Scan lion bitmap btree Index Scan btree default
fetch_c1m (c1m = 12345, 1 row) 64MB Index Scan 0.023 0.025 0.021 Index Scan 0.021
fetch_rare (c20k = 123, ~50 rows) 64MB bitmap 0.058 0.053 0.057 0.046 bitmap 0.058
fetch_medium (c200 = 17, 5000 rows) 64MB bitmap 5.18 3.45 5.04 3.55 bitmap 5.08
fetch_c1m 64kB Index Scan 0.023 0.023 0.025 0.021 Index Scan 0.020
fetch_rare 64kB bitmap 0.058 0.052 0.058 0.046 bitmap 0.059
fetch_medium 64kB Index Scan 3.62 3.44 19.3 (lossy) 3.40 Index Scan 3.40

The plain lion scan is within 10-15% of btree’s on one and fifty rows and level with it on five thousand, and the planner makes the same choices for both indexes in every cell: the plain scan for one row, the bitmap at a normal work_mem (which for these warm, in-memory runs is the slower of the two for both AMs, so the choice is btree’s cost model’s, not lion’s), and the plain scan at 64 kB, where the lion bitmap goes lossy and costs 5.6x. The count pushdown keeps every shape of the benchmark’s scalar cases at 200k rows (checked by comparing each case’s plan with plain index scans enabled and disabled: only the two heap-fetch cases above that a plain scan now serves changed).

30. Lion-filtered, btree-ordered scans (LionOrdered, lion_ordered.c)

The shape is

SELECT ... FROM t WHERE <clauses a lion index answers> [AND <anything else>]
ORDER BY <columns an ordered index on t provides> [LIMIT n [OFFSET m]]

and the two plans core can make for it, when the WHERE is selective and the ORDER BY is not on the lion-indexed columns, are each wrong in their own way:

  1. walk the ordered index (a btree on the ORDER BY column), fetch EVERY heap row it names and filter it - with a 0.25% filter, 400 heap fetches per row returned;
  2. a lion bitmap heap scan and a top-N Sort - which fetches every matching row, 2,500 of them per million, to return ten.

Core cannot filter with one index and order with another. LionOrdered does: it builds lion’s EXACT answer to the WHERE clauses lion can answer, once, as a set of TIDs in memory; walks the btree in order reading only index tuples; tests each TID for membership BEFORE it touches the heap; and fetches, rechecks and returns only the members, until the LIMIT above stops pulling. Its cost is the lion lookups, the btree entries walked (index pages only) and one heap fetch per row returned (plus the members the rest of the WHERE rejects). Because the order comes from the btree, the node works on every supported major (16 .. 20), and DESC, multi-column ORDER BY, NULLS FIRST/LAST, expression indexes and the btree’s own index quals come with it for free. This replaces native ordered lion scans (§29.8, amcanorder on 18+), which stay deferred.

(Since 2026-10-06 the node has an INDEX-ONLY mode, §40: when the btree returns every column the node must produce - (k) INCLUDE (...), as for an Index Only Scan - the row’s values come from the index tuple and the heap is read only for pages the visibility map does not call all-visible. EXPLAIN says Ordered By: <index> (index only). The same walk is then also offered without an ORDER BY, as LionBtreeScan. Everything in this section stands for both modes unless §40 says otherwise.)

30.1 What qualifies

  • The relation. A plain base relation (RELOPT_BASEREL, RTE_RELATION, relkind table or materialized view, not an inheritance parent, no TABLESAMPLE) of the heap table AM (lion_table_am_supported(), §2), or a leaf partition (§30.11), with at least one lion index and one ordered index - or an ordered lion column (§30.11). Declined: a relation with security quals - RLS policies or a security-barrier view, i.e. an RTE with securityQuals or any restriction clause with security_level > 0 - exactly as LionCount declines them (§9, “Row-level security”): the ordinary plan applies them with core’s leakproofness rules. The target relation of an UPDATE, DELETE or MERGE is declined, and so is a relation with LATERAL references (rel->lateral_relids: a PlaceHolderVar it computes for a LATERAL subquery pulled up from under an outer join, la LEFT JOIN LATERAL (SELECT lb.id, la.x FROM lb ...) ON true): every path of it must be parameterized by the relations they name, and every path the node builds is unparameterized - the btree path it builds for its ordered side where core built none (§40.2) trips core’s assertion that it names them (get_baserel_parampathinfo()). (Since 2026-10-23 a nested loop’s probe is parameterized (§40.10), by its join clauses' outer relations; a LATERAL relation stays declined, its other paths being what they were.)
  • The WHERE. The relation’s restriction clauses, an implicit AND. Some of them are answered by lion indexes and the rest stay a heap-side filter. WHICH of them lion answers, and how, is not decided here but by core’s own index matching: every shape the bitmap path supports - =, IN / = ANY, the ranges of §28, IS [NOT] NULL, multicolumn indexes (§24), arrays and tsvector (§17), a partial index whose predicate the query implies, an OR across lion-indexed columns (a BitmapOr), several indexes ANDed (a BitmapAnd), Params wherever a Const may stand - is accepted, because the node’s lion side IS a bitmap path’s bitmapqual tree (§30.2).
  • The ORDER BY. Any pathkeys an ordered index scan of the relation provides: core’s own ordered IndexPaths, taken from the relation’s path list, so that btree ordering (direction, NULLS placement, multi-column prefixes, expression indexes, opclass orderings, the btree’s own index quals) is never re-implemented. An index scanned with ORDER BY operators (KNN) is not an ordered path in this sense and is skipped, and a lion index has no order (§29.1). In core this means a btree. Since 2026-09-29 the order may also be a lion index’s own ordered scalar column, walked by the node itself, for the ORDER BY’s first pathkey (§30.11).
  • LIMIT is optional: the node is priced like any path, with a start-up and a total cost (§30.3), and core’s LIMIT and sort planning choose. Without a LIMIT it can still beat a Sort for a selective result, and loses for a large one; the cost decides.

30.2 Planner integration

set_rel_pathlist_hook, chained to any previous hook and calling it FIRST, as the upper-paths hook does (§23, “hook coexistence”). GUC pg_lion.enable_ordered_scan (bool, default on). For a qualifying relation the hook runs after core has built the relation’s paths, and:

  1. The ordered side. Every unparameterized IndexPath in rel->pathlist - an Index Scan or an Index Only Scan path: the node visits the heap anyway, so the same index serves - with non-NIL pathkeys, no ORDER BY operators, a BTREE (§30.4’s early stop and switch rest on a btree returning each heap TID once per scan, in its pathkeys' order; any other ordered AM is declined, 2026-09-25 review), and no RowCompareExpr among its index clauses (v1). Core builds these exactly when the pathkeys are useful to the query (its ORDER BY, or a merge join), and keeps each unless a cheaper path with the same order exists.
  2. The lion side, found by core. create_index_paths() is run once more, on a SCRATCH copy of the RelOptInfo whose indexlist holds only the lion indexes, whose path lists start empty, and with its join clauses, eclass joins and parallelism switched off - so that only unparameterized, non-partial paths are built and the real rel is left untouched. What it builds - bitmap heap paths, whose bitmapqual is a lion IndexPath or a BitmapAnd/BitmapOr tree over lion IndexPaths, and plain lion index paths, each a one-leaf bitmapqual - are the candidate lion accesses. Only core’s matching decides what a lion index can answer (operator families, collations, partial-index predicates, OR arms), so the node accepts exactly what a lion bitmap scan would.
  3. One CustomPath per (ordered path, lion access) pair, each offered to add_path(), which keeps whichever of them is not dominated in start-up or total cost at those pathkeys. The path carries the ordered path’s pathkeys, rows = lion’s estimate of the relation’s rows (lion_probe_rel_rows(), §30.3), param_info = NULL, pathtarget = rel->reltarget, parallel_safe the relation’s consider_parallel (a partial path of the same pair besides, §40.9) and flags = CUSTOMPATH_SUPPORT_PROJECTION: no backward scan (a SCROLL cursor gets a Material), no mark/restore (a merge join restores through a Material).
  4. Which clauses the node still evaluates is decided the way create_bitmap_scan_plan() decides it. The lion side’s ORIGINAL qual, lionqual: the clauses of the leaves' index clauses, plus each partial index’s predicate where they do not imply it, ANDed across a BitmapAnd and ORed across a BitmapOr - create_bitmap_subplan()’s qual. A restriction clause leaves the heap filter when it is (by pointer) one of the ordered path’s non-lossy index clauses, or one of the index clauses of a lion leaf reached from the root through ANDS ONLY, or when predicate_implied_by() the lionqual; the rest is the plan’s qual, the filter core’s ExecScan() applies. Pseudo-constant clauses are core’s gating Result’s. (Fixed 2026-09-25: the first version dropped every leaf’s clauses, OR arms' included. Core builds each arm of a BitmapOr with the other top-level clauses at hand, so for a = 3 AND (b = 5 OR c = 7) over a lion index on (a, b) the arm a = 3 AND b = 5 reuses the top-level RestrictInfo of a = 3; the lion qual ((a = 3 AND b = 5) OR c = 7) does not imply it, yet it left the filter, and at default settings 38 of 50 rows had a <> 3. An arm’s clauses are now never treated as implied; only what predicate_implied_by() proves of the whole lion qual, or an AND-path leaf’s own clause, leaves the filter - which is what create_bitmap_scan_plan() does. Nothing else in the node assumes a leaf’s clause is implied: an exact set is exact for the lionqual as a whole, and the recheck of an inexact one is the whole lionqual.)

The plan carries in custom_exprs - where setrefs.c fixes their Vars and Params, and where SS_finalize_plan() finds the Param ids that make the executor rescan the node - the lion leaves' index quals (key on the left, as IndexClause.indexquals gives them), the ordered index’s index quals, the lionqual and the ordered index’s original clauses; and in a positional custom_private behind a shape marker (§10) the ordered index, its scan direction, the lion tree (AND / OR / leaf, preorder) with each leaf’s index and index columns, and whether a lion index clause was lossy. The executor substitutes an INDEX_VAR Var for each index qual’s key operand, as fix_indexqual_references() would, and builds the scan keys with core’s ExecIndexBuildScanKeys(), so a Param becomes a run-time key exactly as in an Index Scan.

No overlap with LionCount (§10) or the FK join (§27): those replace an aggregate at the upper rel; this is a scan path of the base rel.

30.3 Cost

With T = rel->tuples, the lion access’s selectivity s and index cost C_lion (core’s cost_bitmap_tree_node(): the leaves' indextotalcosts and the BitmapAnd/Or overhead), and the ordered path’s selectivity s_o (the fraction of the index its own quals leave) and index cost C_ord (its indextotalcost: the whole walk, index pages and index tuples):

  • start-up = C_lion + copying the answer into the set, containers x cpu_operator_cost, with containers = min(heap pages / LION_BLOCKS_PER_CONTAINER, s T);
  • the walk = C_ord + s_o T probes at LO_PROBE_TUPLES (2) × cpu_tuple_cost + cpu_operator_cost each (§40.3, “The probe’s price”; one cpu_operator_cost until 2026-10-06);
  • the heap = F = s_o T s members fetched, priced as cost_index() prices heap fetches: index_pages_fetched() at random_page_cost for an uncorrelated order, the members' share of the heap for a correlated one, interpolated by the square of the ordered index’s correlation (asked of its own amcostestimate); plus cpu_tuple_cost and the filter’s per-tuple cost for each of the F, and the target’s cost per output row. A clause that is both one of the ordered index’s index clauses and a lion AND-path leaf’s (g = 5 over a btree on (g, k) and a lion index on g) is counted once: s is divided by its selectivity, since the walk already met only entries that satisfy it;
  • total = start-up + walk + heap; rows = lion’s estimate of the relation’s rows, lion_probe_rel_rows(): rel->rows - computed from an OR of equalities as the IN list it spells - times what the intersection probe measured of the set clauses among the relation’s restriction clauses (§29.11, “Correlated sets”), as the lion side’s s already was. (Until 2026-09-28 rel->rows: core’s product of the clauses' selectivities, which for a correlated filter is a few rows where it holds thousands. Core’s LIMIT planning takes the share of the run cost a LIMIT needs from the path’s rows, so a page of such a filter was charged the node’s whole walk - rows below the LIMIT - and went to the lion scan and Sort that fetch every one of its rows. §29.11, “Wide filters”, has what that changes on the repro there.)

(Fixed 2026-09-25, second review: F was s_o T s whatever the two sides shared, so for WHERE g = 5 ORDER BY k over those two indexes the walk’s 10,100 members were priced as 102 fetches, and the node - a plain index scan of (g, k) plus the lion lookups - cost 721 against 13,909 for the bitmap scan and Sort that the planner had preferred to that same index scan.)

(Fixed 2026-09-30: the same for a clause the ordered index’s PREDICATE implies. A partial btree on (k) WHERE s <> '' walks only rows with s <> '', so a lion access for s <> '' - a lion index clause, or a partial lion index’s own predicate - selects nothing among them. lo_cost() now divides s by the selectivity of the lion qual’s clauses the predicate implies (lo_pred_implied(), core’s check_index_predicates() test asked of the lion side), and so does §30.11’s walk of a partial lion index, whose walked rows are now also scaled by its predicate as add_predicate_to_index_quals() scales an index’s selectivity. ClickBench’s Q15 (5M rows, WHERE SearchPhrase <> '' GROUP BY SearchEngineID, SearchPhrase, the benchmark’s partial btree search2 beside a partial lion index on the same predicate) took the node at 227k against the bitmap scan’s 485k and ran 3.6 s against 1.9 s: 659k rows walked in key order, priced as 86k fetches. ordered.sql §18 has the shape: the node, dearer than core’s own Index Scan of the partial btree it walks, no longer beats it.)

Core’s LIMIT planning scales a path’s run cost by the fraction of its rows the LIMIT takes (adjust_limit_rows_costs()), which is exactly how the node behaves: the walk and the fetches stop when the LIMIT stops pulling, and the set is built in full before the first row. So LIMIT 10 of a 0.25% filter pays the lion lookups, 4,000 btree entries and 10 heap fetches, where core’s ordered walk pays 4,000 heap fetches and bitmap + Sort all 2,500 matches of a million rows. For an UNSELECTIVE filter (c2 = 1, half the rows) the ordered walk needs 20 entries for 10 rows and no start-up at all, while the node first reads half a million TIDs out of lion: core wins, and the model says so. Without a LIMIT and with a large result the node walks the whole btree and fetches the rows in index order, i.e. randomly, while the bitmap scan fetches them in heap order and a Sort is cheaper: the model says that too.

The hazard: a filter correlated with the order (2026-09-25 review). LIMIT scaling assumes the members are spread evenly along the btree, so that LIMIT 10 of a 0.25% filter meets its rows after 4,000 entries. When the filter is correlated with the order - c = 398 ORDER BY k where c is k / 2500, or a zero-row AND of two correlated columns - the members lie at the far end (or nowhere), and the walk reads almost the whole index: 997,510 entries and 41.5 ms on 1M rows, against 0.6 ms for bitmap + Sort. Core’s own ordered walk has the same hazard (it paid 113 ms there), but the node competes with bitmap + Sort as well, and the planner cannot see the correlation (no cross-column or filter-to-order statistics). The model is left as it is; the EXECUTOR bounds the damage instead (§30.4, “When the walk is not paying”): the set’s exact size is known before the walk starts, and once the walk has cost what fetching the remaining members would, it fetches and sorts those. The same query now takes 2.8 ms: 80,000 entries walked, then 2,500 members fetched and sorted (§30.10). Still 3.5x the bitmap plan’s time - the cost of not knowing - but no longer 50x.

Memory. The set is the answer’s containers (§3): at most 4 kB per 64 heap blocks and about 2 bytes per member for a sparse answer, plus 16 bytes of directory per container - some 12 MB for half of 100M rows, 200 to 300 MB for a tenth of a billion. It is built in hash_mem (get_hash_memory_limit(), work_mem x hash_mem_multiplier, what core allows one node’s in-memory hash table), and past that in a temporary file mapped into memory (§30.4, “Spilling”), with the walk’s visited bits, about a bit a member and eight bytes a key (§30.4, “Stopping early”), which the planner counts in the set’s size and prices (lo_spill_cost(): the pages the estimate puts past hash_mem, at 60 cpu_tuple_cost each, the 5 us a page measured) and never refuses. (Until 2026-10-19 the planner offered no path whose estimate exceeded hash_mem, and a set that outgrew it at run time DEGRADED to its container keys, every member rechecked: at a billion rows every wide filter’s set is past the default hash_mem, and a refused plan there is the query failing under pg_lion.vector_postfilter’s default.)

30.4 Execution

  • BeginCustomScan opens the ordered index and every lion index (AccessShareLock; the heap and the plan’s filter are core’s ExecInitCustomScan()’s), checks the table AM again, builds the scan keys (ExecIndexBuildScanKeys(); run-time keys get an ExprContext of their own, as an Index Scan’s do), and initialises two recheck quals with ExecInitQual(): the lionqual and the ordered index’s original clauses. The scan slot is a buffer heap tuple slot. A non-MVCC snapshot is an internal error: every snapshot a SELECT’s executor runs under is MVCC, and §30.5’s argument needs one. Under SERIALIZABLE its lion lookups and walks take the page locks of §9, “SERIALIZABLE”, as every reader’s do.
  • The set is built at the first fetch after a start or a rescan - or started lazily, below - not in BeginCustomScan: an exec Param is set by a nested loop after the node is initialised. Each lion leaf evaluates its run-time keys, opens lion_source_open(index, keys, nkeys, keeppins = false, cxt) (§29.3, §29.8; an MVCC reader holds no pin, §29.5), and copies every container it produces into a LionTidSet: a sorted array of (container key, container copy), searched by binary search and tested with lion_container_contains(). A SETS or UNION source arrives in ascending container key (lion_source_sorted()) and its containers go straight in; a WALK (a range) or a LIST (a long IN list) arrives entry by entry or batch by batch - one PIECE per container key each entry or batch touches, so a range over 400,000 distinct values brings 400,000 one-TID pieces for 34 keys - and its pieces are GATHERED by key (gathering the pieces, below), the keys taken into the set in order once the source is complete. Gathering the pieces. A container key is a heap block / 64, a small integer, so the gather (LoGather) is a table of the keys in chunks of 64 slots, a chunk made when a key of its is first met (4 KB, and a bit a key met beside it), a cache line a key: the key’s container so far, in the set’s memory context, and room for 27 members not in it yet. A piece of up to 27 members is copied into the room; when the room would overflow, or a piece brings more than it holds, the room’s members and the piece are FLUSHED into the container (lo_gather_flush()), a new one taking the old one’s place, charged against hash_mem as the set’s are:
    • while the key has at most LO_MERGE_DIRECT_MAX members together - LION_WIDTH_ARRAY_CARD(1), 256 at 8K, an ARRAY as large as the narrowest NARROW, so that the union is always an ARRAY or a RUN - they are merged member by member (lion_container_or_merge()): each part’s members are set in a two-level bitmap, the words of an image of the key and a summary bit for each word, and read back in order, which sorts them and drops the repeats (a multi-key column’s entries bring a row once per key of it that matches) without a comparison, and the ARRAY that leaves is optimized. Only the words a member falls in are touched - the first member of a word assigns it - so nothing of 4 KB is cleared or scanned, and the result is byte for byte the container an image makes, since lion_container_optimize()’s choice is a function of the members alone;
    • past that the container, the room and the piece are ORed into one bitset image, which is turned into the smallest container (lion_bits_to_container()) - the way of a damaged piece too, whose header hides more values than an ARRAY holds;
    • and once the key has more members than an ARRAY holds the container is kept as a BITSET, whose bits the flushes set in place, no image and no new container, and which the finish makes the smallest container. When the source is done (lo_gather_finish()) the keys come out of the table ascending for nothing: each key’s room is flushed, a BITSET optimized, and the container, the set’s own already, appended; the chunks go as their keys are done. So a piece costs its slot’s cache line and the copy of its members, a key whose members come one at a time a flush per 27 of them, and the gather holds about what the set will - a chunk per 64 keys met and a container a key - at every moment, which is what lets a set that fits hash_mem be built within it. A BitmapAnd is the container-wise AND of its children’s sets, a BitmapOr their OR; a leaf that selects nothing is an empty set. No lossy page exists anywhere in this: every member is a TID some entry held (§29.6). (Fixed 2026-09-25, second review: the pieces were merged only once the source was complete, and pairwise. Every piece kept a directory entry and a container copy until then, so a set whose merged form was a few hundred bytes degraded at a 64 kB hash_mem, and after degrading the directory went on growing one entry per piece - 52 MB at a 1 MB hash_mem for a range over 4M values, 197 kB of a 64 kB one still held after the build of a 15,000-value range, and “invalid memory alloc request size” past 227 pieces; ORing the pieces pairwise cost a pass over the growing container per piece, 4.2 s to build the set of a 250,000-value range that the lion bitmap scan reads in 0.2 s, now 0.33 s.) (Fixed 2026-10-16, the million-row vector benchmark: the directory still doubled only while it stayed more than half full after a merge, so a merge came every few hundred pieces, each one rewriting every key - a range over 100,000 of a million unique values, 223 keys, merged every 800 pieces - and sorted every slot through a comparator that read the keys. That was half of LionOrdered’s 130 ms before its first row under that filter; the set now takes 45 ms, as long as the lion bitmap scan of the same range takes to walk its 100,000 entries. A column whose ranges span that many keys wants summaries (§32): the set then takes 3 ms.) (Fixed 2026-10-17, DBpedia’s 115,000 vectors of one row a heap page, a lionvec KNN under a lion filter through summaries = auto: every key’s pieces went through the image, 8 KB of passes - the clear, the count, the runs and the members of lion_container_optimize() over the 4 KB of a BITSET - whatever the key held. A summary’s buckets and entries each cover every block of a column spread evenly over the table, so u < 0.1 brought 3.5 pieces of 6.5 members to each of 1,800 keys and u < 1.0 25 pieces of 64, and the set build was 52% of a 9.7 ms query at u < 0.1 - its merges 36% - and 80% of 14.6 ms at u < 1.0, against 2.1 ms for the unfiltered lionvec scan: a cost of the keys, a heap block in 64, not of the members, which a million rows read as a fixed 35-45 ms. Merged member by member, 3 pieces of 6 take 40 ns against the image’s 1.2 us, and 25 of 64 0.2 us against 1.4; the queries took 9.5 -> 6.6 ms at u < 0.1, 9.6 -> 6.7 at u < 0.3 and 14.3 -> 11.6 at u < 1.0, with the same rows. That left the sort of the slots, a quicksort, the largest part of a merge: 16% of the query at u < 1.0, whose 45,000 pieces come in 25 runs, which the quicksort took 2.4 ms over and a merge of the runs takes 0.34 (measured apart); the query then took 9.1 ms, and the set build 69% of it.) (Reworked 2026-10-19, the 50M-row vector benchmark: the pieces were kept as they came, a copy and a directory entry each, and the directory sorted by key and its pieces ORed whenever it filled - every key a piece had come for since the last time, nearly every key for the pieces of a range - so a 10% range over a unique double with summaries, 5M pieces of one member for 260,417 keys, built its set in 845 ms, 170 ns a piece, as long as the restricted lionvec scan took under the filter; on 2M rows of ten a page, 198,000 pieces for 3,125 keys took 245 ns each, 70 of them the source’s, 40 the copy, 64 the sort and 70 the four merges. Gathered, the 2M table’s 198,000 pieces take 105 ns each, 70 of them the source’s: the 10% range’s set is built in 21 ms against 55, the 3% range’s in 5 against 12, the 30% range’s 594,000 pieces in 81 against 155; with summaries of 4,096 rows, 17 ms against 29 at 10%, and without summaries, an entry a value, 74 against 113. At 64 kB of hash_mem the sets that were exact still are, the gather holding a chunk and the containers it has made and no more. The 50M table is to be measured again.)
  • Exact, or rechecked. The set is EXACT when every leaf’s lion_source_exact() is true and no lion index clause was lossy. Then every member satisfies the lionqual and nothing is rechecked; otherwise every fetched member is tested against the lionqual (the multi-key strategies of §17, a range beside its column’s sets, §29.6), and one that fails counts as removed by the lion recheck.
  • Spilling (2026-10-19). The set’s memory - its containers, the gather’s chunks, the directory - comes from lo_set_alloc(), which counts its bytes against get_hash_memory_limit() and, at the first byte past it, moves the rest of the build into an ARENA (lion_arena.c): a temporary file of the backend’s (OpenTemporaryFile(), in pgsql_tmp, gone with the query or the transaction) mapped into memory in regions, the first of a megabyte, each next one doubling the file to 64 MB, from which blocks are handed out in size classes - the powers of two from 16 bytes, and one class for the largest container - with a free list a class, so that a container that grows by a few members at each flush goes back to the block it just left. A block never moves, so the set’s containers are probed through plain pointers wherever they are (lo_set_find(), the lionvec scan’s lv_member()), and what was built in memory before the spill stays there: the memory the node holds is at most hash_mem and the file holds the rest, in the page cache while the kernel keeps it there and on disk when it does not, which is the spill. The file is extended through fd.c, written as zeros before a region is mapped (so that temp_file_limit stops it, log_temp_files and pg_stat_database count it, EXPLAIN (BUFFERS) shows its blocks as temp written, and every page exists on disk before the first write to it: a page a full disk could not allocate at that write would end the backend with SIGBUS), and the mappings go with the query’s memory context, the file with its resource owner, however the query ends. A rescan that rebuilds the set reuses the file. The set stays exact, and the answer does not depend on work_mem; EXPLAIN ANALYZE says spilled N kB. The arena is not §27’s LionSpill, which writes a collected set’s containers to a BufFile in key order to be read back one at a time: this set is probed at random, by every TID the walk or the scan meets. Cost: about 5 us a page of 8 kB on the 2M emulation (a 64 kB hash_mem against the default: the 10% range’s set, 1.3 MB spilled, 20.9 ms against 20.1; the 30% range’s, 3.1 MB, 84.3 against 82.9; the 90% range’s, 6.2 MB, within the noise of 250), the price in §30.3. The arena maps its file with POSIX mmap(), which it needs: pg_lion builds where there is one. The walk’s visited bits (below, “Stopping early”) come from lo_set_alloc() too, charged against hash_mem with the set and in the arena past it, so neither the early stop nor the switch depends on work_mem either; so does a LionOrderedSplit’s arm’s narrowed set (§48.4, “The arm’s memory”). (Before this the set DEGRADED past hash_mem: it freed every container and kept the keys - “every TID of these 64 heap blocks may be a member” -, rechecking every fetched member, and the planner refused a path whose estimate exceeded hash_mem; the test tables for that at 64 kB, ordered.sql’s lon and vector_filter.sql’s vfn, now spill.)
  • The set, lazily (2026-09-30). Building the set reads every container of every leaf’s sets, over the whole table, before the walk hands out its first TID - and a LIMIT walk that finds its rows in the first few hundred container keys needs the set at those keys only. When every leaf’s keys are set trees (an equality or a list on a scalar column, and beside one any column’s IS NOT NULL, whose NULL entry the probe takes away from the leaf’s AND, §14 - since 2026-10-19: lo_lazy_ok(); a range is a walk of entries, a multi-key query may need every row, and a list longer than a plain scan opens at once, lion_scan_list_batch(), is read a batch at a time - a stream holds a cursor for each of its sets), the set is LAZY instead: each leaf key’s posting sets are located once (lion_scankey_sets(), as a bitmap scan locates them) and their pins let go (§30.5: the node holds none between rows), and each TID the walk hands out is looked up in a memo of container keys (a simplehash, lo_memo); a key met for the first time is evaluated there - each key’s stream sought to it (lion_stream_at(), the leapfrog seek of §22), the leaf’s keys ANDed and the tree’s ANDs and ORs applied to those containers - and remembered, members or none, with the members this scan met (a 4 kB bitmap, in the per-scan context; the memo is held to half of hash_mem below). A stream only goes forward; a probe behind where it stands starts it again over the same located sets, which a walk in heap order (the TIDs of one entry, a btree correlated with the heap) rarely does and one in random order does at nearly every key. So the set is BUILT, as above, once the probes have cost what the build would - one unit of work for each set a probe seeks, LO_LAZY_RESTART_WORK (4) more for each set a stream starts again over, against a budget of the located sets' recorded container counts, the containers the build reads (ncontainers, never below LO_LAZY_MIN_WORK, 64) - or once the walk has gone as far as the fetch-and-sort switch below would let it go were the set as big as its entries' recorded member counts say (ntids: an AND no bigger than its smallest child, an OR than its children together), or once the walk has met LO_LAZY_WALK_WORK (8) entries for each unit of that budget, and at least the switch’s 10,000: an entry walked costs 30 to 70 ns and a unit of the build about 350 (the 5M-row list query below has a budget of 24,040 and builds in 8.5 ms), so by then the walk has cost about what the build will, and the early stop and the switch, which need the set’s size, can act from there on; or once the memo holds half of hash_mem. The counts are hints, and decide only when the set is built: whether the switch comes is decided on the set built, and in the regression suite it comes at the same entry it came at before. And a walk that does not go in heap order is built for at once: once LO_LAZY_ORDER_KEYS (64) keys have been evaluated, if a quarter of them or more lay behind the key evaluated before them. A btree walked in an order the heap does not follow - a score, a random key: ORDER BY score DESC LIMIT 100 - hands out TIDs at random container keys, half of them behind the last, and each such probe seeks every set again by a descent, at about eight build containers' cost; the budget above counted it at five, and over 5M rows a 0.67% filter under ORDER BY <random numeric> DESC LIMIT 100 probed 2,012 keys before it built, 29.2 ms against 10.0 ms with the set built at the start. With the rule it builds after 64: 11.4 ms against 10.7. A lion column’s walk goes back only between entries (the here IN (...) walk of §30.11 “Lists” probed 468 keys, none behind the one before, and stays lazy), and a btree correlated with the heap seldom goes back. The build takes over what the walk met (lo_lazy_convert()): each memo key’s bitmap is read into the walk’s visited bits (“Stopping early”), a bit for each member there the built set still has, so the members already returned count as met and the switch does not fetch them again; a key the built set has no member at any more lost only TIDs a VACUUM removed, dead to every snapshot. A small set, whose build is cheap, is built at once, before the walk has cost more than a few dozen probes: the worst case costs about twice the build. And the set starts lazily only where the planner expects the walk to stop short: the node alone in its query under a LIMIT (with its OFFSET) below LO_LAZY_MAX_SHARE (a fifth) of the rows it would return, or a walk it expects to end within the switch’s 10,000 entries anyway (the ordered index’s selectivity, or the walked range’s, times the rows, times that share); otherwise the plan says LO_FLAG_BUILD and the set is built at the start. A walk that goes further meets most of the set’s container keys - one in heap order passes over them once for each entry it walks - and a probe of a key costs more than the build’s read of it. Over the 5M rows below, LIMIT 500 (a third of the 1,420 rows estimated) walks 70,000 entries over 4,224 of the 6,010 keys: built at a fixed 40,000 entries it took 22.8 ms warm, and lazy throughout 13.8, against 16.3 built at the start; LIMIT 1000 and no LIMIT took 25.5 and 34.8 ms with the build at 40,000 entries, 31.4 and 43.2 lazy until the probe budget, and 20.5 and 24.3 built at the start. A walk the planner misjudges still meets the rules above: a far-end walk under LIMIT 10, its members all in the last 1% of the order, builds at 10,000 entries and switches at 12,160, 0.63 ms where building at 40,000 took 1.40. A leaf key that selects nothing makes an AND of it empty, as before (lo_lazy_empty()). Exactness is the lion quals' own (every lazy key is an exact set). A rescan keeps the memo unless a Param of the lion quals changed, as it keeps a built set; the members met are the scan’s and go with it. When a rule above builds the set mid-walk, the members met are counted again against the built set rather than carried over: the probes answered from the index as it was, and a VACUUM in between may have taken out the dead TIDs the walk met and marked, which the built set no longer has; carried over, they would let the count reach the set’s members while live members were still ahead, and the early stop would end the scan before them (test/isolation/specs/ordered_lazy_vacuum.spec). EXPLAIN ANALYZE says Lion Set: lazy, K keys probed, exact, or N containers, exact, after K keys probed (why) when it was built, why naming the rule that built it - heap order, probe budget, switch due, long walk (LO_LAZY_WALK_WORK entries a unit of the budget) or memo size (the memo past half of hash_mem) - with a count when rescans built it for more than one; a set the plan builds at the start says N containers, exact, as with the lazy set off. test/sql/ordered_transitions.sql forces the probe budget, long walk, memo size and heap order rules and the plan’s build at the start (its walks that go far bound the walked column by parameters, so that the generic plan expects them short), ordered.sql the switch due one. The heap order rule needs 64 keys evaluated within the budget: there a walk that goes behind at one key in four, over a 72-container table whose first leaf key’s set is empty where the walk starts, so that each key costs a probe of that set alone while the budget counts all three. The planner still prices the set as built (§30.3): the lazy set makes the node cheaper than its estimate, never dearer by more than the build. pg_lion.enable_lazy_set (on) turns it off, for comparing the two and for the tests of the build (a set that spills). test/sql/ordered_lists.sql. Measured on a release PostgreSQL 18, 5M rows of a 3 GB heap, long IN (2 of 20) AND here IN (2 of 50) AND we IN (2 of 30) ORDER BY here LIMIT 50 (§30.11, “Lists”): the set of long and we built is 5,989 containers read before the first row, 8.0 ms warm and 21 ms from disk (direct I/O, 147 pages); lazily the walk probes 468 of the table’s 5,860 container keys, 1.3 ms warm and 6.0 ms from disk (55 pages). A btree on (here, long, we) INCLUDE (...) answers the same in 0.02 ms warm and 0.8 ms from disk (10 pages), for that query shape alone, at 936 MB against the lion index’s 39 MB.
  • The walk. index_beginscan() on the ordered index under the executor snapshot and index_rescan() with its keys, in the path’s direction. On 16 .. 19 each TID comes from index_getnext_tid(), which reads only the index, and a member is fetched with index_fetch_heap(), which follows the HOT chain under the snapshot and lets the btree mark entries whose chains are dead to all, as an ordinary index scan does. PostgreSQL 20 moved the heap side of index scans into the table AM (table_index_getnext_slot(); there is no index_getnext_tid() or index_fetch_heap()), so there the node takes the TID from the index AM through tableam_index_getnext_tid() and fetches a member with table_fetch_tid() (the HOT chain under the snapshot, the TID moved to the version it sees) and table_tuple_fetch_row_version(). A member with no visible version is skipped; a visible one is rechecked when the set is inexact (above) or the index set xs_recheck (against the ordered index’s original clauses), then handed to ExecScan(), which applies the filter and projects. A set with nothing in it ends the scan before the walk starts.
  • Stopping early. Once the walk has met as many DISTINCT members as the set holds (its cardinality) it stops: nothing further along can be one. The walk remembers which members it has met, a bit a member’s PLACE (lion_container_place()): an ARRAY’s or a RUN’s member’s place is its rank among the container’s members, found by a binary search or a pass over the runs before it, and a bitmap’s is its bit in the bitmap, found at no cost, the bitmap’s places being its bits. The set’s keys' places follow one another (places[], eight bytes a key, made with the set), and the bits come in chunks of 32,768 (4 kB), each made as the walk first meets a member of its places, so that a LIMIT walk that meets a few members clears a few chunks; they are lo_set_alloc()’s, within hash_mem with the set and past it in the arena (“Spilling”), and the walk counts at every work_mem. They take an eighth of a byte a member of an ARRAY or a RUN, a sixteenth of the ARRAY at most, and as much as a bitmap takes for a bitmap’s: never more than the set and a bit a member. A 4 kB bitmap for each container key the walk touches, its member’s bit its offset in the key, would take 4 kB for a key of three members: on a 1B-row table of ten rows a page, 1.6M keys, 6.4 GB for a walk that meets a member at each, where a bit a member is 1.25 MB for a 1% filter. A member counts once however often the walk meets its TID. Scans Stopped Early counts the scans the stop ended. On 3M rows whose 3,000 members (one row in 1,000, at every one of the set’s 809 keys) lie in the first quarter of the order, the walk stops after 752,001 entries at 64 MB, 4 MB, 1 MB and 64 kB of work_mem alike (32 to 34 ms, the best of seven runs); 4 kB a key, 3.2 MB, would not fit in 1 MB’s hash_mem, and a walk that stopped counting there reads all 3,000,000 entries (116 ms). test/sql/ordered_transitions.sql counts at 64 kB, with the bits spilled while the set is not. (Fixed 2026-09-25: the first version counted every meeting, on the premise that a btree returns each TID once. Within one scan it returns each INDEX TUPLE once, but under an MVCC snapshot the paused walk holds no pin (nbtree’s dropPin, and lion’s §29.5), so while a cursor waits VACUUM can remove a dead member X the walk has already passed and an insert can take X’s slot with a key further along; the walk then meets X’s TID again, as a tuple the snapshot cannot see. Counted twice, it made the count reach the cardinality one visible member early, and FETCH ALL stopped before B: test/isolation/ordered_recycle.spec. The choice between the two fixes the review offered: remembering the members met, or counting only members fetched VISIBLE (a recycled slot is never visible to the snapshot, so that count cannot overshoot). The second needs no memory but, on a table with dead members, rarely stops early at all; the first stops exactly when the last member has been met, dead or alive, and the switch below needs the same record anyway. A second meeting of a TID is skipped without a fetch, which is right: the tuple it now names was written after the snapshot. The premise that the ordered index hands out each index tuple once is why the node is limited to btree.)
  • When the walk is not paying: fetch and sort. The set’s exact size is known before the walk starts, so the walk is a bet on reaching LIMIT rows early, and the executor can cap the bet. Once this scan has walked at least LO_SWITCH_MIN_WALK (10,000) entries and at least LO_SWITCH_RATIO (32) entries per member it has not met yet - the ratio is the setting pg_lion.ordered_switch_ratio (real, default 32, 1 to 1,000,000; a large value keeps a walk from ever switching, and index-only mode adds a density gate to it, §40.4) - it stops walking, fetches every member it has not met in TID order (the HOT chain under the snapshot, then the version it sees), keeps the visible ones that pass the lion recheck (an inexact set) and the ordered index’s own original clauses (these rows did not come through the index), sorts them by the path’s pathkeys (SortSupport on the pathkeys' own sort operators, collations and NULLS placement) and returns them. This is the ski-rental rule: the ratio is what a member’s heap fetch costs in index entries (0.04 us an entry against 1-2 us a random fetch, §30.10), so by the time the node switches it has spent on the walk about what the fetches cost, and in the worst case pays about twice what fetching and sorting from the start would have; the 10,000 floor keeps a small set’s walk, which costs well under a millisecond, from switching at all. Correctness: every row the walk has not returned yet is among the members it has not met (a visible member’s index entry lies where the walk has not been, and the walk only skips non-members), and none it has returned is (those were met); rows with equal sort keys may come in any order, which is all the pathkeys promise. The switch needs the pathkeys to be plain columns of the relation - an expression index’s order would evaluate its functions where the ordinary plan calls none, which a revoked EXECUTE could tell apart (§30.6). Its sort is a tuplesort of each row’s sort keys and its heap tuple whole - the TID it goes by and the header too, so that a row comes out as the walk would have handed it out, its system columns with it - in work_mem and past it in temporary files, as a Sort node’s is: the switch comes at the same entry and fetches the same members whatever work_mem is, and never leaves the walk to go on. When nothing between the node and its query level’s LIMIT can drop a row the node returns or read one past the rows the LIMIT takes, those rows bound the sort (lo_limit(), LO_PRIV_LIMIT): a constant LIMIT and OFFSET (the planner’s limit_tuples is only an estimate of a Param), not WITH TIES, which reads the rows tied with the last; no row marks, whose LockRows skips a row it cannot lock; one relation; no filter of the node’s own, whose rejects the Limit does not count; and the query’s order the node’s, so that no Sort or Incremental Sort above it reads on. A partition’s walk under a MergeAppend or an Append is bounded too: neither reads more of a child than the rows it returns. The bounded sort keeps, of those rows, the ones the walk has not returned yet, in a heap (top-N heapsort, a few kB for LIMIT 10); any other sorts every row the switch keeps. The planner prices the walk (§30.3) and not the switch, which is the executor’s bound on a walk the model misjudged, there whatever the settings: nothing in the price counts on it. On 3M rows of 135 bytes whose 10,000 members lie at the far end of the order (c = 299 ORDER BY k LIMIT 10, heap mode, a filter correlated with the order), the switch comes after 320,000 entries at every work_mem: 16 to 20 ms at 64 MB, 4 MB, 1 MB and 64 kB, the ten rows kept in 27 kB, and with a filter of the node’s own, which unbounds the sort, 17 to 25 ms, 1.5 MB on disk below 4 MB; walking on to the end of the index instead, 2,990,010 entries, takes 80 ms (98 with the filter). EXPLAIN ANALYZE says Switched to Fetch and Sort: S of N scans, M members fetched and, for the last switch’s sort as a Sort node shows its own, Fetch and Sort Method (top-N heapsort, quicksort, external merge), Fetch and Sort Space Used: K kB and Fetch and Sort Space Type (Memory, Disk). A rescan starts afresh, its sort ended with its files.
  • EvalPlanQual (SELECT ... FOR UPDATE over the node): the recheck method tests the substituted row against the lionqual and the ordered index’s original clauses, as an Index Scan’s IndexRecheck() does; ExecScan() applies the filter.
  • Rescans. ReScanCustomScan restarts the walk (its keys are evaluated again at the next fetch) and rebuilds the set only when a Param of the lion quals changed (chgParam against pull_paramids() of the lion quals, taken at BeginCustomScan). A LATERAL ORDER BY ... LIMIT subquery whose lion filter takes the outer row’s value rebuilds per outer row; one whose Param is only in the btree’s quals keeps its set. A rescan with no changed Param (a cursor rewound) keeps it too: the snapshot is the same.
  • EndCustomScan ends the index scan, closes the indexes and frees the node’s memory.

30.5 Correctness and concurrency

Let S be the executor snapshot (MVCC), taken before the node starts; the set M is built from the lion indexes at a time t1 after S was taken; the btree is walked afterwards. A row is returned iff its TID comes out of the btree walk, is in M, has a version visible to S at the fetch, and that version passes the rechecks and the filter. The claim is that these are exactly the rows the ordinary plan returns, in the same order.

  • A row visible to S that satisfies the lion quals is in M. Its inserting transaction committed before S was taken (or is ours, at an earlier command id); an index entry is written before its transaction commits, so lion held its TID before S and therefore at t1. Only VACUUM removes it, and only once the row is dead to every snapshot, S included.
  • HOT chains. A HOT update changes no column of any index that is not summarizing (HeapDetermineColumnsInfo() against the relation’s HOT-blocking columns; lion sets amsummarizing = false), so every version of a chain has the same lion-indexed values and the same btree-indexed values. Both indexes hold only the ROOT TID of a chain: the btree returns the root, M holds the root, and index_fetch_heap() / table_fetch_tid() find the version S sees from it. An update of a lion-indexed column or of the btree column is never HOT - it inserts new entries into both indexes under the new version’s TID - so “the root is in M” means “the version S sees satisfies the lion quals” (up to the recheck of an inexact set).
  • A TID in M whose row S cannot see - dead, deleted by a transaction committed before S, or inserted by one S does not see, before or after t1 - is filtered by the heap visibility check, the check every Index Scan relies on. Rows inserted after t1 are not in M, and they cannot be visible to S (their transaction was running or not started when S was taken), so missing them is right. Rows deleted after S was taken are still visible to S and still in the lion index (VACUUM cannot remove them while S holds the horizon back), so they are in M and returned, as the ordinary plan returns them.
  • A recycled TID. M is private memory and nothing pins the lion pages it came from (§29.5’s MVCC rule). If VACUUM removes a member’s tuple after t1 and its slot is reused, the new tuple was written after t1 by a transaction S cannot see: the fetch finds nothing visible. The btree side is an ordinary index scan under core’s own interlock (nbtree’s MVCC dropPin rule).
  • Order. Rows come out in the order the btree walk returns them, which is what the pathkeys claim; a non-member is only ever skipped, never reordered. After a fetch-and-sort switch the rest come out sorted by the pathkeys, all of them at or after the last row the walk returned (§30.4).
  • Non-MVCC snapshots are refused (§30.4). Hot standby: nothing here reads the visibility map or relies on a pin, so a standby needs nothing an Index Scan does not.
  • SERIALIZABLE. The btree walk takes its own page predicate locks; each heap fetch takes a tuple lock (heap_hot_search_buffer()); and the lion lookups M is built from lock the pages that stand for their keys (§9, “SERIALIZABLE”). A concurrent write that would change the answer inserts into a lion index (a new matching row, or a non-HOT update that makes a skipped row match: a conflict with the page lock of its key), or writes a row the node returned (its tuple lock), or inserts into a btree page the walk read. A skipped non-member that is deleted, or updated and still does not match, does not change the answer; the ordinary plan’s tuple lock on it is a false positive the node does not reproduce. test/isolation/ordered_serializable.spec is the write-skew case.

30.6 Security

  • Privileges are the range table’s: the node scans the relation (scanrelid > 0), and ExecCheckPermissions() checks it like any scan.
  • EXECUTE (§9, “EXECUTE on what a count replaces”): the node evaluates the plan’s filter (initialised by core), the lionqual and the ordered index’s original clauses, all through ExecInitQual(), which checks EXECUTE on every function and operator for the current user at executor start - the checks the ordinary plans make (an Index Scan initialises its indexqualorig and its filter, a bitmap heap scan its bitmapqualorig). The lionqual is initialised even when the set is exact, as indexqualorig is. Neither ordinary plan checks the btree’s ordering comparison, and neither does the node. test/sql/ordered.sql revokes an equality function from PUBLIC and gets core’s error from both plans.
  • RLS and security-barrier views are declined (§30.1). Leakproofness does not arise beyond that: the membership test is not user code, and the user’s quals run on a row only after the heap visibility check, as in every core scan.

30.7 EXPLAIN

Limit
  ->  Custom Scan (LionOrdered) on fact
        Filter: (length(payload) > 60)           -- core's, printed first by core
        Ordered By: fact_c1m_idx (backward)      -- "(backward)" for a backward walk
        Index Cond: (c1m > 100)                  -- the ordered index's own quals, if any
        Lion Cond: ((c200 = 17) AND (c2 = 1))    -- the lionqual
        Lion Indexes: fact_c200_idx, fact_c2_idx

and with ANALYZE Index Entries Walked, Lion Set Hits (walked entries that were members), Heap Fetches (members with a visible version), Rows Removed by Lion Recheck (printed when the set is not exact or something was removed), and Lion Set: N containers, exact | rechecked[, spilled S kB][, B builds], the kB those of the arena the set was built past hash_mem in, and its walks' visited bits with it (§30.4, “Spilling”), the builds counted when rescans rebuilt it (a LATERAL subquery whose lion filter takes the outer row’s value: one per outer row). A node that never ran (LIMIT 0, an untaken branch) built no set and prints neither of the last two. A node whose scans switched to fetch and sort adds Switched to Fetch and Sort and its last sort’s method, space used and space type (§30.4, “When the walk is not paying”).

30.8 Declined in v1, and why

  • Partitioned tables. A partitioned parent’s ordered plan is a MergeAppend over per-partition ordered paths, and the hook also runs for each partition (RELOPT_OTHER_MEMBER_REL), so offering the path there might just work; but run-time pruning, a partition’s translated security quals and the pathkeys of child rels are a surface of their own to test. The natural v2. (Done 2026-09-29, §30.11: each leaf partition is scanned like a table, under core’s Append or MergeAppend, with its pruning.)
  • The target relation of an UPDATE, DELETE or MERGE, which a merge join’s ordered input could otherwise make the node scan: its row identity and EvalPlanQual rechecks would work as they do for SELECT ... FOR UPDATE (tested), but nothing needs it and nothing tests it.
  • Parameterized paths (the node as the inner side of a nested loop with t.x = outer.y pushed into it): an inner side needs no order, so an ordered filter scan has nothing to offer there. Params inside a subquery (LATERAL, correlated) are ordinary Params and ARE supported (§30.4). (Done 2026-10-23 for the walk that claims no order, §40.10: what an inner side wants of the node is §40’s index-only walk, which leaves the heap out of every probe; an ordered one stays declined.)
  • Backward scans, mark/restore: the shape needs neither; core adds a Material where a caller does. A parallel walk is §40.9’s.
  • RowCompareExpr index quals on the ordered index ((a, b) > (1, 2)): the executor’s key substitution handles one key column per clause, and such a path is skipped.
  • Ordered indexes other than btree: §30.4’s early stop and switch rest on btree’s behaviour.
  • The fetch-and-sort switch for an expression order (ORDER BY k % 1000 over an expression index): the walk never switches there (§30.4).
  • Lion accesses core did not build (for example ANDing in a lion index that core’s choose_bitmap_and() judged not worth its heap savings): the node only considers the accesses core’s own lion index paths contain. A sharper set would save btree steps but no heap fetch.
  • Non-MVCC snapshots, RLS, other table AMs: above.

30.9 Tests

test/sql/ordered.sql, written before the node and failing without it (no plan contains LionOrdered): every query compared with the ordinary plan (pg_lion.enable_ordered_scan = off) row by row IN ORDER - ORDER BY ends in a unique key, so that the order is determined - with its plan checked to contain the node: ORDER BY k [DESC] [NULLS FIRST] LIMIT n under filters of each kind (=, IN, a long IN list, a range, IS NULL, a multicolumn lion index, an OR across two lion indexes, arrays and tsvector with their recheck, a partial lion index), a multi-column ORDER BY, an expression index, OFFSET and LIMIT, no LIMIT, a LIMIT beyond the match count, an empty answer, a filter on the btree column itself, a residual filter, Params under a generic plan, a LATERAL subquery rescanned with exec Params per outer row, cursors (FETCH forward; a SCROLL cursor over a Material fetching backward), a dirty heap (updates, deletes, HOT updates) and the same after VACUUM, a set spilled at a tiny work_mem; the declines (RLS, the GUC off, no lion-answerable clause); EXECUTE revoked (the node and the ordinary plan give the same error, security_exec.sql’s pattern); and the plan choice with nothing disabled - the node for a selective lion filter with ORDER BY a btree column LIMIT 10, core’s ordered btree scan for an unselective filter, and no node for a large result without a LIMIT. Isolation: ordered_cursor.spec (a cursor paused after its set was built while another session inserts matching rows, deletes and updates matching rows and commits: the rest of the cursor returns exactly its snapshot’s rows, and rows inserted between DECLARE and the first FETCH do not appear either; VACUUM runs while it is paused and completes), ordered_recycle.spec (the review’s repro of §30.4’s recycled TID: FETCH ALL returns B) and ordered_serializable.spec (the write-skew pair, both through the node: one of them fails). Added with the 2026-09-25 fixes, each failing before its fix: a = 3 AND (b = 5 OR c = 7) at default settings and forced (the OR-arm residual; 38 wrong rows of 50), the same shape on the main fixture, and a filter correlated with the order - members at the far end, a zero-row correlated AND, three members early and a thousand late (the switch after rows were returned, including through a paused cursor), NULLs first under DESC with members among the NULLs met before the switch, a btree qual beside it, and a LATERAL rescan - each compared in order with the ordinary plan, with the switch shown by EXPLAIN ANALYZE. Added with the second review’s fixes, each failing before its fix: a range over 15,000 distinct values at a 64 kB hash_mem, whose set must stay exact, and whose LionOrdered set memory context, read while a cursor is paused after the build, must hold no more than hash_mem; and g = 5 ORDER BY k over a btree on (g, k) and a lion index on g, where the plan must not be the node (§30.3), beside g = 5 AND h = 3 ... LIMIT 10, where it still is. test/sql/ordered_switch.sql switches at a 64 kB work_mem that the members' rows do not fit in, each query compared in order with the ordinary plan: the sort bounded, in memory, under a LIMIT, with an OFFSET, backward and with system columns in the rows; on disk, unbounded, with no LIMIT and under one it may not keep to - a filter of the node’s own, WITH TIES, FOR UPDATE, an order sorted again above the node, a LIMIT $1 of a generic plan; a LATERAL rescan that switches twice in three scans; two partitions under a MergeAppend; and the walk’s rows, then the sort’s, through a cursor too. make hookcheck’s companion module chains set_rel_pathlist_hook too and checks, in both load orders, whether the LionOrdered path is in the rel when the previous hook returns (§23).

30.10 Measured (2026-09-24, the vacuum slot’s PostgreSQL 20devel, assert-enabled: ratios, not absolute numbers)

One million rows of the benchmark’s scalar table (bench/comprehensive/workloads.py, scalar_data, the columns the queries touch), lion indexes on c2, c20, c200, c20k and c1m and a btree on c1m beside them - the roaring_btree portfolio of the benchmark -, vacuumed and warm. The queries are the benchmark’s ordered_* cases (SELECT id, payload FROM fact WHERE ... ORDER BY c1m, id LIMIT 10; the btree orders by c1m and an Incremental Sort above the scan breaks the ties by id). Median of 15 runs of EXPLAIN (ANALYZE, TIMING OFF)’s execution time, in ms; “default” is the plan with nothing disabled, the other columns force one path each (total cost of the plan in parentheses).

query default LionOrdered core’s ordered btree walk lion bitmap + Sort
ordered_filter (c200 = 17 AND c2 = 1, 0.25%) LionOrdered 0.47 0.51 (207) 2.60 (430) 4.87 (10,944)
ordered_filter_desc (the same, DESC) LionOrdered 0.55 0.61 (207) 2.54 (430) 4.41 (10,944)
ordered_broad (c2 = 1, 50%) btree walk 0.033 0.56 (4,442) 0.032 (2.9) 191 (38,531)

The node is 5x faster than the ordered walk and 8-9x faster than the bitmap and Sort for the selective filter, and the planner picks it; for the unselective one the ordered walk needs 21 entries for 10 rows while the node first copies half a million TIDs (265 containers) out of lion, 17x slower, and the planner keeps the walk. What the node did for ordered_filter: 4,636 btree entries walked, 23 members fetched for 10 rows - core’s bitmap qual answers only c200 = 17 (its choose_bitmap_and() judges c2 = 1’s set not worth its heap savings), so c2 = 1 is the filter and half the members fail it (§30.8’s “lion accesses core did not build”; ANDing c2’s set in would halve the fetches). Without a LIMIT the same query walks 999,767 entries to find all 5,165 members, and the planner keeps the bitmap heap scan and Sort (5.5 ms).

After the 2026-09-25 fixes (same slot and build; median of 15). The three benchmark queries are unchanged within noise - ordered_filter 0.72 ms through the node (default plan 0.55), ordered_filter_desc 0.52, ordered_broad kept on core’s walk (0.032) - because their walks stay under the switch’s 10,000-entry floor. The correlated hazard of §30.3, on a second 1M-row table with k = i (btree) and c = (i - 1) / 2500, g = c % 2 (lion), 0.25% per c value:

query default LionOrdered btree/lion Index Scan + Sort lion bitmap + Sort
c = 398 ORDER BY k LIMIT 10 (members at the far end) LionOrdered 2.83 2.87 (was ~41) 0.90 0.80
c = 150 AND g = 1 ORDER BY k LIMIT 10 (no row) Index Scan + Sort 0.25 4.90 0.25 0.21

The first is still chosen by the planner (it cannot see the correlation) and now switches after 80,000 entries, fetching and sorting the 2,500 members: 3.5x the best plan instead of 50x. The second is not chosen here; forced, it switches after 80,000 entries and fetches 2,500 members that all fail g = 1.

30.11 A lion column’s own order, and partitions (2026-09-29)

A benchmark’s feed page is the latest few fact rows whose fk is in a filtered dimension:

SELECT f.fk, f.kind, f.ts, f.name FROM fact f
WHERE f.kind = 'a' AND <lion-indexed filters> AND f.ts >= <a stable lower bound>
  AND f.fk IN (SELECT d.pk FROM dim d WHERE <dimension filters>)
ORDER BY f.ts DESC LIMIT 100;

fact is LIST-partitioned by kind, and the kind prunes it to one leaf; the leaf has a lion index over its filter columns and ts, with summaries, and btrees that lead with fk, none with ts; the dimension has a unique key and a covering btree for its filter. Core’s plan is a nested loop from the dimension’s kept keys into the leaf’s btree on fk that fetches every matching fact row and keeps the latest hundred in a top-N sort. The fast plan walks the leaf in ts DESC order through the filters, tests each row against the dimension, and stops at the hundredth.

Why the planner made no ordered plan. No path of the fact had the order ts DESC, so core had nothing to put under a nested loop and a Limit: its only order for ts was a Sort.

  1. LionOrdered took its order from a btree only (§30.2), and the fact had none that orders by ts: the column is a key column of the lion index and nothing else. Lion’s directory holds each ordered scalar column’s keys in the column’s order (§21), but nothing read them in that order for a query - native ordered scans were deferred (§29.8).
  2. LionOrdered declined partitions (§30.8), and the relation the kind prunes to is a leaf partition, an RELOPT_OTHER_MEMBER_REL.

Core’s LIMIT costing over the semi join was not the obstacle. Once the ordered path exists, core’s own composition - Limit -> Nested Loop (outer: the ordered fact, inner: the dimension probed by its unique key) - is what it builds (the semi join over a unique key is an inner join to it, reduce_unique_semijoins()), and it wins at core’s own estimates, which are the underestimated ones: the Limit takes 100 / estimated rows of the nested loop’s run cost, and the walk’s share of that is still far below what the nested loop from the dimension costs (the measurements below). The underestimate makes the ordered plan look EXPENSIVE, not cheap - fewer join rows means a larger share of the walk per hundred rows - but not expensive enough: on the data below the Limit over the nested loop is priced at 8.4 + 8,035,000 x 100 / (estimated join rows), against 368,351 for the nested loop from the dimension and its sort, so it loses only below about 2,200 estimated rows - a 320-fold underestimate of the 707,196 rows there are, where core’s was 14-fold there and the benchmark’s 84-fold. The hashed variant - hash the kept keys once, stream the ordered fact through them - is not a plan core can make: a hash join claims no order, since its batches would reorder the outer side. So the change is the ordered path, and nothing about the join.

The walk (lion_order_walk_begin(), _next(), _end(), lion_scan.c). The TIDs of one ordered scalar key column in its order, ascending or descending, bounded by the column’s range keys:

  • Ascending it is the bitmap scan’s leaf walk (§28): the leaf the tightest lower bound lands on (lion_range_first_leaf()), then the right links, each leaf copied under a share lock and let go, and the entries the range selects handed out until the first past an upper bound.
  • Descending it starts on the leaf the tightest upper bound lands on (lion_range_last_leaf(), new: the first entry not below the bound is there, so every entry the range selects is on that leaf or to its left), or where the column’s VALUE entries end (lion_dir_value_end()), hands the leaf’s entries out from its last to its first, passing over those above an upper bound, and goes to the LEFT sibling: the page whose right link is this leaf, found from the left link by walking right (lion_dir_step_left(), which the endpoint probe of §28 uses), since the page the link names may have split since. It ends at the first entry below a lower bound. A leaf with a left link whose sibling cannot be found is reported as corruption: nothing but damage makes the sibling unreachable, and a walk that ended there would return too few rows.
  • Either way it keeps to the column’s VALUE entries (§24: an earlier column’s entries and the column’s reserved ones sort before them, its summaries of §32 and a later column’s after them) and does not use summaries: it needs keys. The NULL entry is walked before the values or after them, as the ORDER BY puts NULLs, or not at all when a clause on the column is strict (a range bound, IS NOT NULL). An entry’s TIDs come in heap order - rows with equal keys in no order the pathkeys promise anything about - from its INLINE payload or its posting tree, a leaf at a time from the leftmost, each leaf copied and let go (lion_emit_chain()’s walk); a posting leaf that no longer belongs to the set ends it, as there.
  • Between two calls it holds no pin and no lock: what it hands out is a heap TID for the caller to look up under an MVCC snapshot (§29.5), and the node refuses any other (§30.4).

Why it misses no row and returns none twice. A row visible to the snapshot was inserted by a transaction that committed before the snapshot was taken, after writing the row’s key into every index; the walk begins after the snapshot. Its entry is removed only once VACUUM finds all its rows dead to every snapshot, this one included. Directory splits move entries only to a new page right of the one they split, and no directory page is unlinked (§21): an ascending walk that follows the right links of its copies meets every entry that existed when it began, and a descending one does too - when the left neighbour of a leaf it has copied splits, the neighbour’s upper half goes to the new page, which is the page whose right link names that leaf, and so the one the walk reads next; when the leaf it holds a copy of splits, the half that moves is the upper one, which the copy holds and a descending walk has handed out or is about to. Entries inserted after the walk began hold rows no older snapshot sees. Each row version is under exactly one key of the column - a HOT chain’s versions share the key and the root TID (§30.5), and an update of the column is not HOT and puts the new version’s TID under its new key - so a TID the walk meets twice is a slot VACUUM freed and an insert took after the walk had passed it, and the row there is invisible to the snapshot, as §30.4’s recycled TID is; with a set, §30.4’s record of the members met keeps the early stop from counting it twice.

Planner (lo_lion_walks(), lion_ordered.c). For each ordered scalar key column of a lion index of the relation - a plain column of the table, under the column’s own collation, whose key type is the column’s and whose comparison is the type’s default btree comparison (the rule by which a GROUP BY walk claims its order, §21 “Planner”): the directory is then in the order of that btree’s < - the ascending pathkey is looked up among the query’s equivalence classes (build_expression_pathkey(), which for a partition finds its member of the parent’s class). When it is the first of the ORDER BY’s pathkeys, the walk gives that pathkey, whatever its direction and NULLS placement. A partial index needs its predicate implied (predOK), as core asks of an ordered index path. Only the ORDER BY: a GROUP BY, a DISTINCT or a merge join over the column wants every row, which the count pushdown counts from the same entries, and a hash or a Sort takes from the heap in its order where the walk would fetch them in key order. (The first version offered the walk for the query’s first pathkey whatever it was for, and the plans of twelve regression tests' GROUP BY and DISTINCT queries changed - most of them forced ones, where the walk under a sorted aggregate replaced the count pushdown.)

The restriction clauses the walk answers exactly bound it: a range comparison of the column - <, <=, >=, > of the column’s operator family, either way round - with a value that does not depend on the row (a Const, a Param, a stable expression such as now() - interval '...': evaluated when the walk starts, as an Index Scan’s run-time keys are; no subplan), under the index column’s collation, and IS NOT NULL. They leave the filter and are the plan’s Index Cond. The collation is core’s rule for an index clause (IndexCollMatchesExprColl(), as lion_match_index() applies it): the walk compares a bound with keys stored in the order of the index column’s collation, so a comparison under another is no bound of it and stays in the filter. Until the 2026-09-29 review any was taken, and t < 'a' COLLATE "C" over an English column returned no rows (‘A0’ is below ‘a’ in “C”, above it in English); test/sql/ordered_collate.sql. The other clauses may be a lion SET: core’s lion accesses are built as for a btree’s walk (§30.2), on a scratch copy of the relation whose lion IndexOptInfos are copied too, with the walk’s clauses taken out of the clauses they may match (lo_lion_accesses()) - a set that collected the range would read all of it, which is what the walk exists not to do. Or no set at all: every row the walk meets is fetched, and the clauses are the filter. One path per (walk, set) and one with no set, and add_path() keeps what is cheaper. The walk needs no restriction clause at all: ORDER BY ts DESC LIMIT 10 over a table whose only index on ts is a lion index is a walk with nothing to filter, where it was a whole scan and a Sort.

Lists (2026-09-30). One list on the walked column, g = ANY (array) or g IN (...), is the walk’s too: it descends to each of the list’s values in turn, in the walk’s direction, and walks only their entries (lion_order_points(), lion_order_point_range()). Before, the list was left to the set, and the walk started at the column’s first entry: here IN ('H10', 'H11') AND <two other lists> ORDER BY here LIMIT 50 over 5M rows of fifty values read every entry of H00 .. H09 without a member, until the fetch-and-sort switch gave up on it at 41,344 entries and fetched and sorted all 1,292 members - 12.2 ms warm and 128 ms from disk, where the same query over a list at the bottom of the column fetched 50 rows. Now it walks 7,816 entries of H10 and fetches 50 (8.0 ms warm with the set of the other two lists built, 1.3 ms with it lazy, §30.4). The values are sorted in the directory’s order - the probe’s sortproc, the column’s comparison - without NULLs, which no key equals, and without duplicates, which would walk an entry twice; each is a range of its own, the range keys' bounds and >= v AND <= v, whose two bounds are the ones the walk descends to (every entry it selects equals v, whatever else bounds it). Only under the column’s own equality for its own type, the opclass’s input type: that is the order the walk can seek in. n = ANY ('{3,5}'::int8[]) over an int4 column stays the set’s, as does a second list on the column. The cost adds a descent, 2 random_page_cost, for each value after the first (estimate_array_length()); s_w has the list’s selectivity in it already. test/sql/ordered_lists.sql.

Cost (lo_cost_walk()). With T the relation’s tuples, s_w the selectivity of the walk’s clauses, nd the column’s distinct values, and a set of selectivity s and index cost C_lion (§30.3):

  • start-up: a descent, 2 random_page_cost of the index’s tablespace, and the set’s C_lion plus its build as every LionOrdered set is priced (lo_set_build_price(): a cpu_operator_cost a container key and its pieces; then its spill, lo_spill_cost()), its members the candidates when it leaves quals to the recheck (lo_set_candsel(), §40.8), and s their share from there on;
  • the walk: W = s_w T TIDs and E = min(W, s_w nd) entries; each entry range_union_entry_cost (a small entry of a walk, §28, “Counting a walk”), each TID cpu_operator_cost, and the directory leaves under them at seq_page_cost - E entries of a key and an INLINE set of two bytes a row, and, for keys past a quarter of a page, their posting pages;
  • the heap: F = s W rows fetched (W without a set), priced as cost_index() prices fetches, the column’s correlation with the heap interpolating (lion_var_heap_correlation()), plus cpu_tuple_cost and the recheck of a lossy set per row fetched, the filter per row the recheck passes and the target per row returned;
  • rows: lion_probe_rel_rows(), as §30.3. Core’s LIMIT scaling of the run cost is the walk’s behaviour: it stops when the Limit stops pulling.

Executor. The node is §30.4’s with the walk in place of the btree scan: BeginCustomScan opens the lion index as the ordered index and builds its scan keys the same way, takes a relation predicate lock on it (the walk reads it without index_beginscan(), as the set does), and - on 16 .. 19 - an index scan’s heap fetch state (table_index_fetch_begin()), through which each TID is looked up as an index scan looks one up (table_index_fetch_tuple(): the HOT chain from its root, the pin on the last heap page kept between calls); on 20 table_fetch_tid() and the row version it moves to. Each start of the walk evaluates its keys, so a rescan with a new Param restarts it with the new bound, and the set is rebuilt only when a Param of the lion quals changed, as before. The walk is exact - its entries are tested with the column’s own comparison - so nothing is rechecked for it but, after §30.4’s fetch-and-sort switch, the rows that did not come through it (its clauses are the plan’s ordorig, as a btree’s are). Without a set there is no set to test, no early stop and no switch; with one, both work as §30.4 says. EXPLAIN reads

Limit
  ->  Nested Loop
        ->  Custom Scan (LionOrdered) on fact_a f
              Filter: ((tags && '{t1,t2,t3}'::text[]) AND (kind = 'a'::text))
              Rows Removed by Filter: 997
              Ordered By: fact_a_tags_ts_fk_idx (ts, backward)
              Index Cond: (f.ts >= (now() - '365 days'::interval))
              Index Entries Walked: 1367
              Lion Keys Walked: 1167
              Heap Fetches: 1367
        ->  Index Scan using dim_pkey on dim d
              Index Cond: (pk = f.fk)
              Filter: (flag AND (lvl < 40) AND (grp = 1))

Ordered By names the lion index and the column walked, backward for a descending walk, and nulls first or nulls last when the NULL entry is walked; Index Cond is the walk’s clauses; with ANALYZE, Index Entries Walked counts the TIDs the walk met and Lion Keys Walked the entries they came from. Lion Cond, Lion Indexes and the set’s counters appear only with a set.

Partitions. A leaf partition (RELOPT_OTHER_MEMBER_REL) is now scanned like a table, a btree’s walk and a lion column’s walk alike: the node is a path of each partition, and core’s Append - or a MergeAppend, for several - puts them together with its own pruning at plan time and run time. A partition of a table being updated is declined as the table is (all_result_relids), and a partition’s translated security quals decline as a table’s do (they carry security_level > 0). The fetch-and-sort switch sorts by the relation’s own member of each pathkey’s class, which for a partition is a child member - 18 keeps those apart from the class’s members and hands them out through setup_eclass_member_iterator() - so lo_sort_keys() asks for the partition’s; it used to skip child members, and the switch would have been off on every partition.

Declined, and not done.

  • The semi join inside the walk (the dimension’s kept keys as a posting-set filter of the fact, or a hash of them tested per row, in the ORDER BY/LIMIT position): core’s composition works, so it was not built. What it would add is run-time protection. The ordered plan follows core’s estimate of the join: a semi join that keeps few rows where core expects many would walk the whole range under the nested loop - the hazard every core plan over an ordered index scan and a LIMIT has (§30.3). A node that held the dimension’s keys could count, or switch to the per-key lookups of §27 once the walk had cost what they would. The measured selective case below goes to core’s plan at core’s estimate.
  • More than the first pathkey: a lion column orders one column; core sorts the rest incrementally. Expression columns, multi-key columns, a column in hash order (§21), and a key type other than the column’s type: no order to walk, or not the query’s.
  • Summaries are not read by the walk (it needs keys). Parallel: the node stays parallel_safe = false.
  • A btree’s walk beside a range its set collects. The lion side of a btree’s walk is still core’s lion access over every restriction clause (§30.2), so a range on the btree’s own column that a lion index also matches is collected into the set - read in full at start-up - although the btree’s walk is bounded by it already; §30.3 only divides it out of the fetches. Leaving the btree’s index clauses out of the set, as a lion column’s walk leaves out its own, would be the same change there; no measured query needed it.

Tests. test/sql/orderedsemi.sql: a fact LIST-partitioned by kind and a plain copy, each with a lion index over (tags, ts, fk) with summaries and a btree on (fk, ts), ts with three rows a value and NULLs, NULL fks, and a dimension with a unique key and a covering btree; the plans chosen with nothing disabled (the feed with its semi join, over the pruned partition and the plain table; the feed alone; a set; a selective semi join, which keeps core’s plan; two partitions under a MergeAppend); answers compared in order with the node off - the semi join as IN and EXISTS, LIMIT with OFFSET, both directions, ranges bounded on one side and two, a stable bound, NULLs first and last each way with no range, IS NOT NULL, NULL fks, a set, an empty set, two partitions and all of them, FOR UPDATE; the walk’s counters and the fetch-and-sort switch after a sparse set; a generic plan with Params in the kind, the tags, the bound (a NULL one too) and the LIMIT; LATERAL subqueries rescanned with the outer value in the bound and in the set; a cursor fetched in pieces, and one paused after fifty rows while 6,000 new keys go into the fifty hours it walks next, both ways - a descending walk that took the left link of its copy of a leaf, not the page whose right link is the leaf, fails it; a dirty heap - ts moved past every key and to NULL and back, tags changed, HOT updates, deletes - before and after VACUUM, and lion_index_verify(); a parallel plan’s shape and answer; a btree’s walk over partitions; and the GUC and a GROUP BY, which get no walk. pinbudget.out changed: the subquery SELECT k FROM lion_pin ORDER BY k LIMIT 100 that builds its list is now a walk of the lion index on k, and the plan the test prints shows it.

Measured (2026-09-29, PostgreSQL 18, packaged build, shared_buffers 3 GB, work_mem 4 MB, warm cache, four cores that other work shared; medians of five runs after one, ms). A synthetic star of the benchmark’s character: a fact of 22,000,000 rows LIST-partitioned by kind, the leaf the kind prunes to 20,000,000 rows (a 2.2 GB heap; the lion index over the filter columns, ts and fk, with summaries, 2.4 GB; a btree leading with fk, 600 MB), ts at random over two years and a range of one, an && filter keeping about a quarter of the rows; a dimension of 500,000 keys whose filter columns are correlated, so that the filter keeps 200,000 keys where core expects 49,823, and whose kept keys hold a quarter of the fact’s rows, most of the rest being on keys the dimension does not hold. Core estimates the semi join at 51,245 rows; it has 707,196, 14 times as many. “Before” is the library before this change; “node off” is this one with pg_lion.enable_ordered_scan = off, and “forced” the plan without core’s Sort (enable_sort = off). Serial, then with two workers:

query before: chosen plan, cost, ms after: chosen plan, cost, ms node off: cost, ms forced walk: cost, ms
semi join, LIMIT 100 nested loop from the dimension, top-N sort: 368,351, 4,672; with a Gather Merge: 155,663, 2,305 walk under a nested loop: 15,689, 2.6; the same with workers, 1.8 368,351, 4,725; 155,663, 2,636 the chosen plan
semi join, LIMIT 10 367,499, 4,854; 155,297, 2,383 1,576, 0.25; 0.24 367,499, 4,993; 155,297, 2,697 the chosen plan
semi join, LIMIT 1000 369,204, 4,666; 156,122, 2,683 156,811, 28.8; 24.9 369,204, 4,739; 156,122, 2,223 the chosen plan
no semi join, LIMIT 100 lion index scan and top-N sort: 775,383, 2,603; parallel seq scan, Gather Merge: 513,920, 3,752 walk: 268, 0.42; 0.42 775,380, 2,637; 513,919, 3,434 the chosen plan
selective semi join (596 dimension keys, 79 rows), LIMIT 100 nested loop from the dimension, sort: 8,052, 1.75; 1.79 the same: 8,052, 1.83; 1.84 8,052, 1.59; 1.70 842,251, 21,946; 29,569

The feed with its semi join went from 4.7 s (2.3 s with workers) to 2.6 ms: the walk met 1,367 rows of 1,167 keys, 370 passed the filter, 100 of those the dimension (EXPLAIN ANALYZE). A thousand rows took 3,699 rows that passed the filter, 29 ms. With two workers the chosen plan is the serial one - the node is not parallel-safe, and at LIMIT 1000 core’s parallel plan is priced 0.4% below it, within add_path()’s fuzz, where the walk’s start-up wins. The selective semi join keeps core’s plan: the walk would meet the whole year, ten million rows of 8.6 million keys, fetch every one and probe the dimension for the 2.7 million that pass the filter - 22 to 35 s for 79 rows - and is priced at a hundred times core’s plan, which reads the 596 keys' rows. That margin is core’s estimate of the join (954 rows for 79); an estimate a hundred times too high would choose the walk, and that is the hazard the semi join inside the walk would guard against (above). At work_mem 64 MB the plans are the same but for a Memoize core puts over the dimension’s probe, which spends 40 to 75 ms of a 50 to 80 ms query setting up its cache - a hash table core sizes from its estimates and does not price - where the plan without it takes 2.9 ms.

31. Cost calibration on the release build (2026-09-27)

Every number in this section is from a RELEASE build - PostgreSQL 18.6 at -O2, no assertions - on a 4-core, 15 GB VM shared with other work, shared_buffers 5 GB with every relation below resident, work_mem 64 MB, and every core cost GUC at its default. §10’s “The units” says why (the constants had all been fitted on an assert build) and what core’s own plans cost there; this section holds the data sets, the constants that moved, and the decision matrix that checks them. The scripts, the data generators and the raw results are outside the tree, in the project’s .local/calib/ directory, with a README on rerunning them.

Data sets

table rows shape
s 5M, 66,667 pages ints of 2, 20, 200, 20,000 and 1M values at random; 200 values stored in order (clustered); a skewed one (90% one value); one 10% NULL; a lion index on each, btrees beside three
w 5M, 102,639 pages a wide shape: an enum of three values (60/20/20%), a two-value text, a boolean, 2,000 ints, 50 countries, 8 sizes, a text[] of one or two tags, a timestamp in order; one lion index over seven columns, btrees over four columns and over one and one
r 5M a date in heap order (2,500 rows a day), a date at random, a 20-value status, a unique timestamp in order; lion and btree on each
fact / dim / dim2 5M / 200k / 2M §27’s shape: two fks (25 and 2.75 rows a key, 1% NULL), a 20-value kind, a 6-value sen, a tsvector; dimensions with a primary key and a lion index on (status, country, size); and fact2 (2M) over dim1k (1,000)
t 2M §28’s range table: an enum, a boolean, 2,000 ints, a unique timestamp and 50 countries under one lion index
sd 2M a copy of s whose heap updates leave dirty, for the recheck terms

Measurement

Medians of five EXPLAIN (ANALYZE, TIMING OFF) runs while the machine was quiet (core’s units, the FK join shapes); after that, because other work shared the machine, the backend’s own CPU time from /proc/<pid>/schedstat over a DO loop that plans a query once and runs it many times, the alternatives interleaved round by round. That measures serial plans only: a parallel worker’s CPU is not the leader’s, and the parallel plans keep their wall-clock measurements (§27’s “Parallel”). Fits are non-negative least squares on what the planner can compute (containers, members, probes, pages, entries, TIDs), in microseconds, converted at §10’s 500 units a millisecond.

What moved

constant before (assert build) now fitted on fit
LION_CONTAINER_COST 2 cpu_operator_cost a container, nothing else 8 35 counts of 1 to 4 sets (§10) median residual 27%; the 35 re-costed span 283 to 3,724 units a millisecond (were 87 to 46,239)
LION_MEMBER_COST (up to LION_MEMBER_CAP, 1,024) - 0.15 cpu_operator_cost same 0.74 ns a member
LION_PROBE_COST / LION_MEMORY_PROBE_COST - 40 / 30 cpu_operator_cost same 210 ns; 100 to 200 ns in memory
LION_AND_MEMBER_COST - 0.8 cpu_operator_cost same 4 to 5 ns
LION_DESCENT_COST (a count’s, an IN list value’s, an FK join’s) 50 (25 in the FK join) cpu_operator_cost 120 23 FK join shapes (§27) 0.6 us a directory page; median residual 9%
LION_ENTRY_COUNT_COST / LION_LIST_GROUP_COST cpu_tuple_cost 50 / 18 cpu_tuple_cost GROUP BY walks of 200 to 20,000 entries 1.0 to 1.3 us / 0.35 us a group
LION_UNION_SET_COST (= LION_FKJOIN_SET_COST) 0 (80 cpu_tuple_cost in the FK join) 100 cpu_tuple_cost grouped counts over OR and IN sources 2 to 6 us a set
LION_RECHECK_TID_COST / LION_RECHECK_GROUP_TID_COST cpu_tuple_cost 1.5 / 6 cpu_tuple_cost sd, clean against dirty 24 to 70 ns a TID; 130 ns in a grouped count
LION_FKJOIN_PROBE_COST 30 cpu_operator_cost 80 23 FK join shapes 0.4 us (0.46 over 2,000-row sets)
LION_FKJOIN_COUNT_COST 50 cpu_tuple_cost 25 same the fit leaves the per-row term at zero
LION_PLAIN_FETCH_ROW_COST 0.5 cpu_tuple_cost 1.0 3 plain against bitmap shapes (§29.11) 11.5, 21 and 20 ns a row
a range entry’s containers (§28) 2 cpu_operator_cost §10’s merge units 35 range shapes provisional: 47 to 2,294 units a millisecond (median 500), were 14 to 8,049
unchanged, and confirmed LION_DISTINCT_TEST_COST 50 and LION_RANGE_UNION_ENTRY_COST 12 cpu_tuple_cost, LION_FKJOIN_COPY_*, LION_BITMAP_ROW_COST 1.0 us; 290 to 300 ns; 374 to 595 units a millisecond; core’s own number

Not refitted: the terms that arrived with work merged beside this pass - LION_WALK_PASS_COST (§29.11’s WALK, 5 cpu_tuple_cost an entry) and LION_FKJOIN_SORT_COMPARE_COST / LION_FKJOIN_SORT_KEY_COST (§27’s non-unique dimension keys, scaled at 250 units a millisecond) - carry their assert-build measurements, and the endpoint probe’s plain-scan remainder (lion_plain_note_remainder()) adds the same per-row terms as the correlation it completes.

The model’s shape changed with them, each in its section: the early exit of an AND is priced (§10, §25); a GROUP BY is charged per entry of its column, with the entry’s set a source (§10, §26); how many containers a key’s rows lie in follows its column’s correlation (§10); an IN list value pays a descent (§21); lioncostestimate() charges the AND of its sets and no operator per index tuple, and the plain scan charges its quals only when it rechecks them (§29.11 - the four-set AND and pushdown.sql’s lion_pdn); LionCount’s recheck reads the corrected correlation that lion_var_heap_correlation() now shares with the plain scan (§10).

The decision matrix

128 queries where lion competes with core - equality, IN, enum and boolean counts (eq); ranges inside, outside and over the whole domain on timestamps, dates and dense ints (range); GROUP BY and count(DISTINCT) (group); FK join counts forward, reverse, anti and distinct (fk); and fetches that go to a plain, bitmap or sequential scan at many selectivities and over the multicolumn index (fetch) - each with every alternative forced (the count node, seq, bitmap, plain and index-only scans, the join methods, the btrees alone by dropping the lion indexes in a transaction). Every distinct plan was timed once in backend CPU time; each build’s choice is what the planner picked with that build’s library loaded, at random_page_cost 4 and 1.1. A mispick is a choice more than 15% (and 0.05 ms) slower than the fastest plan; one query in each column sits between 10 and 15%, which is noise here, and is not counted.

mispicks, time lost before, rpc 4 now, rpc 4 before, rpc 1.1 now, rpc 1.1
eq (34) 3, 10.8 ms 1, 3.8 ms 3, 10.8 ms 2, 4.3 ms
range (26) 2, 20.6 ms 2, 20.6 ms 2, 20.6 ms 2, 20.6 ms
group (19) 4, 1,889 ms 1, 31.2 ms 4, 1,889 ms 2, 203 ms
fk (24) 2, 62.4 ms 3, 37.4 ms 1, 37.2 ms 2, 12.2 ms
fetch (25) 7, 104 ms 8, 94.3 ms 8, 199 ms 8, 199 ms
all (128) 18, 2,087 ms 15, 187 ms 18, 2,157 ms 16, 440 ms
the chosen plans' total (the fastest: 6,434 ms) 8,590 ms 6,723 ms 8,664 ms 6,962 ms

What went: the grouped counts - GROUP BY c20k under a 20-value WHERE ran 1,069 ms in the node against 148 in a hash aggregate, priced at 6,600 against 73,000, and count(DISTINCT c20k) under a 200-value one 342 ms against 28 - which the per-entry charge prices; the four-set AND counted by the node (7.6 ms, costed 90) against a btree index-only scan (2.1 ms); a three-set AND the node answers in 0.03 ms that went to a BitmapAnd (1.6 ms); the four-set fetch that took the lion plain scan (82 ms) and now takes the btree’s bitmap (52 ms). Two FK join choices between the node over a plain and over a bitmap scan of its dimension (fk fwd ks dim10k, fk fwd nofilter dim10k, 18 and 22% slower) are new, and small.

What is left, the worst first:

  • The smallest dimension set over the 2M-key fk (fk fwd tsq dim2 1.2k): the node 36.9 ms (costed 13,700) against a nested loop’s 11.8 (23,800). Core prices its nested loop over warm indexes at 2,000 units a millisecond; it was §27’s one wrong choice too, and at rpc 1.1 the nested loop is chosen.
  • Plain against bitmap lion scans of 20 to 50% of a table (fetch w.country2, w.val2 supp, s.c20): the lion plain scan is 17 to 31% faster than the bitmap heap scan both builds pick, by the 0.5 us a page §29.11 measured and does not credit; at rpc 1.1 both builds take the btree’s plain scan instead, 49 to 87% slower than lion’s (w.country too). Not a regression; the plain-scan costing’s to take.
  • A 1,000-value IN list over 1M keys (eq s.c1m in 1000): the node 5.6 ms (costed 2,019) against a btree index-only scan’s 1.8 (4,294; chosen at rpc 1.1). The executor walks at most LION_LOOKUP_WALK_MAX (8) entries before it searches again, and over 1,000 sparse keys it searches for nearly every one; the model charges a descent a value and still comes out under the btree. Modelling the walk limit broke the contiguous lists that do walk, and was taken out.
  • A one-day range of dates at random beside a 20-value status (fetch r.dr 1d st): a BitmapAnd of both lion sets (5.2 ms, costed 2,019) against the plain scan of the day (1.1 ms, 7,794), whose 2,500 scattered heap pages core’s convention prices as random reads though they are warm, while the BitmapAnd’s 250,000-TID status set is cheap. Lion keeps that convention (§10).
  • range t.c7 pc >= 1000: the node 30.5 ms against a plain lion scan’s 20.2, now costed 24,600 (was 2,700): one of the provisional range sums (§28).
  • At rpc 1.1 only: GROUP BY c20k over the whole table goes to a group aggregate over the btree’s index-only scan (328 ms, costed 105,000) over the node (156 ms, 117,100), and 100- and 1,000-value IN lists over c20k to the index-only scan (1.2 and 11.5 ms) over the node (0.74 and 7.7), priced within 8% of each other: core’s index-only scans fall with random_page_cost, lion’s CPU does not.
  • group s.c20k w c20: a hash aggregate over a bitmap scan (179 ms) against one over a plain scan (148) - the same plain-against-bitmap credit, under core’s aggregate.

The hash aggregate stays priced below every lion path (157 to 309 units a millisecond, §10’s “The reference”), so a grouped count that competes with one on close to equal terms can lose in the model and win on the machine: inlist.sql’s two GROUP BY counts over IN lists, which the node runs 1.7 to 2.2 times faster, force the node rather than pin the aggregate.

Provisional: the range sums (§28) and the plain scan’s heap side (§29.11), whose costing other work is reworking; the matrix’s range and fetch rows are theirs to rerun. Parallel plans were not in the CPU-time matrix.

The settings (2026-09-28)

Every per-operation price of the model is a planner setting, so that it can be calibrated on a workload of one’s own by changing settings, as core’s model is with random_page_cost, rather than by rebuilding. pg_lion.<name>_cost is the multiplier of LION_<NAME>_COST - pg_lion.container_cost of LION_CONTAINER_COST, which is lion_container_cost * cpu_operator_cost - in the unit the constant was always written in: cpu_operator_cost or cpu_tuple_cost, and for the FK-side join’s sort past work_mem seq_page_cost and random_page_cost, one setting each for the two parts of its page (LION_FKJOIN_SORT_PAGE_COST). So lion’s prices still move with core’s CPU and page costs, and a calibration made at one cpu_operator_cost holds at another. Each defaults to the value it was fitted or derived at - this section’s tables and the comment above each macro are why - so a server that sets none plans exactly as it did; the regression suite runs at the defaults, and costgucs.sql checks that each setting is there and that four of them move the plans they price. lion_costs.c holds the table of them and registers them from _PG_init, user settings from 0 up as core’s cost settings are, and shown by EXPLAIN (SETTINGS) when changed; pg_settings lists them among the customized options, as it does any extension’s, since no API puts a custom setting in one of core’s groups. The README lists them with their defaults and units.

Two constants that were written as another are decided by what they price. LION_FKJOIN_SET_COST stays LION_UNION_SET_COST and has no setting of its own: both are a union source’s set rebuilt by a count, in the same code (lion_count_sources_cached()), whether the counts are a GROUP BY’s or an FK-side join’s. LION_FKJOIN_BATCH_ROW_COST was LION_DESCENT_COST by choice, not by identity - a row’s place in a batch priced as a page visit, on purpose on the high side (§27, “Lookups in key order”) - and has its own setting, pg_lion.fkjoin_batch_row_cost, at the same 120. (Since the refit of 2026-09-29 (§27, “The per-key terms, refitted”) the FK-side join’s counts no longer build a union source again - they probe it (§29.11) - and LION_FKJOIN_SET_COST is a set’s cursor set up and sought, with a setting of its own, pg_lion.fkjoin_set_cost, at 11 cpu_tuple_cost; fkjoin_batch_row_cost is 75, beside a new fkjoin_lookup_cost; and fkjoin_probe_page_cost prices the posting pages a probe reads, which fkjoin_probe_cost had in it.) Two inline numbers became constants as well: the collecting merge’s driver containers (LION_FKJOIN_COLLECT_CONTAINER_COST, two cpu_operator_cost) and the sort’s page mix (0.75 seq_page_cost and 0.25 random_page_cost, cost_tuplesort()’s). A count of operations times a core cost - one comparison a container, a row handed up at cpu_tuple_cost - is core’s own price and stays one; so do the constants that are not prices: sizes (LION_BLOCKS_PER_CONTAINER), caps (LION_MEMBER_CAP), the intersection probe’s bounds, fanouts and selectivity heuristics.

Calibrating. On the build and cache state the server runs in - §10’s release build, warm, is what the defaults are fitted to:

  • For a query where a lion path competes with another - the count against core’s aggregate, a lion plain scan against a bitmap one, the FK-side join against a hash join - force each alternative (pg_lion.enable_count_pushdown, pg_lion.enable_plain_scan, core’s enable_* settings) and set each plan’s EXPLAIN total cost beside its EXPLAIN ANALYZE time. Their ratios, units a millisecond, are what the model says each plan does per unit of time; a lion path well above the competitor’s ratio is overpriced, and well below it underpriced, which is the mispick waiting to happen. Core’s own CPU-bound scans ran at 400 to 700 (§10, “The units”). (A lion path’s cost is its own price over pg_lion.pushdown_margin, with §39’s cold-read and Memoize corrections where they apply: set the margin to 1 to read lion’s own.)
  • Which setting to move is the one whose term dominates the path’s price, read against the node’s counters: Containers Visited, Directory Pages Read and Heap TIDs Rechecked of a count are the containers, the descents and the recheck candidates container_cost, descent_cost and recheck_tid_cost price; Distinct Keys Tested the tests of distinct_test_cost. SET it, plan again, and compare again: the price moves and the answer does not. Once the ratios agree, the value goes into postgresql.conf or ALTER DATABASE ... SET, as a tuned random_page_cost would.
  • For the FK-side join, §27’s “Where a key’s time goes” gives the per-key terms directly: Join Lookup Time over Join Keys Looked Up, with Directory Pages Read over the same keys, is a lookup’s price in pages of descent_cost (and fkjoin_lookup_cost and fkjoin_batch_row_cost a key in key order); Join Count Time over the keys found is a count - fkjoin_count_cost or fkjoin_copy_count_cost, and Join Key Containers Read over the keys found times container_cost and member_cost, with fkjoin_copy_probe_cost for each of those containers against a collected copy (Fact Filter Copy Seeks), or probed fkjoin_probe_cost for each container of the filters sought (Containers Visited less the key’s own), fkjoin_probe_page_cost for each of Join Posting Pages Read and fkjoin_set_cost for each set of a union filter; and Fact Filter Collect Time is the once-a-run collection (fkjoin_collect_container_cost, fkjoin_copy_container_cost). A phase’s time per key over the operations a key makes of it is the time of one, and at R units a millisecond a microsecond is R/1000 units: at 500, a lookup of 0.6 us a directory page is 0.3 units a page, which at a cpu_operator_cost of 0.0025 is descent_cost 120. The child’s rows and pages are core’s plan, priced by core.

32. Summary posting sets for ranges (format version 7)

§28 made a range a contiguous run of VALUE entries, and a count over it a walk of that run: one count per entry, 0.3 to 0.5 us for a small one counted with the rest of its leaf and 1.4 to 1.9 us for one counted on its own. That is linear in the KEYS the range covers, and a range over a unique timestamp covers as many keys as rows: a million keys was 350 ms on the assert build, where the rows' posting sets would have been a few thousand containers. §23 listed the structural fix - coarse posting sets per key bucket, so that a range is the union of whole buckets plus exact work at its two edges - as a format change whose maintenance on insert and VACUUM had to be designed first. This is that design, and it is implemented.

What a summary is

A summarized key column keeps, after its VALUE entries, one SUMMARY entry per BUCKET of consecutive keys. Bucket i holds the keys in (K_{i-1}, K_i] and its summary is keyed K_i, its inclusive upper bound: an entry (attno, kind SUMMARY, K_i, LION_SUMMARY_HASH) whose posting set is the union of the posting sets of every key in the bucket. The last bucket is OPEN: its entry carries both LION_ENTRY_SUMMARY and LION_ENTRY_SUMLAST, has kind LION_KIND_SUMLAST (5, above SUMMARY’s 4), sorts after every other summary of the column by its kind alone, and is keyed by the largest key it holds, which inserts raise. A bucket is a set of rows, never a range of heap blocks: a key’s rows are all in one bucket, and buckets close at key boundaries. No PIVOT (§21) is ever of the SUMLAST kind: one made from the open bucket’s entry is a SUMMARY pivot with the key it has then (lion_pivot_kindflags(); “The open bucket and the directory order” below).

A summary is an ORDINARY directory entry in everything but its kind and its hash: INLINE while its payload fits (§13’s items, sparse segments and all) and a posting tree of its own (§22) past that, written by the same builders, the same lion_dir_place() / lion_dir_add_entry() and the same posting-tree code as a key’s set, and split, moved right, vacuumed and deleted (§18) like one. Its hash is LION_SUMMARY_HASH (0) whatever its key: summaries are ordered by key alone (a column’s bucket bounds are distinct under proc 4, and the open one sorts by its kind), and the hash is what the owner stamp of every page of a chained summary says (§18), which must not change when the open bucket’s key is raised - with the key’s hash there, the first raise of a chained open bucket left its pages claiming another entry (found by the insert benchmark below, before this was released).

That is the design’s main choice: no TID ever moves between summaries, and summaries use no WAL record of their own, so §25’s records, their redo, wal_consistency_checking’s masking and the recovery harness cover them as they stand. lion_cmp_entry() treats the SUMLAST kind as keyless for ordering (it compares equal to any search key of its kind), lion_cmp_prefix() compares keys for VALUE and SUMMARY kinds (LION_KIND_HAS_KEY), and every walk that returns keys - the bitmap scan’s LionLeafWalk, the entry scans of GROUP BY and count(DISTINCT) - stops at the first summary exactly as it stops at the next column’s first entry, so nothing that reads keys ever sees one.

Buckets are decoupled from directory leaves. A leaf split moves entries, summaries among them, as for any entry; it never touches a bucket’s boundaries, because the boundaries are keys. The alternative - a summary per leaf, rebuilt at each split - would have made every leaf split rewrite two posting sets, WAL-log them, and pull the summaries into §21’s split protocol; with buckets keyed by value, a leaf split is exactly what it was.

Which columns, and the options

Only an ORDERED scalar key column can have summaries: a bucket is a run of keys, which needs the column’s directory in its order (§21, “The order is the index’s”), and a multi-key column’s entries are extracted keys whose sets overlap. Two reloptions, per index:

  • summaries = off (the default), on, or auto. on gives every ordered scalar key column summaries, even with no rows yet (its first insert opens its first bucket). auto decides per column at build time from the column’s exact counts: it keeps them when the column has at least LION_SUMMARY_AUTO_MIN_BUCKETS (4) buckets' worth of rows and its buckets average at least LION_SUMMARY_AUTO_MIN_KEYS (16) keys - a column of few large keys gains nothing, since its range walk is already short.
  • summary_tids = the rows a bucket closes at (4096; 16 to 224). A bucket closes at the first key boundary at or past it, so one key larger than that is a bucket of its own.

Chosen conservatively, flagged for the owner: off by default, so nothing changes for an existing schema and no insert pays for summaries it did not ask for; auto is the documented alternative a user opts into, and a later release could make it the default once the insert overhead below is judged acceptable. The option is per INDEX, not per column (a multicolumn index with on summarizes every ordered column, low-cardinality ones included - auto is the way to be selective); per-column control would need a column-level reloption the AM interface does not have, or an opclass parameter. auto on an EMPTY table decides no summaries (there is nothing to decide from); such an index gets them at its next REINDEX, or is built with on.

The meta page records which columns have summaries - summary_cols, a bitmap of key columns, and summary_tids - in the two words that were reserved[2], so the page keeps its 56 bytes. An index with at least one summarized column is format 7; one with none is format 6, byte for byte what it was, and lion_read_meta() reads both (and refuses a 6 with summary words set, a 7 without, or a bucket size out of range). The option is read at BUILD time only: ALTER INDEX ... SET (summaries = on) changes what the next REINDEX builds, never an index’s existing entries, and an insert gives summaries only to the columns the meta page names - a column the build left without them stays without them.

Upgrade. Existing indexes are format 6 and keep working unchanged, walking keys as §28 does; REINDEX with the option adds summaries. Downgrade: a build before this one refuses a format 7 index (its version check), so an index with summaries has to be rebuilt with summaries = off before going back. (Since the 2026-09-27 review the hint with which a build refuses a version it does not know says which way the version differs - an index too old for the build is REINDEXed, and one a NEWER build wrote is read by that build or rebuilt by this one - where every unknown version used to be told that it “predates this build”. lion_read_meta() also checks the size of the meta page’s special area before it reads the page flags there: the buffer manager accepts any pd_special up to BLCKSZ, and a damaged one had the flags read from past the page on every open of the index.)

Build (lion_build.c)

The build writes each key column’s entries in key order in one pass over the spool’s merged output (§21), serially or after a parallel build’s workers have sorted their runs - the pass is the leader’s either way, so parallel builds get summaries too. For a summarized column (lion_sum_begin()), every code the pass hands a VALUE entry’s builder is also added to the current bucket, and the bucket closes at the first key boundary at which it holds summary_tids codes or more. A bucket’s codes are collected in an array, which switches to an INT8 tuplesort of an eighth of maintenance_work_mem past an eighth (one key of a column with few keys can be most of the table), freeing the array; the spool leaves the summaries that quarter of its budget (§24 “Build”). A closed bucket is sorted, run through a collecting builder into the same items a posting set is made of, and put aside in a temporary BufFile after a header naming its key; the array’s sort (a sort_template.h sort that checks for interrupts as it partitions, not qsort()) and both loops over the codes answer a cancel, the loops every 64K codes as the pass’s own loop does (lion_sum_close_bucket(), since the 2026-10-18 review: a bucket of millions of codes had no check at all). A full bucket is closed when the NEXT key arrives (lion_sum_key_begin()), not when it fills, so that when the column’s values are done (lion_sum_finish()) there is always a last bucket, and it is closed as the SUMLAST whatever its size - a column with rows always has its open bucket (one that filled exactly at the last key used to be closed as a regular summary, and the column had none); auto decides, and the file is replayed through ordinary builders into entries written after the column’s values - which is where they sort. The build’s DEBUG1 line reports the buckets per column and whether they were kept.

Insert (lion_insert.c)

lioninsert() inserts the key as it always did (lion_insert_one()), and then, for a summarized column and a non-NULL key, puts the row into its bucket’s summary (lion_summary_insert()), with nothing held in between:

  1. Descend to (SUMMARY, key, hash 0) for write (lion_dir_search_first(), which finishes an INCOMPLETE_SPLIT on the way, steps right past a leaf’s end, and steps over any item that sorts before the search key - which a sound directory never lands on, below): the landing is the first summary whose key is at or above the row’s.
  2. A regular SUMMARY there is the row’s bucket: the TID goes into its set, INLINE or chain, exactly as into a key’s (lion_insert_inline() / lion_insert_chain()).
  3. The SUMLAST with a key at or above the row’s: the same.
  4. The SUMLAST with a key BELOW the row’s - the row is above every key the column’s summaries cover. If the open bucket holds fewer than summary_tids rows its key is RAISED to the row’s and the row goes in, in one step under the lock the lookup took (lion_summary_rekey_insert()): an INLINE entry is rewritten with the new key and the row in one record, a CHAIN one gets its key in one record and the row through the chain insert right after (a payload the longer key and the row push past the limit is spilled first). If it is full it is CLOSED (lion_summary_close_last(): the SUMLAST flag dropped in place, which leaves a regular summary keyed by the largest key it holds) and a new SUMLAST holding just the row is added right after it.
  5. No summary at all: a new SUMLAST holding the row, added where the column’s summaries sort. That is a column that never had a row (on over an empty table), or one a crash left between the two records of a close; VACUUM no longer deletes an emptied open bucket (below). The new entry’s key is above every summary of the column - the lookup found none at or above it - and the summary in front of the place it goes is compared with it as well when it is on the same leaf; one that is not below it sends the insert back to step 1, and a second time is reported as corruption.

So middle buckets never split: inserts into the middle of the key space make their buckets grow, and only the open bucket at the top is closed and succeeded. A column whose keys arrive in order (a timestamp, a sequence) gets buckets of summary_tids rows forever; one whose keys arrive in no order keeps the buckets its build made and grows them - REINDEX rebalances. Flagged: a split of a middle bucket would move rows between two summaries, which the reader’s disjointness argument below forbids without a protocol of its own (a two-entry atomic replace under one WAL record); it was left out, and the backlog has it.

The key goes in FIRST. A crash or an error between the two leaves the row under its key and in no summary; its transaction did not commit, so the row is dead, and a dead row missing from a summary is no difference to a count or to lion_index_verify() (below). The reverse order would have left a live-looking key missing a row a summary has, for the same dead row - equally harmless - but the key-first order is what the reader’s argument below uses for the open bucket.

Cost. One more descent and one more posting-set update per summarized column per inserted row, into a set that is larger and hotter than a key’s: the summary of the open bucket takes every appended row of a timestamp column. Measured below - before step 4 was one step: an append that raised the open bucket’s key paid a record for the key, a second descent and a second record for the row (2026-09-28).

VACUUM (lion_vacuum.c)

Summaries are entries, so the bulk-delete pass removes dead TIDs from them as from a key’s set, in the same chain order and under the same cleanup locks (§11), and an emptied summary is deleted as an emptied entry is (§18) - but never the open one, which stays, empty, with its key: still an upper bound of every key it can hold, and the next row above the closed buckets goes into it (step 3). Since its entry stays, a CHAIN open bucket loses its empty leaves as a key’s set that keeps rows does (§22, “Empty leaves”). A CHAIN summary is identified across a concurrent split by its root block (lion_vac_entry_matches()), and found again after one by descending to (SUMMARY, key) and walking right through the column’s summaries (lion_vac_ref_relocate()), as a key’s entry is found by its key.

Readers: the phased walk (lion_range.c)

A summed walk over a summarized column (lion_entry_scan_begin_sum(): the sum of §28, its complement’s BELOW and ABOVE, the sum over all of §14, and the collection of a range source below) walks one part of the column in up to three PHASES:

  1. LOWER - the VALUE entries from the part’s start up to and including K_j, the key of the first summary at or above the part’s lower bound (found by the same descent the part starts with, with the search key’s kind SUMMARY). That summary’s bucket may straddle the bound, so its keys are walked one by one.
  2. SUMS - the summaries after it, as long as each one’s bucket lies wholly inside the part: its key passes every upper bound (lion_scan_bucket_inside()). The last one taken, K_m, is remembered.
  3. UPPER - the VALUE entries above K_m (a descent to it), up to the part’s end.

A part with no lower bound (BELOW, the sum over all, k < v) starts with SUMS, its first bucket being whole. When the first summary at or above the part’s start lies past the part’s end - the range is inside one bucket - the walk is the plain walk of §28 (lion_entry_scan_plan_sum() decides before it starts); when no summary after it is whole, SUMS takes none and UPPER goes on from K_j. Each phase after the first starts with a descent of its own, resumed by key as every walk is (§28, “One read per leaf”), so a split or an entry deletion between phases is what it is between leaves.

Why the counts stay exact. The three phases partition the rows of the part: a key k <= K_j is walked in LOWER, a key in (K_j, K_m] is in exactly one whole bucket and counted through its summary, a key above K_m is walked in UPPER - disjoint sets of one column, so the sum, the complement and the collection take them as they take entries (§15’s argument). Under concurrent writers, with an MVCC snapshot deciding every row:

  • A row’s bucket never changes once the row is in it. Middle buckets' bounds never move; the open bucket’s upper bound only rises; a closed bucket keeps its key. So a row counted through a summary is in no other summary, and its key is outside LOWER and UPPER’s key intervals.
  • The open bucket’s key is raised by an insert AFTER the row is under its own key (key first). A reader that took the open bucket as whole with its old key K_s walks UPPER from K_s, and finds the new key’s entry there - and the summary it read did not have the row yet. A reader that saw the raised key counts the row through the summary, and UPPER starts above it. Either way once.
  • A summary VACUUM empties and deletes had only dead rows. After it is gone the next summary’s bucket reaches down to the previous one’s key; rows inserted into that span afterwards go into it, and the dead rows that were in the deleted one are counted by no one, which is right. The open bucket is never deleted; emptied, it is a summary of no rows, which the walk skips.
  • A row inserted after the snapshot may be anywhere; it is invisible, and the visibility map and the heap recheck decide it as they decide every candidate (§9).

The §9 interlock is the ordinary one: a summary’s containers are read under a pin on the page they came from, like a key’s. (A sum beside a clause with far fewer rows than the walk hands out probes the rest of its walk at that clause’s rows instead of counting set by set, and a range collected as a source grows its union in place: “Summed ranges: dense and probed” below, whose §9 argument is a collected set’s.)

The landing that finds E_j (lion_range_landing()) steps over an item that sorts before its search key, as the insert’s lookup does: on a sound directory it never lands on one, and on an index the bug below damaged, taking a summary whose key is below the bound as E_j counted whole buckets below the range.

Where summaries are not used. Anything that needs keys: a GROUP BY k walk (its groups ARE the keys), count(DISTINCT k), the ordered scans of §30 (they return TIDs in key order), k IS NULL, and the walks of a whole column (k IS NOT NULL in a bitmap or plain index scan). The bitmap and plain index scans of a RANGE do use them since 2026-09-28 - a summary’s containers are TIDs like any, and a scan hands every one of them to the executor: they take the phased walk above without its sets and pins (lion_entry_scan_begin_summed(), lion_entry_scan_next_copy(); §28, “Bitmap scans”), except a plain scan that keeps its pins (§29.5).

The open bucket and the directory order (2026-09-28)

The first release of this design broke the directory order the first time an open bucket that had been split off on a leaf of its own was closed (the 2026-09-28 review; found by three reviewers). A pivot - a leaf’s high key and the downlink to its right sibling (§21) - copied the kind of the item it was made from, SUMLAST included. When a bucket closed and the new SUMLAST did not fit on the rightmost leaf, the append split (k = nitems - 1) put it alone on the right page, so the left page’s high key and the parent’s downlink were SUMLAST pivots - which sort after every SUMMARY of the column by their kind. Closing that bucket later drops the flag IN PLACE, so the page’s first item became a SUMMARY that sorted BELOW the separator routing to it: the left page’s high key above the right page’s first key. From then on a descent to (SUMMARY, k) for any k routed to the left page, stepped right, and landed on that closed summary; the insert took it without comparing keys and filed every later row of the column into that one closed bucket, and range counts summed it where it did not belong - the review traced count(*) WHERE id <= 150000 over 200,000 appended rows to 200,000 (traced from the code; no server could run there). It needs the close to find the leaf with room for the closed entry but not for the new one, which the review put at about one append split in four, so any appended summarized column reaches it. A second path led to the same state: VACUUM deleting the emptied top buckets, the open one included, left SUMMARY pivots above the key of the SUMLAST the next insert then created (step 5) below them.

What changed:

  • A pivot is never of the SUMLAST kind (lion_pivot_kindflags(), in lion_make_pivot() and lion_build_pivot(); every other pivot is made from one of those or from a high key): one made from the open bucket’s entry is a SUMMARY pivot with its key at the time. That is a valid lower bound for the entry whatever becomes of it, because its key only rises and closing keeps it, and it is above every other summary of the column, whose keys are below the open one’s.
  • The open bucket is never deleted by VACUUM, however empty, and the build always leaves one (a bucket that fills at the column’s last key is its SUMLAST): the key of a SUMLAST only ever rises, so no pivot of the column is ever above it. Step 5 is then reached only by a column that never had a row, or after a crash between the two records of a close - and then the new SUMLAST is above every summary the lookup can find, and the summary in front of where it goes is compared with it too (lion_summary_insert()).
  • Nothing takes a landing on trust: lion_dir_search_first() (the insert’s lookup, and verify’s) and lion_range_landing() (E_j) step over an item that sorts before the search key. An index the old release damaged stops getting worse - later rows go to their own buckets - and stops miscounting beyond the rows already misfiled.
  • lion_index_verify() reports a SUMLAST pivot as corruption with the hint to REINDEX, before anything reads past it: the index is damaged, or will be when that bucket closes. Every order violation of the directory now carries the same hint.

An index built by the first release needs REINDEX if verify says so; one it built or wrote with no append split at a bucket close is sound as it stands. test/sql/summary_append.sql appends rows one statement at a time across several leaves at summary_tids = 16 and checks verify and range counts at and around bucket bounds, and deletes the top buckets, vacuums, inserts below the old top and appends again.

Two bounds on one side (2026-09-27)

§28’s walks descended to the FIRST lower bound (INSIDE) and, with more than one upper bound, started ABOVE where INSIDE starts: k < 10 AND k <= $1 did not know which bound is tighter. Now each side takes its TIGHTEST bound (lion_range_side_leaf()): it descends once per bound of the side and keeps the landing that is further in - the rightmost of the lower bounds' landings for INSIDE, the leftmost of the upper bounds' for ABOVE - comparing the landed items by the directory’s own order (lion_cmp_entries()). The descents are one per bound (two, in practice), and the walk then starts where it should: test/sql/rangesum.sql’s k > 2 AND k >= 4 AND k < 996 AND k <= 993 is now taken as its complement, 10 entries against 990 walked before.

A range as a source (lion_plan_count.c, lion_rangesrc.c)

§28 bounded the DRIVING walk with a range and declined a range anywhere else. With summaries a range on any column is cheap to turn into the set of its rows, so the count pushdown now takes a range on a column that does not drive the count as an ordinary positive SOURCE (LION_CLAUSE_RANGESRC):

  • g, count(*) ... WHERE <range on k> GROUP BY g, beside a two-column GROUP BY, beside the decoded walk of three or more (§34, “A range in the WHERE”), and beside a count(DISTINCT) walk of another column;
  • two range columns without a GROUP BY: the WIDEST range drives the sum (the complement and the summaries make a wide walk short) and the others are sources;
  • a range and an equality on one column (k > 100 AND k = 257): two sources;
  • a range as an OR’s arm, or part of one ((k < 10 OR a = 3), (k BETWEEN 5 AND 9 AND a = 1) OR ...): one leaf of the union per arm, whatever bounds the arm has on the column;
  • a range among the fact filters of an FK-side join (§27).

How it is evaluated. Per relation (and per rescan, Params being values then), the bounds of each range on its column - in its conjunction: the top level, or one OR arm - are made one LionRange, and the rows of the range are COLLECTED (lion_range_collect()): the phased walk above hands out entries and summaries, and each set’s containers are ORed into a union kept by container key, then put in key order into one private, pinless set. It is read like any other set from then on - by every group, pair or dimension row the node counts - and it is exact: the same argument as the collected fact filters of §27 (it is only ever counted beside a located set that holds the §9 pin, which settles any TID VACUUM has since removed; it cannot miss a row visible to the snapshot, because it is read after the snapshot was taken).

(2026-10-27, §53: a range on a sliced column may be collected from its slices instead - O'Neil’s descent at each container key of the column’s existence slice E, the same work at 0.1% of the rows as at 90% - into a copy built and spilled as lion_sources_collect()’s is, and ANDed with E located with its pin, so that it needs nothing beside it to carry the §9 interlock. The planner prices it against this collection, the walk and the summaries, in a build of its own.)

Too large to collect. The collections of one relation share one hash table’s memory (get_hash_memory_limit()): a union is at most a container per container key of the heap, about two bytes a row until those fill up. The planner budgets them as the executor holds them, one after the other out of that one memory (lion_cost_range_source()’s LionRangeMem, the node’s rangesrc_held): a range that would not fit what the ranges before it left is collected in WINDOWS into a temporary file (below), and priced so - the walk of the range once, the passes of what the windows do not keep through their overflow, and the file written once and read once - and every count that reads the file back pays for each container it seeks there (LION_SPILL_READ_COST, lion_count_rel_spill_reads()). The file is a collected set like any other, so every way of counting one takes it: the WHERE collected once and the groups counted in batches (§10), the FK-side join’s copy of its fact filters (§27), a sum probed at F’s rows.

A range that does not fit what the ranges collected in memory before it left takes that memory back first (lion_locate_range()): the executor finds it does not fit by trying, those ranges go to files of their own (lion_range_spill_set(), each written once in key order, to hold sixteen bytes a container key as a range collected in windows does), and it is collected in what that freed - in memory, or in windows of nearly all of it. Left to what they left, a range that filled the memory to its last few kilobytes would leave the next one windows of LION_RANGE_WINDOW_MIN, and a pass through its overflow for every halving of its union down to that. The planner prices it so: the try, the files written and read, and every count’s reads of them. An IN list too long to locate at once is collected in the same memory and the same way (§15, “A list too long to locate at once”; lion_collect_within()), evicting and evicted alike; the planner, which cannot see a parameter’s length, prices none of it.

Walked at every count. A range may instead be WALKED at every count it is part of (LionCountSource.rangewalk, lion_node_count()), where the plan says so (LION_FLAG_RANGEWALK): the rows of the range are the disjoint union of the sets its walk hands out, so the count is the SUM over them of the count of each - a leaf’s worth of small entries at a time, as §28 counts - ANDed with the other sources, or, when the leaf-by-leaf race says the outside is shorter, the count of the other sources minus the column’s NULL entry less the sums below and above (§28’s complement). An existence test walks the inside and stops at the first piece with a row. Every count and existence test the node makes goes through lion_node_count(), which expands a walked range and hands everything else to the counting core unchanged - so a multi-key clause answered only as a superset (a phrase, a NULL element) is rechecked there as it always is.

That is a walk of the range per count, which is no plan for many counts, and the planner takes it as a choice, priced (lion_count_rel_walk_save()): the walk at every count the relation makes - its groups, pairs or tests, or the sets of the range it sums - with the count’s own set and the other sources for F (lion_cost_range_sum() with F given: an existence test’s share of the inside, a second walked range at every set the first hands out), against the collection, in memory or in windows, the reads of the files it and the ranges it sent to files would be read from, and the counts' merges, which a walk makes itself. Any range outside an OR may be walked, one that fits as well as one that does not. A summed range beside one is priced at the side it takes beside the range collected - its inside, as often as not, which is what walking costs at most. The plan takes one WAY of walking its ranges (lion_walk_save_choose()): all of them - a second at every set the walk of the first hands out - or one alone with the others collected, the way that saves the most over the relations and runs it counts, and only where the collection is priced at least LION_RANGE_WALK_MARGIN (1.5) times the walks, since a walk’s price is the count’s estimate of its groups and tests times one walk, and a collection’s is one walk bounded by the memory. The choice is the node’s, and names the ranges it priced walked: the first bound of each is LION_CLAUSE_WALK in the plan’s clause kinds, and the executor walks those from the start (lion_locate_range()), collecting nothing of them. Every other range of the node is collected, in windows if its union outgrows the memory: a range nobody priced walked is never walked at every set of another’s walk. Never beside the decoded walk (§34) or in an FK-side join (§27), whose WHERE is one collected copy and whose counts are the dimension rows, nor for an OR’s leaf, which cannot be walked.

Why from the start, and why at any size: a range collected first and walked only once the memory ran out would cost, where it fit, its collection whatever walking it would have cost, and where it did not, the walk of as much of it as filled the memory before it was walked - so the walk’s price, and the walk, would grow with work_mem. That way a count(DISTINCT)’s existence tests over a range of 3M rows took 5 ms walked at 64 kB, 34 ms at 256 kB and 0.45 s collected from 768 kB, and the existence tests of the regression suite’s shapes 15 to 130 times what walking takes. Measured on the release build over a 20M-row table and the suite’s tables: a count(DISTINCT)’s existence tests and a few groups walk, 1.5 to 60 times faster than collecting; 300 groups and more collect; the margin keeps two ranges beside seven groups collected, 11 to 13 ms, where walking one took 16 (the collection priced at 1.2 times the walk), and the ways leave a range too large to collect walked beside a count(DISTINCT) with the other collected, 3 ms, where collecting both in windows took 9.

Measured on the assert-enabled build (2026-10-19): 3M rows on 25,000 pages, r a hash of the row number with summaries, r < 5592405 a third of them, 1.6 MB as its union builds it, at work_mem 256 kB (a 512 kB hash table) and with the plain plan as the reference for every answer. Seven groups count against the range in windows in 93 ms, where walking it at each takes 213; 500 groups in 159 ms, 8.1 s walked; count(DISTINCT h)’s 500 existence tests walk it, 18 ms, where the windows take 101; an OR’s leaf, (r < 5592405 OR x = 3) GROUP BY g, 94 ms, against core’s sequential scan at 726; among an FK-side join’s fact filters, 335 ms against 120 s walked at each of 20,000 dimension rows, and core’s hash join at 710. count(*) WHERE id >= 100 AND r BETWEEN 1000 AND 5592405 is collected in windows (108 ms) where the walk takes 29 to 54, the summed id then taking its complement: the price of pricing a summed range at its inside. Two ranges of a quarter of the rows each, r < 4194304 AND fk < 5000 GROUP BY g, each about 1.3 MB as its union comes out: where the first fits and the second does not fit what it left, the first goes to a file and the second is collected in what that freed - 76 ms at a 1 MB work_mem, 471 at 512 kB - where collecting the second in windows of what the first left took 462 and 1,430, and walking it at each group 740 and 640; at 256 kB, where neither fits, 560 ms, against 98 s walking both. Beside a walked range, 360,000 rows at 64 kB, r < 524288 AND s < 65536 GROUP BY g: s sized to fit, its union outgrowing the memory all the same, is collected in windows - 70 ms - where walking it at each of the 1,041 sets the walk of r hands out took 627.

What it needs beside it. A collected range holds no pin, so a count with nothing else that carries the §9 interlock would recheck every row in the heap. The planner therefore takes a range as a source only beside something that does: the walk that drives the count (a GROUP BY, the sum over another range, a count(DISTINCT) walk), the fk set of a join, or a clause - or an OR of clauses - outside the ranges. count(*) WHERE k < 10 OR a = 3 alone is left to the ordinary plan. An OR’s leaf cannot be walked (the union’s other arms would have to be subtracted from each piece), and is collected in windows past the memory as any other range is (below). What a collection holds is bounded (2026-09-28 review):

  • The union is the set. A collection that fits keeps its containers where the union built them, and the set’s containers[] points at them in key order; there is no second copy. The containers live in a context whose chunk limit is a kilobyte (ALLOCSET_SMALL_SIZES), so a bitset or a long array is a block of its own at its own size, where the default context rounded a 4104-byte bitset up to 8 kB; and what the union holds is counted as the allocator hands it out (GetMemoryChunkSpace()), where it used to be counted at the containers' own sizes.
  • The walk keeps nothing of a set it has added. Each set the walk hands out is read in a context of its own, reset once the union has taken it (lion_range_union_walk()): an INLINE set’s copy of its payload and key, and the stream, plan and cursors it is read through, are the set’s alone, and releasing a set gives back its pin, not its memory. Left in the node’s WHERE context they would stay there for the rest of the query, some 250 bytes for every entry walked, outside the union’s memory and whatever work_mem said: 154 MB over 600,000 keys at 4 MB; over a 20M-row table on the release build, r < 8388608 GROUP BY g (8.4 million keys) at 64 MB takes 26 MB of peak RssAnon and 3.8 s, where keeping them took 1.2 GB and 9.8 s; on the assert build a range over 950,000 of 3M rows takes 4 MB, where keeping them took 242 MB. The node’s other walks - a walked range a batch at a time (lion_walk_range_part()), a probed one’s sets, an IN list’s batches, the FK-side join’s chunks of groups, the inner GROUP BY column’s entries, the key aggregates' entries - already reset a context of their own as they went.

  • Past the memory, a range is collected in WINDOWS, into a temporary file (LionSpill, the same the FK-side join’s copy spills into, §27): an OR’s leaf, every range beside the decoded walk (§34), whose WHERE is one collected copy, and every range of a plan that does not walk them. It gets what every range of the relation gets - what is left of the hash table’s memory, and never less than LION_RANGE_WINDOW_MIN (32 kB, several bitsets, so that a window always gets on). The range is walked ONCE, into the first window. When a window’s memory runs out it gives back the upper half of the keys it holds and ends where they began (lion_range_union_evict()). What it gives back, and every container that comes after at a key past its end, goes to its OVERFLOW (LionUnionSpill): logical tapes, the keys from its first cut to its end split into LION_UNION_FANOUT parts of equal width - a sixteenth of the memory’s blocks, two to 64 - and what a later cut gives back below the first into one part more. The containers go there as the walk hands them out, a set’s at a key, not united; a bitset given back is folded and optimized first. At the end of its input the window’s containers go to the file in key order, and its parts that hold anything wait on a stack in key order, the lowest on top: each becomes a window of its own, which reads its part in place of the walk, keeps what it can and splits the rest the same way before the part above it is taken. So the windows come off the stack in key order, and the file is written in key order. Each part holds a block of buffer while its window is made, out of the window’s memory (LION_UNION_SPILL_BLOCKS), and what is read back is checked before it is used: a size a container can have, read whole, a kind whose header says that size, and a key within the part (lion_union_part_read()). The set reads the file a container at a time and seeks it by the keys it keeps in memory, sixteen bytes a container key. The one walk is read after the snapshot as it always was, and the windows are disjoint ranges of container keys, each container of the walk kept by the one window whose keys hold it, so the file holds every row of the range once and none twice - the collected set’s argument unchanged. A union n times a window’s memory costs one walk and about log n / log fanout passes over the walk’s containers, each written and read back for every window it is not kept by, which is the price the planner puts on it (lion_cost_range_source(), lion_union_overflow_share()). A walk that hands its keys out in order - a column in the heap’s order, or its reverse - makes one pass more: the first window does not know where its keys end, so its last part is open from about where it cut, and every key after that goes there - 10.77 MB of a 10.78 MB overflow over 3M rows at 64 kB, where a scattered range’s went 8.4, 4.5 and 4.4 MB into three - and only the window made of that part splits fanout ways. The planner priced that pass at none before the 2026-10-23 review, and interpolates it by the column’s correlation now, as it does the union’s size. Over small sets in no order - summaries of a few dozen rows, a container each at keys all over the heap - that is many times the union’s own containers, and the walk at every count may then be the cheaper. An IN list collected the same way locates its values once, into the first window (§15). EXPLAIN ANALYZE counts Range Sources Spilled.

    Measured on a PostgreSQL 18.6 build without assertions (2026-10-22), at work_mem 64 kB and hash_mem_multiplier 1, against windows that each walked the range again - or located the list again - for its keys: x, count(*) WHERE k = ANY (50,000 values) GROUP BY x over 3M rows, the list collected in windows, 934 ms and 500,737 buffer hits then, 75.5 ms and 16,901 now, against 54 ms in memory at 4 MB. Priced so, r < 524288 AND s < 65536 GROUP BY g over the 360,000 rows above collects both ranges in windows, 11 ms, where it walked r and collected s, 50; fk = ANY (1,500 values) AND r < 524288 GROUP BY g collects the range beside the list, 66 ms where it walked it, 124; and a range of 64-row summaries in no order over 600,000 rows, r BETWEEN 15000 AND 450000 AND s = 7 GROUP BY g, is still walked, 25.5 ms.

Flagged: a walked OR arm by inclusion-exclusion - |X ∩ ((R ∩ Z) ∪ Y)| = |X ∩ Y| + Σ_B |X ∩ B ∩ Z − Y| over R’s pieces B - would answer such a leaf without its union or its windows, which for few counts would be the cheaper.

EXPLAIN names each range with all of its bounds (lion_src_ur.u (u > 100 AND u <= 900)), and EXPLAIN ANALYZE counts Range Sources Collected and Range Sources Walked when there are any, and of the collected, Range Sources Spilled: each range once, however many partitions, rescans and participants of a parallel plan took it (§10, “A count in parallel”). An FK-side join’s collection is timed, with TIMING, in Fact Filter Locate Time (since 2026-09-29, “What collecting a range costs”).

Summed ranges: dense and probed (2026-09-28)

A column whose rows lie in no heap order - a timestamp on a table loaded in some other order, a key that is a permutation of the row number - gives every summary a member or two at nearly every container key the heap has: a bucket of summary_tids rows over a heap of more container keys than that is that many one-member containers, stored as the pairs of sparse segments (§13). The summaries still do what they are for - a range reads a set per whole bucket, not a set per key - but each of those sets now costs its ROWS, not its keys, and two places paid for them many times over:

  • A sum beside other clauses F. The sum counts each set of its walk ANDed with F (§28, “Counting a walk”), and a scattered summary is a large entry, so each is a count of its own: a merge of its thousands of tiny containers with F, the smaller side driving and the other sought at its keys, with a count’s set-up around it. Over T whole buckets of S rows, with F in F_c containers, that is some T × min(S, F_c) merge steps - a seek, a container AND and, for what survives, a visibility-map question each - however few rows F has, and F’s containers read again for every summary. A benchmark found such counts slower than the bitmap scan the planner (correctly) preferred.
  • A range collected as a source (“A range as a source” above). Each container of each set was ORed into the union with a lion_container_or(): a merge of the whole union while it was an ARRAY, and past 2048 members a 4 KB image filled, counted, optimized and copied back - per incoming container, which for such a column is per member.

The probed sum (lion_sum_walk(), and lion_walk_range_part() for a range source walked because it would not fit in memory). A summed walk over a summarized column that has come to its summaries, and whose counts have visited LION_PROBE_SWITCH (2) times as many containers as F’s smallest positive source can lie at, turns around for the rest of its sets (lion_sum_probe_due(); until 2026-10-19 it waited until it had handed out twice that source’s ROWS, “The price of a probe that does not fit” below):

  1. F is collected once (lion_sources_collect(): the intersection of its positive sources minus the negated ones, pinless, in at most half of work_mem).
  2. Each further set of the walk is read only at F’s container keys (lion_range_probe_add()): a cursor that holds no pin is sought from one of F’s keys to the next - §22’s seek, a comparison per pair inside a sparse segment, a descent across posting pages - and what the set holds at a key of F’s is MARKED against F’s members there: a bit per member of an ARRAY container of F (lion_container_mark_members()), a bitset image the set’s container is ORed into for any other. What the marks of all of F can take - the ARRAYs' bits allocated at once, an image per other container when something first lands there - is checked with the copy against work_mem before the walk goes on, so the probe is never given up halfway.
  3. When the walk ends the marked rows - F’s rows the range holds - are one set, and it is counted ONCE, ANDed with F (lion_range_probe_count()), through the ordinary merge with its visibility map, heap recheck and row filter.

What the rest of the walk costs is then a cursor per set stepped to F’s keys and the members it marks: no count, no merge and no set-up per set, and F’s containers read once instead of once per summary. The step count is still up to T × min(S, F_c), but a step is a comparison or two - a pair compared, a key found by a galloping search - where a merge step was a probe, an AND and a map question. The switch is bounded regret: the probe costs one count of F to collect it and one to finish, and is only begun once the walk’s own counts have visited twice F’s containers - about what those two counts visit - so it can at most double what the walk spent.

Why it is exact. The sets of a summed walk are disjoint and their union is exactly the rows of the part (“Readers” above), so the sum of their counts is the count of their union ANDed with F; the sets counted before the switch and those probed after it are disjoint too, so the two add up. The marked set holds every visible row of the probed sets' union that F holds - F’s copy is read after the snapshot was taken, and a visible row was in every index before its transaction committed (the collected set’s argument, lion_sources_collect()) - and nothing outside that union, since a member is marked only when a set of the walk holds it. ANDed with F again it is the count those sets' counts would have added up to.

§9, unchanged in what it asks. The probed sets are read without pins, as a collected range’s are, and the marks and F’s copy are copies. So the count of the marked set is made beside F AS THE COUNT READS IT - copies of F’s sets in which a CHAIN set an earlier count of the walk copied into memory (lion_posting_set_materialize()) is walked from its root again, under its cursor’s own pins, and none is copied - and the probe is only begun when a positive source of that view carries the interlock under the open budget (lion_source_pinned()). Otherwise the walk goes on counting set by set, where each set carries it. Every member the final count takes from the map is in the container of that source it holds a pin on, which is all §9 asks of a candidate (the argument on lion_posting_set_materialize(), and “A range as a source” above). The walk pins what it pinned before - a leaf while its INLINE copies are handed out, each INLINE set until it is released - and the probe’s cursors pin nothing. In recovery the marked set counts as a set of the range’s index (lion_sources_all_rmgr()), as each set did when it was counted on its own.

Not probed: a walk that reads no summary (a column without them, a range inside one bucket: §28 and its measurements stand), an existence test (it stops at the first set with a row), F with a range still to be walked, F with no positive source, or none that carries the interlock, and F whose smallest positive source would not fit work_mem as a probe - its copy within half, the copy, the marks and the count’s set within all of it (lion_range_probe_fits(), asked before anything is collected, and of the same source’s estimate by the planner). Nor the range ALONE: count(*) WHERE <scattered range> still counts each summary on its own, a map question per container under the summary’s own pin. A dense union with one question per container key would need, at each key, a pin on every page its members came from - every summary of a window at once, which the pin budget (§15) forbids.

Why not a source the merge seeks. The other way to let F drive - the summed range as one source of the merge, sought by container key across its summaries and edges - opens a cursor per set at once: thousands of posting sets, past the open budget (§15, “Bounded cursors”), so the merge would read it as a windowed union with no pins, rebuilt window by window and each summary sought at every key F drives it to: the same seeks, with the window’s images and its re-reads on top. The probe makes those seeks one set at a time, with one cursor open. It is the count’s form of what the bitmap scan does with a range beside other columns (§28, “Bitmap scans”, lion_emit_intersect()): the range’s sets - summaries and all, by the same phased walk - read only at the other side’s container keys, and what they hold there kept. The scan keeps it in the caller’s bitmap, a window of the other columns' containers at a time, and needs no pin; the count keeps F’s rows marked, all of F at once within work_mem, and counts them once beside F under its pins.

The collected union, dense (lion_range_union_cb()). A union keeps the form it grows in, as the OR node’s image does (§15, LION_OR_BITSET_MIN): once it is a BITSET, containers are ORed into it in place (lion_container_or_inplace()), and it is optimized once, when the walk is over (lion_range_union_finish()); while it is an ARRAY, the members of incoming ARRAYs of at most 32 members wait in a buffer and are folded in through one image (lion_container_add_many()) once half as many as the union holds have come. Since the union grows by at least half between two folds, each member is moved a bounded number of times; the buffer - at most half the union, counted in the collection’s memory as the union is - may put a collection over its memory a little sooner than before, which then walks or spills as any other. A RUN on either side is widened (below). The windowed union of §15 (lion_wide_fill()) was not reused: a window of container keys would walk the range once per window, where the union either fits the memory it is allowed or is spilled past it. It is also what a bitmap scan collects of the first of several range columns with nothing else beside them (§28, “Bitmap scans”), and grows the same way there.

Widened (2026-09-30). Anything else went through an image, as before: a RUN union, or a RUN or a BITSET coming to an ARRAY one - and a RUN is what a DENSE union optimizes to, a heap block’s rows being a run, and what every union of a column in heap order is. Each container that came to a RUN union was folded into it: an image filled, counted, optimized and copied back, a few microseconds however few members it brought, which over a range of scattered rows is per member. count(DISTINCT lo) WHERE hi BETWEEN 10 AND X over 2M rows, hi a permutation of 200,000 keys of ten rows, took 19 ms at X = 10,000, 1.26 s at 50,000 and 12.7 to 17 s at 190,000, 22 times core’s plan - and the model, which took a union to be ORed into in place once it had 2,048 members, priced it at a third of a second (2026-09-29 review). Now a union that would take a container through an image is WIDENED to a BITSET instead (lion_range_union_widen()) and ORed into in place from then on, and optimized with the others when the walk is over; ARRAY unions keep their buffer and their merges, which are bounded. A widened union takes a bitset’s 4 kB where its optimized form may take a few bytes - a RUN of a key in heap order - so the unions widened since the last time are COMPACTED, optimized back (lion_range_union_compact()), whenever that memory runs short: before a union is refused a widening, which then folds as before, and before the collection gives up or a window gives back keys, which therefore happens only where it did before. Each widening is made where a fold would have been and compacted at most once, so a collection short of memory spends about what it did - a scattered range at work_mem 64 kB widened and compacted 6,916 times in 495 compactions and took as long as it did - and one that is not spends a fold or two a union key and a few nanoseconds a container. (A union is widened once and compacted once at most (LionRangeUnionEnt.compacted). Were a compacted union widened again, a collection of scattered rows near its memory would compact to make room for one widening, and the next container would widen the union just compacted, which compacts the one before - a 4 kB optimize and a 4 kB widening a container, 12 s at 768 kB over 3M rows, 60 to 70% of it in the optimize. One that may not be widened takes the members of a small ARRAY into a buffer as an ARRAY union does, folded once they come to a quarter of its size in bytes, and before those buffers would put the collection past its memory they are folded and given back (lion_range_union_squeeze()), and such unions fold as each container comes until a window gives back keys. k < 950000 over 3M rows in no order takes 0.66 s at a 1 MB work_mem, no more than at 64 MB, where widening compacted unions again took 1.82 s, and in windows at 64 kB 1.11 s against 1.93, with the walk’s sets freed as it goes (above, “What it needs beside it”).) On an assert build the series went from 30 ms, 1.09 s and 9.1 s to 21, 101 and 455 ms, against core’s 176, 269 and 492; over 2M rows in heap order, one row a key, 1.9 million keys went from 14.1 s to 1.24, the walk of the keys most of it, and ten rows a key from 951 ms to 109. Measured over 40 collections - two orders, one row and ten a key, with and without summaries, 1% to 95% of the keys - none is slower beyond the noise, and all but six are 2 to 30 times faster. The model’s in-order folds are one fold or two per union key rather than one per container (LION_RANGE_WIDEN_FOLDS, “What collecting a range costs” below).

EXPLAIN ANALYZE prints Range Walks Probed when there were any: each probed walk, so a complement whose walks below and above both turned is two.

Future work, a format change: summaries of summaries. A range over a scattered column reads a set per whole bucket, and each is as many containers as the bucket has rows; a second level of buckets - a summary per run of summaries - would let a wide range read a set per run instead. It changes what VACUUM and inserts maintain and the readers' disjointness argument, and is not done.

The price of a probe that does not fit (2026-10-19)

A count of a range on a near-unique timestamp with summaries, beside an equality and a multi-key overlap, on a heap the timestamp lies in no order on. The count was priced above a plain scan of the same index feeding count(*) and was not chosen. Reproduced on a release build over 20M rows: 729,000 heap pages, 11,392 container keys. Each summary of 4,096 rows is at some 3,440 of them, one or two rows at each. The equality holds half the rows, and the overlap a million, all of them of the equality’s value. The planner, which takes the two as independent, estimates their intersection at 495,000.

What the count cost and what it was priced at did not match in five places:

  1. The memory test. The planner priced the walk probed whenever F’s rows at two bytes each fit work_mem, F being the estimated intersection: 495,000 rows, so probed at 4MB. The executor tested its smallest positive source at the same four bytes a row: the overlap’s 999,065 rows, 3.996 MB, just inside 4MB, so it tried. Then it collected F, gave up past half of work_mem (the copy is 2.23 MB: 88 members a container, 196 bytes each with its pointer and offset), and counted every summary: 930 ms, priced as if probed. At 64MB it probed.
  2. The switch. The executor turned to probing once the walk had handed out twice the smallest source’s rows. A scattered summary’s rows are its containers, so that was after more than half of the window’s 3.7M rows, about 500 summaries counted one by one: 530 ms, where probing from the first took 200.
  3. The counted summary. It was priced as every source of F probed in memory at every one of its container keys: 714 units a summary. A source of a million rows is never copied into memory (more than LION_MATERIALIZE_MAX_CONTAINERS containers and more than LION_MATERIALIZE_MAX_BYTES of members, §9), so it is sought on its posting pages. The merge seeks the second source only where the first left a row, here one key in seventeen. Measured: 272 ns a container of the summaries counted beside both sources, 167 beside the overlap alone, 273 beside the equality alone, and 26 for the walk alone.
  4. The probe’s step was a derived 2 cpu_operator_cost, 10 ns. Measured: 58 ns a container, the slope of probed walks over 1.5M to 5.0M summary containers (30, 90 and 180 days, work_mem 64MB).
  5. The race that picks the side (lion_range_choose_on()) was charged the pages of the smaller side twice, a whole bucket’s summary posting pages among them. It reads directory leaves only: a summary’s entry, not its posting set. That was 56,000 of the window’s 284,000 units.

The plain scan it was set against was priced at 159,000 units and runs in 340 ms warm. An earlier measurement of 6.4 s was the virtual machine’s page faults on a heap untouched for a while, and so was each first run after a restart, 18 s at 365 days. At 4MB the plain scan was therefore the right plan, because the count cannot probe there. At 64MB the count was the right plan, but only with a switch that probes from its first summaries.

What changed.

  • One memory test, asked of the same numbers. lion_range_probe_fits(rows, containers, maxbytes) (lion_rangesrc.c) is lion_range_probe_begin()’s test, taken before anything is collected. It requires the copy within half of maxbytes, as lion_collect_container() accounts it. It requires the copy, the marks and the count’s set within all of it, as lion_range_probe_begin() adds them up: an ARRAY’s members again with a bit each and 16 bytes, and two maximal containers for any other kind. Each container is taken at the average, as an ARRAY while that is no larger than a NARROW of width 1, and at the most a container takes past that. It is asked of F’s smallest positive source: its rows, and the most containers they can lie at, one a member and no more than the heap has keys (lion_containers_for(); an entry’s item count is no use here, as a sparse segment is one item of many containers). The intersection has no more containers and no more members in any of them, and a container is optimized to its smallest form. While the containers are all ARRAYs, or all past them, more containers take more and an even spread is the most the sizes can add up to, so the test is an upper bound of what the probe takes. It is none where the members lie unevenly about the ARRAY limit: a container past it is marked in two maximal containers, 8,240 bytes, where an ARRAY of 256 members takes 568 (Not done, below). The executor asks it in place of the four bytes a row (lion_sum_probe_init()). The planner asks it of its estimate of the same source (lion_cost_range_sum()): each positive clause’s own selectivity, an OR restriction’s at most the sum over its arms of their fewest rows, and the same most containers. So a walk is priced probed when it will be, as far as the single-clause estimates are right. The planner no longer uses the intersection, which is where independence goes wrong. A probe that turns out smaller than the bound only makes the count cheaper than priced; one that would fit only because the intersection is small is not taken.
  • The switch in containers visited. The walk turns to probing once its counts have visited LION_PROBE_SWITCH (2) times as many containers as F’s smallest positive source can lie at, the same most containers. The counter is the node’s Containers Visited, measured around the counts as the FK join’s switch measures it (lion_join_count_key()). The executor asks before every count, not only before every leaf, since a leaf holds a hundred summaries. It does not turn before the walk has handed out a summary: a walk with no whole bucket is counted as §28 measured, as the planner prices it. Visits are what a count’s time goes with. Collecting F visits its smallest source’s containers and seeks the others at them, and the final count does it once more. So once the counts have visited twice the most that can be, the probe can at most double what the walk spent. What a set will cost to count is not on its entry: an entry records its items, and a scattered summary keeps 3,440 containers in a few dozen sparse segments. The planner predicts the switch the same way, two visits a key of whichever of the set and F’s smallest source has fewer.
  • The counted set priced as its merge. lion_cost_range_set_count() prices an entry’s or a summary’s count with lion_merge_cpu_cost_sets(), the model of every AND of sets. The source with the fewest rows drives, the next is sought at each of its keys, and the ones after that only where the AND still has rows. Each source is sought on its posting pages (LION_PROBE_COST) unless the count copies it into memory on its second use (LION_MEMORY_PROBE_COST), as one small enough that fits what work_mem has left is copied. 714 units a summary became 521. That is 303 ns a container modelled against the 272 measured, and 245 against the 167 measured beside the overlap alone, whose seeks land on one posting page after another, where the equality’s land on a page each.
  • The step measured: pg_lion.probe_step_cost 10 (LION_PROBE_STEP_COST). With LION_AND_MEMBER_COST for the member marked it comes to 55 ns a container, against the 58 measured. The probe’s sets are stepped at the keys they share with F’s copy, the fewer of the two: a set with nothing at F’s keys is skipped by the cursor’s seeks, not stepped through.
  • The probe’s overhead. It is F collected (a merge of its sources) plus what the marks hold built and counted with F, where it was two counts of F at a probe into every source a container.
  • The race reads leaves. lion_cost_range_side() hands back the pages the race reads of a side: its keys' pages, and a whole bucket’s summary at its entry’s share of a directory leaf.

Measured (backend CPU, release build, PostgreSQL 18; plain is the lion index scan feeding count(*) with the pushdown off; the count is forced; steady state, the first runs after a restart left out):

window, work_mem count’s price before / after count before / after (ms) plain scan’s price, time (ms) chosen before / after
30 days, 4MB 226,855 / 242,535 392 / 449 84,458, 174 plain / plain
30 days, 64MB 226,855 / 71,294 402 / 99 83,745, 195 plain / count
90 days, 4MB 284,259 / 501,971 854 / 878 159,225, 336 plain / plain
90 days, 64MB 284,259 / 138,190 531 / 186 158,438, 341 plain / count
180 days, 4MB 348,696 / 793,199 1,395 / 1,471 239,322, 476 plain / plain
180 days, 64MB 348,695 / 213,246 594 / 297 238,452, 504 plain / count
365 days, 64MB 457,838 / 340,308 766 / 483 366,216, 676 plain / count

(The counts before are the least of three runs, the rest of six once warm. The plain scan is the same code before and after. At 365 days and 4MB the count, priced at 1,286,487, is the complement.)

Beside the overlap alone at 64MB the count went from 362 ms at 90 days, probed after half its summaries, to 184. It is now chosen over the lion B-tree scan, which takes 860, at a price of 133,080. Beside the equality alone the probe cannot fit at either work_mem: its 10M rows are a NARROW or a BITSET at every key, which the marks take a bitset image of. The count runs in 830 ms and is priced at 483,770, above the lion B-tree scan’s 225,000 for its 1,400 ms. The window alone is unchanged, at 104,381 units for 84 ms. None of these choices got worse. Without summaries the walk is §28’s: it is never probed, and it is priced with the merge model above in place of a probe into every source. test/sql/summary_probe_cost.sql pins the plan and the probe at both ends of the fit on a heap of 112 container keys, and the index rebuilt without summaries.

Not done.

  • A probe in windows. Past work_mem the walk counts every summary, which at 4MB is 2.5 times the plain scan. Probing F a window of container keys at a time would take a walk of the range per window, as the plain scan takes (§29.3). Over 11,392 keys and a copy of 2.23 MB that is two walks, against counting 900 summaries. It needs the walk to be restartable at a window’s first key and the marks per window.
  • The window alone, and the lion B-tree scan. count(*) over a scattered window with nothing beside it is priced at 1.25 times a B-tree index-only scan that takes four times as long (84 ms against 357 at 90 days). One reason is LION_CONTAINER_COST at 41 ns against the 26 a one-member container of a sparse segment takes. The other is a summary’s pages, priced as the pages of the keys it stands for. The lion B-tree scan (§44) of the equality or the overlap beside the window is priced at about 0.4 times its time against the count’s 1.1. Both are outside this change’s model and are left to §31’s next calibration.
  • The seek on posting pages is one constant, 200 ns. It is 120 ns where a source’s containers are small and many to a page, and 250 where each is a page of its own.
  • Marks for an uneven F. The memory test takes F’s smallest source at its average a container. Where some of its containers are past the ARRAY form and the rest well below it, the marks take far more than the average says: two maximal containers for each one past it. Then lion_range_probe_begin() refuses the probe once F is collected, and the walk counts every set, priced probed. That is the walk of before this change at the cost of one collection of F. Marks sized to such a container rather than to the largest one would make the test a bound and let such an F be probed.

Costs (lion_plan_cost.c)

lion_cost_range_sum() prices each side of a summed range with lion_cost_range_side(). On a summarized column a bucket holds max(summary_tids, rows per key) rows and so bucket rows / rows per key keys; a side of n keys covers n / keys per bucket - 1 buckets whole (half a bucket is left at each end on average), walks the keys of the partial buckets as before, and counts each whole bucket’s summary at LION_RANGE_ENTRY_COST plus its containers - bucket rows spread over the heap as the column’s correlation says - merged with the other sources as any AND of sets is priced (lion_cost_range_set_count(), lion_merge_cpu_cost_sets()), as an entry is; it reads about the pages of the keys it stands for, scaled by the containers the union saves. LION_SUMMARY_PHASE_DESCENTS (2) charges the phases' own descents. Without a whole bucket, or without summaries, the model is §28’s. The recheck is spread over the side’s counts rather than its keys. The sum over every entry of a summarized column (k IS NOT NULL alone, §14, or every row, §35) is priced as a range over all of it (LION_RANGED_SUMALL) with nothing outside it: the executor takes a side only of a range, so the inside is the whole price, beside a complement OR’s positive arms too.

Buckets hold summary_tids rows only when keys arrive in order. Keys that arrive in descending order all go into the first bucket (the open one, on an index built empty), and keys in no order grow the middle buckets, which never split; a range then walks the keys of a bucket far larger than the reloption says. So the model reads the rows per summary off the index at plan time (lion_summary_shape(): the entries' counters on at most LION_SUMMARY_SHAPE_LEAVES (4) leaves from where the column’s summaries begin - the first buckets, where descending keys go) and takes that as the bucket’s size once it is more than twice max(summary_tids, rows per key); a single giant bucket makes the average large enough that a range rarely covers a bucket whole, as it rarely does. It used to count whole buckets the data did not have and price such a range as a sum of summaries that the executor walked key by key. A summarized column with no summary at all (on over an empty table) is priced as walked, as it is. (2026-09-28. The alternative - a per-bucket model with a giant bucket’s share of the keys walked - needs the keys per bucket, which the summaries do not carry; the average is what the index can say cheaply.)

A probed side (2026-09-28, “Summed ranges: dense and probed” above; lion_cost_range_side_probed(); revised 2026-10-19, “The price of a probe that does not fit” above). Beside F with a positive source that is not itself a range, whose smallest positive source would fit work_mem as a probe - the executor’s own test (lion_range_probe_fits()) asked of that source’s estimate - a side with a whole bucket whose counts would visit more than LION_PROBE_SWITCH times that source’s containers (a container of the set and one of F at each key of whichever of the two has fewer, for each set) is priced as the executor runs it: the share of its sets before the switch as above, the rest probed - a summary and an entry each LION_RANGE_UNION_ENTRY_COST with no count of its own, each container it shares with F’s copy LION_PROBE_STEP_COST (10 cpu_operator_cost, measured) and LION_AND_MEMBER_COST for each member marked, the fewer of the two sides' per container - and on top F collected (a merge of its sources) and what the probe marked built and counted with F. The heap recheck is then spread over the counts before the switch and the one after it. Every other shape is priced as lion_cost_range_side() prices it. (Until 2026-10-19 the test was F’s rows at two bytes each against work_mem, F being the intersection the rest of the model estimates, the switch was at twice F’s rows, and the step a derived 2 cpu_operator_cost: the walk was priced probed at a work_mem the executor counted every summary at, and the probe at a fifth of what it costs. A scattered range of a thousand whole buckets of 4096 rows beside a clause of 5000 scattered rows, on a heap of 100,000 container keys, was about 400 units a summary before the probe was modelled at all - 4096 probes into F at LION_MEMORY_PROBE_COST - and about 30 with it.)

A range taken as a source costs its collection once per relation - the same walk, with nothing to AND and nothing to recheck, and the union it builds of what the walk hands out (lion_cost_range_union(), “What collecting a range costs” below) - and is then a source read from memory: no page reads for the probes, its rows tuples x selectivity of its bounds. One whose estimated union exceeds a hash table’s memory is priced as walked by every count (the walk times the counts), and one in an OR is priced out. The union’s size is a container per container key its rows lie in, about two bytes a row up to a bitset each when they lie all over the heap and a run per heap block when they lie in its order, the correlation interpolating, and its hash entries. (Before 2026-09-29 it was the scattered size whatever the order: all of a timestamp stored in the order of twenty million rows was sized at 9.4 MB, past the 8 MB hash table of a default work_mem, and priced as walked at every count, where it is collected in 1.3 MB.)

The range’s selectivity is the planner’s (clauselist_selectivity() of its bounds) over the relation it is collected in: over a leaf partition, the bounds are mapped onto the leaf’s columns and estimated with its own statistics, and with the ends of the column read from its lion index (§28, “The endpoint probe”); and a leaf whose bound refutes them at their values when the plan is made - ts >= now() - interval '400 days' beside a leaf of three years ago - selects none of its rows (lion_rel_clauses_selectivity()). Before 2026-09-29 it was the partitioned table’s share of every leaf.

What collecting a range costs (2026-09-29)

Two measurements on synthetic data found a range taken as a source priced wrong in both directions (§27, “A partitioned fact table”, and a dimension semi-joined to a partitioned fact). Reproduced here on a fact of 10,000,000 rows LIST-partitioned by kind into five leaves, one kind sub-partitioned by range on ts into four, ts at random within each leaf, fk skewed (80% of the rows on 75,000 of 500,000 dimension keys), with a partitioned lion index over the filter columns and fk (summaries = auto) and a btree on fk - PostgreSQL 18, packaged build, warm cache, work_mem 64 MB, four cores shared with other work, medians of three runs after one:

  • a semi join of 420 dimension rows to one kind under ts >= now() - interval '730 days': the node, priced at 9,403, was chosen and ran in 90 ms, 87 of them collecting the range in the four leaves of that kind (two million rows, 487 summaries) - in none of the node’s timers. A nested loop, priced at 25,734, runs in 2.2 ms;
  • a semi join of 75,000 dimension rows under a range of 400 days beside an OR with an arm per kind: the node, priced at 512,027, ran in 415 ms, and the hash join chosen at 276,223 in 660 to 700.

What was wrong.

  1. The union was not priced. lion_cost_range_source() charged the walk of a summed range - its entries, summaries and pages, with nothing to AND - and a cpu_operator_cost a container of the union. The walk is the cheap part: the sum over 400 days of a million scattered rows takes 2.3 ms, and collecting the same range 36; stored in order, 0.8 and 5. A collection’s time is its union (lion_range_union_cb()), and counting what the union does shows how: each container of each set the walk hands out is looked up by key and ORed in, or set aside to wait, and a container of the union that is not a bitset yet FOLDS in what came for it - the members set aside, through a bitset image, or a larger container merged in - and is put back at its new size, optimized: some 2 us a fold, where a container ORed in place is a few tens of nanoseconds. A range over 400 days of 20M scattered rows ORed 2,554,479 containers and folded 25,278 times, 11 at each of the heap’s 2,298 container keys; all of 20M rows stored in order ORed 11,043 and folded 8,713 - every one after a key’s first into the RUN that key’s union is, which is no bitset; 400 days of a million scattered rows, whose summaries have 36 members at each of 115 keys, ORed 10,787 and folded or merged 6,077, each merge moving the union’s members too. Over 30 collections - 3 to 1,500 days of a four-year timestamp, in heap order and at random, over 1M, 5M and 20M rows, 0.9 ms to 1.1 s of collecting - the old price came to 28 to 355 units a millisecond, median 184, where the model runs at 500 (§31): up to eighteen times too cheap, the more so the more scattered the column and the smaller its heap.
  2. Every leaf was priced for every key. A partitioned fact’s leaves were each priced for all the dimension rows (lion_cost_fkjoin_path()), but the node looks a key up only in the leaves before the first whose rows match it when the test is an existence test, and a leaf whose fact filters select nothing has no turn at all (lion_join_count_parts()). The second semi join above was priced for 605,200 lookups and counts, eight leaves of 75,650 keys, and made 187,780: two of its eight leaves are years the range does not reach, and a key that matches in one leaf is looked up in no later one.
  3. The partitioned table’s selectivity in every leaf. The range’s selectivity was estimated over the query’s relation, the same share of every leaf: 27% of each yearly leaf, where it is none of the two oldest, a tenth of the third and all of the last.
  4. A partial bucket at each end of a one-sided range. ts >= X has one partial bucket, whose keys the walk reads one by one; its summaries run to the column’s end. Each of those keys is a fold into a column in heap order.
  5. The union’s size was the scattered one whatever the order (the paragraph above on sizes).

The two cases err in opposite directions because the collection is nearly all of the first - 87 of 90 ms, priced at a tenth of that - and a quarter of the second, 110 of 415 ms, where the lookups and counts of eight leaves priced for 605,200 keys (418,000 units, 836 ms at the model’s rate) more than make up for it.

The model (lion_cost_range_union(), on top of the walk). Each container the walk hands out is ORed in at LION_CONTAINER_COST, and the folds are counted as the union makes them, between two shapes the column’s correlation squared interpolates, as cost_index() does:

  • rows in the heap’s order: every container after a key’s first is folded into its RUN - the sets' containers, a set’s rows spanning one container key more than they fill, less the union’s containers. (Since 2026-09-30 the RUN is widened into a bitset by the second container instead and optimized back at the end (“Widened”, above), LION_RANGE_WIDEN_FOLDS (2) folds for each union key that takes more than one container; this is what that measurement found.)
  • rows scattered: each set has a container at nearly every container key (lion_containers_for()), and a key’s union ends up with its share of the range’s rows. Containers of at most LION_RANGE_UNION_PEND_MIN (32) members wait and are folded in once the waiting members come to half the union’s, so the union grows by half at least between two folds: 1 + log1.5(members / 32) folds a key until it is a bitset, past 2,048 members, and takes the rest in place - 11.3, against the 11 measured above. (A union that became a RUN instead - dense, which random rows seldom are at 2,048 members - was folded for every container after that, and is widened into a bitset now.) Larger containers are merged in one at a time until then, a fold each, and each moves the union’s members too, half a bitset’s worth on average, at LION_AND_MEMBER_COST a member. Which are larger is a Poisson count’s tail (lion_poisson_above()): a set’s rows fall at a key at random, and of a summary of 16,384 rows over 575 container keys, 28.5 a key on average, a fifth of the containers hold more than 32.

A fold is LION_RANGE_FOLD_COST, the new setting pg_lion.range_fold_cost: 420 cpu_operator_cost, 2.1 us at 500 units a millisecond, fitted to what the walk’s price leaves of the time of 62 collections, their folds counted - the 30 above, six more over 20M rows in heap order, and 26 over 5M rows with buckets of 1,024 and 16,384 rows and with none, 0.9 ms to 4.1 s. 57 of the 62 are within a factor of 1.5 of it; the five others are columns in heap order folded key by key into a RUN of up to a run a heap block, up to 7 us a fold. With the folds as the model counts them, its price of the 56 still measured is 112 to 1,830 units a millisecond, median 493, 48 of them within a factor of two of 500 and 53 within three (the old price of the 30 with the default buckets: 28 to 355, median 184). All eight outside a factor of two are columns in heap order, whose partial bucket the walk reads and folds key by key: the model takes half a bucket, or every key of a range inside one, and the walk reads anywhere from none of it to all - all of it when the range’s bound lies below the column’s first key, since the walk cannot tell that the first bucket is whole (below) - and a bucket of 16,384 keys, or a column without summaries, folds them into the costlier RUNs. Scattered columns are priced within a factor of 1.6 at every bucket size, and without summaries.

Each leaf of a partitioned fact is priced for the keys it looks up: the dimension rows times the chance that it has a turn - none when its bound refutes the fact filters at their plan-time values (lion_leaf_refuted_now()), and otherwise the rows they are expected to select there, taken as that chance while they are less than one (lion_leaf_turn_share()) - and, for an existence test, times the share of the keys no earlier leaf has matched: a key found in a leaf matches there unless none of its rows passes the leaf’s filters, exp(-rows) for its rows there spread as the leaf’s estimate says. A batch’s place is still a row’s once however many leaves look the row up. The range’s bounds are estimated over each leaf’s own columns and statistics (lion_rel_clauses_selectivity(), the paragraph above), a one-sided range has its partial bucket at one end (lion_range_sides()), and the union’s size follows the column’s order.

Where it is collected, and how often. The same function collects a range in every context (lion_locate_range(), lion_range_collect()), and what differs is how many times:

  • a count’s source, grouped or not: once a run in each relation - each leaf of a partitioned table - before its first count, and read from memory by every group, pair or test after it; the model charges it once a relation (lion_cost_count_rel());
  • the FK-side join’s fact filter: once a run over a plain table, when the filters are located, and then read by every count, or by the collection of the copy of the filters (§27, “The fact filters, collected once”), which reads it as any other set; over a partitioned fact once a leaf, at the leaf’s first turn where the plan collects the copy, and at every turn - every batch of keys
    • where it probes, which the model charges once a batch;
  • in a parallel plan, once in EVERY participant: the filters are each participant’s own, and only their copy is shared (§27, “One copy per query”). The parallel semi join below spent 190 ms of Fact Filter Locate Time over three participants on a range the serial plan collects in 50. The model charges each participant the whole of it, as it charges each the locating of the filters.

EXPLAIN. The locating of the FK-side join’s fact filters is a phase of its own, Fact Filter Locate Time (LION_JT_LOCATE, lion_join_locate_where()), printed with TIMING beside Join Child Time, Join Lookup Time, Join Count Time and Fact Filter Collect Time and, as they are, summed over the participants of a parallel plan and the leaves of a partitioned fact. It was in no phase: of the first semi join above, 87 of the node’s 90 ms were in no timer.

Measured before and after - the library before this change with the timer alone, and this one; the same data, the same session settings, nothing disabled for “chosen”. The partitioned star is the one above; r5m and o5m are 5,000,000 rows of (fk, kind, g, x, ts), fk uniform over the 500,000 keys of the dimension, ts over four years at random (r) or in heap order (o), with a lion index on (fk, kind, g, x), one on ts with summaries, and btrees on fk and on ts; the fact filter is ts >= now() - interval '400 days' (27% of the rows) beside kind = 1 for the inner joins; the dimension rows are the first of a random order. Times in ms, the node’s median over the two libraries' runs (its plan is the same); a node chosen before and after is marked once:

query chosen before, after node cost before, after node ms core’s alternatives: cost, ms
partitioned semi, 420 dimension rows node 90, nested loop 2.4 9,403, 49,968 97 nested loop 25,734, 2.5; hash join 72,241, 1,097
partitioned semi, 75,000 hash join 662, 660 512,027, 316,865 441 nested loop 337,378, 938; hash join 276,208, 667
r5m semi, 100 nested loop 1.4, 0.9 7,100, 17,506 53 nested loop 2,423, 1.0
r5m semi, 1,000 node 9,294, 19,699 55 nested loop 21,580, 11
r5m semi, 10,000 node 28,777, 39,171 84 nested loop 96,829, 78
r5m semi, 100,000 hash join 1,080, 1,116 124,062, 134,360 263 hash join 106,327, 1,116 to 1,293
r5m inner, grouped, 100 nested loop 3.4, 3.5 7,457, 17,861 59 nested loop 5,437, 3.8
r5m inner, grouped, 1,000 node 9,946, 20,350 60 nested loop 49,254, 20
r5m inner, grouped, 10,000 node 32,492, 42,897 97 hash join 172,762, 475
r5m inner, grouped, 100,000 node 155,908, 166,313 326 hash join 926,759, 536
o5m semi, 1,000 node 3,690, 5,956 11 nested loop 21,284, 14
o5m semi, 100,000 hash join 705, 854 118,505, 120,346 201 hash join 70,051, 787 to 854
r5m inner, grouped, 100,000, two workers node 87,933, 98,337 222 hash join 421,483, 338
r5m semi, 10,000, two workers node 21,223, 31,624 96 nested loop 60,741, 54
r5m g, count(*), 30 days node 15,710, 18,926 29 bitmap heap scan 41,064, 187
r5m g, count(*), 400 days node 10,191, 20,597 67 bitmap heap scan 93,093, 568
o5m g, count(*), 30 days node 2,023, 4,030 12 index scan 3,993, 26
o5m g, count(*), 400 days node 2,144, 4,442 17 index scan 56,849, 383
r5m count(DISTINCT g), 400 days node 6,869, 17,275 47 bitmap heap scan 232,347, 654
r5m count(*), 400 days and fk < 50000 bitmap heap scan 227, 270 59,922, 74,697 155 bitmap heap scan 53,097, 270 to 335
20M rows at random, g, count(*), 30 days node 19,542, 36,576 124 bitmap heap scan 164,053, 588

The node’s price per millisecond of its time was 97 to 1,162 units (median 220) - 97 for the first semi join, 1,162 for the second - and is 267 to 719 (median 442). The low end is the grouped counts, whose counts of each group against the range’s bitsets take about twice what §10 prices them at (below), and the joins over a hundred to a thousand keys, whose collection took 48 to 61 ms in these runs, on a machine other work loaded, against 33 in the calibration runs. One plan changes: the first semi join’s, from the node at 90 ms to the nested loop at 2.4. The mispicks left are core’s own prices of its plans against a warm cache (§27, §31): the nested loop of the 1,000-row semi join is priced at 21,580 and runs in 11 ms, 1,900 units a millisecond, above the node’s 19,699 for 55; the hash joins chosen over 100,000 keys run at 80 to 95 units a millisecond, and the bitmap heap scan of the two ranges at 160 to 200. The node over the partitioned star’s 75,000 rows is still priced at 720 units a millisecond - its lookups at two to three times the 75 to 101 ms they took (§27’s LION_FKJOIN_BATCH_ROW_COST is high on purpose) - and loses to a hash join core prices at 400 to 420.

test/sql/rangesource.sql §9 pins both ends on 200,000 rows whose summaries of 1,024 scattered rows have 60 members at each of 17 container keys: ten dimension rows and a range over nearly all of them go to core’s plan (the node was chosen before, and ran three times slower than the nested loop), a hundred and a range over a tenth to the node; and the timer’s presence with TIMING and its absence without. costgucs.sql shows the new setting moving a price; fkjoin_partition.sql’s forward semi join over a leaf whose filter selects none of its rows now descends for each key rather than walking the fk index: no key is looked up there, and the model no longer prices a walk of them.

Found and not done. The executor’s union takes six to fifteen times the sum over the same range, and three things would make it cheaper, each a change of the executor’s own: folding a container of more than 32 members through the same waiting buffer rather than merging it at once; letting a RUN union take the members of a column in heap order through that buffer too, where each one now goes through an image; and taking the first bucket of a range whose lower bound is below the column’s first key as whole, where its 4,096 keys are walked one by one - a range over all of a leaf reads them. The counts of a GROUP BY against a collected range’s bitsets take about twice those against an equality’s set of as many rows, priced the same (§10’s counts). And the node’s lookups over a partitioned fact, as above - since refitted (§27, “The per-key terms, refitted”).

Verification (lion_verify_summary.c)

lion_index_verify() checks every summarized column after its other checks (lion_verify_column_summaries()): bucket by bucket in key order, it looks the bucket’s summary up as it is at that moment (lion_verify_summary_after(), a descent), collects and sorts the codes of the bucket’s keys from one walk of the column’s VALUE entries, and streams the summary’s codes past them in code order. The keys' codes are held in an array up to a quarter of maintenance_work_mem and past that in an INT8 tuplesort of as much, which spills - the array goes when the sort takes over, as the build’s does - and the summary’s are not held at all: one bucket can be most of the table (a column of few keys, or the one bucket descending keys go into), and both sides used to be collected whole, which ran a backend out of memory. It also checks that no row is under two keys, and (in the directory walk before it) that summaries are in order, that a SUMLAST is the last and no pivot is one, and that a summary is of a summarized column and carries LION_SUMMARY_HASH. Like the rest of the function it runs under ShareUpdateExclusiveLock, beside inserts (VACUUM waits for it), so a difference is a CANDIDATE, not an error: an insert may be between its key and its summary. Once the column is walked - or earlier, when the candidates and the records of the buckets they are in fill a quarter of maintenance_work_mem (below) - the writers that could explain them are waited for (lion_verify_wait_for_writers()) and the candidates are SETTLED in one pass over the summaries of their buckets and one walk of the column’s keys (lion_verify_settle()): a row a summary lacked must be in its bucket’s summary now, a row a summary holds must be in one of its bucket’s keys now - and a DEAD row (HeapTupleSatisfiesVacuum() against the oldest non-removable xid, or an unused or dead line pointer) is never a difference, which is what a crash or an error between an insert’s two steps leaves. The first candidate that is still one is reported; nothing else is capped.

What the check holds does not grow with the column. Beside the quarter for one bucket’s codes, the candidates and the RECORDS of their buckets - each bucket’s two bounds, which a settle looks the bucket up by - share a second quarter of maintenance_work_mem (never less than LION_VERIFY_MAX_SUM_CANDS (10,000) candidates' worth), and are settled when they fill it: the record of a bucket with no candidate goes as the walk leaves it, every record but the current bucket’s goes once the candidates are settled, and neither array grows past what its quarter holds. A key’s set and a bucket’s summary are read in memory contexts emptied for the next, since locating a set copies its key and an INLINE payload into the caller’s memory. (Before the 2026-09-28 review every bucket’s bounds were kept to the end of the column, whatever it settled, as were the copies of every key and INLINE payload the walk and each settle read; the array stayed, full, beside the tuplesort; and the candidates' array doubled past its quarter.)

The OPEN bucket is read when the walk gets to it - its key and its summary as they are then - and a key the walk finds above that key belongs to a row that arrived while the check ran: no candidate. So is every key of a column with NO summary above the last one the walk read (on over an empty table, or a crash between the two records of a close): a row whose insert went through both steps before that lookup is in a summary at or above its key, so a row above every summary there was is one whose insert had not, and by the time its writer is waited for it may be in any of the buckets the inserts beside the walk opened and closed since. It used to be a candidate looked for in the first of those buckets only - a load into such a column had the rows of the second reported as corrupt, with the hint to REINDEX a sound index.

Only the largest key that arrived is kept, and it has to be inside a bucket once the writers are done (lion_verify_check_above(): the lookup an insert makes). If it is not - its own insert may have failed between the steps - every key up to it that is not inside a bucket has its rows looked at: a DEAD row is a failed insert; a row whose inserting transaction is still RUNNING may be between the steps now (one that began after the wait, as a load’s next statement does) and is left alone, as is one of a transaction that sits idle after its insert, which only makes the check look at less; and a row whose insert COMMITTED went through both steps before it did, so its key must be inside a bucket once that is seen, or the row is in no summary and never will be - which is reported. That check used to wait a second time and report any row not dead after it, which was any insert that happened to be between its steps then, and it compared the keys with the open bucket’s key as it was before its walk of them began.

A summary at or above the open bucket’s key as the check began was the open bucket then and is taken as the open one, which bounds the walk however fast appends close buckets behind it. Before 2026-09-28 the open bucket’s key came from a copy of its leaf made when the walk began, so every row appended during the check was a candidate that cost a walk of the whole column to settle, and the cap on candidates, meant for a summary of garbage, was applied before any was settled: a column under steady appends was reported corrupt. test/sql/corrupt.sql rewrites a summary’s key on disk to one that still sorts between its neighbours, so that every entry is sound on its own and only the comparison can tell; verify reports the row the summary holds and no key of its bucket does.

lion_index_stats() reports per column the summaries' entries, rows, bytes and posting pages (summary_entries, summary_tids, summary_bytes, summary_pages), which the other counters leave out.

Tests

  • test/sql/summary.sql: range shapes (both sides open, closed, within one bucket, empty, inverted, outside the keys, cross-type bounds past the int4 domain) on a column in heap order, one in no order, a nullable one and a text one, each against a sequential scan with the pushdown off and printed with its evaluation and the summaries it summed; the inside, the complement and the full domain beside other clauses; two bounds on one side; walks that must not use summaries (a GROUP BY, count(DISTINCT), k IS NOT NULL) and the bitmap scan; appends that open buckets, inserts between and below every key; a dirty heap, VACUUM, a run of whole buckets deleted and a column emptied and refilled; a format-6 index (the default), ALTER INDEX ... SET (summaries) having no effect until REINDEX, REINDEX adding and removing them; auto on a multicolumn index, on a multi-key column and on an empty table; on on an empty table with keys arriving out of order; a partitioned table with summaries in some partitions; a unique timestamp under auto. lion_index_verify() after every kind of change.
  • test/sql/rangesource.sql: a range as a source beside a GROUP BY (one and two columns), a count(DISTINCT), another range, an equality on its own column, in OR arms, among an FK-side join’s fact filters, on a partitioned table; collected and - at work_mem’s floor, over 200,000 rows - walked, inside and as its complement, and in existence tests; generic plans with Param bounds, a NULL one, and multi-key Params beside it; a dirty heap and VACUUM. Since 2026-09-29 (§9) the planner’s choice at both ends of a collected range’s price - core’s plan over ten dimension rows and a range over nearly every row, the node over a hundred and a range over a tenth - and Fact Filter Locate Time with TIMING and not without.
  • test/sql/summary_append.sql (2026-09-28): the directory order of “The open bucket and the directory order” - rows appended one statement at a time over several leaves with summary_tids = 16, verify and range counts at and around bucket bounds against a sequential scan, then the top buckets deleted, VACUUM, keys below the old top and appends again.
  • test/sql/summary_verify.sql (2026-09-28): verify of a summarized index built on an empty table before any insert (no summary at all), after appends that open and close buckets a statement at a time, and after one key of 36,000 rows at maintenance_work_mem = 1MB, whose bucket’s codes go from the array to the tuplesort and back to a new array for the buckets after it. What it cannot show single-session - the load beside the walk that §32’s “Verification” is about - is argued there; summary_race.spec keeps verify beside an insert parked between its two steps.
  • test/sql/spill.sql (2026-09-28): an OR’s range leaf planned where its union fits and run at 64 kB, collected into memory and then windowed into a temporary file (Range Sources Spilled), on a clean and a dirty heap; and an FK-side join whose fact filter is = ANY ($1), planned to collect at the ten values a parameter is guessed at and run with 5000, its copy spilled (Fact Filter Copies Spilled) - each against the ordinary plan’s answer.
  • test/sql/summary_scatter.sql (2026-09-28, “Summed ranges: dense and probed”): keys that are a permutation of the row number, on a heap of a score of container keys, one column with INLINE summaries and a timestamp whose summaries are posting trees; sums alone, within a bucket, beside a clause with fewer rows (probed: inside, and each walk of a complement), beside one with more (counted), beside an OR across columns, an IN list and a negated source; on a clean heap, a dirty one (new row versions, keys moved above every key, deletes) and after VACUUM; ranges collected as sources; and a narrow table whose range and clause are bitsets at every container key, so that the probe ORs into images as well as marking an ARRAY’s members, with a range source too large for work_mem’s floor walked, and probed, at each group’s count. Every count against a sequential scan, with whether the walk was probed or the range collected. Since 2026-10-19 the walks within a few hundred keys of a bucket beside a clause of a thousand rows or more probe as well: their counts visit twice that clause’s containers within the first summaries.
  • test/sql/summary_probe_cost.sql (2026-10-19, “The price of a probe that does not fit”): a 42-day window over a timestamp that is a permutation of the row number, with summaries of 128 rows, beside an equality and an overlap of a tag on a tenth of the rows that the equality holds all of, in one multicolumn index, at 64kB (the tag’s probe does not fit: counted, priced so, and the plain scan chosen), 128kB and 4MB (probed and chosen, at under half the counted price, the same price both times); a window short of a bucket (never probed); a tag of a fifth of the rows, whose NARROW containers take a bitset image each in the marks (counted at 128kB, probed at 4MB); a tag of seven tenths beside the equality’s half, which a switch at twice the smaller source’s rows never reached in that window; and the index rebuilt without summaries (walked, never probed, the plain scan chosen). Each against a sequential scan, at random_page_cost 1.1.
  • test/sql/corrupt.sql section 4, and test/sql/rangesum.sql’s two-bounds case; range.sql, distinct.sql and fkjoin.sql keep their formerly declined shapes, now answered, beside a clause no posting set answers.
  • test/isolation/summary_race.spec: a count summing summaries parks with the first summary’s container pinned (lion-count-containers-pinned) and a VACUUM that would remove that bucket’s dead rows waits for the pin (asserted), the count exact with the next bucket’s rows deleted after its snapshot rechecked in the heap; counts park between the phases of their walk (lion-entry-scan-phase, new) while appends close, open and rekey buckets and middle inserts split the leaves the next phase descends into, exact against their own snapshot’s count; and lion_index_verify() runs while an insert is parked between its key and its summary (lion-insert-before-summary, new), waits for it, and comes back clean.
  • The recovery harness (test/recovery/): the table gains lion_rec_sum on (id, k4) with summaries = on, summary_tids = 64, inline_limit = 64 - id is what the writer appends above, so every crash round closes, opens and rekeys buckets (chained ones among them) under WAL while its updates grow the middle buckets and its deletes and the VACUUM loop empty and delete summaries - and range queries that sum summaries and take ranges as sources join the probes compared with a sequential scan on the primary, the standby and the promoted standby. Every check runs lion_index_verify() on it, which compares every summary with its bucket, and a new check requires the summaries to hold every row (exactly, after a VACUUM). No new WAL record type, so no new phase: the summaries ride on the records every phase already replays. Both modes pass with wal_consistency_checking on ('generic', and 'pg_lion' with --mode rmgr): eight crash rounds each, 333,389 and 224,543 index records replayed, 158 checks per round (2026-09-27).

Open for the owner

The choices made conservatively here, each with the alternative:

  • summaries = off by default. Nothing changes for an existing schema and no insert pays for what it did not ask for. The alternative is auto as the default, which would summarize exactly the columns the range walks hurt on (many small keys) at the insert cost measured below.
  • Per index, not per column. on summarizes every ordered scalar key column, a low-cardinality one included; auto is the selective form. A per-column choice needs a column-level option the index AM interface does not offer (an opclass parameter could carry it).
  • Middle buckets never split. Inserts into the middle of the key space grow their buckets, which only REINDEX rebalances; a split would move rows between two summaries and needs its own atomic protocol (both entries in one record) against the readers' disjointness.
  • auto on an empty table decides nothing, and the index gets summaries at its next REINDEX.
  • A range in an OR’s arm is always collected, whatever memory it takes; the planner prices one it expects to exceed a hash table’s memory out of the plan. Walking it would need inclusion-exclusion against the other arms.
  • Downgrade: an index with summaries is format 7, which an older build refuses; REINDEX with summaries = off first.
  • Not used by: GROUP BY k walks, count(DISTINCT k), ordered scans, and the index scans' walks of a whole column (IS NOT NULL) - they want keys, or could take summaries (a whole column) in a later change. The bitmap and plain index scans of a range use them (§28, “Bitmap scans”).
  • The probed sum (“Summed ranges: dense and probed”) turns once its counts have visited twice the containers of F’s smallest source (LION_PROBE_SWITCH), within work_mem for that source’s copy and the marks, and prices its steps at a constant measured on one shape (LION_PROBE_STEP_COST, 2026-10-19). The alternative is a switch the planner decides from its estimates - no regret bound when they are wrong. Past work_mem it does not probe at all, and a scattered range then counts every summary: a probe of F a window of container keys at a time, as the plain scan walks its range, is not done (“The price of a probe that does not fit”). A scattered range ALONE still counts summary by summary; a union with one map question per container key needs pins no budget allows, and summaries of summaries (a format change) are the way to fewer, larger sets.
  • The insert fast path an appending column could have - remembering where the open bucket is, as btree remembers its rightmost leaf - is not there: every summarized row descends from the root (once; an append that raises the open bucket’s key no longer descends twice).
  • An emptied open bucket stays, keyed by the largest key it ever held, and the next row above the closed buckets joins it however far below that key it is. Its key could be lowered to the closed buckets' top by a rewrite of the entry, but not below any pivot a split made of it, which the entry cannot know; keeping it is what keeps the directory order.
  • A crash between the two records of a close leaves a column without an open bucket; the next insert above it opens one (step 5), above every summary the column has, and so above every pivot. The one residue left: if VACUUM empties and deletes that closed bucket before any insert above it, and a row then arrives below its key, the new open bucket is keyed below pivots the deleted one may have left, and closing it would break the order as above - three unlikely events in a row, which lion_index_verify() reports (with its hint to REINDEX). Closing it too means keeping a column’s last summary whatever its kind, which VACUUM can only tell by reading the next leaf under the cleanup lock of this one; it was not worth that here.

Measured (2026-09-27, PostgreSQL 18.6 assert build, generic WAL, CPU time of the backend)

Two copies of one 2M-row table - id, ts (a unique timestamp one second apart, in heap order), g (50 values), r (a permutation of 0 .. 2M-1 in no heap order), a short pad - each with one lion index on (ts, g, r): the BEFORE copy’s built with summaries = off (format 6: what every index was), the AFTER copy’s with summaries = auto (format 7: ts and r get 489 summaries of 4096 rows each, g none, as auto decides). One build, one server, every page all-visible, the pushdown forced for the range timings, the copies interleaved, CPU time of the backend (/proc/<pid>/schedstat) per run, over repeated runs of at least 250 ms, the best of three passes.

Range counts, by keys in the range (ms per count; the “before” walks a key at a time, the “after” walks the edge buckets' keys and sums the summaries between):

keys in range ts range ts range AND g = 7 r range (no order) ts >= X alone
before after before after before after before after
1,000 0.43 0.46 0.69 0.72 0.43 0.43 0.43 0.42
10,000 3.49 0.75 4.56 1.15 3.43 0.86 3.42 0.32
100,000 34.9 0.72 46.5 1.29 34.7 2.11 34.2 0.29
1,000,000 346 3.92 459 4.42 341 16.3 344 3.11
1,990,000 437 2.89 355 3.40 - - 678 5.26

A range within one bucket is what it was (no bucket is whole, so the walk is §28’s). Past a few buckets the time stops growing with the range: it is the keys of the two partial buckets - up to 4096 one-row keys each, 0.3 to 0.5 us apiece - plus a count per summary. The ts summaries are a RUN container or two each (the column is in heap order), so 244 of them cost less than the edges; the r summaries are a container at every one of the heap’s 230 container keys, which is why its million-key range is 16 ms and not 4. ts >= X alone (no other clause, so no complement) is the shape §28 could only walk: 678 ms for 1.99M keys, 5.3 ms now. With a clause beside it the complement already made the near-full range short (355 ms for 1.99M keys was the complement’s walk of the 10,000 keys outside); summaries make both sides short.

Plans (default settings, and the cost units before §31 refitted them on a release build - the ratios are the point): the million-key ts range is priced 132,572 before (a sequential scan at 47,189 is chosen, ~190 ms) and 898 after (the node, 3.9 ms); ts >= X over 1.99M keys 266,885 before (sequential scan chosen) and 1,241 after (the node). A 1,000-key range is priced 134 and 125.

Ranges as sources (after only; “off” is the plan the planner picks with the pushdown off):

query pushdown off pushdown on
g, count(*) WHERE <ts range of 100k keys> GROUP BY g 33.5 (bitmap heap scan) 12.5
g, count(*) WHERE <ts range of 1M keys> GROUP BY g 338 (seq scan) 39.4
g, count(*) WHERE r < 1000000 GROUP BY g (half the rows, no order) 376 (seq scan) 310
count(*) WHERE ts >= X AND r < 100000 (two ranges) 253 (seq scan) 26.7

Insert (CPU of the statement, best of three): 5,000 single-row INSERTs 320 -> 419 ms (+31%, about 20 us a row for the two summarized columns: a descent and a posting-set insert each); INSERT … SELECT of 100,000 rows 3,086 -> 5,342 ms (+73%, 23 us a row). The index work per row is five posting-set updates instead of three, and that is the price: about 10 us per summarized column per row on this assert build. The open bucket of an appending column takes every row, INLINE and small (a RUN container); the r column’s buckets are posting trees of scattered rows.

Size: 265 MB -> 277 MB (+4.5%). The ts summaries are 67 kB of items (RUN containers), the r summaries 4.9 MB of items on 1,465 posting pages; nothing else changes.

VACUUM after deleting a tenth of the rows spread over the table (1.5M rows left; the heap pass is the same work in both): 1,376 -> 1,594 ms, best of two rounds each in alternating order (+16%; the rounds differed by 10 to 20% among themselves). The bulk-delete pass reads every summary as it reads every key, and the summarized columns hold each dead TID twice.

Build (REINDEX, serial, 1.27M rows): 3,335 -> 3,598 ms (+8%), best of two in alternating order; the index is 159 MB -> 169 MB (+6%) at that size.

Not measured here: the rmgr WAL mode’s timings (the records are the same ones, so its ratios should follow), and a production build - these are assert-build CPU times, for ratios only.

33. n_distinct from the directory (implemented)

Core estimates a column’s number of distinct values - pg_statistic.stadistinct - from ANALYZE’s sample of 300 rows per unit of statistics target, with Haas and Stokes' estimator for what the sample did not see. A column with a long tail of rare values is where that falls short: a foreign key whose referenced rows have a handful of rows each, beside a few that have thousands, is sampled as the heavy values over and over and a sliver of the tail, and the estimator puts the column at a small fraction of what it holds. Every estimate that divides by n_distinct inherits the error: estimate_num_groups() for GROUP BY and DISTINCT and the hash aggregate sized from it, eqsel() for a value that is not a most common one, the join selectivity of the column.

A lion index knows the answer. Each VALUE entry of a scalar key column is one distinct value (§21, §24), so a lion index on a plain column counts its keys, keeps the count on its meta page, and the planner is given it in place of ANALYZE’s.

When the keys are counted

  • At build (lionbuild()): every entry passes lion_build_add_entry() once, in directory order, so the count is exact - the VALUE entries of each key column, and the rows: the TIDs under the entries of the first scalar key column, NULL entry included (lion_index_row_column(), which §18’s row count uses too). A parallel build counts on the leader, which writes every entry.
  • At VACUUM, in ambulkdelete’s own walk (lion_vac_count()): lionbulkdelete() visits every leaf left to right already, and every entry once - an INLINE one in pass 1, a CHAIN one at the end of its pass 2 - where it adds the entry’s TIDs to the rows it reports (§18). The same place counts the entry as one of its column’s keys when it is a VALUE entry left with a TID after this VACUUM’s deletions; an entry left without one is the one the leaf’s final step deletes. When the walk is over the survivors are recorded, and lionvacuumcleanup(), which is handed that call’s result, does not walk again. A VACUUM whose dead TIDs take several ambulkdelete calls records each call’s count, and the last stands.
  • At a VACUUM without ambulkdelete, and at ANALYZE - a VACUUM that found no dead row (an insert-only table’s autovacuum), and ANALYZE, which calls the cleanup with analyze_only when it is not part of a VACUUM (do_analyze_rel(), after update_attstats() and the pg_class updates): lion_vacuum_count_keys() walks the directory when it is small, and counts a sample of its leaves when it is not (below). VACUUM (ANALYZE) calls the cleanup once, from VACUUM, and its ANALYZE step does not call it again. A VACUUM with INDEX_CLEANUP off calls neither and does not count.

An entry is counted on the leaf ambulkdelete’s pass 1 found it on, and a split that moves it right afterwards moves it to a page between that leaf and the right link pass 1 read, which the walk does not visit: nothing is counted twice. The cleanup’s walk reads each leaf under a SHARE lock alone, through the VACUUM’s or ANALYZE’s buffer ring, with the cost-based delay (and CHECK_FOR_INTERRUPTS) between leaves and a right-walk guard against a cycle of right links (§21); a leaf that splits after the walk has read it moves entries already counted onto a page between it and the right sibling the walk goes to next, one that split before is read with its new sibling. Either way nothing is missed but what writers add or VACUUM deletes meanwhile. Counted: the VALUE entries of each column with at least one TID. Not counted: the reserved entries (§14, §17), SUMMARY and SUMLAST entries (§32), pivots. A key whose rows are all dead is counted until the VACUUM that empties its entry deletes it (§18), and dead TIDs count in the rows: the count is an estimate, as ANALYZE’s own is.

A multi-key column (§17) is not counted - its keys are elements, not values of the column - and an index whose key columns are all multi-key records nothing. Nor is an rmgr-mode index counted on a server that has not registered the resource manager: the record could not be written (lion_wal_begin() refuses it), and a VACUUM must still go through there (§25, “VACUUM is the one path that could write without lion_wal_begin()”; test/recovery/run.sh phase 1f vacuums such an index). It keeps the counts it has until the preload is back. An unlogged or temporary index is counted like any other and writes no WAL.

What counting costs

For a column of many values the directory is as large as a btree over it: a column of a hundred million keys has a directory of a few gigabytes. Counting it may not cost what ANALYZE was never meant to cost, so:

  • VACUUM counts for nothing. ambulkdelete reads and cleanup-locks every leaf whether or not the count is taken; the count adds a comparison and an increment per entry to a pass that filters its posting set, and one meta page record at the end. The first version of this section walked the directory again in the cleanup, right after ambulkdelete had walked it: that walk is gone.
  • ANALYZE, and a VACUUM with nothing to delete, walk only a small directory. The bound is what ANALYZE’s own sample of the table reads for the index’s columns: 300 rows - from at most as many heap blocks - per unit of the largest statistics target among the index’s key columns (lion_vacuum_count_budget(): a table column’s target is the table’s, an expression’s the index column’s, -1 is default_statistics_target as std_typanalyze() takes it, 0 is nothing). At the default target of 100 that is 30,000 pages, some 230 MB of directory - a column of several million keys. The meta page knows the directory’s size exactly (dirpages, which the build and every split keep, §21), so the cost of deciding is the meta page. A walk that finds the directory has grown past the bound while it walks (a concurrent insert splitting leaves) is abandoned at the page that would pass it, and the index keeps the count it has: a count of part of the directory would be wrong where an old one is only old.
  • A larger directory is counted from a sample, as ANALYZE estimates a table from a sample of its blocks (below, “Counting a sample”), within the same bound.
  • The leaves are asked for ahead of their reads (lion_vacuum_prefetch(), 2026-11-01), as far ahead as the index tablespace’s maintenance_io_concurrency (none at 0): the sample’s from its list, and the walk’s from a list of level 1’s downlinks, read first where level 1 fits in the bound beside the leaves (it is far below a 32nd of them) and charged to it; a list that turns out not to fit is given up within the bound’s slack, uncharged, and the walk made without it. Read one at a time, each leaf waited a full device round trip, where the table’s own sample beside it is read through a read stream: on storage with a millisecond’s latency a bound of 30,000 cold leaves is half a minute of ANALYZE on its own. The ask pins nothing; a leaf a split adds while the walk runs is not listed and is read as the walk reaches it, and one listed and gone since is asked for and not read.
  • Nothing is written that is already there. lion_meta_write_ndistinct() compares the counts under the meta page’s lock and writes no record - and so, the first time after a checkpoint, no image of the meta page - when an ANALYZE of an unchanged table counts what the page holds.

So every ANALYZE - autovacuum’s too, which a table that is only inserted into gets once a tenth of it is new - counts again, whole or from a sample, and so does every VACUUM but one with INDEX_CLEANUP off, on its own walk when it deletes. Between counts the planner scales a count by the table’s growth, uses it while the table stays within a factor of 2 of the rows it was counted over, and past that has ANALYZE’s estimate again, exactly as without lion (below, “Stale counts”).

Counting a sample

lion_vacuum_sample_keys(). Every VALUE entry of a directory leaf is one key and carries its TIDs, so a leaf’s share of a column’s keys, of its Σn² and of the rows is read off the leaf alone, and a sample of the leaves scales up by the number of leaves. The downlinks of level 1 list every leaf; a walk along level 1 reads them (lion_dir_leaves()), a few hundredths of the directory, and with each the key column its lower bound is in.

  • Strata by column. Entries are in the order of their key column first (§24), so a column’s keys are a run of leaves. A short run - a column of a few keys, which a sample of the leaves would find by luck or not at all - lies on the leaf where the run of leaves whose lower bound is in one column ends; that leaf is read whatever the sample, for every column, and the last leaf with them. The other leaves are sampled in strata, one per column their lower bound is in, each its share of what the bound leaves (one leaf at least), and each sampled leaf stands for its stratum’s leaves over its sample. Each column’s estimate is unbiased.
  • The heavy keys are read, not sampled. A few heavy keys carry most of a skewed column’s Σn², and a sample that found the leaf of one or missed it would swing the sum by that key’s square. They are the values ANALYZE has just listed as the column’s most common - the cleanup runs after ANALYZE writes its statistics - so, where the statistics are of the key column’s own type, the leaf each of the first 100 of them is on is looked up (lion_key_ntids()) and read whole too, and only the rest is sampled: the part of the sum the planner uses (below, “The rows of a probed key”). Statistics of another type (an expression ANALYZE keeps as its input’s) mark no leaf.
  • The bound. Every page the walk along level 1 and the lookups read, and every leaf read whole or sampled, counts against it. Each stratum gets one leaf and its share of what is left, so the strata together never read more than the bound leaves them. A directory whose level 1 alone would take half of the bound, or whose pages that must be read leave less than a leaf a stratum, is not counted and keeps the counts it has.

Each page is read SHARE alone through the VACUUM’s or ANALYZE’s ring, the leaves left to right. A leaf that a downlink named and is no longer a live leaf when it is read - deleted since, or reused - counts as empty, and a leaf a writer splits off meanwhile is not in the list: the precision an estimate needs. On a 2,000,000-row test column of 206,679 log-uniformly skewed keys, at a bound of 300 pages over 11,320, three ANALYZEs put the keys at 193,249 to 207,544 and Σn² within 0.2%.

The meta page, and the format

LionMetaNdistinct (lion.h), 528 bytes, right after LionMetaPageData on the meta page at LION_META_NDISTINCT_OFFSET:

valid_cols   uint32   bit i - 1: key column i has a count
bulkdeletes  uint32   the ambulkdelete calls finished, mod 2^32 (§37; `unused` until then)
rows         uint64   the rows the index held when counted (above)
ndistinct    uint64[32]  the VALUE entries of each key column
sumsq        uint64[32]  Σn² of each key column: the square of each such entry's TIDs, summed
                         (below, "The rows of a probed key")

Nothing in it says when it was counted. A build writes exactly the bytes every other build of the same rows writes, whatever its memory, its workers or its snapshot - test/sql/build.sql and buildspill.sql compare the files - and a time would differ; the planner needs the rows it was counted over, which it has, and not the moment.

The reserved words of LionMetaPageData are all spent (§25’s wal_mode, §21’s order record and §32’s summaries took them), and its 56 bytes are asserted not to change: a meta page written by any build since version 4 reads as the others. So the counts do not go inside it; they follow it, and they are there only where pd_lower covers them (LionMetaHasNdistinct()). Every meta page written before this section ends at LionMetaPageData, which is what its pd_lower says, and reads as a meta page without counts - “absent”, which supplies nothing, exactly as a zero wal_mode reads as generic. lion_init_metapage() now sets pd_lower past the area, zeroed (no column valid); a build fills it in; an older index gets it from the first VACUUM or ANALYZE that counts, which moves pd_lower. The array has 32 slots whatever INDEX_MAX_KEYS the server was built with, so that the layout does not depend on it (ordered_cols already caps a lion index at 32 key columns).

The format version does not move, by §14’s test for a bump: does a build that predates the change read the index wrongly? It does not. Nothing before this section reads past LionMetaPageData; the counts are advice to the planner, and an older build that ignores them plans as it always did. The other direction - this build reading an older index - finds no area and supplies nothing. An older build writing an index that has the area leaves it alone in generic mode (GenericXLog diffs the whole page below pd_lower, and pd_lower is the page’s), and in rmgr mode its redo of the META operation sets pd_lower back to the end of LionMetaPageData, which hides the area rather than corrupting it: the next count writes it whole again. A version number that moved would have made every index built from now on unreadable to the builds before, for nothing they could get wrong.

WAL

lion_meta_write_ndistinct() writes the area in a record of its own, through the shim (§25): the meta page locked EXCLUSIVE, lion_wal_begin(), the area copied onto the registered page and pd_lower raised to cover it, a LION_OP_NDISTINCT operation carrying the 528 bytes, and lion_wal_finish(LION_XLOG_META) - the record type §25 had reserved for the meta page and never written. In generic mode the operation is a no-op and GenericXLog’s diff of the image carries the change, pd_lower included; generic redo zeroes what lies between pd_lower and pd_upper, and the area is below pd_lower by then. In rmgr mode redo copies the payload into the area and raises pd_lower the same way. The WAL mode is known without reading the meta page under the lock (lion_index_meta_wal_mode()), because the caller - lionbulkdelete() or lion_vacuum_count_keys() - built the index state first. (Since §37’s 2026-10-02 review the area also carries bulkdeletes, which every lionbulkdelete() call adds one to at its end in this same record - lion_meta_count_bulkdelete(), written even when the counts are the page’s own - and which lion_meta_write_ndistinct() takes from the page rather than from its caller, so that an ANALYZE never takes it back.)

Two redo rules changed with it. The META operation of a directory split used to SET pd_lower to the end of LionMetaPageData; on a meta page with the area that would cut it off on the standby while the primary kept it, which wal_consistency_checking compares (pd_lower is header, not masked, and mask_unused_space() masks from the page’s own pd_lower). It now only raises pd_lower to at least that, which is what it always did on a page without the area. A full-page image of the meta page carries the area, because the page is registered REGBUF_STANDARD and the hole is pd_lower..pd_upper. A standby therefore reads the count as the primary wrote it, and the recovery harness checks that it does (standby_build_replay(): the counts of a build, of an ANALYZE and of a VACUUM that deletes on the primary, each read back on the standby through lion_index_stats() and compared with the exact counts, under wal_consistency_checking in make recovery-check-rmgr).

The hook (lion_selfuncs.c)

get_relation_stats_hook is lion’s already, for the endpoint probe’s scope (§28); the count comes through the same hook, and through get_index_stats_hook for an expression. lion_relation_stats() is, in order: the probe scope’s corrected row when a scope covers the column (unchanged), the previous hook when it takes the column, then lion_ndistinct_stats(), then nothing. It supplies, for column attnum of the base relation rte names, the column’s pg_statistic row with stadistinct replaced, and everything else - null fraction, most common values, histogram, correlation - as ANALYZE left it (but for the frequencies of the values the query compares the column with, since 2026-10-23: “The frequencies of the listed values”, below), when all of these hold:

  • pg_lion.enable_index_ndistinct is on (the default; PGC_USERSET);
  • a plain column (attnum > 0) of a table’s own statistics: not rte->inh, which asks for the statistics of an inheritance tree, and not a partitioned table, which has no index storage of its own - one index of one table does not count a tree;
  • core has a row for it: a column never analyzed gets nothing, and the index’s count waits for the first ANALYZE (a count without the rest of the row would be an estimate out of nothing);
  • the user has not set the column’s n_distinct (ALTER TABLE ... ALTER COLUMN ... SET (n_distinct = ...), which ANALYZE puts in the row): get_attribute_options() says so, and the row stands as ANALYZE wrote it;
  • the relation has a lion index - valid and usable (the planner’s own index list), not partial (it counts a part of the table), not hypothetical (there is no meta page to read) - one of whose key columns is exactly that column;
  • whose opclass decides the equality the planner counts by: its strategy 1 is the column type’s default equality operator (TYPECACHE_EQ_OPR), under the column’s collation or, where the two collations differ, two deterministic ones (they have one equality, the bytes'). Every scalar opclass pg_lion ships passes; a multi-key one has no strategy 1 (and no count); a user opclass whose = is coarser than the type’s (a case-folding one, as in test/sql/pushdown.sql) counts classes the planner would not, and is passed over;
  • with a count on its meta page for that column, and a count above zero;
  • and a count the table has not outgrown (stale counts, lion_nd_current()): the rows it was counted over are within a factor of 2, either way, of the rows the planner finds the table to have (rel->tuples, pg_class.reltuples scaled to the table’s current pages). Between counts the value is scaled with the table (below), which is right for a column whose values grow with its rows and wrong for one whose new rows bring no new values - by as much as the table has grown. A factor of 2 bounds that error to the one a table doubling between two ANALYZEs would give ANALYZE’s own fraction, and past it the planner has ANALYZE’s estimate, which the ANALYZEs that could not count the directory have kept up to date.

Of several such indexes the one counted over the rows nearest the table’s now is taken (the least |ln(rows / tuples)|), and of those the largest count: every VACUUM and ANALYZE reaches every index of the table, so the counts differ only where one index was built since, or where one directory is past ANALYZE’s bound and another is not, and the one whose rows match the table’s is the one that has seen its present.

The value. ANALYZE’s own convention (compute_scalar_stats()): a count above a tenth of the rows the index held when it was counted is supplied as the negative fraction -(count / rows), which get_variable_numdistinct() multiplies by the rows the planner finds the table to have when it plans, and any other count as the count. The choice follows the one ANALYZE makes because the question is the same - does the number of values grow with the table? - and because the fraction is what keeps a count useful between two counts: a table that has grown by a tenth since its last count plans a foreign key of distinct-per-row values with a tenth more of them, where a fixed count would have gone stale at exactly the rate autovacuum tolerates. A column of a fixed set of values keeps its number. The fraction’s denominator is the index’s own rows at the same moment - the TIDs of its first scalar column - not pg_class.reltuples, which ANALYZE passes to the cleanup call from the relation’s cache and which may be the previous ANALYZE’s; the one cost of that choice is that dead row versions not yet vacuumed count in the rows, so between VACUUMs a column whose rows were updated in place reads as having somewhat fewer values per row than it has (a unique column without a unique index, ANALYZE’s -1, reads as the live share of its TIDs; one WITH a unique index is -1 whatever the row says, vardata->isunique).

acl_ok is core’s. Whether the user may read every row of the column decides whether a non-leakproof operator may be handed the row’s values (statistic_proc_security_check()), and it is computed from the column’s privileges, the table’s row-level security and any security-barrier view above it. Nothing here decides it: lion_nd_core_lookup() asks core’s own examine_variable() for the column - a Var of the same range table entry, found by its position in root->simple_rte_array, which is where core took the entry it handed the hook - with both of lion’s statistics hooks standing aside for that one lookup (lion_ndistinct_nested, reset on error), and the row it returns is the one supplied, with its acl_ok. As lion_probe_begin() does for the probe’s row. The lookup also checks that core’s row is the column’s own (starelid, staattnum, not stainherit).

Caching. Each index’s meta page is read once per planner run (one pin, a SHARE lock and 528 bytes, lion_nd_index()), and each column’s corrected row is made once per planner run and handed out as is, with a release function that frees nothing: both live in the memory of the planner run, forgotten when it is reset (lion_nd_cache_for(), as lion_probe_cache_for()). The next plan reads the meta page again, so a count written by a build, a REINDEX, a VACUUM or an ANALYZE is what the next plan sees, and there is no relcache state to invalidate: a relcache callback was the alternative, but the count changes without any catalog change (an ANALYZE inside a transaction that rolls back has written it all the same, and a transactional invalidation would be rolled back with it), and backends have ten relcache callback slots in all.

Expressions. An expression a lion index holds gets its count through get_index_stats_hook, where core asks, for an expression it has matched to an index column, whether a hook supplies that column’s statistics (the ones ANALYZE keeps for an expression index): lion_index_expr_stats(), once the hook installed before it has declined, applies the same rules to the index core asks about - lion, not partial, not hypothetical, an expression column, the expression’s type’s equality and collation, a count the table has not outgrown - and takes the row by the same nested lookup, which walks the same indexes to the same one. A lookup that another index’s row stopped at is left to core. Only the index core asks about is considered; it asks in the order of the relation’s index list and stops at the first with statistics.

Tests. test/sql/ndistinct.sql: a column of 200,000 rows, half in ten heavy values and half two rows each of 50,000 others, sampled at statistics target 10 - ANALYZE puts it at about 3,000 values - planned at the exact count after CREATE INDEX, after inserts and an ANALYZE of its (small enough) directory, after deletes and the VACUUM whose ambulkdelete counted them - at a statistics target of 1, whose bound the directory is past, so that nothing else could have - and for an expression index. At that target, an ANALYZE and a VACUUM with nothing to delete leave the count as it was. A smaller table of the same shape plans with its count while it grows to 1.9 times the rows counted, and with ANALYZE’s estimate at 2.25 times. ANALYZE’s number with the setting off and with the user’s n_distinct; nothing for a partial index or a multi-key column; and a leaky operator given the most common values exactly when core would give them: to the owner, not to a role without SELECT on the column, with the count or without it, and to that role once it has the column. lion_index_stats() shows the stored counts (ndistinct, NULL where none).

What moved in the rest of the suite, measured by running it with every supplied estimate logged beside ANALYZE’s and the column’s true count at plan time: every supplied estimate is the true count, or nearer it than ANALYZE’s, except for columns planned between a DELETE or UPDATE and the VACUUM after it, where both are off by the dead rows (a spill test’s column 13% over where ANALYZE’s is 9% under, a range test’s 3% over where ANALYZE’s is 3% under, a count test’s both a third over), and ndistinct.sql’s grown table, on purpose. One plan changed: test/sql/partition.sql’s stale-statistics section, whose partitions' indexes count the 20,010 groups ANALYZE has not seen. The planner now expects them all and scans the two partitions under a HashAggregate, where it chose the pushdown under a Finalize HashAggregate expecting ten groups - and the scan takes about half the pushdown’s time there. The section is about the planner that does not know, so the rest of it runs with the setting off.

The rows of a probed key (2026-10-23)

n_distinct is the number of distinct keys, from which core takes the rows a key has on average, (1 - nullfrac) N / n_distinct of a table of N rows. A probe does not meet an average key. Core prices a key the plan does not know when it plans at (1 - nullfrac) / n_distinct of the table (var_eq_non_const()), capped by the most common value’s share: an equality to a Param, a nested loop’s outer row, a correlated subquery’s, a prepared statement’s generic plan. That is the rows of a key drawn at random from the column’s keys. The keys a query probes are not drawn that way. An outer row names the key its own row references, and a set of outer rows draws keys as rows draw them, each key with the chance of its rows. Of keys of n_1 … n_K rows, N = Σn, a key drawn by key has N / K rows on average and one drawn by row has Σn² / N. On a column whose keys are skewed the two are orders of magnitude apart, and every probe of such a key is priced for a sliver of what it reads.

Measured (a private workload, a fact table’s key column into a dimension): a key column of a few rows a key on average, with most of Σn² in the keys of ANALYZE’s MCV list, and a key drawn by row outside them with orders of magnitude more rows than the mean (the formula below). The keys the workload’s queries probed met:

  • from small outer sets, hundreds to thousands of rows each. Core priced each of those probes at the per-key mean, one to two orders of magnitude too few rows; the formula is within a few times above them.
  • from outer sets covering most of the keys, about the per-key mean, as core has it. They are planned as hash joins, which no probe’s rows price (Limits, below).

The statistic. Σn² of each key column is summed in the same pass that counts its n_distinct and over the same entries: the VALUE entries with at least one TID, each adding the square of its TIDs. That pass is one of:

  • at build, lion_build_add_entry();
  • in VACUUM’s ambulkdelete, lion_vac_count(), with the TIDs the entry keeps;
  • in the walk of a small directory at ANALYZE and at a VACUUM that deletes nothing, lion_vacuum_count_keys();
  • or, past that walk’s bound, estimated from a sample of the leaves, lion_vacuum_sample_keys() (above, “Counting a sample”), which reads the leaves of the heavy keys whole where the statistics are of the key’s type.

It is a uint64 that saturates at its maximum (lion_sumsq_add()), which reads as unknown, as zero does. It costs a multiply and an add per entry, in a pass that reads every entry already. Dead TIDs count in it until VACUUM, as they do in the rows.

The meta page. sumsq uint64[32], Σn² of each key column, follows ndistinct in LionMetaNdistinct (above, “The meta page, and the format”). It is written and logged with the counts, in the same LION_OP_NDISTINCT record, under the same lock and the same rule that nothing already there is written again. A sum is known where it is above zero and below the saturated maximum, over a count of some rows (LionMetaSumsqKnown()).

The figure. Of a table of N rows, its null fraction and the rows n_i of each of ANALYZE’s most common values, the keys outside the MCV list hold N (1 - nullfrac) - Σn_i rows and Σn² - Σn_i² of the sum, so a key drawn by row from them has

(Σn² - Σ n_i²) / (N (1 - nullfrac) - Σ n_i)

rows on average, and Σn² / (N (1 - nullfrac)) with no MCV list. Each n_i is the index’s own count of the value’s rows, one lookup each (lion_key_ntids()) for the first 100 values of the list - its heaviest - and ANALYZE’s share f_i N for any past them. Statistics of another type than the index column’s are not looked up, and use the shares throughout; so do counts that add up to more rows than the table’s estimate leaves the column (dead TIDs a VACUUM has yet to take, rows added since the estimate), which would leave the rest none. The shares will not do for the heavy ones: on a column whose few heaviest keys carry most of Σn², a share off by a few percent moves Σ(f_i N)² by more than the rest’s whole sum. On a 2,000,000-row test column of log-uniformly skewed keys, whose keys outside its 100 heaviest meet 197 rows a probe, five ANALYZEs in a row gave the figure from the shares as -311, 194, -700, 248 and 568 - below zero, and so core’s 10, as often as not - and from the index’s counts as 203 to 207. The figure is never below core’s rows per key nor above N; a unique column has Σn² = N and keeps its one row, and a column whose MCV list holds every value has no key outside it and keeps core’s rows. Σn² was summed over the rows the count was taken over: it is scaled as the count is (lion_ndistinct_value()), as the square of the growth for a fixed set of keys (a count of a tenth of the rows or fewer) and as the growth otherwise, and past a factor of 2 it is not used (lion_nd_current()). The index is the one n_distinct’s would be (“The hook”, above), and the column needs ANALYZE’s statistics, whose null fraction and MCV list the figure takes (lion_skew_rows_of(), lion_skew_compute()).

Where it prices. A nested loop’s inner scan, a path of a base relation parameterized by a join clause that equates one of its key columns to the outer row’s value. Once every hook has added its paths (set_rel_pathlist_hook, lion_skew_reprice_params()), each such parameterization whose figure is at least twice core’s rows per key has its rows grown by that factor, up to the relation’s own rows, and so do its paths' rows. A path whose index answers the clause - an index scan with it among its index clauses, a bitmap heap scan with it under its bitmap, a LionBtreeScan whose B-tree has it among its index clauses - reads the rows that clause finds, and its run cost grows by that clause’s factor, up to the rows' own (where the parameterization equates two skewed columns, a path whose index answers only one grows by that one’s), and a bitmap heap scan’s startup by what each index scan under its bitmap reads more by the factor of the clauses it answers; any other path reads what it read before. Below a factor of 2 the column is near enough to uniform that core’s figure stands: the two agree exactly where every key has the same rows. The join’s rows are not touched: they are the outer rows' share of the column’s, which core’s rows per key price right.

A semi or anti join’s probe ends at its key’s first match, no later at a heavy key, and one with no match ends at once where the index answers every join clause. So where the relation is on the inner side of a semi or anti join and is probed by the join’s outer side, a parameterization whose clauses all equate a column to an outer value keeps core’s rows, and each of its paths whose index answers every one of those clauses and every restriction of the relation keeps core’s price for its run (answers exactly: a B-tree clause that is not lossy, since other methods can ask for a recheck at run time); a path that filters one its index does not answer reads the rows of the keys it does answer to find a match, or that there is none, and its run is repriced as above. A bitmap heap scan’s bitmap is whole before its first row, so its index scans are repriced either way. A parameterization with a clause beyond those, which a probe with no match reads every row of its key for, is repriced whole. A probe whose outer values do not all come from the join’s outer side - from another relation of the inner side, say - is an ordinary probe there, and is repriced.

Only parameterized paths are repriced. A correlated subquery’s scan and a generic plan’s Param are restrictions of the relation, not parameterizations, and keep core’s rows; so does every estimate core makes from the column. pg_lion.enable_skew_estimate (on) turns it off. lion_debug_skew_rows(text) plans a query and returns, for each column whose figure a probe’s pricing computed, repriced or not, the figure and core’s rows per key; lion_debug_key_skew(regclass) returns each key column’s rows, ndistinct and sumsq from the meta page (sumsq NULL where it is unknown). Both are C functions for the tests, superuser only, that the extension’s SQL does not create (test/sql/skew.sql).

Limits

  • The count is as fresh as the last build, VACUUM that deleted, or ANALYZE of a directory within its bound: inserts of new values in between are seen through the fraction’s scaling, not counted, and past a factor of 2 in the table’s rows the count is not used at all.
  • Inheritance trees and partitioned tables are left to ANALYZE’s estimate of the tree; their partitions and children, planned as relations of their own, get their indexes' counts.
  • A column indexed only as a later key column of a multicolumn lion index is counted like the first: each key column’s run of entries is its own (§24). Its directory is the whole index’s, so the bound is met sooner than for an index of the column alone.
  • Dead row versions count until VACUUM (above).
  • An index whose meta page predates this section supplies nothing until a build, a VACUUM that deletes, or an ANALYZE within the bound has counted it.

The frequencies of the listed values (2026-10-23)

Serving the exact count made one estimate worse. Core estimates col = c for a value that is not among ANALYZE’s most common values as an equal share of the rows those leave: the rest of the rows over the rest of the distinct values (var_eq_const()), no more than the least common listed value. col IN (...) and col = ANY (array) add that up value by value (scalararraysel()), and an OR of equalities on one column is estimated as the IN list it spells (§29.11). On a skewed column - a few values with thousands of rows, most with a handful - ANALYZE’s list holds some of the heavy values, a statistics target’s worth, and a heavy value it left out is estimated at a rare value’s share, which the exact count, larger than ANALYZE’s estimate of it, makes smaller still. In test/sql/index_mcv.sql a value with 1,750 of 30,000 rows is planned at 7, an IN list of five heavy values (11,250 rows) at 34, and the planner reads them through an index scan and filters, joins or groups them, where with their rows it counts them from the index (LionCount, with its FK-side join or Top K).

A lion index on the column knows each value’s rows: the header of the value’s entry counts its TIDs. So the row that lion_relation_stats() serves for a plain column carries the values the relation’s clauses compare the column with among its most common values, at their exact frequencies, and core’s estimate is then each value’s rows.

What is served. For each value, the TIDs of its entry (lion_key_ntids(), one descent of the directory under a SHARE lock, no predicate lock - planning reads no rows for the transaction - and no pin kept) over the rows the planner finds the table to have (rel->tuples), 0 for a value the index has no entry for; core multiplies the frequency back by the same rows, so its estimate is the entry’s TIDs. A value ANALYZE’s list has already gets the exact frequency in place of the sampled one, a value it does not have is added, and the list is sorted again by frequency, the most common first, as ANALYZE leaves it: var_eq_non_const() and the hash join’s bucket estimate read the first entry as the most common, and eqjoinsel_semi() cuts the list from its end. ANALYZE’s other entries stay, and so do the histogram, the correlation, every other slot, the null fraction and stadistinct (the index’s count, as served above, or ANALYZE’s). A column whose row has no most common values - every value sampled once - gets a new slot, the first free one, with the type’s equality and the column’s collation; a row with no free slot is served without the frequencies.

Scaled to fit. The values' frequencies and the null fraction are shares of the same rows, and core counts on their adding up to no more than all of them: past it, scalararraysel() takes the values of an IN list to be independent of each other rather than exclusive. An entry’s TIDs count its dead row versions until VACUUM, and ANALYZE’s other entries are a sample’s, so where the merged list adds up to more than the rows that are not NULL, every entry is scaled down by the same factor to fit - by a millionth more, as the sum core makes of the float4 frequencies may round up. Without it, test/sql/countresid.sql’s list of every value of a column, on a table with dead versions of some 6% of its rows, added up to 1.05, came out at two thirds of the rows, and turned the plan from the sequential scan to a bitmap scan of the whole table. A list shorter than the column’s values still counts its values' dead versions, less what the scaling takes off.

The values are:

  • the constants in the relation’s restrictions (rel->baserestrictinfo) and in its join clauses (rel->joininfo), each clause taken at its top level or inside ANDs and ORs nested to any depth, at every leaf that lion’s count pushdown reads as col = c (either side), col IN (...) or col = ANY (array) for this column (lion_analyze_leaf(), src/lion_plan_match.c): a Const, the elements of a Const array, or the Const elements of an ARRAY[...];
  • compared by the index’s own equality (lion_index_equality_op()), under its collation or a deterministic one where the index’s is deterministic too, on the index’s key type (or one stored the same way, as varchar is under text’s class; an enum is under the class of every enum). NULL elements are skipped: = takes no rows for them.

All or none. Core takes a value outside the list to be no more common than the least common value in it, and a listed value of a few rows, or of none, lowers that bound to almost nothing. So no constant compared with the column by an equality may be left outside the list: a comparison of the column with constants that cannot be taken - a list of more values than the cap, a constant of another type (int4col = 5::bigint is int48eq, not the index’s =), another collation, an expression the executor evaluates and the planner folds to a constant for its estimate (a stable function’s result) - has the column served without the frequencies, for the whole relation. A Param (a generic plan’s $1, or an InitPlan’s) is estimated by var_eq_non_const(), whose bound is the first entry’s frequency, not the last’s, and is left as it was, beside the constants of the same list: kind IN (1, 2, $1) has the frequencies of 1 and 2 and the share of the distinct values for $1. A custom plan’s value is a constant, and gets its frequency. <>, NOT IN and IS DISTINCT FROM are not looked up and do not refuse the column: their estimates are one minus the equality’s.

The cap. At most LION_MCV_MAX_KEYS distinct values of one column of one relation, 1,000 (LION_MAX_ARRAY_ELEMS, §15). A column whose clauses name more is served as before. The bound is on two costs: one descent per value, and core’s estimate of each listed value, which walks the list once per value with the clause’s operator - its time grows with the square of the list. Measured on the PostgreSQL 17 packages, best of five, planning SELECT * (a count, a join) of the test’s table: an IN list of 228 values 0.88 ms (1.19, 0.90) with the frequencies, 0.11 ms (0.21, 0.29) without; of 1,000 values 5.4 ms (9.4, 8.9) against 0.46 ms (1.0, 1.2). Building the row takes 0.22 ms of that at 228 values and 1.1 ms at 1,000: the rest is core’s walk of the longer list.

The cache. Both live in the cache of the planner run that serves the count (lion_nd_cache_for(), forgotten when the run’s memory is reset):

  • each served row, per relation of the query (its RelOptInfo) and column: made the first time the column is asked about - when the relation’s size is estimated - and handed out as it is for every estimate after, restrictions, joins and GROUP BY alike, with the same release function that frees nothing; made again when the relation’s clauses have changed in number since. A self-join’s two scans of the table are two relations with two rows, each with its own values;
  • each value’s TIDs, per index, key column and the hash of the value’s bytes, with the value to settle a collision (cache->mcvkeys, a dynahash in the run’s memory): a value is looked up once per planner run however many relations or subqueries name it in that column.

The values are collected and the row merged in a memory context of their own, deleted before the hook returns; only the finished row is copied into the run’s memory.

The switch. pg_lion.enable_index_mcv (on; PGC_USERSET), beside pg_lion.enable_index_ndistinct and independent of it: off, the row is served without the frequencies (with the count while that is on); on with the count off, the frequencies go into core’s row as ANALYZE wrote it. Both off, the hook supplies nothing.

The probe scope. The endpoint probe’s scope (§28) makes its row through core’s examine_variable(), which comes through this hook: the scope’s corrected histogram ends sit in a copy of the row served here, frequencies and all, so the two corrections are merged, and the scope’s row still comes first while it lasts.

Who may see them. The frequencies count the rows a row-level policy or a security-barrier view would hide, and EXPLAIN shows the estimates they make: they are served only where core’s acl_ok says the user may read every row of the column, as the intersection probe asks too (lion_isect_readable()). Elsewhere the row is the one served without them.

What is not done.

  • Expression columns (get_index_stats_hook, lion_index_expr_stats()): their count is served, not their values' frequencies.
  • A partial lion index (it counts a part of the table), a hypothetical one, a multi-key opclass (its entries are elements, not values) and a user opclass whose = is not the type’s: the indexes the count is not taken from either.
  • Inheritance trees and partitioned tables, as for the count; each partition is planned with its own index’s frequencies.
  • Values of a join: a join clause between two columns names no constant. Core’s join estimate reads the row, and so the listed values with their exact frequencies.
  • An entry’s TIDs include its dead row versions until VACUUM, as the count does. Where they make the merged frequencies add up to more than the share of rows that are not NULL, every entry is scaled down by the same factor (“Scaled to fit”, above).
  • Nothing while PostgreSQL 16’s old_snapshot_threshold is set, when no lion index may be read (§9): the column is served without the frequencies.
  • Nothing is stored: pg_stats shows ANALYZE’s list.

Tests. test/sql/index_mcv.sql: a table of 30,000 rows in no order, eight values of 1,750 to 3,500 rows and 3,000 of one to five, analyzed whole at statistics target 3 for the column, so its list is three of the heavy values and every number is the same at every run. Each estimate is shown with the switch on and off beside the rows the query returns, from EXPLAIN (FORMAT JSON): = c for a heavy value in the list and out of it, a rare one and an absent one; IN lists mixing them, = ANY of an array constant and of ARRAY[...], NULL and repeated elements, two lists of one column; ORs of equalities, with another column, with ANDs in their arms; the second column of the two-column index; a generic plan’s Param alone and beside constants, and the custom plan; a self-join; the cap at 1,000 and 1,001 values and two lists of 600; a cross-type constant and a <>; the plan flips (a bitmap scan, LionCount under the join to its FK side and with Top K, in place of index scans and a sort or a merge join) with the same answers; a partial index and an expression index served without them; a role under a row-level policy served without them; varchar and an enum; a sampled column whose count is served beside the frequencies; and a table with dead versions of a fifth of its rows, whose list of every value is the table’s rows. The tests around the hook plan as they did: ndistinct, partition, countresid, and the tests of the probes, the count pushdown and the FK joins.

34. GROUP BY three or more indexed columns: the decoded walk (implemented 2026-09-30)

SELECT a, b, c, count(*) FROM t [WHERE ...] GROUP BY a, b, c, up to LION_MAX_GROUPCOLS (8) columns - and two, where it is cheaper than §20’s nested loop - each a column of a scalar lion index - the same one (§24) or several - under the same per-column rules as §20’s two: its own index, collation, grouping equality and value gate.

Why not pairs. §20 counts a pair of groups as the AND of their posting sets: one AND per combination, each over every container key the two sets share. A third column multiplies the ANDs by its values, and each AND is not a word but a walk of two sets across the table - about a microsecond a container key, so a millisecond a combination at 5M rows. That is the work that grows with the product of the columns' values.

What a key knows. At a container key (64 heap blocks) every row of the table is in exactly ONE entry of each scalar column - its value’s, or the column’s NULL entry (§14). So instead of trying every combination, the key’s rows are told apart: each column’s containers there are DECODED - a store per member of value index into an array of the key’s 32,768 row positions - and then the rows the WHERE keeps are visited once, each row’s values read off the arrays and its combination counted. Two passes over the key’s rows, whatever the number of combinations. A throwaway benchmark of one key (three columns, 2,176 rows a key; AVX2 for the ANDs):

values a column combinations ANDs of the combinations decode and count
2 8 15 us 8 us
8 512 80 us 7 us
16 4,096 556 us 8 us
32 32,768 3.8 ms 19 us

Decoding won at every size: the ANDs of array containers first build bitset images, which costs what decoding does, and then do the ANDs on top.

The walk (lion_count_groups_decode(), lion_count_decode.c). One PASS takes, for each column, a set of its values (a CHUNK, below), and counts every combination of them:

  • column 0 DRIVES: its cursors - one per value - stand on a heap by container key, and the walk goes to the smallest key column 0 stands at that the collected WHERE (§10, “The WHERE sets, collected once”) holds, or every key of column 0 without a WHERE. The rows of the pass are column 0’s rows: its values in the pass are the only ones counted;
  • column 0’s containers there are decoded into val0[lo] and a bitset of its rows, which is ANDed with the WHERE’s container (its bitset image);
  • the other columns' cursors, on a second heap, are sought to the key (§22: a step, or a descent past the keys between) and decoded into valc[lo], each store stamped with the key’s GENERATION (genc[lo], 16 bits), so that no array is ever cleared: a stamp of another key is no value here, and a row some column holds no value of in the pass is no row of any combination in it;
  • the visibility map is asked once for the key’s blocks with rows (a 64-bit mask), and each row is counted under its combination’s CODE - v0 * radix1 * ... + vn, radix c being column c’s values in the pass - in the TALLY, or queued for the heap with its code when its block is not all-visible (lion_recheck_visible(), which gives the heap’s answer per TID: the rows come key by key and in row order, so the list is in TID order already).

The tally is an array of every combination while that takes at most a quarter of hash_mem (256k combinations at the default 8 MB), and a simplehash of those with rows past it. A hash table that outgrows hash_mem writes its entries to a temporary file and starts again, so the node’s memory is bounded whatever the planner estimated. The same combination may then come out more than once, which is why the node’s rows are PARTIAL counts under a Finalize HashAggregate, as a partitioned table’s are (§16): core adds them up, and spills to disk itself if the groups do not fit. The hash holds at most 231 entries, whatever hash_mem says (LION_DECODE_TALLY_MAX_ENTS: a simplehash counts its members in a uint32): its bound was hash_mem over twice an entry’s 24 bytes, cast to a uint32, which wrapped past 192 GB - at exactly that it came out 0, and the tally spilled at every new combination, right but at a write a row (2026-10 review). The rows the heap has to decide wait in a list of their TID, combination and answer, 15 bytes a row, which holds at most work_mem of them (§9, step 5); sized by the TID alone and doubled past that, it took two and a half to five times work_mem.

Chunks and passes (LionDecodeRun, lion_exec_count.c). Each column’s values are read off its entry walk (§21) a chunk at a time: column 0’s as many as a batch of groups may pin (lion_count_groups_batch(): the pin budget of §15, work_mem of cursors, at most 256), every other column’s a share of a work_mem of cursors (a page image each for a CHAIN set), and all of them so that a pass’s combinations fit a 60-bit code. The chunks advance like the digits of an odometer, the last column fastest, so that every combination of values is in exactly one pass. The columns are ordered by their estimated distinct values, fewest first, so column 0’s chunk is the one most likely to be whole: the usual GROUP BY - a few dozen values a column - is one pass. Between two rows nothing is pinned: a chunk’s sets are kept unpinned (an INLINE one gives its leaf’s pin up as it is read), each entry walk lets go of its leaf once the chunk is read (lion_entry_scan_pause()), and column 0’s sets are located again for each pass with their pins and released when it ends.

Why it is exact (§9). Every row the walk counts is in exactly one of column 0’s sets, each located for the pass under a pin of its own - an INLINE one keeps its leaf’s pin, a CHAIN one is read by a cursor that pins each page it copies - which is the pinned source a count needs, one per row. The map is asked at a key after every column-0 cursor standing there has copied its container and before any moves past it, lion_count_container()’s order. The other columns and the WHERE copy are read with no pin, and are safe on the terms of lion_sources_collect()’s copies: they only narrow or label a row of a pinned set. A dead TID a pinless column still lists is either gone from column 0’s container, and no row of the pass, or in it - so column 0’s page was read before VACUUM’s ambulkdelete got past it, VACUUM has not set its block all-visible, and the row goes to the heap, which does not show it. A row a pinless column has lost already is dead to every snapshot. A live row cannot be missing from any column: the pass is read after the snapshot was taken, and a visible row was in every index before its transaction committed. A column-0 set located past the pin budget (NOPIN) or read from a copy carries nothing, and its rows go to the heap; so does every row under a row filter (§17). Under SERIALIZABLE the all-visible blocks counted are predicate-locked page by page, as an index-only scan locks them, and each index read is locked as a relation.

The plan. LION_FLAG_DECODE and the GROUPN member of custom_private (member 15, shape 18): the columns in the walk’s order, and their indexes. The target list takes LION_TL_GROUPKEYN(g) and LION_TL_COUNT_GROUPCOLN(g) for the third and later columns. EXPLAIN names every column’s index under Lion Indexes, prints every column as the Group Key, Group Strategy: Decoded, and under ANALYZE Decoded Passes, Decoded Keys, Decoded Rows, Decoded Rows Rechecked, Partial Rows and Tally Spills.

Two columns are §20’s nested loop or the walk: the planner builds both paths, the walk’s on the same terms as three columns' (lion_count_path_decodable()), and add_path() keeps the cheaper. The loop wins where the pairs are few - c4, w, 8 pairs, 25 ms on the data below - and the walk from a few dozen on: c4, c10, 40 pairs, 143 ms as the loop and 51 as the walk; c10, c25, 250 pairs, which the loop was priced out of (a HashAggregate over the table, 792 ms), 51; c25, c100, 2,500 pairs, 871 ms under a HashAggregate and 57 as the walk. pg_lion.enable_decoded_walk off (on by default) leaves two columns to the loop and three or more to the ordinary plan, as before.

Cost (lion_cost_decode_path()): the grouped count of column 0 alone against the WHERE, as lion_cost_count_rel() prices it (its walk, the map, the rechecks), and at each key the WHERE’s rows lie at - nkeys * (1 - exp(-rows / nkeys)) - each column’s containers there (LION_DECODE_CONTAINER_COST, a seek where the keys are few), a store per row a column (LION_DECODE_MEMBER_COST), and each of the WHERE’s rows counted (LION_DECODE_ROW_COST, plus LION_DECODE_ROW_COL_COST a column after the first, plus LION_DECODE_ROW_MISS_COST once the tally’s array is past a core’s cache, or LION_DECODE_HASH_ROW_COST for the hash table), times the passes the estimates make. Fitted to the release build below at 313 to 453 units a millisecond.

Measured (2026-09-30, PostgreSQL 16 as packaged, 5M rows of 120 a page, all-visible, one lion index over c4, c10, c25, c100, w - 4, 10, 25 and 100 values and a boolean - work_mem 64 MB; medians; core is the plan with the pushdown off, a HashAggregate over a sequential or bitmap scan):

GROUP BY decoded core
c4, c10, c25 (1,000 combinations) 77 ms 900 ms
c4, c10, c25 ... WHERE w (30% of the rows) 61 ms 438 ms
c4, c10, c25, c100 (100,000) 179 ms 1,947 ms
c4, c10, c25, c100, w 343 ms 2,851 ms
c4, c10, c25 ... WHERE c100 = 7 (1%) 49 ms 60 ms

and on a table of 200,000 of those rows, 4.7 ms against 41 for three columns. A WHERE of a few thousand rows (c100 = 7 AND c25 = 3) is left to core, 11 ms against the walk’s 46: every key the rows lie at still decodes every column there. The four-column form first took 440 ms, with a hash table for its 100,000 combinations; as an array, 179.

test/sql/group_decode.sql checks three to five columns - the NULL group and count(col) in it, another index’s column, a text column - against a sequential scan with the pushdown off: WHEREs of an equality, an IN list, a NOT, an OR across columns, an IS NULL, a range and one that selects nothing; a HAVING and an ORDER BY above the Finalize Agg; a work_mem of 64 kB (fourteen passes); a generic plan; a rescan per outer row; a cursor; SERIALIZABLE; a heap the map no longer vouches for; and the shapes it declines.

A range in the WHERE (2026-10-11)

SELECT a, b, c, count(*) FROM t WHERE k < 5000 GROUP BY a, b, c: with two grouping columns or more no range bounds a walk (§28), so every range of the WHERE is a source (§32), and the walk reads it as part of the WHERE’s collected copy. A source of that copy is never WALKED: a range too large for the memory is collected in windows into a temporary file, as an OR’s leaf is (lion_locate_where() asks lion_locate_range() for a walkable range only where there is no decoded walk), and the copy is then made of it with the WHERE’s other sources, spilling past the memory itself. It is exact on the copy’s terms above: the range is pinless and only narrows the rows of column 0’s pinned sets. The price is the range’s walk and union once, as §32 prices a range that fits, and past the memory a walk for each window - two for each hash table’s memory the union takes - and the file written and read once (lion_cost_range_source(), LION_RANGESRC_SPILL), where a count beside no decoded walk would walk the range at every count. A range on a grouping column is the same: a source, which leaves that column’s NULL group out of the copy.

5M rows, three grouping columns of 2, 20 and 200 values, a range on a fourth column of 20,000 values, against the plan with the pushdown off (a HashAggregate over a bitmap scan): 25% of the rows, 361 ms to 153 ms, and 167 ms with work_mem at 4 MB; 75%, 606 ms to 307 ms; 2.5%, 95 ms to 84 ms; and 0.1%, which the bitmap scan wins, stays core’s. Two grouping columns of 20 and 200 under the 25% range, which §20’s nested loop took at 344 ms, take the walk in 134. test/sql/group_decode.sql compares ranges against the sequential plan on a heap with rows updated across a range’s bounds, deleted and inserted since its VACUUM - a range outside the grouping and on a grouping column with NULLs, beside another clause and another range, as an OR’s leaf, empty and contradicted, under a work_mem of 64 kB - and a range collected into a file.

Not done in this version

  • A partitioned table: each partition would be a relation of its own, as §16’s partial rows already are; the planner declines it.
  • Parallel: the walk is container-major, so it divides by ranges of container keys exactly as §10’s parallel GROUP BY does, under the Finalize Agg it already has.
  • A hot standby, where no WHERE is collected (§10): the planner declines it there.

35. <>, a count of every row, and a filter on a walked column (implemented 2026-09-30)

ClickBench on a 5% sample (5M rows, PostgreSQL 18, the benchmark’s own settings) found three gaps. Twenty of its 43 queries filter on col <> c, which no lion access answered: not the count, not a bitmap or plain scan, not LionOrdered’s set. count(*) with no WHERE was left to core. And WHERE SearchPhrase <> '' ORDER BY SearchPhrase LIMIT 10, walked by the column’s own lion index (§30.11), fetched and rejected the 87% of the table whose phrase is '' before its first row: 3.1 s against the sequential scan’s 1.1. A fourth, the price of a partial btree’s walk, is §30.3’s.

The operator: strategy 10, a range with a hole

Strategy 10 is <>, added beside the range comparisons (§28) of every class and type pair that has them, citext’s included: 36 operators in the install script, one in pg_lion_citext’s. lionvalidate() takes it where it takes a range, beside support function 4 for the same pair (LION_STRAT_IS_WALK()). k <> c is every VALUE entry of the column but c’s, which is a walk of the column with a HOLE: a LionRange bound of strategy LION_STRAT_NE that fails the one entry whose key compares equal to c and bounds the walk on neither side. So:

  • lion_range_test() skips the hole’s entry and never ends a walk on it; the descents to a lower or an upper bound, and the walk ABOVE a range, pass holes by (lion_range_side_leaf(), lion_range_fails_upper()); a DESCENDING walk ends at the first entry that fails a lower bound, and passes over one skipped for anything else (lion_range_fails_lower(), §30.11 - it ended at any entry that did not fail an upper bound, which a hole would have made it do early).
  • Summaries (§32) are not read by a walk with holes (lion_entry_scan_plan_sum()): a bucket’s summary may hold the hole’s rows. Nor is the complement (§28) raced: the entries outside a range with holes are not what it subtracts.
  • Scans: a <> scan key is a range key (lion_scankey_is_range()), so everything the bitmap and the plain scan do with a range - a WALK or a WINDOW beside other columns' sets, one walk for every range key of the column, dropped and rechecked beside the column’s own sets - they do with it. The NULL entry is never walked: <> is strict.
  • k <> ANY (array): core hands the AM an op ANY (array) for every operator of the family. It is true of a row whose k differs from SOME element - every row with a value once two non-NULL elements differ, every row but one value’s with one, no row with none - so the scan rewrites the key as the one it means before anything classifies it (lion_scan_keys_rewrite()): IS NOT NULL, the <> of the one element, or a <> of NULL. The elements are told apart with the probe’s comparison of their type among themselves, under the column’s collation, where two values it calls equal are one value to the index too ('AB' and 'ab' under citext). Where the probe has none - a column in hash order (§21), an element type the family orders against the column’s but not against itself - equality tells them apart: the column’s own for its type, the family’s strategy 1 for the element type’s pair (lion_ne_any_equality()).
  • The price (lioncostestimate()): a walk with holes reads the keys of its RANGE - n_distinct times the other bounds' selectivity - less the holes, and no summaries (lion_cost_ranges_entries()). Priced by its selectivity, SearchPhrase <> '' was 13% of 477,418 keys where the walk reads all but one. The rows a run of <> on one column keeps are estimated as the one exclusion it is (§29.11, “A run of <> is one exclusion”).

The count: a negated source

In the pushdown k <> c is LION_CLAUSE_NE, the second NEGATED clause kind beside IS NOT NULL (§14): the rows it rejects are c’s entry and the NULL one, and the source is the union of those two, subtracted at every container key the positive sources have, exactly as IS NOT NULL’s NULL entry is. Either set may be missing - no row has c, the column has no NULLs - and then there is less to subtract; a NULL c (a parameter) makes the count 0. It is taken wherever IS NOT NULL is: beside positive clauses, under a GROUP BY and a count(DISTINCT), a partitioned table’s and an FK-side join’s fact filters, and as an arm of an OR by the rewrite below. The clause’s operator has the column on its left (a commuted c <> k is rewritten), and it is what a candidate is tested with when a multi-key query narrowed nothing and the heap is read (lion_count_scan_filtered()). EXECUTE on the operator’s function is checked as for any WHERE function (§9).

Every row

With nothing in the WHERE that selects rows - count(*) alone, or beside IS NOT NULL and <> clauses only - there was nothing to drive the merge, except §14’s sum over the entries of the first IS NOT NULL column. Now any lion column drives it: the entries of one SCALAR column are disjoint and, the NULL entry included, hold every row of the table, so the sum over them - each less what the negated sources subtract - is the count, visibility and all, as §14’s sum is. The planner takes the column with the fewest entries (lion_count_all_driver(), by n_distinct), a column whose own IS NOT NULL is a clause when it is no dearer, since its NULL entry is then skipped and its clause dropped (§14). A scalar, whole, non-expression index’s column only (lion_find_roaring_index()): a multi-key column’s entries overlap and miss the rows with no keys, a partial index holds only its predicate’s rows - all the count needs once the query implies the predicate, so in an attempt for a partial index (§45) its columns are preferred, and their walk holds the count to its rows. A partitioned table’s candidates are its first live leaf’s, which lion_collect_targets() then asks of every partition.

The plan names the column in a new member of custom_private, LION_PRIV_ALLROWS (shape 19), whichever column it is: the executor never looks for it among the clauses, where the first IS NOT NULL may be an OR’s leaf or a filtered count’s and say nothing of the driver. EXPLAIN prints the driver as (all rows), or (all keys) when it is the column’s own IS NOT NULL that drives. count(*) WHERE k IS NOT NULL over a column of many keys is driven by one of few: lion_null_t (all rows), lion_null_k (k IS NOT NULL) in null.sql.

A summarized driver (§32) hands its summaries to the sum, and they hold no NULL row: its NULL entry is in no bucket. So the sum over every row counts that entry beside the walk, ANDed with the negated sources like any other entry, unless the column’s own IS NOT NULL drops it (lion_sum_walk(), lion_sum_null_entry()); the lookup takes the entry’s pin (§9) and, under SERIALIZABLE, locks the entry or its absence, since the summed walk never reads the leaf it lives on. In a parallel count each range counts the entry at its own keys, as it counts every set (§10), so the ranges add its rows up once. summary.sql counts every row over a summarized driver with NULLs, and countparallel.sql does in parallel.

Complements: IS DISTINCT FROM, NOT of a multi-key query, and an OR

Two more negated kinds join IS NOT NULL and <>, each the union of entries it rejects:

  • k IS DISTINCT FROM c (LION_CLAUSE_DISTINCT, either side) rejects c’s entry ONLY: unlike <> it is true of a NULL k. Its operator is the = it is built on (DistinctExpr), which has to be the index’s strategy 1 for the pair, and EXECUTE on that function is checked (§9). A NULL literal c makes it IS NOT NULL; a NULL parameter rejects the NULL entry.
  • NOT (q) of an EXACT multi-key query (LION_CLAUSE_NOTMULTI: @>, &&, @@ the extractor answers from posting sets alone, with literal keys and no positions or prefixes) rejects q’s rows and the NULL entry, since q over a NULL array or document is NULL. A query the sets only bound - an empty array, a NULL element, a phrase, a run-time value - is declined: its complement would need a recheck. At run time the query is located as §17’s; one that comes out other than exact is an error, not a wrong count.

Both are taken wherever <> is, the count of every row above included. So is §17’s LION_CLAUSE_NOTRESERVED (“Non-empty arrays”), every row outside some of an array column’s empty entries, with its NULL entry or without: cardinality(a) IS DISTINCT FROM 0, cardinality(a) > 0, a <> '{}'.

Complements in an OR. n <> 17 OR n IS NULL, c200 = 17 OR c20 <> 3: an OR whose arms are each all positive or all negated clauses. Let P be the union of the positive arms (a §19 tree, or no row when there are none) and M the INTERSECTION over the complement arms of each arm’s rejected set N, the union of its clauses' rejected entries (an arm a <> 1 AND b <> 2 is true where neither rejects, so its N is rej(a<>1) ∪ rej(b<>2)). The OR is then P ∪ ¬M, and with F the rest of the WHERE:

|F ∩ (P ∪ ¬M)| = |F \ M| + |F ∩ M ∩ P|

two counts the engine already makes: M as a negated source, then M and P as positive ones (lion_compl_count(), lion_exec_count.c). It is exact under NULLs because N is built from what each clause REJECTS, which for <> includes the NULL entry and for IS DISTINCT FROM does not. The second count is skipped when P has no row, or an existence test is answered by the first. A complement arm whose clauses reject nothing (<> of a value no row has) makes M empty and the OR every row; a <> of a NULL parameter is false and drops its arm. With nothing else selecting rows, F is the count of every row’s driver (above).

The OR travels as any other (LION_PRIV_ORS); its leaves are §19’s, never materialized, and each arm’s kind is read off its first leaf (armcompl). lion_locate_or() locates the complement arms' sets as a second source hung off the OR’s (LionCountSource.compl), which nothing but lion_node_count() reads; the count engine refuses one. So everything that reads the WHERE other than as a count declines an OR with a complement arm: the decoded walk (§34), the parallel GROUP BY, an FK-side join (§27), the range probe (§32), the WHERE collection and the sum probe. A complement leaf is no selecting clause of a partition’s bounds (lion_leaf_drops()). At most three such ORs (LION_MAX_COMPL_ORS): each doubles the counts in the worst case. The cap is asked before the OR is noted anywhere, so a fourth is a restriction no posting set answers like any other - residual, or left out where every partition’s bounds imply it - and leaves nothing behind in ors that would make the clauses after it its arms. A complement OR beside a multi-key query that may narrow nothing - one whose value is known only at run time, a literal tsquery with a prefix or a jsonpath (§17, “Prefix lexemes”; §43.2) - is declined where that query may leave the heap scan of lion_count_scan_filtered() as the count, which is one count: with no clause positive whatever its value beside it, and no walk of a GROUP BY (but the decoded walk, §34) or a count(DISTINCT) to find each count’s rows. So is a negated clause on an expression column (§41): the scan tests the negated clauses of a heap column alone (lion_filter_with_negated(), lion_where_scan_untested()). A complement in a filtered count’s condition (§10, “Filtered counts”) is counted by that filter’s run alone, which carries the OR among its items, and answers to the same terms, checked again once the filters are analysed (lion_count_path_filters()).

The price (lion_cost_count_rel()) is the sum of the two counts': the WHERE with the complement ORs left out, as negated clauses are free in the model, and the WHERE with only their positive arms kept, an upper bound on the second count’s sources. Neither is collected once (nocollect). The count of every row is priced as §35’s driver always is, by its entries' members: over a narrow table whose cheapest driver has a few dozen sparse entries it can come out dearer than the sequential scan while running faster.

A filter on the walked column

A lion column’s walk (§30.11) now answers <> on its own column as a hole, which takes the ‘’ entry out of SearchPhrase <> '' ORDER BY SearchPhrase LIMIT 10 without reading it: 13 entries walked and 16 buffers. Any other filter on the walked column alone - s LIKE 'p1%', length(s) > 3, k % 7 = 3, n IS NULL OR n > 4 - is decided once per ENTRY (lo_walk_value_rinfos()): when the index’s stored keys are the rows' own values byte for byte (lion_index_can_emit_value(), the value-representation rule of §10), every row of an entry holds one value, so a clause that reads no other column and is immutable is true of all of them or of none. The first VISIBLE row the walk fetches of an entry decides for the entry; one that fails takes the rest of it with it, unread (lion_order_walk_skip_entry()). EXPLAIN shows the clauses as Filter per Value and ANALYZE counts Lion Keys Filtered; they stay in the node’s filter too.

Deciding on the stored key alone was the first design, and it is wrong twice over. The key may be a DEAD row’s, so a function the query never calls on a visible row would be called on it: 100 / k > 3 with k = 0 only in deleted rows raised a division by zero the ordinary plan does not. Only leakproof functions promise not to, and those are the comparisons the walk’s bounds answer already. And over a type whose equal values may differ - numeric’s 1.0 and 1.00 are one entry - the representative is not every row’s value, which the image rule excludes. Non-volatile is not enough either: a stable function may read how a row stores its value rather than the value, and pg_column_size(s) > 1000, pg_column_compression(s) IS NULL differ between two rows of one value, one compressed and one not, so the first row would decide wrongly for the rest. Hence immutable. The price (lo_cost_walk()): of the entries the filter rejects, the walk reads the TIDs up to the first member and fetches it; of the rest, everything.

Tests

test/sql/notequal.sql: the opclass census and amvalidate(); <> through the bitmap and the plain scan against a sequential scan - commuted, cross-type, nullable, beside a range, a second hole, an equality on the same column and another column’s sets, a column without the value, a summarized column, a multicolumn index, citext, and <> ANY with zero, one and two distinct elements; the pushdown of each shape, GROUP BY and count(DISTINCT), an OR it declines, the plans, a generic plan with a NULL parameter, EXECUTE revoked on int4ne; the driver of a count of every row and the indexes that cannot be one; a dirty heap and a partitioned table; the walk’s hole in both directions, with NULLs first and last, beside a range and a list; and the per-entry filter, the deleted row it must not evaluate 100 / k on, and the numeric column it must not be taken on. test/sql/countcompl.sql: IS DISTINCT FROM and NOT of a multi-key query alone, and complement arms in ORs - beside equalities, ranges and IN lists, under GROUP BY of one and two columns, HAVING, top k and count(DISTINCT) - each against the sequential plan on a table with NULLs, negative values, NULL and empty arrays, and deleted and updated rows; what is declined; generic plans with NULL parameters; a LATERAL rescan; a partitioned table, with a fourth complement OR its bounds imply. node_contracts.sql revokes int4eq under IS DISTINCT FROM, int4ne under a complement arm and ts_match_vq under NOT.

36. The top k of a GROUP BY ordered by its count (implemented 2026-09-30)

ClickBench’s GROUP BY x ORDER BY count(*) DESC LIMIT 10 queries (UserID, SearchPhrase, ClientIP) counted every group to keep ten: 3.4M UserID entries in 1.7 s, where the answer is the ten largest.

The bound

An entry’s header records ntids, the TIDs its posting set holds, kept exact in the record that changes the set (§22, §18; verify() checks it). Every row a snapshot sees has its TID there - a row is in the index before its transaction commits, VACUUM takes out only what no snapshot sees, a TID is one row at most - so ntids BOUNDS the group’s count under any WHERE, which can only take rows away. It is read after the snapshot was taken; an entry inserted later holds no row the snapshot sees, and one VACUUM deletes held none.

The walk (lion_topk_run(), lion_exec_count.c)

One walk of the driving column’s entries - the walk the groups would be counted in, under the same range - reads their headers alone (lion_entry_scan_next_copy(): no set located, no pin kept) and keeps the topkcand largest, with maxout, the largest count it left out: candidates gather in an array of twice that, which is cut back to the largest half whenever it fills, and an entry no larger than the smallest kept is not copied. The candidates are then counted exactly, largest bound first - each looked up again by its stored key, VACUUM having possibly deleted it (it holds no row then), and counted as any group of a walk is, against the WHERE collected once it pays (lion_group_count()). The walk stops once the k-th largest count so far is AT LEAST the next candidate’s bound, or maxout after the last: no group not yet counted can then pass it. When the groups tied with the k-th must all come out - a second ORDER BY key, which may pick any of them, or WITH TIES - it stops only once the k-th count is LARGER. The groups counted, with a row or more, go up; core’s Sort and Limit above the node order and cut them. Running out of candidates short of that means a count left out might still be large enough: the node then counts every group as it would without a k (Top K Walked Whole in EXPLAIN ANALYZE), which is the price of a WHERE whose counts fall far below their entries' - WHERE light in topk.sql, which leaves the largest entries nothing.

Entries the WHERE takes out whole are not candidates: g <> c subtracts c’s entry and the NULL one, g IS NOT NULL the NULL one (§35), so a group of either has no row, and SearchPhrase <> ''’s '' is the largest entry there is - counting it would read 87% of the table for nothing. Its entry is the one whose stored key the clause’s lookup found, one entry to an equality class, so the same bytes (lion_topk_excluded()).

The planner (lion_count_path_topk())

The grouped rel’s paths are read by nothing but the Sort and Limit when the query has no DISTINCT, window function, set-returning function, set operation or grouping sets, so a path that leaves out groups the Limit would cut answers the query. Nor does the path claim the key order a walk of the entries has (lion_count_path_make()): it emits its candidates largest bound first, for that Sort to order by their counts. The shape is one grouping column of one table walked a group at a time - not an IN list driving the groups, a coalesce group, a count(DISTINCT), the decoded walk, a partitioned table’s or a parallel plan’s partial counts - with no HAVING, which could reject the groups the k were counted from. The first ORDER BY key is the group’s count, descending: an aggregate the node answers as the group’s rows (count(*), or count(col) of a column no row of the group has NULL - not count(g) beside g’s NULL group, which is 0 there), sorted by int8’s > of the integer btree family. k is the LIMIT and the OFFSET, both constants, at most 10,000; a LIMIT known only at run time is declined.

topkcand is 2k/s for a WHERE that keeps a share s of the rows, leaving out what it says of g alone: where the rows it keeps are spread evenly over the groups, the k-th count is some s of its bound, and the bounds after it have to fall below that. Never fewer than k + 64, never more than 16,384, and only when that is under half the column’s entries - past that every group is counted anyway. The price (lion_cost_topk_path()): every entry’s header at two operator costs, the leaves of the walk, a lookup per candidate, and the candidates counted as the groups of a walk of that many entries are - their rows the column’s most common values' frequencies, largest first, leaving out the values a clause of the WHERE on g alone rejects (g <> '' passes the '' entry by in the walk, so it is no candidate; only g op constant is evaluated against the values, any other clause keeps them) (lion_topk_rows()) - with their rows rechecked. Nothing comes out before the last is counted.

The plan carries k, the candidates and the tie rule in a new member of custom_private, LION_PRIV_TOPK (shape 20). EXPLAIN prints Top K: 10 by count, 74 candidates (with ties when they must all come out); ANALYZE adds Top K Entries Walked, Top K Groups Counted and, when it happened, Top K Walked Whole. pg_lion.enable_topk turns it off.

Expressions of the grouping column

GROUP BY ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3 was declined: a grouping clause that is not a column. An expression of the grouping columns alone splits none of their groups, when it is immutable and made of nothing but those columns and constants - no aggregate, window function, set-returning function, subquery or parameter (lion_group_dependent()) - and the rows of a group hold the same column values, not merely equal ones: the value rule of §10, so each column it names is one the node prints (groupvalueout). The node emits the columns; lion_plan_custom_path() puts the expression’s columns in custom_scan_tlist, and setrefs.c makes the expression part of the plan’s projection. One table and its finished groups only - not a partitioned table’s, the decoded walk’s or a parallel plan’s partial counts, which a Finalize Agg would group by the expressions again - and the order of the entries is not claimed as the query’s. Under a top k an expression is computed for the groups that come out, so an error only another group’s key would raise (ClientIP + 1 at the type’s maximum) is not raised; the ordinary plan computes it for every row.

The keys of a multi-key column (lion_index_facets(), §7)

The bound holds for a multi-key column’s entries as for a scalar one’s: a row is filed under each distinct key of its value, once, so the TIDs of an entry are at least the visible rows whose value holds its key, under any filter. lion_index_facets() takes the walk above to such a column, where the planner has no grouped rel to offer it (§7, “Facet counts”): the candidates with the largest ntids, counted largest bound first in batches against the filter’s collected copy, then a walk of the whole column when they run out. It always stops on the strict rule - the k-th count LARGER than the next bound - because its output is ordered by count and then key, so a key tied with the k-th may precede it and must be counted. The reserved entries and a positions column’s length key are not candidates. When the filter’s copy is small next to the keys there are to count, the facets are read from its rows instead and the walk is not made.

Measured (ClickBench, 5% sample, PostgreSQL 18 release build, warm)

query before top k groups counted
SearchPhrase ... WHERE SearchPhrase <> '' (Q13) 280 ms 53 ms 10 of 477,418
UserID (Q16) 1,664 ms 160-190 ms 10 of 3,407,288
ClientIP, ClientIP - 1, ... (Q36, a lion index on ClientIP) 4,000 ms 130-150 ms 10 of 2,280,454

What is left is the walk of every header: some 50 ns an entry, most of it copying the leaves.

Tests

test/sql/topk.sql: every answer against a sequential scan. The NULL group largest; ties at the cut under a second ORDER BY key and WITH TIES; OFFSET; the count unprinted; count(col) beside IS NOT NULL; a WHERE on another column, and one that leaves the largest entries nothing (every group counted, and not without a tie-breaker); g <> c and g IS NOT NULL taking entries out; the declines - ascending, ordered by the group, HAVING, DISTINCT, a window function, count(g) beside the NULL group, few groups, a generic plan’s LIMIT, the setting off; deleted rows still in the entries' counts, before and after VACUUM; a rescan under each outer row’s WHERE; expressions of the grouping column, and the volatile one, the one of another column and the one alone it declines.

37. Aggregates over the entries of lion columns, weighted by their rows (implemented 2026-09-30)

ClickBench’s table-wide aggregates - SUM(AdvEngineID), COUNT(*), AVG(ResolutionWidth), ninety SUM(ResolutionWidth + k), AVG(UserID) - read every row where the answer is in the entries: with no WHERE and no GROUP BY, every row of the table is in exactly one entry of a scalar lion column (the NULL one for a NULL), and holds that entry’s key when the keys are the rows' own values. So sum(f(x)) is the sum over x’s entries of f(key) times the entry’s rows, avg that over their total, and min or max the first f(key) in the aggregate’s order among the entries with a row.

What is taken (lion_wagg_classify(), lion_count_path_wagg())

  • The aggregates: sum and avg of an int2, int4 or int8 argument, and every aggregate with a sort operator - min and max of any type, bool_and, bool_or, every - which planagg.c already answers as the first value in that operator’s order. No DISTINCT, ORDER BY or FILTER. The counts of the target list stay what they were: the sum over every row (§35) answers them beside.
  • The argument: an expression of ONE column of the table - immutable, with no parameter, subquery, aggregate, window or set-returning function - since it is evaluated once per entry, on the key. sum(ResolutionWidth + 89) is; sum(s + i) is not.
  • The column: a whole (not partial) scalar lion index on it whose keys are the rows' values - the value rule of §10 (lion_index_can_emit_value()), which numeric’s 1.0 and 1.00 fail - in the column’s own representation (a binary-coercible type of the same width).
  • The query: one table, not partitioned, with no WHERE and no GROUP BY, beside nothing but counts; each column is walked once however many aggregates are taken over it.

The rows of an entry (lion_wagg_run())

An entry’s rows are its visible TIDs. On a heap the visibility map calls all-visible they are its header’s ntids, which the walk reads without locating a set (lion_entry_scan_next_copy()) - and that is decided as a whole, not page by page: every page is all-visible before the first header is read and after the last, the heap has as many pages both times (lion_heap_all_visible()), and no index walked has finished a bulk delete in between - its meta page’s bulkdeletes, read before the first look and after the second, has not moved (lion_wagg_bulkdeletes()).

  • At the first look every TID in the index is a row every snapshot sees: core never marks a page that has a dead item, which is the promise an index-only scan rests on.
  • A change after it - an insert, an update, a delete - is by a transaction this snapshot cannot see: one it sees made its change before the snapshot was taken, so before the first look. The change takes the mark off its page before its TID reaches any index.
  • If that transaction COMMITS, the page cannot be marked again while this snapshot’s xmin holds VACUUM back, and a new page is not marked at all: the second look finds it unmarked.
  • If it does not - an aborted insert or non-HOT update, a subtransaction rolled back inside one that commits, a speculative insert that INSERT ... ON CONFLICT killed - the tuple is dead to every snapshot whatever the horizon, and nothing holds VACUUM back from it. VACUUM prunes it to LP_DEAD, takes its TID out of the index and marks the page again, and all of that may happen after the walk, which holds nothing between two leaves, has read a header that counted the TID: the second look cannot tell. What it cannot do is mark the page before the line pointer is LP_UNUSED, which only VACUUM’s second heap pass makes, after a bulk delete of this index with the TID among its dead ones has RETURNED (§11, “On-access pruning”). That call ended after the header was read: the TID was still in the entry then, so the call either took it out and ended later, or found it gone and began after an earlier call that took it out - later still. Every call counts itself on the meta page as it ends (§11, “Every bulk delete counts itself”), so the count moved between the two reads. Its read takes the meta page’s share lock after the second look, and its write the exclusive lock before VACUUM locks the map page to mark: a reader that sees the mark sees the count.
  • So when the second look finds every page marked and no count has moved, nothing changed in between, and every ntids read counted exactly the rows of the first look, which are this snapshot’s.

Otherwise - a page not marked, at either look, or a count that moved - the aggregates start again and each entry is counted as a group of a walk is (lion_node_count()), the NULL entry included. So is every walk during recovery, where a standby’s snapshot holds nothing back on the primary.

What the walk locks. It reads every row of the table, so it takes the predicate lock a sequential scan takes, PredicateLockRelation() on the heap, on every run and before the first look. Locks on the index pages it reads (§9, “SERIALIZABLE”) would not stand in for it: they would see every insert, since the sum over every row reads a whole index of the table, which every insert reaches, but a DELETE reaches no index. The counted walk’s page locks (§9) saw it; the walk from the entries' counts visits no heap page and took none.

An expression of the key waits for a look. Giving an entry to its aggregates evaluates their arguments on its key. Until a look has vouched for its header, though, its ntids may count the TID of a row this snapshot does not see, inserted after the first look by a transaction that rolls back or commits after the snapshot, and its key may be one no row the snapshot sees has, and an expression evaluated on it can fail where the sequential scan, which never meets the row, does not: sum(100 / k) on an inserted k = 0. So the walk of a column with an aggregate of an expression of its key keeps each entry with a row - its key and ntids - in a batch (LionWBatch), and gives the batch to its aggregates only after a look that passes. Nothing in the argument above is particular to the last header: a look that finds every page marked, as many pages and no count moved says of every header read before it what the second look says of all of them. So when the batch fills work_mem the walk looks there and then, gives the batch its aggregates if the look passes and stops, to count every entry, if it does not; the look after the walk gives the rest. A column whose aggregates all take the bare key adds each entry as it reads it: a sum of keys is 128-bit arithmetic and an extreme of them the column’s own sort operator, neither of which fails on a value of the column’s type. The counted walk needs neither: an entry with no row is given to no aggregate (lion_wagg_add()).

The rows of an aborted insert (2026-10-02 review; the 2026-10-01 soak found it, and test/soak/findings/wagg_aborted_insert.spec on its branch was the reproducer). The section first said of every change what the COMMITS bullet says: “the page cannot be marked again while this snapshot’s xmin holds VACUUM back”. That holds of committed changes only. With the walk parked between two directory leaves at lion-entry-scan-leaf, an insert of one row for every key rolled back, the rest of the leaves read, and VACUUM (INDEX_CLEANUP ON) run before the second look, the heap kept its size, every page was marked again, and SELECT sum(k), max(k % 50) answered 147969514 | 49 for 97980400 | 48 - the keys of every leaf after the first once more, and a k % 50 that no row has (97987880 | 49 in the soak’s reproducer, which let the walk read one leaf before the VACUUM). ROLLBACK TO SAVEPOINT in a transaction that commits did the same; a killed speculative insert leaves the same kind of TID. Two fixes were weighed:

  • The entry’s own visibility under its leaf’s pin - each INLINE entry’s member blocks asked of the map while the leaf is pinned, the interlock of §9. It is the count’s argument and holds without any new state, but it is a visibility-map read per heap block of every entry, which is the cost the header walk exists to avoid, and it cannot answer a CHAIN entry without locating its set: every column of few keys and many rows would be counted outright. Counting each entry, the walk’s fallback, already rests on that interlock.
  • A count of finished bulk deletes on the meta page, taken. It lives in the one word of §33’s area that was spare (LionMetaNdistinct.bulkdeletes, the old unused), so the meta page keeps its size and the format version stays where it is; it rides in the NDISTINCT record that lionbulkdelete() already wrote at its end, so both WAL modes log it, redo copies it, a crash keeps it and a standby has the primary’s number (checked under wal_consistency_checking = 'pg_lion,Generic'); and the fast path pays two reads of the meta page per walked index per execution. A meta page with no area reads as zero, and the first bulk delete writes the area.

The walk counts every entry when the count moved, which is also what happens when an unrelated VACUUM finished a bulk delete during the walk: a slower answer, never a wrong one. On a hot standby nothing changes: the walk counts every entry there already, its snapshot holding nothing back on the primary. test/isolation/wagg_aborted_insert.spec is the reproducer made deterministic - it parks the walk a second time at lion-wagg-walked, between the last header and the second look, so that the VACUUM falls exactly there - with the rollback and the savepoint as its two permutations; without the count’s comparison it fails with the numbers above. keyaggs.sql reads the count off the meta page (pageinspect): 0 after a build, kept by a VACUUM with nothing dead and by ANALYZE, one more for each bulk delete. A killed speculative insert is not among the permutations: the kill needs a conflicting tuple inserted concurrently, which either commits, and keeps its page unmarked while the walk’s snapshot holds VACUUM back, or aborts, and the killer’s retry then inserts its own row and has to abort too.

SERIALIZABLE (2026-10-02 review). The walk from the entries' counts took no predicate lock on the heap: two SERIALIZABLE transactions that each read sum(k) through it and each deleted, by ctid, a row the other’s sum counted both committed, where the sequential scan’s relation lock fails the second with a serialization failure. Inserts were caught, by the index’s own lock, and so were deletes under the counted walk, by its page locks. “What the walk locks” above is the fix; it also makes the counted walk’s page locks redundant, a coarser lock covering them. A standby needs nothing: it refuses SERIALIZABLE. test/isolation/wagg_serializable.spec is the schedule: both commit without the lock.

The key of a row the snapshot does not see (2026-10-02 review). The walk evaluated an aggregate’s argument on each entry’s key as it read the entry. Parked after the first look (lion-wagg-looked), an insert of k = 0 that rolled back, or that committed after the snapshot, made SELECT sum(100 / k), min(1000 / k), max(k) fail with “division by zero” where the sequential scan answers 964 | 0 | 5000: the second look would have sent the walk to counting each entry, where k = 0 has no row, but the error came first (test/isolation/wagg_unseen_key.spec). “An expression of the key waits for a look” above is the fix. A tuplestore of the keys and their rows, read back after the second look, was weighed: it keeps the one look, but spills to a temporary file past work_mem, where the batch takes another look instead - a read of the map’s pages and of each walked index’s meta page.

What the dirty-key walk needs (claude/lion-walkcost-wagg, not merged): its LION_WALK_DIRTY walk takes an entry’s ntids when none of its members lies under a container key that held a page not all-visible at the first look, and keeps the walk when no clean key turned dirty by the second. That is this argument made per container key, and it has the same hole: an aborted insert onto a clean key’s page, pruned and marked again between the two looks, passes both tests (wagg_aborted_insert_dirty.spec on the soak branch). The same comparison closes it - each walked index’s bulkdeletes read before the first look and compared after the second, falling to its LION_WALK_LIVE walk when it moved; that walk asks the map under the leaf’s pin, so it needs nothing. Its entries taken from ntids need the batch of “An expression of the key waits for a look” as this walk’s do.

The results, as core computes them

  • sum(int2), sum(int4): an int8 that core adds up without an overflow check, wrapping; the weighted sum is kept in 128 bits and cut to 64, which is the same sum modulo 264 in any order.
  • sum(int8): a numeric; the 128-bit sum made a numeric exactly (lion_int128_numeric(): in parts of 1018 past int8’s range). No table has rows enough for it to overflow 128 bits.
  • avg: int8_avg() and numeric_poly_avg() alike divide the sum by the count as numerics (numeric_div) - the wrapped int8 sum for int2 and int4, the exact one for int8.
  • min, max and the rest: the aggregate’s sort operator, under its input collation, over each entry’s f(key) - the NULL ones skipped.
  • No rows: every one NULL; the counts 0.

Without 128-bit integers only the extremes are taken.

The plan

The path is LionCount’s sum over every row (§35) with the columns the aggregates are taken over in a new member of custom_private, LION_PRIV_WAGG (shape 21): their attnums, indexes and key columns. lion_plan_custom_path() adds what each aggregate computes and appends its argument to custom_exprs, after the clause values, where setrefs.c points the column at a place in custom_scan_tlist the executor fills with the key (LION_TL_WKEY); the aggregate itself is a LION_TL_WAGG column. With no count in the target list the sum over every row is not made. The price (lion_cost_wagg_path()): each column’s leaves and two operator costs an entry where the table’s allvisfrac is at least 0.999, the grouped count of the column (§10) elsewhere, and the arguments' evaluation. EXPLAIN prints Aggregates Over Keys: <index> (<column>), ...; ANALYZE adds Keys Aggregated, Key Walks From Entry Counts and, when it happened, Key Walks Counted.

Measured (ClickBench, 5% sample, PostgreSQL 18 release build, warm)

query before over keys ClickHouse
SUM(AdvEngineID), COUNT(*), AVG(ResolutionWidth) (Q3) 1,043 ms 8-15 ms 14 ms
AVG(UserID) (Q4) 540 ms ~200 ms 19 ms
ninety SUM(ResolutionWidth + k) (Q30) 2,085 ms 9-14 ms 30 ms

Q4’s 3.4M entries are the walk of every header, as for the top k of §36.

Not done in this version

  • A heap with a few pages not all-visible counts every entry. The proof is all or nothing, so one insert since the last VACUUM sends each column’s walk to the count of each of its entries - cheap for a column of a few thousand keys, a GROUP BY walk’s price for one of millions. The price follows allvisfrac, which only VACUUM and ANALYZE refresh, so a table written to since is still priced as the header walk. Correcting the headers' sums by the dirty pages alone would need each dirty TID’s key, which a dead item no longer has.
  • A WHERE, a GROUP BY, a partitioned table: each entry’s rows would be its count against the WHERE, per group - the walks of §10 with an argument per entry, not yet written. (2026-10-26: for an integer, date, time or timestamp column a lion index slices (§51), §52 answers sum, avg, min and max under a WHERE and per group of a GROUP BY from the slices instead, and with no WHERE competes with this walk on price. The walk stays for every other column and type.)

Tests

test/sql/keyaggs.sql: every answer against a sequential scan. int2, int4 with NULLs, int8 whose sum passes int8’s range, text, date and bool columns and a two-column index’s second column; sums, averages, extremes and bool_and/bool_or/every of the column and of expressions of it, NULL for some keys; a HAVING on the one row; the entries' own counts on an all-visible heap, and each entry counted after a DELETE, beside this transaction’s own insert and delete, and again after VACUUM; an empty table and one of NULLs only; and the declines - a WHERE, a GROUP BY, two columns in one argument, a volatile argument, DISTINCT, FILTER, ORDER BY, a numeric column, the aggregates no sum of keys answers, and a partial index.

test/isolation/wagg_serializable.spec: the write skew of “SERIALIZABLE” above, through deletes under the walk from the entries' counts and under the counted walk, and through inserts.

test/isolation/wagg_unseen_key.spec: the insert of “The key of a row the snapshot does not see”, rolled back and committed, and rolled back again with work_mem at 64kB, where the look the walk takes once about a thousand keys wait finds the page unmarked and stops it. keyaggs.sql takes the walk at 64kB too, through some thirty looks that pass.

38. NARROW containers: a bitmap as wide as the heap’s offsets (format version 8, implemented 2026-10-01, any width 2026-10-02)

A container covers 64 heap blocks of 512 offsets each (§2), so its BITSET is 4104 bytes, one to a container page. But a heap page holds 291 tuples at most, and most tables far fewer: 226 for rows of a few small columns, 185 for three ints, 61 for rows of 132 bytes, the tuple header and the line pointer included. Every word of a BITSET past the highest offset its blocks have is a zero it can never use - five eighths of it at 185 rows a page, seven eighths at 61. A NARROW keeps only the words the offsets need: k words a block, the width k chosen per container, from 1 to 5 at 8K.

The first version (2026-10-01) had one width, two words a block for offsets below 128, which left every table of more than 127 rows a page with BITSETs: those are the narrow rows, whose indexes are the densest. This one makes the width a property of the container, in the header byte that was reserved, and redefines version 8’s NARROW in place: the version number did not change, and a NARROW of the first version - flags 0 - is refused as a width this build does not have (“Format version” below).

Format

A NARROW is container type 5 (LION_CT_NARROW) with the usual 8-byte header, its width k in the flags byte, and a payload of LION_WIDTH_WORDS(k) words, 64 k at 8K: word j of block b of the container is word b * k + j and holds the block’s offsets 64 j .. 64 j + 63, so that member lo is bit (lo >> 9) * 64 k + (lo & 511), for an offset below 64 k. A BITSET is the same layout at the full width, LION_BITSET_WIDTH = 8 words a block, with flags 0, and its bit for lo is lo itself; so the code has one representation, a BITMAP of width k, with BITSET as the full width (lion_container.h, “widths”), and a BITSET of any index ever written keeps working as one.

The widest NARROW is the narrowest bitmap that holds every offset a heap page can have, from a formula rather than a constant, so that every block size has its own:

LION_HEAP_MAX_OFFSET  = (BLCKSZ - 24) / (MAXALIGN(23) + 4)   (MaxHeapTuplesPerPage; asserted)
LION_NARROW_MAX_WIDTH = LION_HEAP_MAX_OFFSET / 64 + 1
BLCKSZ offset bits blocks a container BITSET width most offsets a page NARROW widths NARROW of width k
8K 9 64 8 291 1 .. 5 8 + 512 k bytes
16K 10 32 16 584 1 .. 10 8 + 256 k bytes
32K 11 16 32 1169 1 .. 19 8 + 128 k bytes

At 8K a NARROW is 520, 1032, 1544, 2056 or 2568 bytes - fifteen, seven, five, three or three to a container page where one BITSET fills it - and as large as an ARRAY of LION_WIDTH_ARRAY_CARD(k) = 256 k members. Every payload bit is a member the container may hold, so a damaged NARROW can claim two things wrongly: its cardinality, which is treated as every BITSET’s is (§3, “Damaged containers”), and its width, which sizes it: a width outside 1 .. LION_NARROW_MAX_WIDTH is refused by every reader of a page before it reads the item (“item N on container page N is a NARROW of width N, not one of 1 .. 5”) and reported by verify() (“narrow container width is out of range for the block size”); every function of the library clamps it into range, and the readers check that the item has the bytes the width sizes before any of them reads it.

Rules

  • Who makes one. lion_container_optimize() considers the NARROW of the narrowest width that holds every member (lion_container_min_width(): an ARRAY’s or a RUN’s highest offset, a bitmap’s highest non-zero word of any block) and takes the smallest of the four, ties going to ARRAY, then RUN, then NARROW, then BITSET. A NARROW is always smaller than a BITSET at 8K, so the build never writes a dense container as a BITSET any more; an ARRAY of up to 256 k members and a RUN of up to 128 k - 1 runs (a NARROW’s bytes or fewer) stay what they are. Every container an index gets is optimized so: the build’s, the one an insert writes as a new item - a container key a sparse segment promotes, and a container it adds a member to that it cannot take in place - and the one VACUUM’s re-optimize writes for an item it changed. An insert that adds the member in place (“Growth in place”) keeps the kind, so an ARRAY grows past 256 k members only by its slack, until the next insert that writes it anew makes it a NARROW: under inserts, as at a build, a dense set’s containers are NARROWs, not the ARRAYs of up to 2048 members and BITSETs past them that add() alone would leave. lion_container_optimize_ext(c, false) leaves NARROW out; nothing in the write paths asks for that.
  • Widening. add() of a member at an offset of 64 k or more makes the NARROW the narrowest width that holds it (lion_lo_width(lo)), copying each block’s k words to the start of its new ones; past LION_NARROW_MAX_WIDTH - an offset no heap tuple has, which only a made-up or damaged member can be - a BITSET. That changes the item’s size, so it never happens in place (“Growth in place”).
  • Shrinking. remove(), remove_if() and remove_range() turn a bitmap of width k that has LION_WIDTH_ARRAY_CARD(k) members or fewer into an ARRAY, 256 k for a NARROW and 2048 for a BITSET, as a BITSET always did. They never narrow it: a NARROW whose widest members went is wider than it needs until something optimizes it, which VACUUM does to the items it changed.
  • The set algebra. The raw forms (§15) keep the width of what they build: an AND is an ARRAY if either side is one, a RUN if both are and it fits, else a bitmap of the NARROWER bitmap operand’s width - no member of the result is past it; an ANDNOT is an ARRAY when its left side is one, else a bitmap of the left side’s width; an OR is a bitmap of the WIDER bitmap operand’s width when both are bitmaps, else a BITSET (an ARRAY with an ARRAY whose members fit one aside). So no member of a result is one its width cannot hold, and an operand’s members past it change nothing. An operand of another width is first copied into the result at the result’s width - each block’s first words, and zeros past them (bitmap_rewiden()) - and one kernel pass then makes and counts the result with the operand that has the width. The optimizing forms choose among all four, at the result’s narrowest width. and_cardinality() counts two bitmaps of one width with the AND_COUNT kernel, and of two widths the wider one’s words gathered to the narrower width, the same pass.
  • The kernels (§3, “Whole-bitset kernels”) take a word count, a multiple of 64; each implementation runs the BITSET’s 512 words as the constant-count pass it always was, and every narrower width through one pass that takes the count as it comes, so a NARROW gets the same fused AVX2, POPCNT and portable loops. The loops that go member by member or block by block - extraction, iteration, block masks, re-widening - are compiled for each constant width from 1 to 4 and for the BITSET’s (LION_WIDTH_DISPATCH), and run with the width as it comes for the others, 5 at 8K and more at larger blocks. The RUNS pass counts runs of the words as they lie, where offset 64 k - 1 of one block and offset 0 of the next are adjacent bits but not adjacent members, so below the full width a block that starts with a member where the block before ends with one is one run more (bitmap_count_runs()).

Growth in place

lion_container_inplace_need(c, lo) is the one table that both the in-place insert (lion_insert_container_inplace()) and its redo (LION_OP_CONTAINER_ADD, §25) read: the bytes the item must have for add(c, lo) to stay inside it, or 0 when it would change the representation. A bitmap of width k - a BITSET, or a NARROW - needs its own size when lo’s offset is below 64 k (every lo, for a BITSET), and 0 otherwise, which sends the insert down the general path: the container is copied out, widened, and written back as the item of its new size, exactly as a full ARRAY becomes a BITSET there. The redo asks the same table of the item it replays onto, and a record no item could take in place is a PANIC, so a replay never widens an item past its bytes. The ARRAY and RUN rows are what they were. lion_item_alloc_size() gives a NARROW no slack: a member it holds changes no byte count, and one past its width needs another 512 bytes for every word a block it gains, which no slack an item carries would hold.

Images stay at the full width

The count engine’s images - a container key’s union of many containers (or_into_bitset()), the range union that grows in place (lion_container_or_inplace(), §32), the image a GROUP BY’s groups are tested against (§10) - stay BITSETs of 512 words. A NARROW is ORed into one at its words' places; the image’s width would otherwise have to follow the widest container that ever joins it, which is not known until the last one has, and every consumer of an image (extraction, the block masks, and_image_count(), the recount) would take a width it now takes as a constant. The cost is the words a narrower image would not have: one pass over 512 words rather than 64 k, per image and container key. The unit benchmark, in ns, the minimum of five runs, on the machine of the measurements below:

operation BITSET NARROW(1) (2) (3) (4) (5)
OR into a full image 160 55 77 100 117 139
AND, two of one width, counted 110 20 35 42 65 81
AND, two of one width, raw 138 25 41 50 76 92
optimize (copy and choose) 263 71 94 117 132 171

So a full image costs at most some 100 ns more a container key and operation than one of a NARROW’s width would, against the 3,900 to 18,600 rows a container key holds at 61 to 291 rows a page: a few hundredths of a nanosecond a row. A container read from a page is a NARROW whatever the image is, which is where the bytes are saved.

Format version (lion.h, lion_meta.c)

Version 8 is version 7 plus NARROW items, with or without summaries (summary_cols says, as in version 7), and its NARROW is the one above. Every version since reads and writes the same NARROW. An index of version 14 may hold one wherever a container goes, whoever wrote it - the build, an INSERT or VACUUM - and its meta page records nothing about it: order_flags has no bit for it. Version 13 had one, 0x0004, set when the build had written a NARROW, and VACUUM made NARROWs only in such an index. An older version than 14 is refused with the “not a valid lion index” ERROR and its REINDEX hint (lion.h).

The first version’s NARROWs had flags 0 and two words a block; this build reads flags 0 as no width it has, and refuses the item as above. No build of the first version was released, so nothing reads it in place: a REINDEX under this build rewrites such an index at the widths its heap needs.

verify(), statistics, EXPLAIN, costs

  • lion_index_verify() checks a NARROW’s width (1 .. LION_NARROW_MAX_WIDTH), that the item has the bytes its width sizes, its cardinality against its payload, its members' offsets against the heap block’s line pointers like any container’s (“narrow container member is not a heap tuple offset” for a bit past the block’s last), and that every other container’s flags are 0.
  • lion_index_stats() counts NARROWs of every width in narrow_containers, its last column, and in containers; its signature did not change, and there is no column per width.
  • EXPLAIN has no counter per container kind; Containers Visited counts NARROWs with the rest.
  • The probe heuristics of the expression evaluator (lion_or_probe_pays(), lion_tree_probe_pays(), lion_expr.c) price a NARROW of any width as the BITSET it replaces - a bit test a lookup, and the BITSET’s extract and OR, which a NARROW’s k eighths of the words overstate. No cost constant changed; the planner’s byte estimates (lion_plan_cost.c) still bound a dense container by a BITSET’s 4 kB.

Measured (2026-10-02, assert-enabled PostgreSQL 18.6 at -O1, four shared cores)

The numbers are for their ratios, not their absolute values. Five synthetic tables of 1M rows, (i int, k int2, g int2, pad text) with pads that make 61, 97, 136, 185 and 226 rows a page; k has two values and g twenty, both a hash of the row number, each under a lion index. Before is the first version’s build (NARROWs of width 2 only), after this one, the same rows in two clusters on the same machine.

rows a page heap pages k before k after g, both
61 16,394 5.11 bits/row: 512 NARROW(2), 1 ARRAY, 1 RUN 2.62: 514 NARROW(1) 19.60: 5,140 ARRAY
97 10,310 3.28: 322 NARROW(2), 1 ARRAY, 1 RUN 3.28: the same 19.79: 3,240 ARRAY
136 7,353 15.34: 230 BITSET 3.28: 230 NARROW(3) 18.61: 2,300 ARRAY
185 5,406 11.40: 170 BITSET 2.49: 170 NARROW(3) 19.79: 1,700 ARRAY
226 4,425 9.31: 138 BITSET, 2 ARRAY 3.41: 138 NARROW(4), 2 ARRAY 19.79: 1,400 ARRAY

g’s containers hold a twentieth of a container key’s rows, at most 723 members, which an ARRAY holds in fewer bytes than any bitmap: those indexes did not change. k at 97 rows a page was already the first version’s NARROW(2). Timings, the two builds alternating query by query in eight rounds, each the median of 15 runs after 3 warm-ups, with every scan but the count pushdown disabled (the plan is LionCount in both, every heap page all-visible), in ms, before / after, the median of the eight rounds:

rows a page count(*) WHERE k = 1 WHERE g = 3 k, count(*) GROUP BY k GROUP BY g g, count(*) WHERE k = 1 GROUP BY g WHERE k = 1 AND g = 3
61 0.19 / 0.19 0.21 / 0.20 0.37 / 0.35 3.58 / 3.64 2.46 / 2.39 0.30 / 0.27
97 0.12 / 0.13 0.17 / 0.15 0.23 / 0.23 2.67 / 2.67 2.14 / 2.15 0.25 / 0.25
136 0.20 / 0.11 0.13 / 0.14 0.41 / 0.20 2.22 / 2.24 2.16 / 2.04 0.28 / 0.24
185 0.13 / 0.09 0.11 / 0.11 0.29 / 0.15 1.93 / 1.88 1.83 / 1.94 0.23 / 0.21
226 0.10 / 0.08 0.11 / 0.10 0.24 / 0.13 1.74 / 1.78 1.94 / 1.88 0.19 / 0.19

Where k’s BITSETs became NARROWs of width 3 or 4 (136, 185 and 226 rows a page), its count and its GROUP BY take half the time or less, the words they read being three eighths or half as many; where a NARROW(2) became a NARROW(1) (61), half the bytes of an index that was already small save nothing these timings resolve. Everything that g’s ARRAYs dominate is the same, within the spread of the rounds.

The first version’s own measurement (2026-10-01: 2M rows of 65 to a page, whose BITSETs at 30.93 bits a row became NARROW(2)s at 4.65, against GIN’s 8.91) stands: those are NARROW(2)s still.

Not done

  • VACUUM narrows only the items it changed. A NARROW whose widest members are deleted narrows when VACUUM takes them out, since it re-optimizes the item it rewrites; nothing narrows an item VACUUM has no dead rows in.
  • The cost model prices a NARROW of any width as a BITSET (above).
  • Images stay at the full width (above), and with them the GROUP BY’s test of its groups against a collected WHERE: the WHERE is optimized when it is collected, so it is a NARROW wherever its rows are dense, and is tested through a full image of it; the path that tests a BITSET WHERE as it is now runs only for a WHERE that is a BITSET, which an optimized one never is at 8K.
  • No switch: a build always makes NARROWs where they are smallest; there is no reloption or setting to build an index without them.

Tests

test/unit/container_test.c: for every width k, every transition - add below 64 k and at it, widening to each wider width and past the widest, removes to 256 k + 1 and 256 k members, optimize() from every kind at 256 k - 1, 256 k and 256 k + 1 members and around the RUN that is a NARROW’s size - the set algebra of every type and width pair against the bit-at-a-time reference with the result kinds and widths above, inplace_need() against what add() writes, the kernels forced to each implementation at every width, and random damaged NARROWs of every width and of widths out of range, in exact-size heap buffers under make unit SANITIZE=1. The unit tests also pass built against a PostgreSQL configured for 16K and 32K blocks (widths 1 .. 10 and 1 .. 19).

test/sql/narrow.sql: tables of 35, 61, 97, 136, 185 and 226 rows a page, and one of 272, whose two-value columns are NARROWs of widths 1, 1, 2, 3, 3, 4 and 5; lion_index_stats() per kind, counts, GROUP BY, IN lists and <> over columns of several widths in one query, bitmap and plain scans, lion_index_count() and the count pushdown, every answer against a sequential scan; a table whose keys hold one band of offsets each, so that one index has NARROWs of four widths; inserts within a width, past it (UPDATEs that widen a NARROW from 1 to 4 a width at a time), into new keys, where the ARRAY an insert writes past 256 k members is a NARROW and never a BITSET, and into the slots VACUUM emptied, three rounds of which leave the containers a build of the same rows writes, kind for kind and byte for byte; DELETE and VACUUM that narrow back, make an ARRAY, and narrow the NARROWs the inserts into an index built on an empty table wrote, as a REINDEX does; summaries, whose items widen and narrow back the same way; and lion_index_verify() throughout. test/sql/bitset.sql runs one set of queries over a table of 136 rows a page (NARROW(3)), one of 226 (NARROW(4)) and the same rows inserted under an index built before them, whose inserts write the NARROW(4)s the build does: AND, OR, <>, GROUP BY, bitmap and plain scans, inserts that the bitmaps take in place and that make NARROWs of the ARRAYs they take past a NARROW’s size, and an UPDATE, a DELETE and a VACUUM that shrink them; no index holds a BITSET, and the ones the set algebra makes of unions are its own. test/sql/corrupt_items.sql §6 writes widths 0, 6 and 2 into a NARROW of width 1: readers refuse 0 and 6, verify() reports them, and width 2, which the item has no bytes for, is refused by both as an item too short.

Every test that sized a set to outgrow a 64 kB work_mem lost it, because a dense container of narrow rows went from 4104 bytes to 1544: groupwhere.sql §7, range_cost.sql §2, spill.sql, rangesource.sql §4, summary_scatter.sql §4, ordered.sql §10, fkjoin_perkey.sql §3, fkjoin_semi.sql and both of fkjoin_parallel.sql §6’s copies. Each now has more container keys: most pad their rows to 65 a page, just past the 63 offsets of width 1, which makes a NARROW the largest for the rows it holds (0.25 bytes a row at width 2 against 0.13 at 185 rows a page), and grow their row counts to keep the old margin; groupwhere.sql §7 keeps its rows and has 600,000 of them instead of 240,000; summary_scatter.sql keeps narrow rows and takes a larger prime for its permutation; fkjoin_parallel.sql sets pg_lion.parallel_range_keys so that its copies are still one chunk and three; range_cost.sql runs its 64 kB cases on rcostw, rcost’s rows at some 25 a page over 63 container keys, and fills rcostn’s memory with 39 container keys of v = 0 NARROWs of width 3 where 15 of BITSETs did. The tests that grow a posting set until a leaf splits need leaves their inserts fill, and a dense set is NARROWs, a few to a leaf, that an insert within their width does not grow, whether the build or the inserts wrote them. Most build their index over their rows at posting_fillfactor = 10, which gives every container of more than some 800 bytes a leaf of its own, and then insert rows that go onto the set’s last leaf, or into heap slots that VACUUM emptied in container keys a leaf’s range already covers, until it splits: flat_trees.sql, insert.sql’s lion_mid, walrecords.sql’s lion_wl, count_root_pushdown_race.spec, vacuum_retry_pushdown.spec, vacuum_regrow_pushdown.spec, verify_concurrent.spec, verify_incomplete_split.spec, posting_split_repair.spec, and phases 1d and 1g and the push-down case of phase 3 in test/recovery/run.sh. count_split_race.spec and the split case of test/recovery/run.sh’s phase 3 build full leaves (posting_fillfactor = 100) of ARRAYs of 15 rows a page, which the inserts make NARROWs of twice their bytes. posting_split_repair.spec keeps key 1’s RUNs of five rows in sixty between 97 other keys, so that VACUUM’s filtering grows a RUN into an ARRAY larger than the leaf’s spare bytes. The other expected changes follow from smaller containers and change no answer: a switch to the empty copy one key later (fkjoin_switch.sql), a range summed from its complement (rangesum.sql) or with another mix of summaries (summary.sql), a union probed at keys where it was built (unionprobe.sql), posting trees on fewer blocks (verify.sql, corrupt_walk.sql, gettuple_pause.spec), and the stats of NARROWs where there were BITSETs (scanand.sql, basic.sql). The first version’s changes to treeprobe.sql and fkjoin_adaptive.sql stand, and so does count_split_race.spec’s wider pad.

39. The competitor, and a margin for the pushdown (2026-10-01)

§10’s “The reference” fits every CPU constant of lion’s model at 500 cost units a millisecond, the middle of core’s sequential scans (400 to 700) and index-only scans (415 to 535), so that lion is honest against them - and says what that costs: core’s other plans run at rates of their own, and against them the same 500 is dishonest. A hash aggregate runs at 157 to 208 units a millisecond, a hash join at 57 to 283, a bitmap heap scan anywhere from 70 (TID-bound, §22) to 4,400 (page-bound), a nested loop into a warm btree at 1,057 (§10) to 2,000 (§31). Reproduced on a release PostgreSQL 16, 2M rows of §31’s s shape, 51% of the heap pages all-visible:

plan cost ms units/ms
LionCount, GROUP BY c200 86,261 169 510
sequential scan and HashAggregate 68,232 325 210
LionCount, count(*) WHERE c2 = 1 25,988 59 440
sequential scan and Aggregate 65,494 249 263

The node ran the GROUP BY 1.9 times faster and lost it by 26% on cost: priced in the units of a scan, it competed with a plan priced in units 2.4 times cheaper. And nothing stood between a near tie and the choice: lion_count_path_add() priced the path and called add_path(), which takes the cheaper by its fuzz factor of 1%. The outcomes are not symmetric - where lion wins it wins by 10 to 100 times, where it loses it loses by 1.5 to 13 - and the losses of the last two passes were not constants: ClickBench Q15 (0451360, a clause a partial index’s predicate implies priced twice) and a count(DISTINCT) priced as a walk it did not make were formulas, which no setting of the 37 multipliers of §31 corrects. This section changed nothing in a formula and refit nothing; it corrected the units a lion path is compared in, and gave a site a margin a lion path has to win by - which it first set to 0.8, and which the matrix set back to 1 (“The margin”, “The matrix”).

Without the rates (2026-10-29). The units are gone: a lion custom path is priced at its own price, in the units lion’s constants are fitted in, over the margin. The rates (one for each kind of competitor, the hash aggregate’s of its hashing alone), the skew estimate that priced a probed key’s competitor at the rows such a key meets (§33), and pg_lion.competitor_pricing, the switch that turned them all off (§49), went together. What is left of the competitor is what it reads that lion’s path does not, which no constant of lion’s model can price: its uncached reads (“A nested loop’s cold reads”, “A bitmap heap scan’s cold reads”), within pg_lion.cold_max_discount, and a Memoize’s misses under a LIMIT (“A Memoize under a LIMIT”). A probe of a skewed key is priced at the rows it meets on core’s own parameterized paths instead (§33, “Where it prices”), so that every plan that probes the key, lion’s or core’s, is priced at the same rows. Why:

  • A private workload ran every statement faster warm with the switch off (rates, skew estimate, cold reads and Memoize scale all off). Cold, the switch off lost a statement whose hash join read a table larger than effective_cache_size by a bitmap heap scan, the case the cold reads price - which is why they stay.
  • The rates were constants for plans whose units a millisecond spread four to six times within one kind (a hash aggregate’s 121 to 799, “The matrix”), fitted on an assert build. Each fixed some plans and broke others: the held-out set’s two GROUP BYs under a selective WHERE took a rate of the hashing alone to give back.
  • What they bought, measured: with hashagg_rate at 1 the held-out set’s six whole-table GROUP BYs went to a hash aggregate 1.0 to 2.4 times slower (“Held out”); with hashjoin_rate at 1 a LionProbeJoin under a NOT EXISTS of 150,000 keys at scale went to the hash anti join, 2,021 ms against 879 warm (§49, “On the fixtures' shapes at scale”). Those are now lost, to be won back by pricing the node’s own work, not its competitor’s.
  • In the regression suite, an FK-side count(DISTINCT) GROUP BY went to a hash join, the faster plan (0.47 ms against 0.64), and a count of a position filter at a hundred keys went to a hash join, the slower (3.2 ms against 1.4); so did index_mcv.sql’s FK-side GROUP BY over 11,250 rows on PostgreSQL 16 and 18, while 19 keeps the node (5.8 ms against the node’s 4.5, release build), so that test no longer shows its plan; the bitmap heap scan test’s FK-side join needed a wider fact for its cold reads to tip it; and the count against the AM’s own scan of the same sets is pinned through cpu_operator_cost, the rates having been the one lever that moved the count’s price without the scan’s.

The subsections below that describe the rates, the matrix’s and the held-out set’s measurements of them, are the record of what they measured.

The competitor (lion_plan_units.c)

A lion custom path’s competitor is the cheapest core path of the relation it is added to, found when the planner hook that adds it starts. Its price does not move a lion path’s; what it reads does (its cold reads, a Memoize’s misses under a LIMIT, below), and its kind decides where the margin applies (“The margin”):

  • Which path. The upper rel’s, for LionCount and the FK-side join (create_upper_paths_hook runs after core has added every grouped path); the join rel’s, for LionSemiJoin and LionAntiJoin (set_join_pathlist_hook runs after core’s paths for the join order and join type it is called for, and the ones before it). Of those, the cheapest (fewest disabled nodes, then total cost) that is not parameterized and has no lion custom path anywhere inside it (lion_path_has_lion()): an Agg over a LionOrdered or LionBtreeScan scan, or a hash join over a LionSemiJoin, is lion’s plan as much as core’s. A path is lion’s by its CustomPathMethods, which are the library’s own - every one of LionCount, LionSemiJoin, LionAntiJoin, LionJoinAgg, LionOrdered, LionBtreeScan and LionBm25 - not by the name EXPLAIN prints, which another extension’s path may carry. None - nothing of core’s yet - prices at the reference, and so does a relation whose every path of core’s is disabled (DISABLED, below).
  • Once for each hook (lion_units_pin()). add_path() frees the paths a new path dominates, so the core path the first lion path of a relation was priced against can be gone when the next is priced - the decoded walk after the nested loop of the same GROUP BY (§34), the parallel FK-side join after the serial one, the second way round of a join - and the next would be priced against whatever core had left, of another kind or none: two of lion’s own paths compared in two units. Each hook finds the competitor before it adds its first path and prices every path it adds against that one, and forgets it when it returns (and when it starts, so that an error inside a pricing leaves nothing behind).
  • What kind of plan it is (lion_competitor_kind()): down from its top, through the nodes that pass a scan’s or a join’s rows up (projections, sorts, gathers, materializations, a LIMIT), to the first that says what its units are. A join is its method, under any aggregate: a hash join, a merge join, or a nested loop whose inner side is an index or bitmap scan parameterized by the outer rows (§10’s and §31’s nested loops; one over a materialized inner side is CPU, and prices at the reference). An aggregate that HASHES anywhere above a scan - the Partial HashAggregate under a Finalize Agg included - is a hashed aggregate. A bitmap heap scan is one under a plain or sorted aggregate or none. A plain or sorted aggregate over a sequential, index-only or index scan is an aggregate over a scan; the scan alone is its own kind. An Append is its dearest child’s kind. In an upper rel’s plan, a semi or anti join over a join that is most of its cost is that join’s kind (“The join that reads the dimension”, below). (A hash aggregate over a join was first a hashed aggregate: “What running it changed”.)
  • The price: the path’s own price, as lion’s model makes it (lion_units_price()), times the factor its competitor’s cold reads and Memoize give it, over the margin. Every choice the model makes inside a price - walk or probe, collect or seek, the inside of a range or its complement, the decoded walk or the nested loop - is made before, so that the competitor changes how dear a lion path is and never which of its forms the executor runs.

Only the node’s own price. What core puts above or below a lion node is core’s plan, priced by core: the Finalize Agg over a partitioned count and the Gather and Finalize Agg over a parallel one; the dimension scan an FK-side join reads its rows from and the rows a semi or anti join emits (a cpu_tuple_cost each, as core’s joins charge them); the HAVING the node evaluates (cost_qual_eval(), as cost_agg() charges it).

Nothing to compete with. When every path core has for the relation is disabled (DISABLED: disabled_nodes on PostgreSQL 18, disable_cost before it) - a plan forced with core’s enable_* settings, as most of the regression suite and the matrix’s lion arm force one - or there is none yet, the path is priced at its own price, with no margin.

The margin

pg_lion.pushdown_margin, default 1, no margin, range 0.01 to 1000. Set below 1, a lion custom path’s own price is divided by the margin before the path is offered to add_path(), so that a lion path is chosen only where its price comes to at most that share of the best core plan’s - at 0.8, a near tie within 25% goes to core. It applies to every lion custom path - LionCount (serial, parallel, partitioned, the FK-side join), LionSemiJoin, LionAntiJoin and LionOrdered (startup and total alike) - and to the node’s own price only: core’s nodes around it are not marked up. The AM’s own scans compete inside core’s path machinery on their plain cost, unmarked. This section first made 0.8 the default; the matrix made it 1 (“1, from the matrix”, below).

Divided, not declined. Not adding the path would be cheaper and would leave EXPLAIN’s costs alone, but it decides once, against the paths the relation has at that moment, and two of lion’s paths meet their competitors later:

  • a join rel gets paths of core’s after set_join_pathlist_hook has run - the right semi join, the unique-ified inner join, the same rel reached by another split, the Gather over its partial paths - which a decline made earlier never saw;
  • a path whose worth is its order or its first rows - LionOrdered under ORDER BY ... LIMIT, a GROUP BY’s groups in key order, a semi join under a LIMIT - is not dominated by the cheapest path of its relation at all; it meets its competitor above the relation, in a Sort it saves or a LIMIT’s fraction of a path, long after it is added. Declined against the cheapest total cost it would lose every LIMIT; declined only by dominance, the margin would never apply to it.

A divided price carries the margin into every comparison it is in, whenever and wherever that is made, and a disabled path still loses to an enabled one whatever the margin: core’s enable_* settings disable core’s paths and lion’s alike (“Core’s switches”, below), and pg_lion.enable_* leave lion’s out. What it costs is that a lion path’s cost in EXPLAIN is its price over the margin, where one is set.

Not where it hedges nothing. The margin is a hedge against the error of lion’s price alone; three prices carry no such error, and are offered at no margin whatever it is set to:

  • Nothing to compete with - NONE or DISABLED above: the choice is not between lion and core.
  • The AM’s own scan of a lion index (lion_path_is_lion_scan(): a plain, index-only or bitmap scan of one, or an Append of a partitioned table’s partitions in which such scans cost at least ten times the rest - an empty partition, or one of a page or two, is scanned sequentially at next to no price, and a partition scanned another way at a price that counts keeps the margin - under the nodes that pass its rows up), for a count: that price is lioncostestimate()’s, lion’s model with lion’s errors, and a margin would only tilt a choice between two of lion’s plans - and where that scan reads every set the count reads, the count is priced below it whatever the margin (“The AM’s own scan of the same sets”, below). At 0.8 three counts of the regression suite went from the node to such a scan: corrupt_walk.sql’s, 3.9 ms against the node’s 1.75; indexscan.sql’s clustered value, 0.19 against 0.06; unlogged.sql’s, a tie. LionOrdered keeps its margin against one (lion_units_margin_for()): what it is chosen over is an ordered btree scan or a Sort under a LIMIT, not the cheapest scan of its relation, which under a lion filter often is the AM’s bitmap scan.
  • A query unknown until run time (lion_where_query_unknown()): a generic plan’s tags @> $1 is priced as every row (§17), the node’s own sequential scan - the most the count can cost, not an estimate that may fall short of it. At 0.8 the generic plans of array.sql and countmultikey.sql went to the sequential scan they tie at worst: 3.2 ms against the node’s 0.03 for the value they run with, 3.3 against 2.8 for the worst value, '{}'.

1, from the matrix. The case for 0.8 was this: taking the cheaper of two plans minimizes the expected time when the estimates are exact; when one carries a log-normal error of spread sigma - lion’s, whose fit §10 left at a median residual of 27% - the rule that minimizes it takes that one only below exp(-sigma^2 / 2) of the other’s price, 0.78 to 0.88 for sigma 0.5 to 0.7. That assumes the other price exact, and the matrix (below) says core’s is not: its plans of one kind run at 142 to 795 units a millisecond (an aggregate over a sequential scan), 121 to 799 (a hash aggregate), 167 to 2,258 (a nested loop), as widely spread as lion’s counts' 118 to 1,219; and the ordered btree scan under a LIMIT that LionOrdered competes with ran at 40 to 115, four to twelve times below the reference. Where both prices err alike the correction cancels, and what is left is which way the near ties fall. In the matrix they fall to lion. Seven of its 39 queries have lion’s price, in its competitor’s units, within a third of core’s best; lion was the faster plan in six, in both runs:

query lion’s price over core’s lion’s time over core’s, run 1, run 2 core’s plan
eq.c20k 1.13 0.82, 0.74 aggregate over an index-only scan
eq.c1m 1.17 0.93, 0.66 aggregate over an index-only scan
group.c20k 1.09 0.70, 0.68 sorted aggregate over an index-only scan
range.c20k 0.96 0.77, 0.75 aggregate over an index-only scan
fk.status.kind 0.91 0.85, 0.86 hash join
ord.c200 0.86 0.12, 0.15 ordered index scan under the LIMIT
eq.c20k.in100 0.78 1.31, 1.40 aggregate over an index-only scan

At 0.8 the margin refused the three of these lion was priced below core in and won - range.c20k, fk.status.kind and ord.c200, the last 7 to 8 times faster - and kept the one it loses, priced below 0.8. Those four are all the margin moves, at any value from 0.9 to 0.5, in either run: the three wins go to core at 0.9 or 0.8, and the loss only at 0.6, the three wins long gone by then. 1 had the fewest mispicks and the least time lost of the sweep in both runs (“The matrix”, below). So the default is 1; the setting stays, with the exemptions above, for a site whose own matrix says otherwise. A margin above 1 - lion favoured - would have taken three more of the seven (the first three rows), and the default stays at 1: on 39 queries that would be a fit, not the correction of a known bias. Above 1 the setting is a lever for tests and calibration: at 1000 it hands a lion path every comparison with an enabled core plan of the competitor’s kind, which is how a test forces one over core’s index scans: enable_indexscan cannot switch those off without disabling the lion paths too (below).

Core’s switches (lion_path_core_switches())

Each lion custom path stands in for an index scan, and core’s enable_indexscan, enable_indexonlyscan and enable_bitmapscan are what a user turns off to keep index scans out of a plan. So the first two disable the lion paths that do an index scan’s or an index-only scan’s work, by core’s own mechanism: the path is kept and marked, disabled_nodes incremented on PostgreSQL 18 and later, disable_cost added to its startup and total cost before 18, so that it loses to any enabled path and still wins where every path left is disabled, as core’s do. Removing it instead would leave a query whose every other path is disabled without a plan of lion’s, which core’s disabled paths never do. The mapping follows what the node reads:

  • Index-only (LION_CORE_INDEXONLY, disabled when enable_indexscan or enable_indexonlyscan is off, as core’s index-only scan is): LionCount in every form (serial, parallel, grouped, partitioned, the FK-side join’s count) and LionSemiJoin and LionAntiJoin answer from the index alone - the heap is read only to recheck a row, as an index-only scan reads it for a page not all visible.
  • Index (LION_CORE_INDEXSCAN, disabled when enable_indexscan is off, as core’s plain index scan is): LionOrdered (over a B-tree, or walking a lion or lionvec column), LionBtreeScan and LionBm25 return the table’s rows in an index’s order or by an index’s TIDs. LionOrdered and LionBtreeScan in index-only mode are, as before, not offered with enable_indexonlyscan off (§40.1); their heap mode stays.
  • enable_bitmapscan disables nothing of lion’s: no custom node builds a TID bitmap for a heap scan of core’s, and the AM’s own bitmap scans are core’s paths, which the setting already disables.

The disabled mark goes on the node’s own cost, outside the margin: the price is divided by the margin first, then marked, so that no margin turns a disabled path into an enabled one. pg_lion.enable_* are unchanged and still leave a lion path out altogether. To force a lion path, a test switches off what it competes with that is not an index scan (enable_seqscan, enable_bitmapscan, pg_lion.enable_plain_scan for the AM’s own plain scans) and sets pg_lion.pushdown_margin to 1000 for the core index scans it leaves on.

The AM’s own scan of the same sets (2026-10-19)

A private workload’s count went to an Aggregate over a plain scan of the same lion index, though the count was priced below it: less than 1% below, inside add_path()’s fuzz of 1%, where two paths of the same pathkeys and parameterization are kept by parallel safety first. The Aggregate over the scan is parallel safe; LionCount is not (it counts the whole relation, so it may not run inside a worker), and lost the tie. The scan’s heap side was priced from the conjunction’s estimate, far below its rows (“A sample of the driver’s own rows”, §29.11): a fair estimate would have priced the scan far above the count, and a worse one below it.

Neither the estimate nor the margin is the point: such a scan does what the count does and more. A plain or bitmap scan of a lion index whose index clauses are every restriction of the relation, each read from the index the count reads it from, reads the sets the count reads, ANDed the same way (§29.11, “The AND, as the count makes it”), and then fetches every row it finds from the heap, where the count tests the visibility map and fetches only the rows of pages that are not all-visible. Whatever the rows are estimated at - and both prices take the rows from the same estimate - the count is the cheaper plan. So lion_count_path_add() prices the count below the cheapest such path of the upper rel (lion_count_ownscan_cost()), by LION_COUNT_OWNSCAN_EDGE (1.02): past the fuzz, so that add_path() keeps the count on cost and never comes to the tie. The count’s own price stands where it is lower; DEBUG2 prints both.

Only where the dominance is plain:

  • the count: one table’s plain count - no GROUP BY, DISTINCT, range, filtered count, HAVING, top k, FK-side join, partitions, residual clause, complement or negated clause, or multi-key query known only at run time - its WHERE answered by the sets alone, each restriction of the relation read from one index;
  • the path: an Aggregate, plain and whole (not partial, no HAVING), over projections only, of an unparameterized plain or bitmap scan of the relation - a bitmap’s tree of quals ANDs only, no BitmapOr - whose index paths are all lion’s, with every index clause one of the relation’s restrictions read from the index the count reads it from, and every restriction such a clause: nothing of the WHERE left to a heap filter, which would make the scan read fewer sets than the count;
  • not a parallel plan: a Gather over a parallel bitmap scan divides the heap’s side among the workers, which the serial count does not, and keeps its price; nor a path an enable_* setting disabled, which loses whatever its cost.

test/sql/isectstretch.sql pins it with the count’s own price set to 0.995 and 0.9995 of the scan’s, for counts of a row or none, whose scan fetches next to nothing: a higher cpu_operator_cost brings the two prices together, the count being more operators than the scan (found by bisection). Without the edge the scan’s Aggregate wins both.

Resident index pages: §22’s third rung

§22’s open item, taken: a page of a lion index that a custom path reads is priced as the buffer hit it is when the index is resident, lion_index_page_cost(), a third rung below seq_page_cost.

  • Why only lion’s index. lion_heap_page_cost() argues a page from random_page_cost down to seq_page_cost by residency and stops there, because for a HEAP page the competing plan reads the same page and pays the same convention. Lion’s index is not read by its competitor. A BitmapAnd of btrees is charged CPU per TID for the same rows, a hash aggregate its rows; the node was charged a device read for each page of a small hot index, and per microsecond of real time ran at about thirty times its competitor’s rate on §22’s c20k = 77 AND c200 = 17 AND c2 = 1 (117.6 against the BitmapAnd’s 62.1, for 0.25 ms against 0.88).
  • Residency is core’s measure of what stays cached, as index_pages_fetched() prorates it: effective_cache_size over the query’s table pages (root->total_table_pages) plus the index’s own, at most 1. A page costs device - resident x (device - hit), where device is what it was priced at before - seq_page_cost for a page walked in order, lion_heap_page_cost()’s price for a page a probe touches - and hit is pg_lion.resident_page_cost, 120 cpu_operator_cost (0.3 units): §10’s warm page read in order, 0.6 us against the 2 us that seq_page_cost stands for at 500 units a millisecond (the posting leaves a probe’s seeks cross fit at 0.7, a directory page descended at 0.6 with its search). Never above device: a page core prices lower keeps core’s price.
  • Walked and probed alike, as §22 asked: the driver’s walk of its sets and the other sources' probes of theirs (lion_cost_set_pages()), a GROUP BY’s entry scan and a second column’s (lion_count_rel_entry_scan(), lion_count_rel_pair_cost()), a range’s leaves read again by a parallel walk, the top k’s descents (§36), the aggregates' walk of entry headers (§37), and the decoded walk’s pages (§34).
  • Not here. The one directory leaf of a single key’s lookup keeps random_page_cost, as btcostestimate() charges btree’s leaf and lion’s own scans charge theirs; so do a collected range’s sums and the heap pages of a recheck. The FK-side join’s probed posting pages already are priced as the warm pages they were measured as (fkjoin_probe_page_cost, 270 ns a page, §27). The AM’s own scans (lioncostestimate()) compete with core’s index scans, which charge their own pages as device reads; they keep the convention.

What it moves. Every count’s estimate, by the pages it reads: the selective counts most, whose price is mostly pages (§22’s case comes to about 48 by the arithmetic, against the BitmapAnd’s 62.1) and the large GROUP BYs least, whose price is per-group work. Both refusals §22 named stay refusals: §10’s 20000 groups at 5M rows cost 502,000 against the hash aggregate’s 142,000, their entry pages a few hundred units of it, so 211,000 in the hash aggregate’s units; §20’s 200 x 20 is refused by its pairs' member term, and its two entry scans are a few pages. test/sql/pushdown.sql pins both at 100,000 rows, and they hold, by EXPLAIN on this build at the defaults: the 20000 groups of the table never vacuumed (40.8 ms through the node against 28.0, §10) are 21,187 in lion’s units, 8,898 in the hash aggregate’s, against its 4,642; the 200 x 20 with the decoded walk off (60.7 ms against 36.8, §20) 30,259 and 12,709 against 4,915, and the 20000 x 2 beside it 43,989 and 18,476 against 5,075. Each stays refused while lion’s price is more than 2.4 times the hash aggregate’s (1 over 0.42), and they are 4.6, 6.2 and 8.7 times; the margin at 0.8 made that 1.9.

What running it changed (2026-10-01, the same day)

Everything above was first written without a server, and is given as it stands after the run. Run - the regression suite first, 17 of its 104 tests failing, then the matrix below, then the suite again - it changed in seven places, each with the measurement that moved it. Timings here are the assert build’s (-O1) on a shared 4-core VM, medians of five to six runs: ratios between two plans of one query, not rates.

  • One competitor for each hook (“Which path”, above). Priced against whatever add_path() had left, pushdown.sql’s GROUP BY c20, c2 over 100,000 rows took its nested loop, priced against the hash aggregate at 0.42, over its decoded walk, priced against the sorted aggregate left after the nested loop had freed the hash aggregate, at 1: 7.1 ms against 1.3. Seventeen of the FK tests' parallel joins went serial the same way, in six files.
  • Lion’s own units against a disabled plan (“Not converted at all”).
  • No margin against the AM’s own scan of a lion index, nor for a query unknown until run time (“Not where it hedges nothing”).
  • The price converted whole. The first version converted lion’s CPU terms and left its pages in core’s convention, summing the price with every page divided by the rate (lion_page_scale) and multiplying by the rate after. A rate is a ratio of whole plans' units a millisecond, so that applied it to a part it was not measured on; and it reached inside the price, where lion’s forms read pages and CPU in different shares - priced CPU-only, the decoded walk and the nested loop, a walk and a probe, a collected filter and a sought one changed order with the competitor’s rate, and the executor follows the form the price picks. The matrix’s 39 queries chose the same plans either way (one run of each); the regression suite differed in one forced plan. lion_page_scale, its two macros, lion_units_begin() and lion_units_end() and lioncostestimate()’s wrapper that reset the scale are gone with it.
  • An aggregate over a join is the join’s kind (“What kind of plan it is”). It was a hashed aggregate wherever the aggregate hashed, and rangesource.sql’s GROUP BY over a nested loop into the fk btree - ten dimension rows, the hash aggregate over them a few cost units of the plan - had the node priced at the hash aggregate’s 0.42 against a plan whose time is the nested loop’s, and with the margin at 1 the node was chosen: 4.7 ms against 0.6. A join’s rate was measured with the aggregate over it (§10’s and §31’s are counts over joins), and its rows are what the aggregate is fed.
  • The margin’s default, 0.8 to 1 (“1, from the matrix”).

The matrix (bench/calib/)

§31’s decision matrix, in the tree this time and on synthetic tables only: bench/calib/setup.sql builds §31’s s shape (calib_s: ints of 2, 20, 200, 20,000 and 1M values at random, 200 values in heap order, a skewed column, a 10% NULL one, lion indexes on each, btrees beside three and on (k, id) for ORDER BY) and §27’s FK shape (calib_f over calib_d, 20 rows a key, and over calib_dk, 1,000 keys) at 2M rows by default, optionally with a percentage of calib_s’s rows updated after the VACUUM (--dirty). bench/calib/matrix.py (Python’s standard library and psql) plans each of 39 queries - counts under equality, IN, AND, NULL and skew (eq), ranges (range), GROUP BY, two-column GROUP BY, count(DISTINCT) and the top k (group), FK-side joins and semi and anti joins (fk), ORDER BY ... LIMIT under a lion filter (ordered) - under eleven arms: the planner’s choice, lion forced (every core scan and join method off), lion off, and each core alternative forced (sequential, bitmap, index, hashed and sorted aggregates, the three join methods). Each distinct plan is timed once per round with EXPLAIN (ANALYZE, TIMING OFF), the median of five rounds after a warm-up, interleaved, serial and without JIT. It prints the chosen and fastest plan of each query, the mispicks by §31’s rule (15% and 0.05 ms), each kind of plan’s cost units a millisecond, and for --sweep NAME=V1,.. - by default the margin from 1 to 0.5 - the planner’s choices and mispicks at each value; matrix.py compare A.json B.json sets two runs side by side.

The run. PostgreSQL 18.6, an assert-enabled build at -O1; 2M rows, warm, serial, five rounds; a 4-core VM shared with other work, whose load average ran from 2 to 15 during the runs. The load moves a plan’s time between runs by up to twice - the before run’s GROUP BYs, widest range and FK joins took twice what the same plans took after - but not the ratio of two plans timed in the same rounds, which is what a mispick is. The build after dc7cd29 was the branch’s as it stood at each run; what changed after run 2 changes no plan of the 39 (EXPLAIN of each). The commands:

python3 bench/calib/matrix.py --setup-only --rows 2000000
# dc7cd29's library installed, the server restarted
python3 bench/calib/matrix.py --label before --out before.json
# this branch's library: run 1 had the margin's default at 0.8, run 2 at 1
python3 bench/calib/matrix.py --label after --out after.json
python3 bench/calib/matrix.py --label own --out own.json --set pg_lion.pushdown_margin=1 \
    --set pg_lion.hashagg_rate=1 --set pg_lion.agg_rate=1 --set pg_lion.hashjoin_rate=1 \
    --set pg_lion.mergejoin_rate=1 --set pg_lion.bitmap_rate=1    # settings since removed
python3 bench/calib/matrix.py compare before.json after.json

Six runs: before; after with the margin’s default at 0.8 (run 1), and the same build with the first version’s CPU-only conversion; after with it at 1 (run 2); and own twice, the first under a load of 10 to 15 and used here only for the choices its sweeps of the rates make, which are costs, not times. Each query’s chosen plan and its median time in milliseconds, a mispick in bold, and the fastest plan of run 2 (“own” is the second own run: every rate and the margin at 1):

query before (dc7cd29) after, margin 0.8 after, margin 1 own fastest
eq.c2 LionCount 0.96 LionCount 0.85 LionCount 0.93 LionCount 0.93 LionCount
eq.c20 LionCount 0.39 LionCount 0.38 LionCount 0.36 LionCount 0.39 LionCount
eq.c200 LionCount 0.14 LionCount 0.15 LionCount 0.14 LionCount 0.16 LionCount
eq.c20k agg, IOS 0.06 agg, IOS 0.04 agg, IOS 0.05 agg, IOS 0.06 LionCount
eq.c1m agg, IOS 0.04 agg, IOS 0.04 agg, IOS 0.05 agg, IOS 0.05 LionCount
eq.c200.c20 LionCount 0.33 LionCount 0.34 LionCount 0.34 LionCount 0.33 LionCount
eq.c20k.c200.c2 LionCount 0.07 LionCount 0.07 LionCount 0.08 LionCount 0.08 LionCount
eq.c20k.in100 LionCount 1.28 LionCount 1.30 LionCount 1.32 LionCount 1.37 agg, IOS
eq.c1m.in1000 LionCount 6.54 LionCount 6.76 LionCount 6.97 LionCount 7.74 agg, IOS
eq.skew.hot LionCount 0.71 LionCount 0.84 LionCount 0.90 LionCount 0.82 LionCount
eq.skew.cold LionCount 0.10 LionCount 0.10 LionCount 0.10 LionCount 0.10 LionCount
eq.n10.null LionCount 0.49 LionCount 0.50 LionCount 0.46 LionCount 0.49 LionCount
eq.cl200 LionCount 0.04 LionCount 0.04 LionCount 0.05 LionCount 0.05 LionCount
eq.c200.in2.c2 LionCount 0.91 LionCount 0.97 LionCount 0.93 LionCount 0.87 LionCount
range.cl200 LionCount 0.07 LionCount 0.08 LionCount 0.08 LionCount 0.08 LionCount
range.c20k LionCount 0.50 agg, IOS 0.63 LionCount 0.46 LionCount 0.46 LionCount
range.c20k.c2 LionCount 88.66 LionCount 41.57 LionCount 41.93 LionCount 42.01 LionCount
group.c20 LionCount 14.78 LionCount 6.73 LionCount 6.67 LionCount 6.81 LionCount
group.c200 LionCount 44.94 LionCount 19.36 LionCount 20.51 LionCount 21.32 LionCount
group.c20k agg, IOS 302 agg, IOS 133 agg, IOS 143 agg, IOS 145 LionCount
group.c200.w.c2 LionCount 23.56 LionCount 11.26 LionCount 11.46 LionCount 9.17 LionCount
group.c20k.w.c20 HashAgg 90.15 HashAgg 87.68 HashAgg 73.44 HashAgg 90.10 HashAgg
group.c20.c2 LionCount 17.11 LionCount 17.29 LionCount 21.19 LionCount 21.48 LionCount
group.c200.c20 LionCount 42.40 LionCount 26.79 LionCount 24.31 LionCount 27.15 LionCount
distinct.c20k.w.c200 agg, lion idx scan 26.66 agg, lion idx scan 8.23 agg, lion idx scan 9.30 agg, lion idx scan 8.99 agg, lion idx scan
distinct.c200 LionCount 0.82 LionCount 0.92 LionCount 0.95 LionCount 0.82 LionCount
topk.c20k LionCount 8.74 LionCount 3.87 LionCount 3.99 LionCount 4.07 LionCount
fk.status LionCount 58.80 LionCount 39.43 LionCount 37.59 LionCount 40.02 LionCount
fk.status.country LionCount 2.29 LionCount 2.29 LionCount 2.23 LionCount 2.22 LionCount
fk.status.kind hash join 69.86 hash join 62.29 LionCount 47.59 hash join 56.67 LionCount
fk.group.country HashAgg 81.41 HashAgg 95.84 HashAgg 95.72 HashAgg 97.10 HashAgg
fk.small.grp LionCount 1.69 LionCount 1.90 LionCount 1.83 LionCount 1.81 LionCount
fk.semi.country LionSemiJoin 10.86 LionSemiJoin 6.71 LionSemiJoin 7.59 LionSemiJoin 7.80 LionSemiJoin
fk.semi.count hash join 89.00 hash join 31.30 hash join 33.38 hash join 32.25 merge join
fk.anti.country LionAntiJoin 14.44 LionAntiJoin 7.13 LionAntiJoin 7.32 LionAntiJoin 7.36 LionAntiJoin
ord.c200 LionOrdered 0.51 index scan 4.19 LionOrdered 0.59 LionOrdered 0.54 LionOrdered
ord.c20k lion idx scan 0.25 lion idx scan 0.20 lion idx scan 0.21 lion idx scan 0.20 lion idx scan
ord.c200.c2 LionOrdered 1.75 LionOrdered 1.68 LionOrdered 1.63 LionOrdered 1.50 LionOrdered
ord.c2 index scan 0.42 index scan 0.41 index scan 0.37 index scan 0.42 index scan

At the defaults the matrix chooses as dc7cd29 did in 38 of its 39 queries. The one it changes is fk.status.kind, whose LionCount the hash join’s rate, 0.5, prices at 0.91 of the hash join where lion’s own units priced it at 1.86: 47.6 ms against 59.6, and 47.7 against 62.3 in run 1. Its other mispicks are dc7cd29’s, and none is the rates' or the margin’s:

run mispicks time lost, ms lion’s in them
before, dc7cd29 4 116.7 eq.c20k.in100, eq.c1m.in1000, group.c20k
after, margin 0.8 (run 1) 6 62.4 the same three, range.c20k, fk.status.kind, ord.c200
… the CPU-only conversion 6 71.2 the same six
after, margin 1 (run 2) 4 54.4 eq.c20k.in100, eq.c1m.in1000, group.c20k
own: every rate and the margin 1 5 83.3 the same three, fk.status.kind

The fourth of dc7cd29’s and of run 2’s, fk.semi.count, is a hash join chosen over a merge join it ran 25% and 15% slower than (8% in run 1), and own’s fifth, group.c20k.w.c20, a hash aggregate over one scan chosen over the same over another priced 1.5% dearer: core’s choices between core’s plans, which lion’s paths do not enter. The three that are lion’s are formulas, the kind §31 named and this section does not refit: the two IN lists price the union of 100 and 1,000 sets at 0.78 and 0.60 of the index-only scan’s aggregate and run 1.3 to 1.4 and 2.1 to 2.2 times slower (§31’s eq s.c1m in 1000), and group.c20k’s 20,000 entries are priced 1.09 times the sorted aggregate over an index-only scan and run 1.4 times faster.

The margin’s sweep, mispicks and time lost, in the two runs that swept it:

margin run 1 run 2
1 3, 44.0 ms 4, 54.4 ms
0.9 5, 58.7 ms 6, 66.6 ms
0.8 6, 62.4 ms 7, 69.9 ms
0.7 6, 62.4 ms 7, 69.9 ms
0.6 5, 62.1 ms 6, 69.6 ms
0.5 5, 62.1 ms 6, 69.6 ms

Cost units a millisecond, by the top of each plan, from the own run (lion in its own units):

kind of plan plans median min max median over LionCount’s over 500 its rate
LionCount 33 344 118 1,219
aggregate over a sequential scan 23 244 142 795 0.71 0.49 agg_rate 1
aggregate over an index-only scan 9 396 95 1,089 1.15 0.79 agg_rate 1
hash aggregate 18 212 121 799 0.62 0.42 per plan, from hashagg_rate 0.1
hash join 19 298 175 558 0.87 0.60 hashjoin_rate 0.5
merge join 7 547 213 675 1.59 1.09 mergejoin_rate 1
nested loop 12 516 167 2,258 1.50 1.03 the reference, 1
aggregate over the AM’s bitmap scan 20 870 52 20,242 (lion’s model)
index scan in order, under a LIMIT 4 54 40 115 0.11 (LionOrdered’s competitor)
LionOrdered 6 908 276 4,294
LionSemiJoin, LionAntiJoin 4 869 819 897

The rates stay. The definition above makes a kind’s rate its median over the lion paths' median in such a run, and this run says hash aggregates 0.62, hash joins 0.87, nested loops 1.5, merge joins 1.6 - not the defaults. Two things say not to take them. This build’s lion runs at 344 units a millisecond where §10’s release build ran it at the 500 its constants are fitted at, against core’s scans at 244 to 396 here and 400 to 700 there: assertions and -O1 slow lion’s code and core’s by different factors, so a ratio measured on this build is not a release build’s, and measured over 500 instead the same medians say 0.42, 0.60, 1.03, 1.09. And the matrix’s choices barely depend on them: in the first own run’s sweeps, no choice of the 39 moves with hashagg_rate from 1 to 0.3 or nestloop_rate from 1 to 4, and one moves with hashjoin_rate - fk.status.kind, which 0.5 decides for lion, 1.2 to 1.3 times faster, and 0.87 would give back to the hash join. Refit on a release build, over the near ties rather than the medians, they may move; not on this evidence.

Held out

The rates were fitted on the matrix’s own 39 queries, whose choices barely depend on them (above). bench/calib/heldout.sql and heldout.py are 41 queries the model was never fitted to, on a different schema: 3M rows with 10% dirtied, a column of geometric skew, 24 runs of uneven length in heap order, a 30% NULL column, 300 text tags, and dimensions of 64, 300,000 and 1.5M keys. A release build of PostgreSQL 18, five repetitions, the margin at 1; a mispick is a chosen plan 15% and 0.05 ms slower than the fastest of the matrix’s arms.

python3 bench/calib/matrix.py --setup-only --rows 3000000 --dirty 10 --setup-file bench/calib/heldout.sql
python3 bench/calib/matrix.py --queries bench/calib/heldout.py --sweep pg_lion.pushdown_margin=1 \
    --repeats 5 --label default --out ho_default.json
rates calibration set (39): mispicks, time lost held out (41): mispicks, time lost held-out plans changed
hash aggregate 0.42, hash join 0.5, nested loop 2 3, 81 ms of 514 10, 509 ms of 4,549
all three at 1 3, 76 ms of 539 13, 1,182 ms of 5,254 9
only hashagg_rate at 1 13, 1,188 ms 8 GROUP BYs
only hashjoin_rate at 1 10, 562 ms 1, h.fk.semi.d10
only the nested loop’s at 1 10, 557 ms none

Differences of about 50 ms in time lost with no plan changed are the fastest arm’s noise.

  • hashagg_rate carries most of the layer: at 1, six GROUP BYs over the whole table go to a hash aggregate 1.0 to 2.4 times slower (h.group.c5 261 ms to 461, h.group.c50.c5 197 to 476). The calibration set cannot show it: none of its GROUP BYs sits near the boundary.
  • hashjoin_rate decides two near ties, fk.status.kind in the calibration set and h.fk.semi.d10. Timed seven times each on their own, the node is the faster in both (47 ms against 58; 52 against 73), though one matrix run had the hash join ahead in the second. It stays.
  • The nested loop’s rate moved nothing, and is gone.

What the held-out set found that no constant rate can fix: two GROUP BYs under a selective WHERE (h.group.tag.w.c50, h.group.c1k.w.geo) where the node, chosen at any of these rates, takes 148 and 256 ms against a hash aggregate’s 46 and 85.

A nested loop’s cold reads (2026-10-09)

A nested loop into a parameterized index scan is priced by core at random_page_cost for each page index_pages_fetched() says its probes fetch. On a private workload that price was right warm and far off cold: a semi join probing a table several times effective_cache_size was a little faster than the FK-side join warm and orders of magnitude slower cold, and the two were priced alike. A cold page costs a probe a synchronous read - nothing reads ahead of a parameterized scan - which is tens of times what random_page_cost stands for, set low as it is for an SSD.

So against a nested loop whose inner side is one parameterized index scan, under a Material, a projection or a Memoize (lion_nestloop_cold(), lion_plan_units.c):

  • its cold reads are the pages core counts it fetching, a leaf a probe and the heap pages of the rows a probe fetches (index_pages_fetched() over the loops, as core counts them; over a Memoize’s misses where one sits over the inner scan), in the share the cache cannot hold: one less the residency lion_index_page_cost() argues from, effective_cache_size over the query’s tables and the index. Each is charged pg_lion.cold_page_cost, 50 seq_page_cost (100 us at the reference: a local SSD’s synchronous read with the kernel’s share), less the random_page_cost core charged it. In an upper rel’s plan the loop is the one below a semi or anti join over it (“The join that reads the dimension”, below);
  • the FK-side join’s own cold reads are counted the same way (lion_cost_fkjoin_path()’s reads): the directory leaves and fk sets of its lookups, the fact filters it locates, the heap of a recheck;
  • the path is priced with both added (lion_units_cold()): (own x rate + its cold reads) over (the loop’s cost + the loop’s cold reads), of the loop’s cost - a factor of the rate, each path of the relation with its own reads. The loop’s cost is less its outer side where the path reads that side as its child at core’s price (LionProbeJoin’s fact: “The join that reads the dimension”). It is at most 1, so a path that reads more than the loop keeps its price, and at least 1 / pg_lion.cold_max_discount, 2 by default (“The cap” below).

The floor is what this costs warm. With the inner table’s working set in the cache the loop runs at core’s price, and a lion path chosen over it can take up to pg_lion.cold_max_discount times its time (2 by default). Nothing changes where the tables and the index fit in effective_cache_size, which every table of the regression suite does but those of the tests that set it lower (fkjoin_cold.sql, competitor_cold.sql and others), nor against any other kind of plan. Not counted: the probes a semi join stops early, which the loop’s reads overstate; the count paths other than the FK-side join’s, which compare with the loop at its price.

A bitmap heap scan’s cold reads (2026-10-23)

Core prices a bitmap heap scan’s heap pages at random_page_cost falling to seq_page_cost with the square root of the share of the table one scan reads (cost_bitmap_heap_scan()), as if each page were cached or a device’s cheap random read. On a private workload, over a fact heap several times effective_cache_size, a covering walk of a B-tree on the fact’s key, index-only over the lion set (LionOrdered, §40), was priced at about twice a parallel bitmap heap scan and lost; cold it ran about twice as fast. The walk’s price was right; the scan’s heap pages, which core charged about one unit each, took four times the walk’s rate. They are not a loop’s reads: a bitmap heap scan asks for its pages ahead of itself (effective_io_concurrency), so that the reads overlap one another and the scan’s own work. On the same device a bitmap heap scan’s uncached pages took 33 to 40 us each, where a nested loop’s synchronous probe paid 270 to 300 us: 0.11 to 0.15 of it.

So against a competitor that reads through bitmap heap scans (lion_bitmap_cold(), lion_plan_units.c), and the same way §39’s loop is priced:

  • the competitor is the cheapest core path of the relation the lion path is added to, lion_competitor_path()’s, pinned for the hook’s paths as before - for a count and the FK-side join, lion_units_for()’s, where the loop’s cold reads are not taken already (both are never added); for a base relation’s covering walks, which have no rate, lion_units_scan_cold()’s, the cheaper of that path and the relation’s cheapest unparameterized partial path under the Gather core puts over it (parallel_setup_cost and a parallel_tuple_cost a row, as cost_gather() prices it), taken before the walks are added, since add_path() frees what they dominate. A plain or parallel bitmap heap scan; an aggregate over one; a hash or merge join with one on either side; a nested loop’s parameterized inner one, over the loop’s outer rows; a Material scans its input once, a Memoize once a cache miss, at the hit ratio cost_memoize_rescan() prices it at (lion_memoize_misses()), a subquery’s scan is followed into its plan under the subquery’s own root, and a custom path reads nothing of core’s;
  • its cold reads are each scan’s heap pages as compute_bitmap_pages() counts them, over the loops (loop_count, which pro-rates them, times the loops), and each bitmap leaf’s index pages, its selectivity’s share of the index and a page at least, over the loops as index_pages_fetched() counts them; in the share the cache cannot hold, one less the residency lion_index_page_cost() argues from - effective_cache_size over the query’s tables and the bitmaps' indexes - each charged pg_lion.cold_stream_page_cost less the price core charged it: the heap page the scan’s random_page_cost - (random_page_cost - seq_page_cost) x sqrt(pages / T), the index page random_page_cost;
  • the lion path’s own reads are counted at the same per-page charge (cold_stream_page_cost less the largest scan’s random_page_cost), since they too are read in an order the device or the kernel reads ahead of - posting chains, a B-tree’s leaves in order, a recheck’s heap through a read stream: an index-only covering walk’s B-tree pages as its amcostestimate() counts them, its set’s build (each lion leaf’s share, as the scan’s bitmap) and its heap fetches as priced (lo_cost()’s reads); a count’s directory leaves, posting and summed pages and heap rechecks as its price counts them (LionRangeCost.reads), and the heap pages its residual clauses read, each count’s candidates in block order as compute_bitmap_pages() counts the scan’s (lion_residual_pages()); the FK-side join’s as §39 counts them;
  • the path is priced with both added as against the loop: (own x rate + its cold reads) over (the competitor’s cost + its cold reads), of the competitor’s cost, at most 1 and at least 1 / pg_lion.cold_max_discount - a factor of the rate for a count and the FK-side join (lion_units_bitmap_cold()), of the walk’s own startup and total for a covering walk (lo_cold_price()). A parallel walk’s partial cost is one participant’s share where the scan’s is a whole plan’s, so it is priced under the Gather core puts over it - parallel_setup_cost and a parallel_tuple_cost a row, as lion_units_scan_cold() adds to the scan’s partial path - and the factor taken of that discounts the walk’s own startup and total, never the Gather’s overhead, which core charges only where a Gather takes the walk’s rows (lo_cold_price_partial()). The margin divides the result as it divides any price.

pg_lion.cold_stream_page_cost is 6.25 seq_page_cost, an eighth of cold_page_cost’s 50: the middle of the measured 0.11 to 0.15. The device measured is some three times slower than the reference’s local SSD under both kinds of read (270 to 300 us against cold_page_cost’s 100); the ratio is what carries over, and 6.25 is 12.5 us at the reference. 0 turns the charge off.

Which paths: the index-only covering walks, LionOrdered’s and LionBtreeScan’s, serial and parallel; the count of one relation, of its WHERE or its groups' sets, with its heap rechecks and residual clauses; and the FK-side join, where its competitor is a hash join over a bitmap heap scan rather than a loop. Not a walk in heap mode, which fetches each member’s tuple as it meets it, a synchronous read the stream’s price would under-charge, nor §40.10’s probes, nor LionProbeJoin (§46), which reads the fact through core’s own path as the competitor’s scans of it do and the dimension a descent at a time; it is priced against a loop’s cold reads alone.

The cap is pg_lion.cold_max_discount (2 by default, 1 to 10), the loop’s and the bitmap’s alike. It is the warm floor: where the cache holds what both plans read after all - effective_cache_size set below the real cache, a hot working set - a lion path chosen for those reads can take up to the cap times its competitor’s time. The measured case (§39’s private workload above), prices, times and pages as shares of the bitmap heap scan’s:

the walk the bitmap heap scan
priced 1.82 1
ran, cold 1 2.2
its pages, uncached an estimated 0.16 of the scan’s, the B-tree’s 1, the heap’s
their cold reads, x 0.74 uncached 0.16 x 0.74 x (6.25 - 1.1) = 0.60 1 x 0.74 x (6.25 - 1.08) = 3.8

The factor, uncapped, is (1.82 + 0.60) / ((1 + 3.8) x 1.82) = 0.28: the walk at 0.50 of the scan’s price. The walk’s pages are an estimate, and move it little: none would make it 0.21, three times as many 0.41. The case needs a cap above 1.82. At 1.5 the walk is priced at 1.21 of the scan and loses; at 1.82 it ties within add_path()’s fuzz; at 2 it is priced at 0.91 and wins by 9%; at 2.5, at 0.73. The model’s own ratio, 4.8 against 2.42, is 2.0 against the measured 2.2.

The default is 2, Chinmay’s choice after the measured cold and warm comparison (1.5 was proposed, 2 recommended). It is the smallest round cap that takes the measured case with room past the fuzz, and what it costs is bounded: only where effective_cache_size says the tables do not fit, and there a third more than 1.5’s worst case where they do after all - twice the competitor’s time where 1.5 allowed one and a half. A cap past 2 changes nothing in that case, and lets a path priced 2.5 times core’s through on the model’s say alone.

Nothing changes where the tables and the indexes fit in effective_cache_size, and nothing against a sequential scan, an index scan or an index-only scan. Not done:

  • A sequential scan’s cold reads: its pages are read ahead in long runs, near seq_page_cost even cold, and none are charged; a lion path against one keeps its price.
  • Core’s plain index scans read their heap pages synchronously outside a loop too; only the loop’s are priced (§39 above).
  • A loop with a bitmap heap scan beside it: a competitor with both kinds of cold reads takes the loop’s alone.
  • The counts whose prices leave their reads out - the decoded walk (§34), the top k (§36), the aggregates over entries (§37), a partitioned table’s, beside an anti join, the cells' - and LionSemiJoin and LionAntiJoin, KNN and lion-column walks, keep their price against a cold scan.
  • One residency for every table: a hot dimension and a cold fact are taken alike, as for the loop. A parallel scan’s pages are counted whole, as core counts its I/O; several workers prefetching overlap more than one does (33 us a page against 40 above), which the one price does not see.

Tests: bitmap_cold.sql, with a cache of a megabyte as fkjoin_cold.sql has: a covering walk taken over a bitmap heap scan cold and not where the tables fit, with the charge at 0 or the cap at 1; the walk’s price over its price against the cold scan exactly the cap where it binds (1.5, 2) and past 2 uncapped, and a parallel walk’s the same under its Gather; the margin dividing the cold price and still a threshold on it; a GROUP BY’s count the same way, over an all-visible heap and with rows added since the VACUUM, which it rechecks; a count whose residual clause reads more heap pages than the scan, not taken; the FK-side join against a hash join over the fact’s bitmap heap scan; the answers the same under each plan; and a GROUP BY over a subquery whose plan holds a Memoize, estimated under the subquery’s root. costrates.sql lists both settings and their ranges, and costgucs.sql lists cold_stream_page_cost among the cost settings.

A Memoize under a LIMIT (2026-10-23)

Core prices a Memoize over a nested loop’s inner scan at one hit ratio for all its calls (cost_memoize_rescan()): of calls rescans over ndistinct keys, of which the cache holds est_entries, hr = (calls - ndistinct) / calls x est_entries / ndistinct hit, each priced at the lookup alone, and the rest at the inner scan’s total cost. Under a LIMIT the planner takes the loop’s startup and the LIMIT’s share of its run, and that share counts the hits of the whole walk. But the loop stops after the share’s first calls, which repeat only the keys among themselves. On §47.9’s bench a medium page’s walk makes 625,513 calls over 188,097 keys, and core priced 70% of them as hits; the LIMIT’s 3,030 calls hit 0.8% of the time. The loop’s price of its first 20 rows was 2,847, where the same model with those hits is 3,982.

So against a nested loop over a Memoize (under a Material or a projection) in a query with a LIMIT (limit_tuples, which the planner leaves -1 under a grouping, an aggregate or a window), whose order gives the query at least its first sort key and of whose rows the LIMIT takes a share s < 1 (lion_memoize_limit_extra(), lion_memoize_limit_scale(), lion_plan_units.c):

  • n = s x calls calls are made; of ndistinct keys drawn about uniformly they hold d(n) = ndistinct x (1 - (1 - 1 / ndistinct)^n) distinct ones;
  • they hit hits(n) = (n - d(n)) x min(1, est_entries / d(n)) times: every repeat where the cache holds every key met, and est_entries / d(n) of them where it holds fewer, as core takes est_entries / ndistinct of the whole walk’s;
  • the loop’s price lacks extra = n x (hr - hits(n) / n) x the inner scan’s total cost, where that is positive.

The figures are core’s own: calls (est_calls from PostgreSQL 19), est_entries, ndistinct as cost_memoize_rescan() estimates it (estimate_num_groups() of the cache keys, or the calls where that falls back on a default; est_unique_keys from 19) and the hit ratio from them (est_hit_ratio from 19). As with a cold loop’s reads, the lion path’s price is put into the loop’s units: it is multiplied by memoscale, the loop’s price of the share (its startup and s of its run) over that price plus the extra (lion_units_price()). That scale is not bounded by pg_lion.cold_max_discount, since it is the competitor’s own price, corrected; the cap bounds only the cold reads on top, which take the loop at nlcost / memoscale (lion_units_cold()). Every lion path priced against such a loop through lion_units_for() sees it, and LionOrderedJoin weighs its own competitor the same way (§47.3). A LionOrdered walk is priced against a scan of its relation, never a loop, and its price does not change. pg_lion.enable_memoize_limit (on) turns it off.

Not counted: keys drawn other than uniformly. A skewed key column repeats more early, and a key correlated with the loop’s order (each run of the walk meeting the same few keys) hits more than this model and core’s both. A cache that evicts is taken at core’s share of entries, not at the hits its eviction order leaves. Tests: ordjoin.sql’s page that is core’s loop over a Memoize with the setting off and LionOrderedJoin with it on.

The join that reads the dimension (2026-10-24)

A NOT EXISTS over a small table puts its anti join at the top of core’s plan, above the join of the fact to the dimension: a Hash Anti Join hashing the small table over the Nested Loop into the dimension’s B-tree, or a Nested Loop Anti Join into the small table’s B-tree over it. The plan’s time and pages are the loop’s, and the anti join is the first join down from the top. So in an upper rel’s plan (lion_competitor_through(): the paths of the count, the FK-side join and LionProbeJoin, which take every join below the aggregate):

  • past a semi or anti join over a join (lion_join_filters(), lion_competitor_below(), lion_plan_units.c): a semi or anti join - JOIN_SEMI and JOIN_ANTI, and JOIN_RIGHT_SEMI (PostgreSQL 18) and JOIN_RIGHT_ANTI, which emit their inner side - whose emitted side reaches a join, down through the nodes that pass rows up, and costs at least what the semi or anti join adds over it, is passed through to that side. The competitor’s kind is the join’s below it (lion_competitor_kind()), and the nested loop whose cold reads and outer side are weighed is the first one below it (lion_nestloop_cold(), lion_nestloop_outer()). A semi or anti join over a scan, or one that is most of the plan’s cost itself, is its own kind. A join rel’s lion paths - LionSemiJoin and LionAntiJoin as join paths - compete with that one join, their outer side core’s child at core’s price, and take the semi or anti join as it is.
  • through a Memoize (lion_nestloop_cold()): a Memoize over the loop’s inner index scan (one, under a Material or a projection) probes it once a miss, at the hit ratio cost_memoize_rescan() priced it at (lion_memoize_misses(), as the bitmap heap scans' reads count it): a leaf and the heap pages of a probe’s rows for each miss.
  • in a subquery’s plan (lion_nestloop_cold()): a loop below a subquery scan is the subquery’s, and its pages, the residency over its tables and a Memoize’s keys are looked up in the subquery’s planner info, whose range table its Vars refer to.
  • the loop less its outer side (lion_units_cold(), LionUnits.nlshared): where the lion path reads the loop’s outer side as its child, at core’s price beside its own price - LionProbeJoin’s fact, the same rels as the loop’s outer side (pj_units(), lion_probejoin.c) - its own price is weighed against the loop’s cost less that side. The factor scales the path’s own price alone, and the same scan inside both warm prices draws it towards 1 by its cost, which neither side’s cold reads change. With the loop’s own part L, the outer side S and the path’s own price P, the path at P x f + S against L + S is (P + its reads) against (L + the loop’s reads): the two plans' cold prices, short of the cap.
  • each LionProbeJoin path its own reads (pj_units()): a participant of the parallel form probes its share of the fact’s rows and makes every build, so the partial path’s factor is of its loops' reads over its price with the builds whole. The serial node’s factor would weigh the cold reads of every participant’s probes against the builds of one. Its probes are priced at its own loops too (lion_ordered_probe_price()), as core prices a parallel loop’s inner scan: fewer probes revisit fewer pages, each a little dearer.

Against a hash anti join over the loop, the loop’s rate with its cold reads prices a lion path at no less than the hash join’s rate would: that rate, 0.5, is the cold factor’s floor at the default pg_lion.cold_max_discount. So where the cache holds the tables the path is the dearer for it, and the dimension’s cold reads count wherever an anti join of either method sits over the loop.

Measured (2026-10-24; PostgreSQL 18, the dev cluster - a VM of four cores shared with other work; EXPLAIN (ANALYZE, TIMING OFF); warm, shared_buffers 2 GB, the median of seven runs; cold, the server restarted and the relations' files dropped from the OS cache before each run, the median of three). A 1,000,000-row dimension (a 503 MB heap, a 21 MB primary key, lion indexes on twelve columns) and the 42,858 fact rows of one grp of 3,000,000, under §46’s shape with a CASE of nine predicates - ranges among them, which the node takes as cells with #178 - and a NOT EXISTS over 60,000 rows, at an effective_cache_size of about a quarter of the query’s tables and a random_page_cost of 1.1, in parallel: core’s plan is a Hash Anti Join over the Nested Loop into the dimension, cost 57,893. Against the loop, whose cold reads are 796,568 over its own 35,613 (57,893 less the fact’s parallel bitmap heap scan, 22,280) - the serial node’s 1,156,102 over its 86,138 (factor 0.62), the parallel form’s 410,937 over 49,590 (0.40 before the cap, 0.5 applied) - the node is 53,934, the plan, as it is at the hash join’s rate. With the loop’s whole cost the parallel form’s factor would be 0.63, and with the serial node’s factor 0.62: core’s plan either way. The node runs it in 895 ms cold against core’s 1,666 (404 against 1,274 with the dimension’s heap alone dropped), and in 295 warm against 184. The thirteen other settings and forms the query was measured at keep their plans, and their prices but one: serially, with the cache holding the tables at random_page_cost 1.1, core’s plan is a hash anti join over the loop with a Memoize, 86,032, and the node 77,053 at the loop’s rate (51,805 at the hash join’s), still the plan.

test/sql/competitor_cold.sql, at the same settings and timed the same way (an 80,000-row dimension of 1,231 pages, not all-visible until the last case; effective_cache_size ‘4GB’ or ‘1MB’; ms, the node; core’s plan):

query settings was is warm cold
300 fact rows, the anti join of 300 4GB the node hash anti join over the loop 2.72; 1.91
… 1MB the node the node 17.9; 37.1
…, hash and merge joins off 1MB nested loop anti join over the loop the node 2.05; 2.68 23.5; 35.2
4,500 rows over 3,000 keys, a Memoize 1MB, the same the loop over a Memoize the node 9.26; 9.65 40.8; 111.0
300 rows of a 200,000-row scan 1MB, the same nested loop anti join over the loop the node 11.5; 11.1 40.7; 44.2
10,000 of them, the anti join of 30,000, 2 workers 1MB, the same, all-visible core’s plan, in parallel the node, in parallel 46.8; 36.3 64.8; 93.9

The cold column is a cache emptied once, each page read from the device at most once: the test’s 1 MB stands for a cache a tenth of the dimension, which re-reads pages a cold run of a small table does not, and the node’s advantage there is the larger. Without the loop less its outer side the fifth query is core’s plan, and the sixth too; the sixth keeps the node with the serial node’s factor, which the 1,000,000-row fixture’s does not. On the same tables with int keys, the warm change of units moves the first query’s plan from the node to core’s loop at 1,000 and 3,000 fact rows and random_page_cost 1.1, where the two tie (6.10 ms against 6.12; 11.6 against 11.4), and at 300 rows at either random_page_cost, where core’s is the faster (2.1 and 2.2 against 3.3).

The regression suite’s other plans are unchanged: make installcheck passes on PostgreSQL 18 and 19.

Tests

test/sql/costrates.sql: the settings and their ranges; the margin as a threshold - a GROUP BY the node wins by far is chosen at a margin just above the ratio of its price to core’s best and is not just below it - that divides the node’s and a LionOrdered’s cost by exactly the margin, that core’s enable_* settings still force at the dearest margin (0.01) and lion’s switches still leave out with none; no margin and no rate where they hedge nothing - a count, a GROUP BY, a semi join and a LionOrdered forced with core’s settings cost the same at every margin and every rate, and a count whose cheapest competitor is an aggregate over the AM’s own index scan the same at every margin while its rate still converts it; each kind of competitor made the cheapest by core’s settings (a plain aggregate over a sequential scan, a hash aggregate, the join methods) moving the cost of the count, the GROUP BY, the FK-side join and the semi and anti join paths linearly with its own rate and not at all with any other, and deciding the GROUP BY at the lowest and the highest rate; the AM’s scans unmoved by any rate or margin and LionOrdered by any rate; and every plan’s rows against the pushdown off and sequential scans only. costgucs.sql lists 38 cost settings with resident_page_cost.

The expected outputs regenerated. No answer changed. Eight plans did, each timed against the plan it replaced (assert build, the median of five, two or three times over):

test query was is ms, is against was
pushdown g, count(*), 1,000 groups, never vacuumed hash aggregate LionCount 13.8 to 14.3 against 15.0 to 15.8
pushdown a, b, count(*) WHERE c = 'c1', decoded walk off hash aggregate LionCount 2.4 to 2.9 against 3.2 to 3.5
pushdown … WHERE c = 'c1' OR n = 2 hash aggregate LionCount 2.2 to 2.5 against 3.9 to 5.1
multicolumn a, b, count(*) WHERE c = 'c1', one two-column index hash aggregate LionCount 2.2 against 4.2 to 5.5
multicolumn … over single-column indexes hash aggregate LionCount 1.7 to 1.8 against 4.2 to 5.2
fkjoin_mixed d.grp, count(*), f.x = 3 AND d.attr = 2 hash aggregate, hash join partial counts 1.0 to 1.25 against 1.35 to 1.66
fkjoin_mixed … , max(d.name) hash aggregate, hash join counted rows 1.28 against 1.62 to 1.74
fkjoin_distinct d.region, count(DISTINCT f.fk), f.x = 3 AND f.t = 't4' sorted aggregate, hash join LionCount 0.63 to 0.65 against 0.42 to 0.51

The first seven are the hash aggregate’s and the hash join’s rates finding the node where it is the faster plan and dc7cd29 refused it; the tests' comments say so. The eighth is a loss, accepted: the hash join’s rate prices this small join’s node at 0.82 of the hash join, and it runs 1.35 times slower, where the matrix’s fk.status.kind is priced at 0.91 and runs 1.25 times faster. To give each to its faster plan the rate would have to be at least 0.57 and below 0.555 at once (EXPLAIN at each); 0.5 keeps the larger, 12 ms against 0.17. At 0.8 the margin had moved four more - fkjoin_semipath.sql’s GROUP BY to a sorted aggregate (a tie) and its two picks to a nested loop (1.56 ms against 1.36) and from a LionCount (1.20 against 1.36), orderedsemi.sql’s LionOrdered under a LIMIT to an index scan three times slower (0.13 to 0.15 ms against 0.04 to 0.05) - which at 1 are dc7cd29’s plans again.

make installcheck passes in both WAL modes, 104 tests and 35 isolation tests in each, and so does make unit.

Not done

  • Three of lion’s mispicks in the matrix are formulas, not units: the IN lists' unions (eq.c20k.in100, eq.c1m.in1000, priced at 0.78 and 0.60 of an index-only scan’s aggregate and 1.3 to 2.2 times slower) and the 20,000-entry GROUP BY (group.c20k, priced 1.09 times a sorted aggregate over an index-only scan and 1.4 times faster). The margin is not the tool for them, and this section refits none of the 37 constants they are made of.
  • The parallel GROUP BY’s participants are priced with the serial node’s competitor; core’s partial paths are not searched for one of their own.
  • One residency for both sides' common reads. A B-tree leaf a probe is counted for the loop and for LionProbeJoin alike, at the share of effective_cache_size the key’s size gets among the query’s tables (index_pages_fetched()): the 1,000,000-row fixture’s 21 MB key, probed 13,467 times by a participant, is some 10,000 reads of its 2,745 pages, which any cache keeps while the probes revisit them. Both sides' alike, they draw the factor towards 1: without them the serial node’s 0.62 and the parallel form’s 0.40 would both be far below the cap. A residency a relation, an index the probes revisit as hot as its size allows, would weigh the heap pages the node avoids alone.
  • The FK-side join’s factor weighs its own price against the loop’s whole cost, the dimension scan it reads as its child at core’s price inside it: nlshared is LionProbeJoin’s alone, and the same pull towards 1 is not measured there.
  • The other readers of the loop. A Memoize under a LIMIT (lion_memoize_limit_scale()) is looked for only on the plan’s top join.

40. Lion-filtered walks of a covering B-tree, index-only (LionOrdered’s index-only mode and LionBtreeScan, lion_ordered.c; 2026-10-06)

The shape is

SELECT <columns a B-tree holds> FROM t
WHERE <clauses lion indexes answer> [AND <the B-tree's own quals>]
[ORDER BY <the B-tree's columns> [LIMIT n]]

over a B-tree the user created to cover it - (k) INCLUDE (...), as for an Index Only Scan - and lion indexes on the filter columns. §30’s node already filters a B-tree’s walk through lion’s set before it touches the heap; what it fetched from the heap was the row’s VALUES. When the B-tree returns every column the node must produce, the values come from the index tuple and the heap is read for visibility alone, on pages the visibility map does not call all-visible, exactly as core’s Index Only Scan reads it (nodeIndexonlyscan.c): a LIMIT 10 of a selective filter then costs the lion lookups, the entries walked and ten visibility-map tests. And since the walk pays without an ORDER BY once the heap is out of it, the same node is offered for any B-tree path of the relation whose index covers the query - a full walk of (grp) INCLUDE (x, tags) under WHERE tags @> '{t}' GROUP BY grp included - under the name LionBtreeScan when it claims no order. Lion’s part is the membership test, which answers what a B-tree cannot hold: multi-key columns, a column behind a high-cardinality leader, ANDs and ORs across independent indexes. This is what replaces the window store (format version 9, reverted 2026-10-06): a value store partitioned by heap window cannot beat a B-tree’s clustering on a pinned key, and the B-tree is the user’s to create.

40.1 What qualifies

  • The relation, the WHERE and the lion side: §30.1 and §30.2, unchanged (lo_rel_ok(), lo_lion_accesses()). The target of an UPDATE, DELETE or MERGE, a relation with security quals, another table AM: declined as there. So is a relation the planner has proven empty (IS_DUMMY_REL(): a constant-false or NULL restriction, constraint exclusion) - core builds no index path for one and make_one_rel() leaves its pages out of root->total_table_pages, which cost_index() asserts they are in; before LionBtreeScan the hook never reached create_index_paths() for such a relation, because its path list held only the empty Append and there was no ordered path to match, but lo_btree_paths() runs it whenever a B-tree covers the target, and ordered.sql’s EXECUTE lo_p(NULL, 0) tripped the assertion (2026-10-06).
  • Coverage. The B-tree covers the node when every column it must produce is a plain column of the index - a key column (indexkeys[i] != 0) or an INCLUDE column - that the AM can return (canreturn[i], true of every B-tree column). The columns the node must produce are decided per (ordered path, lion access) pair (lo_needed_attrs(), lo_index_only()): every Var of the relation in rel->reltarget->exprs, in the residual filter (lo_residual(): the clauses the node still tests itself, a partial lion index’s predicate when the query spells it included), in the ordered B-tree’s own clauses (its recheck under xs_recheck) and - only when a lion index clause is lossy at plan time (IndexClause.lossy), so that every member would need the recheck and heap mode is the honest plan - in the lion qual. The lion filter’s columns are otherwise NOT required, where core’s check_index_only() would ask for every restriction clause’s: whether a set needs rechecking is a run-time property (an inexact source), and a member of such a set the B-tree cannot recheck is rechecked on its heap tuple, fetched for that alone (§40.4) - heap mode, the alternative, fetches every member’s. So (created_at) INCLUDE (id, title) under WHERE tags @> '{t}' is covered. Lion’s planner marks no clause lossy today (a tsquery prefix, NOT or weight widens the set to a superset the executor rechecks, §17), so the lossy branch is dormant until one does. The pre-filter that decides which B-trees get unordered paths at all (§40.2) tests the target’s Vars alone, a necessary condition; the pair decides. An expression column matches no Var: it may still give the order, as in §30, but no value. A system column makes the index non-covering, as in core: SELECT ... FOR UPDATE adds the row mark’s ctid to the target, so a locked relation is always heap mode, and EvalPlanQual never meets an index-only plan. So does a partial B-tree’s predicate column when the query spells the predicate: the clause stays in the node’s filter (§30.2 step 4 - core’s index scan drops it, but the fetch-and-sort switch rechecks rows that did not come through the index, which the predicate does not hold for), so (k) INCLUDE (x) WHERE flag under WHERE flag AND ... is covered only with flag among the INCLUDE columns. The mapping of index columns to relation attnos goes into custom_private (LO_PRIV_IOCOLS), so that the executor looks nothing up per row.
  • enable_indexonlyscan: off, there is no index-only mode - the ordered node keeps its heap mode, and no unordered path is offered.

40.2 Planner integration (lion_ordered_set_rel_pathlist())

  • Ordered paths (§30.2): as before, from rel->pathlist; each (ordered path, lion access) pair decides coverage (§40.1) and is priced in index-only mode when covered, in heap mode as before otherwise.
  • Unordered paths. For each B-tree of the relation that holds the query’s target columns (a necessary condition; the pair decides the rest, §40.1), create_index_paths() is run on a scratch copy of the RelOptInfo that sees that index ALONE (lo_scratch_rel(), the helper lo_lion_accesses() now builds its copy with; lo_btree_paths())
    • alone, so that its paths compete with nothing: in the relation itself the full walk of a covering index without a qual, which core builds as an index-only scan whenever check_index_only() passes, loses to the sequential scan and never reaches rel->pathlist, and a path with quals may lose to another index’s. Core builds, for that index, the paths with its matching quals - with pathkeys where the query finds them useful, without otherwise - a backward one where that order is useful, and the no-qual one; an IndexPath that add_path() dropped for a bitmap scan of the same index is taken from that scan’s one-leaf bitmapqual. Core builds the no-qual path only where its order is useful or the index alone answers the query (check_index_only(), every restriction clause’s columns among its own), which a B-tree lacking the lion filter’s columns - a primary key under a lion filter on other columns - never does; when core built no unparameterized path of the index, lo_btree_paths() builds that walk of the whole index itself (create_index_path() with no clauses; a partial B-tree only where the query implies its predicate), and the pair decides coverage as for any other. Each resulting IndexPath - a B-tree, not hypothetical, param_info == NULL (the parameterized ones are §40.10’s nested loop probes), no ORDER BY operators, lo_indexpath_ok() - whose twin is not among the ordered paths (the same index, direction and pathkeys: offered above already) is paired with each lion access from lo_lion_accesses(), as the ordered loop pairs them, less the pairs in which the lion access adds nothing: every clause of its every leaf, an OR arm’s too, is one of the B-tree path’s own index clauses (lo_lion_adds()), so every entry walked would be a member - the same RestrictInfo, or an equal() clause: 18 and later match an OR of equalities to each index as a ScalarArrayOpExpr RestrictInfo of that index’s own (match_orclause_to_indexcol()), so the B-tree’s k = ANY and the lion index’s are two objects spelling one clause (lo_same_clause(), which lo_cost()’s shared-clause correction uses too, or the OR’s selectivity would be squared and the node underprice itself against the plain Index Only Scan). A leaf with no index clauses at all - a partial lion index walked whole for its predicate, which lo_lion_tree_ok() allows - adds that predicate’s selectivity and counts as adding; lo_cost()’s lo_pred_implied() correction prices it. The pair must cover (§40.1) and is priced in index-only mode. The CustomPath carries the IndexPath’s pathkeys when it has them - a free order - under the name LionOrdered, and none otherwise under the name LionBtreeScan: a second CustomPathMethods/CustomScanMethods pair over the same plan function, state struct and executor functions. pg_lion.enable_ordered_scan keeps gating the ordered paths and the lion columns' walks of §30.11, and with it off the unordered paths claim no order whatever core found useful - a LionOrdered must not appear with its switch off - and are all LionBtreeScans; pg_lion.enable_btree_scan (bool, default on) gates the unordered paths. Core’s enable_indexscan disables both, as it disables the B-tree’s own index scans (§39, “Core’s switches”): the path is kept and marked disabled after it is priced, since its price comes from indextotalcost, which carries no disable_cost; a partial path is disabled as a parallel index scan is.
  • A clause core’s AND leaves out (lo_lion_cover()). The lion accesses are core’s bitmap paths, and choose_bitmap_and() leaves out of a BitmapAnd an index whose bitmap costs more than the heap fetches it saves - c3 IN (0, 1), two thirds of the rows, beside c20 IN (1, 2) - since the bitmap heap scan’s Filter tests such a clause on the tuple it fetches anyway. Paired with such an access, a walk of a B-tree that lacks the clause’s column has the clause in its residual, needs the column, and is not covered: from core’s accesses alone no index-only walk is offered, and the plan reads the heap (a bitmap heap scan of every member of c20 IN (1, 2)). So when every one of core’s accesses leaves one of the clauses a lion access answers - those whose columns a lion index’s key has (lo_knn_own()) and some access of the pool answers - the AND that answers all of them is added to the accesses: built from core’s and those of each lion index alone, greedily (lo_cover_pool(), lo_cover_and()), as §44.6’s post-filter builds a lionvec order’s. It is paired only where it makes the walk index-only - every unordered path, and an ordered one whose B-tree then covers; a walk in heap mode reads every member’s tuple for its values anyway, and keeps core’s accesses, whose sets are the cheaper to build. The price decides between the pairs: a clause that keeps most rows makes a set costly to build (on a 60,000-row table the c3 set is 40,000 members), and under ORDER BY k LIMIT 50 the walk over core’s smaller set, fetching the few members' tuples, is the cheaper; for the aggregate the index-only walk is chosen over the heap. A clause no lion access answers (c3 + 0 = 1) is the Filter’s whatever the set. On a synthetic 3,000,000-row table of 340-byte rows (a 1.1 GB heap, a 64 MB primary key), SELECT sum(id) ... WHERE a IN (<10 of 20 values>) AND b IN (<2 of 3>) over core’s accesses alone is a bitmap heap scan of a’s half of the rows with b in its Filter, 1,049 ms (median of five, warm; 687 ms parallel); over the added AND it is an index-only walk of the primary key, 307 ms serially and 202 ms with two workers.
  • Partial paths (§40.9): each pair is offered a second time as a partial path when the walk may be divided among a parallel plan’s participants (lo_parallel_ok()), and every serial walk is parallel_safe when the relation is (rel->consider_parallel): whole, it may run in a worker - the inner side of a join a worker runs, or below a non-unique semi join’s unique-ification (§27, “Forward semi joins over a non-unique key”), which takes a whole path under a Gather.
  • Partitions: as §30.11, each leaf is scanned like a table under core’s Append or MergeAppend.

40.3 Cost (lo_cost(), index-only)

§30.3’s start-up (the lion lookups and the set), walk (indextotalcost plus a membership test per entry) and shared-clause correction stand. The heap term changes: the F members' pages are priced as cost_index() prices an index-only scan’s - index_pages_fetched() for an uncorrelated order and the members' share of the heap for a correlated one, EACH scaled by (1 - rel->allvisfrac), the fraction of pages the visibility map does not call all-visible, and interpolated by the squared correlation as before - plus a visibility-map test per member at cpu_operator_cost; cpu_tuple_cost per member (now the index tuple’s deform) and the filter’s cost were charged already. lo_margin applies as §39 applies it. For an unordered path core has no LIMIT to scale by, so the planner prices the whole walk, and a path whose walk is the whole index competes on indextotalcost + T × cpu_operator_cost against the bitmap heap scan’s heap pages, which is right: on btreescan.sql’s 100,000 rows, whose heap is six times the covering B-tree, the covering walk is chosen over the bitmap scan for a filter of a tenth of the rows, the bitmap scan over it where only the wide (k) INCLUDE (...) B-tree covers the columns, and the lion index’s own scan for a filter of three in a thousand (§40.7, “the plan choice”). The recheck from the heap (§40.4) is not priced: an inexact source is not knowable at plan time, and the fetches it costs are bounded by the members - the bound heap mode pays for every member. (Since 2026-10-19 a qual the set leaves to the recheck at PLAN time is priced: the lion access’s IndexPaths are asked what a plain scan’s source of them leaves (lo_rechecked_quals(), lion_path_rechecked_quals(); a BitmapOr’s arms are not followed), the set’s members become the CANDIDATES, the rows of the quals it does answer, and every candidate the walk meets is charged its recheck and - in heap mode, and index-only when the btree does not return a column the lion qual reads (LO_FLAG_HEAPRECHECK) - its heap fetch, all-visible or not, since the recheck reads the tuple before the visibility map is asked. IS NOT NULL used to be such a qual (§14): priced by its members, such a set chose an index-only walk that fetched 1.41M heap rows to throw half of them away. A set made lossy by a lossy qual at run time stays unpriced.) The planner also hands the executor the entries the walk is expected to visit, indexselectivity × rel->tuples, for the switch’s density gate (§40.4; LO_PRIV_EXPECTED, a float8 Const in custom_private, 0 when the ordered path is a lion column’s walk).

The probe’s price (2026-10-06). The walk is CPU-bound. On bench/quick.py’s 5M-row fact table (its covering B-tree (c20k) INCLUDE (c1m), 107 MB, under WHERE c20 = 3 GROUP BY c20k, a 5% filter) the covering walk took 830 to 925 ms for 5M entries, 170 ns an entry - the btree step, the probe (a binary search over the container keys and a membership test) and the met-before mark - against 85 ns a row for the sequential scan with its filter, which the planner charges cpu_tuple_cost + cpu_operator_cost (0.0125). Priced at indextotalcost’s cpu_index_tuple_cost plus one cpu_operator_cost for the probe, a fifth of that for twice the work, the walk came out at 97,072 against 102,912 for the lion index’s own scan and HashAggregate, which run in 200 to 250 ms, and was chosen at random_page_cost 1.1 as well (57,055 against 101,486). Each walked entry’s probe is now LO_PROBE_TUPLES (2) × cpu_tuple_cost + cpu_operator_cost, 0.0225, 1.8 times the sequential scan’s row, which puts that walk at 197,072 (157,055 at random_page_cost 1.1, against the lion index scan’s 101,486) and sends the query to the lion index’s scan at either setting, while the LIMIT walks keep the node (§40.7, “On the benchmark’s tables”). The lion column walk of §30.11 keeps its own pricing (lo_cost_walk()). Where the covering B-tree holds the filter’s column as well, core’s own Index Only Scan with the filter walks the same entries for one cpu_operator_cost each, and is now priced below the node; on btreescan.sql’s 100,000 rows the two run in the same time for c200 = 17 ORDER BY k LIMIT 10 (0.5 ms either way), and core’s filtered scan of tags @> '{t3}' over 94,000 entries takes 18 to 28 ms against the node’s 13 to 17, the containment operator costing core more than the one cpu_operator_cost it is charged at.

Warm and cold. A full covering walk reads the B-tree (107 MB here) in place of the heap (751 MB) but spends those 170 ns an entry, so warm it loses to the lion index’s own scan of a 5% filter by four times and to the sequential scan by two; it pays when the heap would be read cold, or the filter is far wider than the B-tree’s share of the heap. The planner’s I/O constants decide, pg_lion.enable_btree_scan turns the unordered walks off and pg_lion.enable_ordered_scan the ordered ones. Since 2026-10-23 an index-only walk against a bitmap heap scan whose heap the cache cannot hold is priced with both plans' cold reads (§39, “A bitmap heap scan’s cold reads”).

40.4 Execution

  • The slot. The mode is fixed per plan, so the scan slot’s type is: TTSOpsBufferHeapTuple of the relation’s descriptor in heap mode, as before; TTSOpsVirtual of the relation’s descriptor in index-only mode - the relation’s, not the index’s as core’s node uses, so that the plan’s Vars (scanrelid, no custom_scan_tlist) and the rechecks read it unchanged. lo_create_state() picks it from the plan’s flags, not BeginCustomScan: ExecInitCustomScan() initialises the projection and the quals on the scan slot’s type before BeginCustomScan runs, and the expression machinery specialises on it, so one plan never returns slots of two types.
  • A row. xs_want_itup is set on the B-tree scan before index_rescan(). For a member TID (lo_fetch_indexonly()): VM_ALL_VISIBLE(heapRelation, blkno, &vmbuffer), the map’s buffer kept across calls and released at rescan and end, as ioss_VMBuffer is. All-visible, the row is visible to every snapshot, this one included, and the heap is not read. Not all-visible, the heap tuple is fetched into a slot of the table AM’s kept for this alone (index_fetch_heap() on 16 to 19, table_fetch_tid() and table_tuple_fetch_row_version() on 20 - lo_fetch_btree(), as heap mode fetches), counted as a heap fetch and kept in that slot until the row’s rechecks are done (below); a TID with no visible version is skipped. Either way the values are the INDEX TUPLE’s: xs_itup deformed with xs_itupdesc (or xs_hitup, had the AM returned a heap-format tuple) and laid into the virtual slot through the mapping, every other attribute NULL - the visible version reached from the root TID has the index’s values for indexed and INCLUDE columns, since those are HOT-blocking (§30.5). A name column a B-tree stores as a cstring (name_ops’s storage type) is copied back into a Name of the node’s, reused per row, as core copies it. The “met before” bitmap of §30.4 stands: a TID met twice is skipped before any of this.
  • The rechecks. ordrecheck under xs_recheck and EvalPlanQual’s lo_recheck evaluate on the scan slot as before; coverage guarantees the columns are there. So does lionrecheck, for an inexact set, when the B-tree holds the lion qual’s columns. When it does not (LO_FLAG_HEAPRECHECK, decided at plan time from the lion qual’s Vars against the mapping; st->lionheaprecheck), the member is rechecked on its heap tuple (lo_lion_recheck()): fetched now into the table AM’s slot if the visibility test did not fetch it already - counted as a heap fetch, a TID with no visible version skipped - slot_getallattrs(), and ExecQual(lionrecheck) with that slot as ecxt_scantuple, which is sound although the qual was initialised for the virtual scan slot (§40.5). A row the heap was read for holds its tuple predicate lock (fromheap). The heap tuple is let go at the end of the row’s iteration. The switch’s rows, in index-only mode, are heap tuples copied whole into the scan slot, so their recheck reads the slot as before.
  • The fetch-and-sort switch (§30.4) fetches members from the heap in TID order: in index-only mode each lands in the table AM’s slot and is copied into the virtual scan slot (ExecCopySlot(), a deform) before the rechecks and the sort, so the slot type stays fixed, and each copy carries its row’s TID. Its ratio is LO_SWITCH_RATIO (32) in both modes, the setting pg_lion.ordered_switch_ratio (§30.4). In index-only mode the walk would not have read the members' heap pages at all, so a switch pays for pages the walk never touches: a uniform full walk under no LIMIT - the aggregate shape - must never switch (it would fetch every remaining member from the heap; under the heap-mode rule alone a filter of a tenth of the rows switched three quarters of the way through the index and fetched the 2,300 members left), while the hazard the switch exists for (§30.3: a LIMIT walk whose members sit at the far end of the order) must switch, and a ratio large enough to spare the former - some 530 at the planner’s page and entry costs - spares the latter too for any set over a fifth of a percent of the index. The two are told apart by density instead: at both decision sites (the lazy set’s conversion trigger and the switch itself, lo_switch_due()) the switch is considered, in index-only mode, only when the walk has met markedly fewer members than a uniform spread predicts - distinct < LO_SWITCH_DENSITY (0.5) × members × scanwalked / expected, expected the entries the walk was expected to visit (§40.3; 0 when unknown, and a gate with no expectation passes); the lazy site uses lazymembers. A uniform walk meets members × scanwalked / expected members and never passes; the correlated hazard keeps distinct near 0 while scanwalked grows and switches at the measured 32 (§40.8 has the gate’s blind spot). In heap mode the gate does not apply: the members the switch fetches would have been fetched by the walk anyway. Under ANALYZE a switch’s heap visits count in Heap Fetches in index-only mode as the walk’s do (§40.6). An unordered path has no sort keys and never switches, as a path whose pathkeys were not plain columns never did; the planner priced the full walk.
  • Unchanged: the lazy set, the early stop, the spill, rescans with changed Params, the predicate locks of the lion lookups.

40.5 Correctness

§30.5’s argument stands for which TIDs come out of the walk and are members. What changes is how a member’s visibility is decided and where its values come from, and both are core’s Index Only Scan’s.

  • The visibility map. A page’s all-visible bit says every tuple on it is visible to every snapshot. It is cleared, under the page’s lock, before a row is inserted on the page, and the B-tree entry for that row is inserted after; the walk reads a TID under the B-tree page’s shared lock, serialised with the insert’s exclusive lock, and the lock released between the clearing and the index insert is a full barrier - so a TID the walk meets whose row was inserted after the bit was set is met with the bit cleared (nodeIndexonlyscan.c’s “memory ordering effects” note, which visibilitymap_get_status() reading without a lock rests on). A delete clears the bit too and does not touch the index, but the deleted row stays visible to the snapshot until its deleter commits, and a snapshot that sees the commit was taken under ProcArrayLock after the bit was cleared. Lion adds no TID the B-tree did not return, so nothing new arises on its side.
  • A recycled TID, and a hot standby. xs_want_itup makes nbtree keep its pin on the leaf page while the executor tests the map (its dropPin rule, nbtree’s README): VACUUM cannot remove the page’s TIDs from the index, and so cannot free the heap slots they name, until the walk has moved on - the interlock core’s Index Only Scan relies on, and now the node’s. The set’s own hazard (§30.5, “A recycled TID”) is unchanged: a member whose slot was reused names a tuple the snapshot cannot see, and on a page still marked all-visible no such tuple exists. On a hot standby the same pin rule protects the walk as it protects core’s scan, and the lion side still reads no visibility map and holds no pin (§30.5).
  • SERIALIZABLE. A row returned without a heap visit has no tuple to predicate-lock - a tuple lock needs the tuple’s xmin, which was not read - so the node takes PredicateLockPage() on its heap page as core’s Index Only Scan does, after the rechecks pass, for each row it returns; a row the heap was visited for holds its tuple lock from the fetch. The B-tree page locks and the lion relation locks are §30.5’s. test/isolation/btreescan_serializable.spec is the write skew of two HOT updates of rows the other transaction read index-only: it must fail, and without the page lock neither update would conflict with anything the other read (a HOT update writes no index) and both would commit.
  • The recheck from the heap (§40.4) evaluates lionrecheck, initialised for the virtual scan slot, on the table AM’s slot. That is sound because the scan slot’s type is fixed (ExecInitScanTupleSlot() sets scanopsfixed), so ExecInitQual() emitted no EEOP_SCAN_FETCHSOME step for it - ExecComputeSlotInfo() drops the step for a fixed virtual slot, and with it the slot-type check that step makes (CheckOpSlotCompatibility) - and the qual’s Var steps read tts_values[] and tts_isnull[] directly, which slot_getallattrs() fills for the heap tuple. The tuple it tests is the version the snapshot sees of the TID the walk returned (the HOT chain followed from its root, as the visibility fetch follows it), so the recheck tests the values the row has for this snapshot, the same version whose indexed and INCLUDE columns the index tuple returned (§40.4).
  • EvalPlanQual never meets an index-only plan: a row mark’s ctid is in the target (§40.1). lo_recheck() asserts the test tuple’s slot type is the scan slot’s; the recheck from the heap is index-only mode’s alone and never reaches it.

40.6 EXPLAIN

Custom Scan (LionBtreeScan) on bs
  Index: bs_grp (index only)
  Lion Cond: (tags @> '{t3}'::text[])
  Lion Indexes: bs_tags

Limit
  ->  Custom Scan (LionOrdered) on bs
        Ordered By: bs_k (backward, index only)
        Lion Cond: ((c200 = 17) AND (c20 = 3))
        Lion Indexes: bs_c200, bs_c20

LionOrdered’s lines are §30.7’s; index only joins the Ordered By line’s parenthesis in index-only mode (heap mode prints as before, so no existing output changes), and a LionBtreeScan names the index it walks under Index, with the same qual and counter lines. Under ANALYZE, Heap Fetches counts in index-only mode the heap visits: the members whose page the visibility map did not call all-visible, the members an inexact set had rechecked on their heap tuple (§40.4) and the fetch-and-sort switch’s, visible or not, as core’s Index Only Scan counts its visits (in heap mode, as before, the members with a visible version); Heap Fetches: 0 on a vacuumed table under an exact set, or an inexact one the B-tree can recheck.

40.7 Tests

test/sql/btreescan.sql, on a 100,000-row table - its heap six times its covering B-trees - with covering B-trees (k) INCLUDE (id, grp, x, c200, c20, nm) and (grp) INCLUDE (x, tags), two partial ones and an expression one, and lion indexes on c200, c20 and tags: every query run through the node (every core scan off) and through the ordinary plan (both of the node’s settings off) and compared - in order under an ORDER BY, as multisets otherwise - with the node and its mode named. A covered =, ranges and IN lists on the B-tree side with lion filters, lion IN lists, a name INCLUDE column filtered and returned, DESC with a LIMIT, OFFSET, an empty answer, a LIMIT beyond the match count; the aggregate shape under a GROUP BY and a plain aggregate, with a scalar filter and a multi-key one (exact as well, §29.6); the counters (Heap Fetches: 0 after VACUUM, Rows Removed by Lion Recheck evaluated on the index tuple’s values, the switch in index-only mode with its heap fetches counted); rows inserted in the same transaction and a dirty heap (HOT and non-HOT updates, deletes) before VACUUM, their heap fetches counted, and after VACUUM with none; FOR UPDATE (heap mode); cursors fetched in pieces in both modes; a generic plan and LATERAL and correlated subqueries rescanned with the outer value in the lion filter and in the B-tree’s quals; the partial B-trees, with the predicate column covered and not; the expression B-tree (the order, no values); each setting off; and the plan choice with nothing disabled. Then a B-tree lacking the lion filter’s columns (§40.1): an exact set (Heap Fetches: 0), an inexact one (each member fetched for its recheck), rows not all-visible on an inexact set (fetched once), a set that spills past hash_mem through a kept plan (as ordered.sql’s section 10 does, on a 400,000-row table of its own: Lion Set: exact, no heap fetch and nothing removed) and tsquery prefix and NOT filters (supersets, rechecked from the heap); the switch in index-only mode (§40.4): the far-end shape switches and matches the ordinary plan, a large pg_lion.ordered_switch_ratio keeps it walking, the uniform GROUP BY walk and a uniform backward walk to the end never switch and read nothing from the heap, and the same backward walk in heap mode (a ctid in the target) switches as before; NULLs in key and INCLUDE columns, a NULL name among them, over (grp DESC NULLS FIRST, id) INCLUDE (x, nm) walked both ways, IS NULL filters, on a table with a dropped column (the mapping with a gap); ctid, tableoid and a whole-row Var (heap mode), both settings off (no node), a SCROLL cursor fetched backward; and the pairs that add nothing (§40.2) - k = 1 OR k = 7 with a lion index on k and a B-tree leading with it, with core’s scans on and off: no node, its presence alone printed since the plan’s text differs between majors - against a partial lion index walked whole, which is offered; and a clause core’s AND leaves out (§40.2), on a 60,000-row table of its own: the AND’s index-only walk chosen for the aggregate with nothing disabled, Heap Fetches: 0, the answers of the aggregate, ordered walks both ways, a B-tree filter beside the AND, the OR spelling of the IN list and a <> keeping nearly every row; a LIMIT 50 in order that takes core’s smaller set in heap mode; a column the B-tree lacks in the target (heap mode, core’s access) and a clause no lion access answers (the Filter’s). test/sql/btreescan_parallel.sql is §40.9’s. test/isolation/btreescan_serializable.spec is §40.5’s write skew, and passes only with the page lock; a step shows the read ran index-only with Heap Fetches: 0, so the conflict is the page lock’s.

On the benchmark’s tables (2026-10-06): bench/quick.py’s fact (5M rows) and docs (1M), warm, through the node, with both of its settings off, and in heap mode (enable_indexonlyscan = off), EXPLAIN (ANALYZE, TIMING OFF) three times each. SELECT c20k, sum(c1m) FROM fact WHERE c20 = 3 GROUP BY c20k: the lion index’s scan and HashAggregate at random_page_cost 4 and 1.1 (the node forced: the 5M-entry walk, §40.3). ... WHERE c200 = 17 GROUP BY c20k: the same plan. SELECT id, payload FROM fact WHERE c200 = 17 ORDER BY id DESC LIMIT 2000 over (id) INCLUDE (payload): the node, index only, 400,000 entries walked and no heap read, against core’s backward index scan with a filter. ... WHERE c200 = 17 AND c20 = 3 ORDER BY id DESC LIMIT 10: the node. SELECT id, payload FROM docs WHERE tags @> ARRAY['t1','t17'] ORDER BY id DESC LIMIT 10 over (id) INCLUDE (payload): the node, index only, the set exact (§29.6) and Heap Fetches: 0. Timed on the dev cluster after the probe was priced: the c20 = 3 GROUP BY 217 to 247 ms through the lion index’s scan (the node forced: 973 to 1,002 ms), the c200 = 17 one 40 to 48 ms (the node forced: 722 to 746); the LIMIT 2000 walk 28.8 to 29.6 ms through the node against 44 to 49 for core’s backward scan with a filter; the two-filter LIMIT 10 2.5 to 2.9 ms against 5.2 to 7.5; the tags LIMIT 10 over docs 1.5 to 1.7 ms against 4.0 to 4.5.

40.8 Not done

  • The density gate’s blind spot (§40.4): members dense early in the order and the rest at its far end, under a LIMIT the early ones do not satisfy (a residual filter removes them, say), keep distinct at the uniform rate while the walk crosses the empty middle, so the gate never opens and the walk goes to the end where heap mode’s rule would have switched.
  • The recheck from the heap is unpriced (§40.3) for a source made inexact at run time (a lossy qual, a non-MVCC snapshot); its fetches are bounded by the members, every one of which heap mode fetches. A qual the set leaves to the recheck at plan time is priced since 2026-10-19.
  • Pending CI on 20. index_beginscan()’s third argument and the IndexScanDesc fields the index-only walk reads (xs_want_itup, xs_itup, xs_itupdesc, xs_hitup) are taken as 16 to 19 have them; an index-only walk whose xs_want_itup went unhonoured errors with “no index tuple returned”, which the master job’s btreescan run will tell.
  • A partial B-tree’s predicate clause stays in the filter (§40.1); dropping it as core’s index scan does would need the switch’s recheck to take the predicate on.
  • A faster probe. Each entry’s probe is a binary search over the set’s container keys and then a membership test, about half of the 170 ns an entry (§40.3); a directory indexed by container key would find the container in O(1), and a bitset container’s test is O(1) already where an array container’s is a second binary search. The price above would then come down with it.
  • Expression columns returning a value for a matching expression of the target.
  • The lazy set, the switch and the early stop in a parallel walk (§40.9): the first two are off, the third is each participant’s own and rarely fires.
  • Other heap reads a B-tree could spare. The count pushdown’s residual clauses (§10, “Residual clauses”) are tested on each candidate’s heap row even where a B-tree holds the clause’s column; the plan competes on price with an aggregate over the walk, which wins where the heap reads cost more (on the synthetic table of §40.2, count(*) ... WHERE b = 1 AND id % 7 = 3 ran in 252 ms over the walk against 800 ms for the pushdown’s rechecks, medians of five). A partial B-tree’s predicate column must be among its columns (§40.1) because the switch rechecks the predicate; an unordered or parallel walk never switches and could drop the clause. Reads by TID - the plain and bitmap scans of lion indexes, lion_index_facets()’s read of a few rows' keys, a BM25 scan - have no B-tree to turn to, since a B-tree is not searched by TID.

40.9 Parallel walks (2026-10-20)

A walk of a B-tree probed against lion’s set is offered as a partial path too: the participants of a Gather or a Gather Merge share core’s parallel scan of the B-tree, which hands each of them whole leaf pages in the walk’s direction, as a Parallel Index Only Scan’s participants get theirs, and the set, built once.

Finalize Aggregate
  ->  Gather
        Workers Planned: 2
        ->  Partial Aggregate
              ->  Parallel Custom Scan (LionBtreeScan) on t
                    Index: t_pkey (index only)
                    Lion Cond: ((a = ANY ('{0,1}'::integer[])) AND (b = ANY ('{0,1}'::integer[])))
                    Lion Indexes: t_a, t_b

Gather Merge
  ->  Parallel Custom Scan (LionOrdered) on t
        Ordered By: t_k (index only)
        Lion Cond: (c = 3)
        Lion Indexes: t_c
  • What qualifies (lo_parallel_ok()): the relation may be read in a worker (rel->consider_parallel: its restriction clauses, which the node evaluates, and its target are parallel-safe, and it is not a temporary table), and the B-tree’s access method has a parallel scan (amcanparallel). The lion column walks of §30.11 and a lionvec order (§44, which has a parallel scan of its own) are not divided. Every pair of §40.2, heap mode and index-only, ordered and not, is offered once more through add_partial_path(), its rows a participant’s; core then builds the Gather and the Gather Merge above it, the partial aggregates, the parallel hash joins with the walk on either side, and the parallel anti joins - §40.2’s unordered paths are what these needed. A whole walk is parallel_safe with the relation, so that a worker may run it too.
  • Cost (lo_cost() with workers), as cost_index() prices a parallel index scan: the workers are compute_parallel_worker()’s for the B-tree’s pages and - in heap mode, or where the set’s members are rechecked on the heap tuple - the heap pages the fetches read, an index-only walk’s left out as core leaves an Index Only Scan’s; the per-row CPU - the probe of each entry walked, the visibility map’s test, the quals and the target - is divided by core’s parallel divisor; the lion lookups and the set (the start-up), the B-tree’s own cost and the heap’s pages are each participant’s whole. The partial path’s rows are divided likewise.
  • The set, built once (lo_shared_set()). A LoShared in the Gather’s dynamic shared memory, the B-tree’s parallel scan descriptor after it, holds the set’s state. The participants that need the set build it together, a part of its container keys each (“Built in parts”, below), as the serial node builds the whole, and copy each part into the query’s DSA as one image while it fits the room a Parallel Hash’s table would have - get_hash_memory_limit() for each participant planned, the leader among them (lo_image_write(): the container keys, each container’s offset, the containers, and whether the part is exact); one with no part left sleeps on a condition variable while the others finish theirs (the wait event LionOrderedSetBuild on 17 and later, the generic Extension on 16), and then each reads the containers where they are (lo_set_attach()), with a directory, the container keys and the walk’s visited bits of its own. A set larger than the room is built by each participant for itself, spilling as the serial node’s does (§30.4), and so is every set under pg_lion.enable_shared_set = off. A builder’s error ends the query: the leader rethrows it at its next interrupt check, which its wait makes, and its abort ends every worker.
  • What a participant does with an entry is the serial node’s, on the entries it is handed: the probe, the visibility map’s test in index-only mode with a heap fetch for a page it does not call all-visible, the rechecks - the B-tree’s under xs_recheck, and an inexact set’s on the index tuple or, under LO_FLAG_HEAPRECHECK, on the heap tuple fetched for it (§40.4) - and the page’s or the tuple’s predicate lock.
  • Not in a parallel walk: the lazy set (§30.4) - one set serves every participant, and each would reach other ranges first - and the fetch-and-sort switch, whose fetch of the members left would need every participant’s; a parallel walk keeps the B-tree’s order to its end, which a Gather Merge needs of each participant anyway. The early stop (§30.4, “Stopping early”) is each participant’s: it counts the members it met against the set’s, which a participant meets all of only when the others met none, so it rarely fires; a count shared among them would need the met-before bits shared too.
  • Rescans. Before a Gather’s workers are launched again (ReInitializeDSM), the B-tree’s parallel scan starts over (index_parallelrescan()) and the set’s shared parts are freed, the leader’s view of them let go first, to be built again by the next run, whose parameters may be others; the run is counted (gen), and a set a participant needs again in the same run - a lion Param changed under it, which no plan of this section’s has - is its own. A Gather Merge that a LIMIT shut down and that is rescanned initialises its memory afresh.
  • EXPLAIN names the node Parallel Custom Scan (LionBtreeScan) or (LionOrdered). Under ANALYZE the counters are the leader’s own and the workers' sums, which each worker adds as it shuts down and the leader takes as it does (a LIMIT that stops the Gather early leaves the workers' out, as §27’s parallel node does); Lion Set shows the set as the leader had it, or its builders' when the leader took no part, shared when the participants read one copy, and in N parts when it was built in more than one. A walk that is not parallel-aware but ran in a worker only - under debug_parallel_query, or the inner side of a join a worker ran - prints no counters: they stayed in the worker, since a CustomScan’s DSM hooks are called for a parallel-aware node alone, and the leader’s zeros would mislead.

Visibility. §40.5’s argument is per TID and holds in every participant. The snapshot is the leader’s, which every worker restores. The visibility map’s bit is tested for each member after the entry was read under the leaf page’s lock, as in the serial walk, by whichever participant was handed that page; the participant keeps its pin on the leaf while it tests the map (xs_want_itup, nbtree’s dropPin rule, which a parallel scan follows as a serial one does), so VACUUM cannot free the heap slots the page’s TIDs name until it has moved on. The set is built after the query’s snapshot was taken, as the serial node builds it, by whichever participant builds it: §30.5’s argument for a recycled TID rests on that order, not on the builder. Every participant that reads the set reads the same members and exactness, and rechecks as the serial node would. Under SERIALIZABLE the predicate locks a worker takes are the leader’s transaction’s (core’s parallel SSI): what the set’s build reads of the lion indexes is locked as it is read (§9), by whichever participant builds it, and each participant takes the heap page lock of each row it returns without a heap visit. test/isolation/btreescan_parallel_serializable.spec is §40.5’s write skew read by parallel walks, and must end in a serialization failure.

Tests. test/sql/btreescan_parallel.sql, on a synthetic 40,000-row table whose heap is several times its B-trees, with every parallel cost at 0: each query run as CREATE TABLE AS - a plpgsql cursor never runs in parallel - through the partial walk below its Gather or Gather Merge and through the ordinary plan (the walks off, sequential scans only, serially), the answers compared in order under an ORDER BY and as multisets otherwise, the node, its mode and the Gather named; and the counters of each parallel walk, summed, against the serial walk’s (the lazy set and the switch off, a serial run first so that both meet the same index entries killed by an earlier heap fetch): the members met, the heap fetches, the rows the recheck removed. ANDs, ORs, IN lists and multi-key filters, a B-tree qual beside them, aggregates and a GROUP BY, both orders, a LIMIT, an empty answer and heap mode; a vacuumed table (no heap fetch, the set built once and shared); rows inserted in the transaction, HOT, INCLUDE-column, lion-column and key updates and deletes before VACUUM, fetched by every participant; sets made inexact by a tsquery NOT, rechecked on the heap tuple in every participant; the set each participant’s own (pg_lion.enable_shared_set = off), and shared at hash_mem’s least; rescans of a Gather on a nested loop’s inner side and of a Gather Merge a LIMIT shut down, a generic plan’s and an initplan’s parameter in the lion filter; the leader taking no part; the walk on either side of a parallel hash join and below an anti join; and §27’s FK-side count over the walk of its dimension, serially and as a parallel LionCount, for the inner, semi and anti joins and a GROUP BY of a column the B-tree includes, chosen over the dimension’s bitmap heap scan at random_page_cost 4 and 1 and with the dimension’s pages not all-visible.

Measured (2026-10-20, PostgreSQL 18.6 release build, a VM of four cores shared with other work, shared_buffers 1 GB, warm; medians of five unless said). A synthetic table of 3,000,000 rows of 340 bytes - a 1.1 GB heap, 142,858 pages, and a 64 MB primary key of 8,228 pages - with lion indexes on a (20 values), b (3) and h (2); SELECT sum(id) ... WHERE <filter>, the walk of the primary key forced (every core scan off) and the plan without the walks:

filter walk, serial 2 workers 3 workers without the walks
h = 1 (50%) 239 ms 177 136 749 (bitmap heap scan); 244 (parallel sequential scan, 2 workers)
a = 3 (5%) 184 150 119 124 (the lion index’s scan)

The walk is CPU-bound - 60 to 80 ns an entry, the probe and the row - and divides less well than its entries: 1.2 to 1.4 times faster on two workers and 1.5 to 1.8 on three, the set built before the walk and the four cores busy. Planned with nothing disabled, the 50% filter takes the walk, parallel at two workers; the 5% one takes it serially as well, warm the slower by 35 ms, the heap’s pages priced as reads (§40.3, “Warm and cold”).

The set shared against one per participant, under pg_lion.enable_shared_set, for §40.2’s a IN (<10 of 20 values>) AND b IN (<2 of 3>), a set of 2,233 containers that takes 26 to 32 ms to build: 192 ms shared against 214 each its own on two workers, 152 against 197 on three (medians of seven). Built in every participant at once, the set costs the same time on idle cores but as many times the CPU and memory, which on four busy ones shows; shared, it is built once and held once. So it is shared, and each builds its own only when it would not fit.

Built in parts (2026-10-26). The first participant to need the set built it whole while the others waited, idle for the whole build - seconds on a dimension far larger than memory (§40.10), every worker of the Gather waiting on one. An AND or an OR of sets is decided one container key at a time, so the set’s part at a range of container keys is the AND or the OR of its leaves' parts there, and the whole set is its parts in key order: the participants can build it a range each, with nothing to merge.

  • The parts (lo_parts_plan()). The first participant to need the set cuts the relation’s container keys - its blocks now, 64 to a key at 8 kB - into LO_PARTS_EACH (2) for each participant planned, at most LO_SHARED_PARTS (64), each of at least pg_lion.shared_set_part_blocks (16,384 blocks, 128 MB at 8 kB); the last part is open-ended, for the rows added since. A relation of fewer than twice that has its set in one part, built whole by one participant as before. Two for each participant, because the leader starts building before the workers have started and a participant that finishes early can take another; not more, because a part has a cost of its own: each walks a range’s entries again (“Measured”, below).
  • A part’s build (lo_build_part()). Each leaf’s source is bounded to the part’s keys (lion_source_bound()): a SETS stream is sought to the first key and left at the last; a LIST’s batches and a WALK’s entries are each sought as they open and left at the last key; a UNION’s windows start at the first key and stop at the last; a WINDOW’s other columns are sought likewise. A sparse segment can start below the key sought, and its keys below are skipped. The leaves are combined as for the whole. A part is exact when every leaf’s source was; the set is exact when every part is. An index’s scan is counted once, by the first part’s build.
  • Taken, built, put in the DSA (lo_share_part()). A participant takes the next part nobody has (nextpart, under the struct’s spin lock), builds it in its own memory, copies it into the DSA when the room still has it, and lets its own copy go; the last part done ends the build (LO_SHSET_SHARED) and wakes the waiters. A part that does not fit the room ends the building: the parts nobody has taken are counted done, the last part done frees the ones put in the DSA, and each participant builds its own set (LO_SHSET_OWN). With no room at all (pg_lion.enable_shared_set off) nothing is built in parts. A participant’s view (lo_set_attach()) reads each part’s containers where they are, with the parts' container keys copied in order into a directory of its own - a set of one part reads its keys in place too.
  • What a waiter can wait on. A participant takes a part only from inside the build, and finishes it without waiting on anything: the build reads lion indexes and its own memory, and no tuple queue. So every part taken ends, and the waiters with it, or the query ends with its builder’s error.
  • The probe’s set (§40.10, “Parallel query”) is built the same way: below a Gather a nested loop’s probe is parallel-aware for its set alone.
  • The price is not changed. A walk’s (above) charges each participant the whole build, which a set built in parts takes about the participants' share of; a probe’s spreads the build over the loop’s scans, which at a participant’s share of the scans is the build divided by core’s parallel divisor - what the parts take of each, as long as the set has parts enough for every participant. A set of one part (a relation under 256 MB) is built by one participant while the others wait, and that probe’s price is low by the divisor, on the cheapest builds.

Measured (2026-10-26, PostgreSQL 18.6 release build; the four-core VM shared with another benchmark, load 2 to 3; a timer around lo_shared_set() in each participant, the longest participant’s time until the set is in hand, medians of five). A 4,000,000-row dimension of 665 MB - 85,107 blocks, 1,330 container keys - probed by a parallel nested loop of 10,000 outer rows, two workers planned, at 1,024-block parts so that the parts are LO_PARTS_EACH for each participant:

set whole, one builder 1 part each (3) 2 each (6) 4 each (12)
tags overlap of four, a bool, two IN lists (a SETS source each) 10.1 ms (7.6 to 26.8) 10.7 7.9 7.6
a range over 3,000 of a column’s 10,000 values, beside a bool 119 (90 to 142) 94 106 105

The workers start a few milliseconds after the leader, so on a build of ten the leader builds most of the parts; a range’s every part walks its 3,000 entries again, 25 to 30 ms a part on this table, so that more parts cost more than they divide. At the default part size the table’s set is 5 parts, 88 to 104 ms for the range against 130 to 171 whole. A build of seconds - a dimension far larger than memory - leaves the workers' start a smaller share of it; this machine holds no such table to measure. The parts take the memory one set took, where each participant building its own took it as many times over.

Tests. test/sql/btreescan_parallel.sql, section 11: at 64-block parts - six for the three participants planned - the answers and the counters of parallel walks against the serial walk’s over a SETS source, a WALK of a range, a LIST longer than a batch, a UNION of a query that needs every row, an OR, an empty answer and a WINDOW of a long range beside another column’s set in one index; the leader taking no part; no room; a Gather run again, its parts built again each run. test/sql/btreescan_param.sql, section 8, the same for a parallel nested loop’s probe, and its answers with Memoize enabled (chosen over the probe or not by a near tie of the prices, which 18 and 19 break differently).

40.10 A nested loop’s probe: the walk parameterized (2026-10-23)

The shape is

SELECT d.id, count(*) AS n FROM dim d JOIN fact f ON f.dim_id = d.id
 WHERE <clauses lion indexes on d answer> AND <a selective filter on f>
 GROUP BY d.id ORDER BY n DESC LIMIT 100

where the fact side’s filter leaves few rows against the dimension’s, so that core’s plan is a nested loop over them into the dimension’s primary key: an Index Scan that fetches each outer row’s dimension row from the heap to test the filter on it, and throws most of them away. On a dimension of this shape far larger than memory, with several filters, that is hundreds of thousands of heap fetches, most of them rejected by the Filter, read cold. Nothing of §40 so far could be the loop’s inner side: every path the node built was unparameterized (§30.8 declined parameterized paths, “an inner side needs no order”), and the walk of the whole primary key under a hash join reads every entry of the key where the loop wants six hundred thousand. What the inner side needs is not an order but §40’s index-only half: the outer row’s key looked up in the B-tree, the entry tested against lion’s set of the dimension’s filter, the visibility map asked, the values taken from the index tuple, and the heap read only for pages not all-visible.

Nested Loop
  ->  Index Scan using fact_tsv on fact f
        Index Cond: (tsv @@ '''w7'' & ''x3'''::tsquery)
  ->  Custom Scan (LionBtreeScan) on dim d
        Index: dim_pkey (index only)
        Index Cond: (d.id = f.dim_id)
        Lion Cond: ((d.tags && '{t1,t3,t5,t7,t9}'::text[]) AND d.verified AND (d.hue = 'green'::hue) AND (d.grade = 'alpha'::grade))
        Lion Indexes: dim_tags, dim_verified, dim_hue, dim_grade

The probe is a LionBtreeScan parameterized by the outer relations, as core’s parameterized Index Scan is: its B-tree quals include the join clauses (id = outer.fk), and each rescan is a descent to the outer row’s key. Its set is built from the inner relation’s restriction clauses that lion answers - never from a join clause or an outer row’s Param - once per execution of the plan, and kept for every scan the loop makes. A scan then costs a descent, a membership test per entry, a visibility-map test per member and the values from the index tuple.

  • Which paths (lo_btree_paths(), the probe loop of lo_set_rel_pathlist()). Core’s own: the scratch copy of the relation that sees one B-tree (§40.2) now keeps the relation’s join clauses (lo_scratch_rel()’s joins), so create_index_paths() makes for that B-tree the parameterized IndexPaths it makes for an inner side - the join clauses as index clauses, beside the restriction clauses the B-tree matches: (grp, k) under grp = f.g AND k = 3 probes both columns. (Its equivalence-class joins it made all along: match_eclass_clauses_to_index() asks index->rel, the relation itself, not the copy. Those paths were then dropped - and, being paths, stopped §40.2’s whole walk of a B-tree core made no unparameterized path of, which kept a primary key’s whole walk out of every join on it; the fallback now looks for an unparameterized path, and that bug is gone.) The parameterized paths kept are forward ones (a backward twin walks the same entries), one per parameterization and set of index clauses (lo_same_indexclauses(): an IndexClause has no equal() support), that pass lo_indexpath_ok(). Each is paired with every lion access of §40.2, which are unparameterized - built from the restriction clauses alone - and priced, less the pairs that add nothing (lo_lion_adds()); the residual (lo_residual()) is taken over the restriction clauses and the parameterization’s join clauses (ppi_clauses), a join clause the B-tree’s index clause answers left out - the same RestrictInfo, or one of the same equivalence class (lo_ord_answers()), as create_indexscan_plan() drops it as redundant - and the B-tree must cover what is left (§40.1), the outer side’s Vars being Params. The path carries get_baserel_parampathinfo()’s ParamPathInfo and its rows, claims no order (an inner side’s order is no use to a nested loop, and a merge join takes unparameterized inputs) and is never partial. Core then builds the nested loops, and the Memoizes above them, as for its own parameterized scans. §30.1’s precondition that the relation have no LATERAL references stands: a LATERAL relation’s paths are parameterized by what it references, not by join clauses, and the node’s others are not.
  • The set is the relation’s, as for an unparameterized walk: the lion access’s lookups, built after the snapshot is taken, held in the query’s memory, and rebuilt on a rescan only when a Param of the lion quals changed (§30.4) - which an outer row’s never is, since no join clause reaches the lion side. A correlated subquery’s or a generic plan’s Param in the lion filter still rebuilds it for each value, counted in Lion Set’s builds. It is built lazily (§30.4, “The set, lazily”) when the loop’s scans together are expected to meet fewer than LO_SWITCH_MIN_WALK (10,000) entries - a few outer rows probing a few keys, the memo of the keys probed kept across the scans - and at once otherwise (LO_FLAG_BUILD).
  • Cost (lo_cost() with loops and scans; lo_probe_scans()). A scan’s price is §40.3’s with the walk the parameterized IndexPath’s: core’s indextotalcost for one scan (the descent, and the B-tree’s pages amortized over the loop count by core’s amcostestimate), each entry’s probe at LO_PROBE_TUPLES × cpu_tuple_cost + cpu_operator_cost, a member’s visibility-map test at cpu_operator_cost, the residual and the target per row; and the heap pages as cost_index() prices a parameterized index-only scan’s - the fetches of all the loops' scans through the Mackert and Lohman formula, scaled by 1 - allvisfrac, every fetch random, divided by the loops - which on a vacuumed table is nothing, where core’s Index Scan pays a random page per outer row. The loops are what core’s get_loop_count() makes of the parameterization: the fewest rows of the outer relations, a semi join’s inner side as its distinct values. The set’s build (the lookups, the copy, the spill) is the start-up, divided by the scans S it is spread over: core’s nested loop charges an inner path’s start-up again on every rescan (cost_rescan() of a CustomPath), so S scans of build / S charge it once. S is the loops when enable_memoize is off, and otherwise the distinct values the outer side hands the probe, Min(D, loops) with D from estimate_num_groups() over the outer side of each join clause (a clause Memoize cannot key on gives the loops): a Memoize rescans its inner side for a value it has not cached alone, and cost_memoize_rescan() charges a scan’s price per expected miss, so a build spread over the loops would be charged only misses / loops of it in the plan core likes best. Spread over the distinct values, the Memoize’s plan pays the build once and the plan without it loops / D times, though it builds once too: an overcharge that tips the choice only where the values repeat, and there the Memoize’s D scans against the loops' are the faster plan anyway. lo_margin (§39) applies as to every LionBtreeScan: the price divided by pg_lion.pushdown_margin, 1 by default. Where the outer side is small the build is not spread far enough: three or twenty outer rows take core’s Index Scan, which fetches their rows, eighty the probe, and a dimension of a few hundred rows takes a hash join of a sequential scan (test/sql/btreescan_param.sql, section 10).
  • The build’s price (lo_build_excess()). §30.3 prices the lookups as the lion access’s bitmap paths (cost_bitmap_tree_node(), lioncostestimate()), a cpu_index_tuple_cost a TID - what a lion index’s scan hands a bitmap. The set’s build hands nothing over a TID at a time: each leaf’s posting sets are read and combined a container at a time (lo_build_leaf(), lo_set_combine()), a dense container’s members costing nothing apiece. For the walks of §40 the walk is most of the price and this did not decide; for the probe the build is the whole fixed cost, and it decided wrongly: four dense filters on a 6M-row dimension of 4,688 containers were priced 88,000 and built in 57 ms, on a 50M-row one of 12,808 containers 715,000 and 80 ms - 1.5 and 0.8 us a container of a posting set, against the 10 ns a TID stands for at the reference’s 500 units a millisecond, and the error grows with the rows a container holds. So the probe charges a leaf at most LO_BUILD_PIECE_TIDS (80) TIDs a container of each entry it reads - 0.8 us a piece, the lower of the two measurements, where a tie should fall - and takes its TIDs past that back out; a sparse leaf keeps its price, and the other walks keep §30.3’s.
  • Cold reads, declined. §39’s “A nested loop’s cold reads” prices a loop’s pages the cache cannot hold at pg_lion.cold_page_cost, and a probe saves core’s scan one heap page an outer row. Taking the probe against core’s scan that way (its leaf, its members' pages not all-visible and the lion indexes' pages once for the set, against core’s leaf and heap page) was built and measured: with the build priced as above it moved none of the plans measured below - the probe’s own price is the lower wherever its cold reads are the fewer - and on the regression tables, where the lion indexes' pages over a few dozen scans outweigh core’s heap pages, it never applied. It was taken out rather than kept untested.
  • Execution (LO_FLAG_PROBE, st->probe). Each rescan restarts the walk with its keys evaluated again (core’s run-time keys, ExecIndexBuildScanKeys()), so a NULL outer key makes a strict operator’s key unsatisfiable and the descent finds nothing, as in core’s Index Scan. A probe marks nothing: the met-before marks of §30.4 are counted for the early stop and the fetch-and-sort switch, neither of which a probe has, and the visited bits' directory, remade for every scan, is as long as the set’s places - O(the set) a rescan for a scan of a few entries. A TID met twice in one scan is then fetched twice; the second time it names a tuple written after the snapshot (§30.4, “A recycled TID”), on a page whose all-visible bit the write cleared, which the heap’s visibility check skips. The lazy set’s memo is not walked at each scan’s reset for the same reason. The visibility map’s buffer is kept across rescans, as core’s Index Only Scan keeps ioss_VMBuffer: the next outer key is often on the same map page. lo_begin() refuses a plan that marks a probe parallel-aware, a lion column’s walk, a lionvec order, setless or sorted.
  • Memoize, and rescans with no Param changed. A nested loop puts its parameters in the inner side’s chgParam for every outer row. Under a Memoize the probe’s chgParam accumulates while the cache answers, and the probe is rescanned by its next ExecProcNode() with all of them; a rescan of the Memoize with none changed - its parent rescanned with the same values, a cursor moved back
    • rescans the probe directly (ExecReScanMemoize()) with chgParam empty. Either way the set is kept unless a lion Param is in chgParam (lo_rescan()): the probe restarts its walk with the same keys and the same set. Memoize’s own purge on a non-key Param is untouched by any of this.
  • EvalPlanQual and row marks. A row mark of the dimension - FOR UPDATE OF d, FOR SHARE, or the ROW_MARK_REFERENCE every other relation of a locking query or an UPDATE gets - puts its ctid in the target, which no index returns (§40.1): no probe is offered, and EvalPlanQual never meets one. lo_recheck() would evaluate the original clauses, the join clause with the outer row’s Param among them, and the lion qual, as for any walk.
  • Parallel query. The probe is parallel_safe when the relation is (rel->consider_parallel), so a partial nested loop below a Gather runs it whole in each participant, the outer side divided
    • a scan is a descent for one key, with nothing to divide. Core calls a CustomScan’s DSM hooks for a parallel-aware node alone, so the probe is parallel-aware for its set: the join hook (lion_ordered_join_probes(), 2026-10-26) gives each partial nested loop core made over a probe
    • or over a Memoize over one - a copy of the probe marked parallel_aware, its own path left as it was for the serial loops. Its participants then build the set together and read it from the Gather’s shared memory (§40.9, “Built in parts”), where each had built its own; the struct has no parallel scan of the B-tree, each participant descending it for its own outer rows. EXPLAIN names it Parallel Custom Scan (LionBtreeScan), and its ANALYZE counters are the participants' sums, as a parallel walk’s. A lazy set (above) stays each participant’s own - the containers its own outer rows reach - and is built whole for itself when its probes have cost a build.
  • Partitions. A partitioned dimension’s Append takes each partition’s probe when its parameterization is the Append’s (reparameterize_path() gives a CustomPath none). A partitionwise join’s nested loop joins a fact partition to the dimension partition of the same keys, whose probe is parameterized by the fact’s parent and is moved to the partition by core (reparameterize_path_by_child(): at path time on 16, at plan creation on 17 and later) through ReparameterizeCustomPathByChild (lo_reparameterize()): the B-tree’s IndexPath is reparameterized by core’s own code; the lion access is the partition’s and unparameterized, and the column mapping the B-tree’s.
  • EXPLAIN is §40.6’s LionBtreeScan with core’s Index Cond: the index quals with the key on the left, which core’s Index Scan prints (its indexqualorig is that list) - (d.id = f.dim_id) however the query wrote the join. Every walk prints them now (LO_EXPR_ORDQUALS in place of the original clauses, LO_EXPR_ORDORIG, which stay the recheck): 100 > k as (k < 100), a prefix LIKE as the range it is searched by, with the LIKE in the Filter, as core’s. No expected output of the suite changed; the old display printed such a LIKE in both. Under ANALYZE Index Entries Walked, Lion Set Hits and Heap Fetches are the sums over every scan, and Lion Set shows the set built once - 48 containers, exact, with no builds count - or N builds where a lion Param made it rebuild; a Memoize above it prints its own hits and misses.
  • Correctness. Per scan, §30.5’s and §40.5’s arguments hold unchanged: the B-tree’s entries and the set’s members are both HOT chains' root TIDs, the visible version reached from a root has the index’s values, the visibility map’s bit is tested under the leaf’s pin, and a row returned without a heap visit takes the predicate lock of its heap page. Across scans the snapshot is the execution’s, so a set built after it serves every scan of it, a cursor’s pieces included. test/isolation/btreescan_param_serializable.spec is §40.5’s write skew read through the loop’s probes (two HOT updates of rows the other transaction read index-only): it must fail.
  • Settings. No new one. pg_lion.enable_btree_scan gates the probe with the other unordered walks; core’s enable_indexscan disables it as it disables the B-tree’s own scans (§40.2), and enable_indexonlyscan = off takes it away, since the probe is index-only or nothing. pg_lion.enable_plain_scan and pg_lion.enable_ordered_scan do not touch it: it is neither a scan of a lion index nor an ordered walk. enable_memoize decides S above. With enable_nestloop = off the probe is still offered, and on 18 it then costs a parallel plan its partial hash join: 18’s add_partial_path_precheck() compares a new partial path with the join relation’s existing ones by cost alone, so the disabled partial nested loop over the probe, cheaper than the partial hash join, keeps that join out, and the plan is a hash join of two Gathers; 19 and later compare the disabled nodes first and take the hash join (test/sql/btreescan_parallel.sql’s section 9 switches the probe off with the loop). Not offering the probe while nested loops are disabled would avoid it, and was not done: a disabled path is still the plan where nothing else joins, and the quirk is 18’s alone.
  • A count over the join (§27, §39). A LionCount meets core’s plan above the join in the competitor’s units, and the competitor (lion_competitor_path()) is core’s cheapest path with no lion path inside: where the probe’s nested loop is the join’s cheapest path, every aggregate core builds over it holds lion, the competitor is none, and the count is priced at rate 1 where a hash join’s 0.5 had been. test/sql/rangesource.sql’s join of a ten-row dimension then planned the probe’s loop over its count, 2.1 ms against 1.8; that section now runs with the walks off.

Declined, and why.

  • A heap-mode probe (a B-tree that does not cover): a scan would fetch every member’s heap tuple, which is core’s Index Scan with lion’s test in place of its Filter - the heap fetches of the members saved only where the set rejects most of them, and a plan that loses to core’s wherever the set does not; the index-only probe is the one that removes the heap from the loop.
  • A parameterized ordered walk (LionOrdered), a lion column’s walk and a lionvec order: an inner side’s order is no use to a nested loop, a merge join takes its inputs unparameterized, and a lion column has no descent to a key.
  • A set from the join clause - the outer row’s key in the lion side: a set per outer row, which is §30.4’s rebuild per Param at a loop’s scale; the B-tree’s descent does that part.

Not done.

  • The price of the build does not shrink with a LIMIT above the loop: a fraction of the loop’s scans still pays its share of a build that happened whole on the first.
  • Duplicate outer keys with enable_memoize on but no Memoize chosen are charged the build loops / D times; it is built once.
  • A parallel nested loop’s participants build the set together (§40.9, “Built in parts”), and are charged a participant’s share of it; a set of one part, a relation under 256 MB, is built by one while the others wait.
  • The container price of the build is the probe’s alone. The walks of §40 keep §30.3’s TID price, so a hash join over the dimension’s whole covering walk carries a build priced several times the probe’s for the same set: at 600,000 outer rows into 6M dimension rows the two plans tie on price and the probe’s loop, chosen, runs twice as long (below). Refitting §30.3’s price moves every walk’s plans, each to be timed against the plan it replaces.
  • A build is not repaid by warm heap fetches: 14,000 outer rows into the 6M-row dimension take the probe (a 57 ms build) and run 1.7 times as long as core’s loop with its heap fetches in the cache. The probe is for a dimension the cache does not hold, and the planner’s page costs are the user’s statement of that.

Tests. test/sql/btreescan_param.sql, on a 100,000-row dimension - its heap several times its primary key - with lion indexes on an array, a bool and two enum columns and a (grp, k) INCLUDE (id) B-tree, and a 200,000-row fact table with a lion tsvector index and every fiftieth key NULL: every query run through the probe (hash and merge joins off, lion’s paths priced at a thousandth) and through the ordinary plan (the walks off), the answers compared as multisets, the plan’s shape and Index Cond named. The target shape, chosen with nothing disabled (Index Entries Walked: 714, Lion Set Hits: 135, Heap Fetches: 0, the set built once), and core’s plan with the switch off; a few outer rows, NULL outer keys, a LEFT JOIN and an empty set; repeated outer keys with a Memoize (Hits: 26572 Misses: 2000, two thousand walks) and without (28,572 walks), the set built once either way; the two-column B-tree with a parameterized leader and a constant second column; a generic plan’s Param in the lion filter, a subquery rescanned with its Param in the fact’s filter (built once) and in the lion filter (built for each value); a cursor fetched in pieces; pages not all-visible - inserts, deletes, HOT updates and lion-column updates in the transaction, then after it and after VACUUM (Heap Fetches: 0); a parallel nested loop (CREATE TABLE AS, Gather over a nested loop over a Parallel Seq Scan and the probe, parallel-aware: its set built in parts and its counters the participants' sums, against the serial probe’s; a Gather run again; no room to share; few outer rows, each participant’s lazy set; Memoize enabled); a partitionwise join of hash-partitioned tables, the probe reparameterized per partition; the price with nothing disabled (above): 3, 20 and 80 outer rows and a 300-row dimension; and the settings - enable_indexscan and enable_indexonlyscan off, pg_lion.enable_plain_scan and pg_lion.enable_ordered_scan off (the probe stays), a column no B-tree with the key covers, FOR UPDATE OF f and FOR SHARE OF d (no probe). Section 2 also has a semi and an anti join, the Index Cond printed key first however the clause was written. test/isolation/btreescan_param_serializable.spec is the write skew above.

Measured (2026-10-23; PostgreSQL 16.15, the dev cluster: shared_buffers 2 GB, a VM of four cores and 15 GB shared with other work; warm, the tables in the OS cache; EXPLAIN (ANALYZE, TIMING OFF), medians of five or seven). Two synthetic dimensions with the target shape’s filters - an array overlap of five tags of ten, a bool true in two thirds of the rows, enum equalities of 70% and 80%: 18.7% of the rows pass - each joined from a fact table whose tsvector search picks the outer rows, GROUP BY d.id ORDER BY n DESC LIMIT 100: 6M rows of 410 bytes (a 2.3 GB heap, a 129 MB primary key) under 4M fact rows, and 50M rows of 134 bytes (6.4 GB, 1 GB) under 2M.

dimension outer rows planned its time core’s plan, the walks off core’s nested loop
6M 14,286 the probe 142 ms its nested loop, 84 ms (the same)
6M 100,000 the probe 536 hash join over a bitmap heap scan, 3,206 655
6M 300,000 the probe 1,573 hash join, 3,420 2,199
6M 600,000 the probe under a Memoize 3,322 hash join, 3,662 5,302
50M 7,143 core’s nested loop 54 (the same) (the same)
50M 50,000 the probe 352 its nested loop, 318 (the same)
50M 150,000 the probe 1,016 its nested loop, 1,045 (the same)
50M 300,000 the probe 1,874 hash join over a bitmap heap scan, 12,510 -

The 600,000-row line’s other plan with the walks on, a hash join over the dimension’s whole covering walk, ran in 1,627 ms (“Not done”). The set took 57 ms to build on 6M rows and 80 ms on 50M, warm (84 and 170 ms the first time); a scan of the loop, the descent, the probe and the visibility map’s test, 4 to 5 us, within a microsecond of core’s Index Only Scan rescanned for the same keys without a filter (2.56 s for 600,000). Warm, then, the probe ties core’s nested loop at 0.1 to 0.3% of the 50M rows and is 1.1 to 6.7 times faster where core’s plan reads the dimension’s heap whole. What it is for is the cold case, which a table in the OS cache cannot show: at 150,000 outer rows of the 50M table core’s Index Scan read 139,693 pages from outside shared_buffers and the probe 61,753, the primary key’s leaves and the lion indexes; on a dimension far larger than memory each of core’s is a synchronous read from the device, and the probe reads none of them. A 500M-row dimension was not built: its set’s build grows with the heap’s containers, 0.8 to 1.5 us each of each posting set, seconds for a dimension far larger than memory.

40.11 A correlated subquery’s calls (2026-10-23)

The shape is

SELECT d.id,
       (SELECT count(*) FROM fact f
         WHERE f.dim_id = d.id AND f.doc @@ '<a tsquery>'::tsquery) AS n
  FROM dim d WHERE <a filter on d>

or the same subquery as an EXISTS, or a scalar sub-select of another aggregate (max(f.id)): a subquery that refers to the query around it, which the executor runs again for every outer row, the outer row’s value arriving as an exec Param. Core plans such a subquery as a query of its own, with no loop count, prices each of its scans for one call, and cost_subplan() charges the plan’s price again per call. For core’s scans that is the convention a heap page is charged in. For lion’s it charged every call the pages of the first: the posting trees a sought set reads, the directory leaves its keys are looked up in, a set that §40.10’s probe builds once and keeps across rescans - the parts of a nested loop’s inner side that core’s parameterized paths spread over their loops (index_pages_fetched() with the loop count, cost_index()), and that a correlated subquery repeats exactly as a loop does. On a fact table with a lion index on the key and a tsvector and a covering B-tree on the key, a correlated count over a thousand outer rows was priced at a call as the same count in a prepared statement’s generic plan, and the subquery’s lion paths lost to core’s B-tree scans they beat inside a join of the same rows.

So a lion path whose quals hold an outer query’s exec Param is priced as the inner side of a nested loop over an assumed number of calls.

  • The calls (lion_outer_param_loops(), lion_selfuncs.c). The subquery’s exec Params are found in its quals (RestrictInfos and IndexClauses looked into) and each is looked up in the plan_params of the query levels above, where replace_outer_var() lists the outer Var it stands for while the subquery is planned, with that level’s range table index. The calls are the rows of the Var’s relation, and of several relations the fewest, as get_loop_count() takes the fewest rows of a parameterized path’s outer relations. The outer level is still being planned and its relations not yet sized, so the rows are the relation’s own - pg_class.reltuples scaled to its pages now (estimate_rel_size()), before the outer query’s restrictions, read once per planner run (the run cache of §33). A filter on the outer relation makes the calls fewer than that, and the price a call’s share of more calls than there are: index pages are spread further than they will be, which, as for a parameterized scan, matters only where the index is larger than effective_cache_size says the cache holds. Restricting the outer relation would need its clauses' selectivity before its level has its paths; it was not done.
  • What is not a loop. A Param that is not an outer relation’s Var - an InitPlan’s output, an outer aggregate or PlaceHolderVar, a prepared statement’s own (PARAM_EXTERN, its generic plan’s $1) - is evaluated once, or once per execution, and its scan keeps the price of one call. A relation of no known size (a subquery’s, a function’s, a table neither vacuumed nor analyzed that estimate_rel_size() makes nothing of) gives no calls either.
  • The lion index’s own scans (lioncostestimate()). genericcostestimate() takes the loop count times the calls, so the directory leaves and the pages it prorates are spread over both; the price of a set AND (§29.11, “One price for the AND of sets”) spreads its leaves and posting pages the same way (setand.leafpages + setand.setpages), its CPU a call’s. The heap is core’s, which prices a subquery’s heap pages for one call, and lion_plain_heap_correlation() keeps core’s loop count.
  • The walk of a covering B-tree (lo_cost(), lion_ordered.c). The subquery’s walk is a probe, as §40.10’s: its B-tree’s descents and pages priced by the B-tree’s amcostestimate at the calls times the loops, the build priced by containers (lo_build_excess()), and the build’s start-up divided by the calls as well as by §40.10’s scans: a set no Param reaches is built once and kept for every call (lo_rescan() rebuilds it only when a lion Param changed). A set whose Lion Cond holds the outer Param - the subquery’s key is in a lion column too - is built again at every call, as Lion Set’s N builds shows; its lookups' pages alone are spread over the calls (lo_lion_amortized(): each leaf of the lion access priced again at the calls, the difference taken off), the build’s CPU a call’s. Beside it the walk is offered the set without the Param, built once (§40.12). An InitPlan’s Param in the Lion Cond is set once per execution, before the calls, and rebuilds nothing: only a Param the levels above hand down (lion_has_outer_param(), their plan_params) counts. The heap’s pages are a call’s, as core’s.
  • The count (lion_cost_count_rel_one(), lion_count_rel_calls_share(), lion_plan_cost.c). A LionCount whose WHERE holds the Param is located again at every call (its rescan re-locates everything), so nothing is built once; the pages it reads - the directory leaves at random, an IN list’s buckets, the walked and the sought posting pages - are spread over the calls as fetches at random from the indexes its WHERE reads (index_pages_fetched() of the pages times the calls over those indexes' pages, divided back), the CPU of the lookups, the merges and the counts a call’s, and the heap a recheck reads a call’s.
  • Unchanged. cost_subplan(): a SubPlan’s price is still its plan’s per call, and the whole of the change is what a call costs. A join’s paths, which the planner sees inside their loop, carry Vars and no exec Params and are priced as they were; a parameterized path’s own loop count is core’s as before, the calls multiplying it only inside a correlated subquery. No setting: the price follows the shape, and the switches that remove the paths (pg_lion.enable_count_pushdown, pg_lion.enable_btree_scan, pg_lion.enable_plain_scan) remove them here too.

Measured (2026-10-23; the test’s tables: 100,208 dimension rows, 300,000 fact rows - eight keys of 5,000 rows, two hundred of 300 and the rest one to three - a lion index on the key and a three-word tsvector, a B-tree on the key including id; the subquery called for the 1,003 keys of one group; warm, effective_cache_size the default 4 GB). A call’s price, before and after, the generic plan’s being what a call cost before:

subquery’s scan a call, generic plan a call, correlated chosen there instead
LionCount 11.6 2.3 to 3.0 -
the lion index’s Index Scan (the count off) (core’s bitmap scan of the B-tree, 16.3) 6.3 core’s bitmap scan
LionBtreeScan (the count and the lion index’s scans off) (core’s Index Scan, 16.5) 2.8 core’s Index Scan

The table being warm and small, the plans' times are close: 16 to 20 ms for the count, 19 ms for the walk against core’s Index Scan’s 22, and 37 ms for the lion index’s Index Scan against 17 for a bitmap scan of it and 22 for core’s Index Scan - a lion index scan’s rescan costs a few dozen microseconds of CPU that §29.11 prices, and that the cache hides from nothing here. What the change prices is the index pages a cold loop reads once rather than at every call.

Not done. The outer relation’s restrictions (above). A subquery’s heap pages stay a call’s each: core’s convention for its own scans, which lion’s are compared with. A SubPlan core hashes (hashed SubPlan, its Params none) is run once and is not a loop; a correlated subquery under a Memoize does not exist in core’s plans.

Tests. test/sql/correlated.sql: the correlated count, its price at a call under half the generic plan’s and an InitPlan’s Param priced as the generic plan’s; the lion index’s Index Scan and the walk chosen in the subquery where the generic plan takes core’s bitmap and index scans; a scalar max() sub-select and an EXISTS through the walk; the walk with an InitPlan’s tsquery in its Lion Cond, its set built once and kept through every call; every answer compared with core’s plan, the lion paths off; and the join of the same rows, its plans unchanged.

40.12 A set without the key (lo_set_rel_pathlist(), 2026-10-24)

In a correlated subquery the walk is offered, beside the lion accesses of the relation’s restrictions, those of its restrictions without the clauses that hold the outer query’s Param (lion_has_outer_param()). The walk keyed on the Param answers the key; the set of the rest holds no Param and is built once and kept for every call (lo_rescan(), §40.11), its build divided by the calls (lion_outer_param_loops()), where the set with the key is built again at each. Lion Set shows no builds count for it; the test’s ten calls show 85 containers, exact, against 50 containers, exact, 10 builds for the set with the key.

41. Expression columns in the count pushdown (2026-10-09)

A lion index on an expression - (doc->>'status'), ((doc->>'qty')::int), (lower(email)) - is a scalar column whose values the table does not store. Its entries are those values, one per row, exactly as a column’s are, so everything the count pushdown does with a column’s entries holds for it: a clause on the expression is a posting set, a GROUP BY on it is the walk of its entries, a count(DISTINCT) of it is the number of entries with a visible row. The pushdown treats each expression such an index holds as a column of its own, an expression column, numbered past every heap column (LION_VCOL_ATTNO(i), MaxHeapAttributeNumber + 1 + i, lion_customscan.h).

Matching (lion_vcol_collect(), lion_vcol_subst(), lion_plan_match.c)

  • Which expressions: those of the table’s valid, non-partial lion indexes - and, in an attempt for a partial index (§45), that index’s - under a scalar opclass, deduplicated, at most LION_MAX_VCOLS (64), collected when the pushdown is tried on one table that is not partitioned. A multi-key opclass over an expression (to_tsvector(...)) is not one: its entries are keys, not row values. Nor is a binary-coercible cast of a column, (v::text) of a varchar: stripped of its relabel it is the bare column, which the plan cannot carry as an expression column, and which an index on the cast does not answer as a column either (a clause on it declines, as before).
  • The query’s expressions: the GROUP BY items, the WHERE clauses, the HAVING and the target list are rewritten with every subexpression equal to an expression column - as core’s match_index_to_operand() compares them, relabels stripped - replaced by a Var of that column’s number. Everything downstream then sees Var op Const, a grouping Var or an aggregate’s Var argument and takes the paths a column takes; an expression that is not one of them (upper(doc->>'s') = 'A' against an index on doc->>'s') stays as it was and declines as before.
  • The index: lion_find_roaring_index() answers an expression column with the first key column, in index order, whose expression is equal to it, as a column is answered by the first key column on it. The expression must be written as the index writes it, under its collation.

The rewrite lives only while the path is built: the scope is entered in lion_try_count_path() after the relation is known and left in a PG_FINALLY, and anything that asks the planner’s statistics - estimate_num_groups(), examine_variable(), clause_selectivity() - gets the expression back first (lion_vcol_unvar()), so that core finds the index expression’s own statistics as it would for the ordinary plan. The expression-statistics hook (§33) declines for a partitioned table or an inheritance parent, as the column hook does: one index of the parent counts no partition’s rows, and a partitioned index has no pages to read a count from.

The plan

  • VCOLS (custom_private member 19, shape 22): the expression columns' expressions, renumbered to varno 1 when the plan is made (lion_plan_custom_path()); setrefs.c does not look into custom_private. The codec checks each is an expression of the table’s columns, and lion_count_priv_check() that every expression column a member names is one of them, and that there are none with partitions or a join, and that a coalesce group is not one. A coalesce of a column beside a clause on an expression column (GROUP BY coalesce(c, 0) ... WHERE doc->>'a' = '1') is planned: the clause is a posting set like any column’s.
  • custom_scan_tlist holds the expressions themselves, not the virtual Vars: setrefs.c matches the plan’s target list and HAVING against it by equal(), and finds doc->>'s' there as it would a column. A HAVING that names an expression column is wanted as that expression, not as the columns inside it (lion_vcol_refs()), and so is the partial target of the decoded walk, a parallel GROUP BY or a partitioned one (lion_make_partial_target()): core’s make_partial_grouping_target() would pull the columns out of a HAVING like count(*) > 100 OR doc->>'a' = '2', or out of a printed expression that a WHERE equality took out of processed_groupClause, and the node cannot emit doc.
  • The executor finds each key column the same way (lion_index_col_for() with VCOLS, lion_index_col_is_vcol()), comparing with the index’s own expressions, which are of varno 1.
  • EXPLAIN prints the expression, deparsed against the table, where it prints a column name.

Privileges

The ordinary plan computes a grouped or counted expression in its scan’s projection, so it checks EXECUTE on the expression’s functions; a WHERE clause’s functions are in baserestrictinfo already. The plan adds the functions of each expression column it uses to exec_funcs (lion_expr_functions()), and test/sql/node_contracts.sql checks both a filtered and a grouped expression column against the ordinary plan.

Not done

  • Partitioned tables: each partition’s index would have to be matched to the parent’s expression through the partition’s own attribute numbers.
  • FK-side joins (§27), aggregates over keys (§37) and coalesce() (§10) of an expression column.

Tests

test/sql/exprindex.sql: jsonb fields and a cast, filtered (=, IN, <>, ranges, IS NULL, OR), grouped by one, two and three columns (with a HAVING on a grouped expression, and with one pinned by the WHERE), count(DISTINCT), HAVING, a coalesce group beside an expression clause, an index on a binary-coercible cast of a column next to the expression indexes, a generic plan’s parameter, through LionCount and through the ordinary plan before and after updates and deletes, only differences printed; what declines; and EXECUTE on a function an expression calls.

42. jsonb containment and existence: jsonb_contains_ops (lion_jsonb.c, 2026-10-09)

A multi-key opclass (§17) for jsonb that answers @>, ?, ?| and ?& with the rows a sequential scan and GIN return, from the posting sets alone wherever the keys make that exact. It is the DEFAULT for jsonb, as GIN’s jsonb_ops is GIN’s: USING lion (doc) is this class. Lion’s own jsonb_ops - whole documents under = and the ranges, btree’s meaning of the name - is kept beside it, not the default, for an index that answers =, the ranges and count(DISTINCT doc), which a multi-key column cannot (§17). Strategies 11 .. 14 are its operators; it names its own extractValue and extractQuery (lion_jsonb_extract_value(), lion_jsonb_extract_query()), which take lion’s strategy numbers (lion_gin_strategy() passes them through). STORAGE is bytea with byteacmp as the ordering, so keys compare bytewise and the directory is ordered (§21).

42.1 Keys

Each key is a bytea whose first byte says what it is:

  • R + o | a: the document is an object, an array. A raw scalar has none.
  • L names value marks: a scalar at a path. names is each object key on the path from the root as 0x01, its length (unsigned LEB128) and its bytes, then 0x00; value is z (null), t, f, n + length + numeric_normalize()’s text, or s + length + the string’s bytes; marks is, for each gap before, between and after the names, the number of array levels in it. Array indexes are not on the path, because @> matches an array element wherever it is.
  • C names type marks: a container (o, a) at a path below the root.
  • E name: a top-level object key, a string element of a top-level array, a raw scalar string, which is what ? tests (jsonb_exists()).
  • H + SHA-256: any of the above whose varlena would pass LION_MAX_KEY_SIZE.

A raw scalar document is a one-element array to the iterator, and gets the L key a scalar element of a top-level array gets: '["foo"]' @> '"foo"' and '"foo"' @> '"foo"' are both true, '"foo"' @> '["foo"]' is false (no R a). Numbers go through numeric_normalize(), as GIN’s jsonb_ops does, so 1, 1.0, 1e0 and -0/0 agree; strings are bytes, as jsonb equality compares them. The names come before the value and the array levels after it, so one run of the ordered directory holds every array nesting of one path = value, which a lax jsonpath search would want.

42.2 Queries

@> extracts the query’s L keys, a C key for each EMPTY container in it (a non-empty one is implied by the paths below it), and an R key when the query is an array (which must not match a raw scalar) or an empty object, and ANDs them. The AND is exact unless one array element of the query contributes two or more keys: {"a": [{"b": 1, "c": 2}]} must find both in the SAME element of the row’s a, and {"a": [{"b": 1}, {"c": 2}]} has the same keys. Then, and when a key was hashed, extractQuery answers LION_SEARCH_MODE_LOSSY, a search mode of lion’s own beside GIN’s: lion_extract_query() calls the query not exact, and lion_extract_query_superset() builds the same AND with LION_QMODE_LOSSY, which every caller already rechecks (§17, “Literal queries through the superset”). Duplicate query keys are dropped.

? is one E key, ?| their OR and ?& their AND, NULL array elements skipped as jsonb_exists_any()/_all() skip them; ?| '{}' and ?| '{NULL}' select nothing, and ?& '{}' every document, which no key names, so it answers GIN_SEARCH_MODE_ALL.

Every document has at least one key (R, or a raw scalar’s L), so nothing goes to the EMPTY entry.

42.3 Elsewhere

  • lion_match_index() let an index with no collation match a clause with one (doc ? 'k': the text operand has the default collation), which is the planner’s own IndexCollMatchesExprColl().
  • lionvalidate() accepts the pre-18 hashvarlena for a multi-key class’s bytea keys, as it does for a scalar bytea class.
  • LION_STRAT_IS_MULTI() replaces the 2 .. 5 range checks; doc @> ANY (array) and doc ? ANY (array) reach the count pushdown’s OR like the other multi-key operators.

42.4 Size and speed

On synthetic corpora (1M documents each, PostgreSQL 18; a catalog of low-cardinality fields, an event log with ids, timestamps and short text, orders with arrays of item objects): 15 to 21 keys per document; the index 1.0 to 1.4 times GIN jsonb_ops and 1.5 to 1.9 times jsonb_path_ops; a serial build 1.5 to 2.7 times faster than jsonb_ops and from 1.4 times slower to 1.5 times faster than jsonb_path_ops, and with three workers faster than either. Counts over @>, ?, ?| and ?& were 25 to 500 times faster than through GIN, and twice as fast for one matching 4 rows (GIN rechecks every @> in the heap; the count pushdown reads no heap on all-visible pages), and the lossy {"items": [{"sku": .., "qty": ..}]} 17 times, rechecking only the rows its keys select.

42.5 Tests

jsonb_contains.sql: random documents (nested objects and arrays, raw scalars, empty containers, numbers written several ways, NULL) and random queries, half of them pruned copies of a document, through bitmap scans, plain index scans and the count pushdown with the query a literal and a parameter, against a sequential scan, GIN jsonb_ops and jsonb_path_ops, before and after updates and deletes; fixed edge cases; keys over 2000 bytes; the plans. Disabling the lossy answer makes it fail.

42.6 A field equal to a string (2026-10-20)

(doc ->> 'k') = 'v' is the usual spelling of “field k is v”, and no operator of the class: the count pushdown left it to the heap in a WHERE, and declined a count with it in a FILTER, whose conditions have to be answered from posting sets (§10, “Filtered counts”). It is the same test as doc @> '{"k": "v"}' when v is a string no other JSON value prints as. ->> gives the text of the field’s value: a string’s own characters, a number’s digits, true or false, a container’s JSON text, and NULL for a JSON null, for a missing key and for a document that is no object; the containment holds exactly where the document is an object whose field k is the string v ({"k": ["v"]} and [{"k": "v"}] contain no such field, and their ->> is no v either). So the two agree on every row, NULL counting as false as it does in a WHERE and a FILTER, when:

  • = is texteq() under a deterministic collation, which compares the characters themselves;
  • the key and v are constants (text or varchar), so that v can be looked at when planning;
  • v is the text of no number - numeric_in() refuses it, which also turns away NaN, Infinity and padded digits, though no JSON number prints as those -, is neither true nor false, and starts with neither { nor [, as every container’s text does.

lion_jsonb_field_eq() (lion_plan_match.c) builds the containment - the same expression as ->>’s operand, pg_catalog’s @>, and v as a JSON string under the key, escaped by escape_json() - and lion_analyze_leaf() analyses that in the clause’s place, as the multi-key clause the posting sets answer exactly (§42.2: one L key, whose array marks keep {"k": ["v"]} apart); the plan and its EXPLAIN show the containment. Not under a NOT, whose ->> is NULL where the containment is false, and not as <>, which the planner makes of NOT (... = ...). A value that is a parameter, IN (...), doc -> 'k' = '"v"' and a nondeterministic collation stay the heap’s. The estimate is core’s, of the ->> expression. An expression index on (doc ->> 'k') is an expression column first (§41), and answers the clause as one.

Tests: jsonb_fieldeq.sql: documents whose field is the string, a number, a boolean, null, an array or object holding the string, strings that print like those, other cases and padding, escapes in the key and the value, top-level arrays and scalars, nested fields, NULL documents; every spelling taken and every one declined, in a WHERE, a FILTER with doc IS NOT NULL AND doc ? 'k' beside it, an OR, a GROUP BY and a generic plan, against the pushdown off.

42.7 Not done

  • Multi-leaf array elements exactly: store each leaf’s outermost array-element ordinal as a per-(key, row) payload, as positions are (§17), and keep candidates with an ordinal all the element’s keys share.
  • jsonpath ranges over the ordered L keys (§43.6).
  • More than 1000 query keys answers every row; dropping keys from the AND would still be a superset.
  • Opclass options to leave chosen paths or value types out, which keeps every query a superset.

43. jsonpath @? and @@ through jsonb_contains_ops (lion_jsonb.c, 2026-10-09)

Strategies 15 (jsonb @? jsonpath) and 16 (jsonb @@ jsonpath) of jsonb_contains_ops, on the keys of §42 unchanged. A jsonpath is never answered exactly: the index selects a superset, which every caller rechecks, as GIN does for every jsonpath. doc @? path is doc @@ exists (path).

43.1 What a jsonpath requires (lion_jp_bool())

The query is a tree of AND and OR over key PREFIXES, true for at least the documents the jsonpath is true for:

  • An accessor chain (lion_jp_chain()) from $, or from @ in a filter, is followed as the object key names it passes; array accessors ([*], [n], [last]) add none, since array levels are not part of a key’s names, and lax mode’s unwrapping of arrays only changes how many levels there are. .*, .** and item methods cut the chain: its items lie at or below the names so far. A chain that starts anywhere else (a variable, a literal, arithmetic) says nothing about the document.
  • chain == scalar is the L keys that start with the names, 0x00 and the value, at any array levels. jsonpath compares strings by their bytes and numbers by value, which is what the L key stores. A cut chain only says something is under its names. The scalar is a bare literal: with an accessor after it ($.a == "2".double()) it stands for another value, and the chain is only required to yield an item.
  • A chain that must yield an item (exists (chain), either operand of !=, <, <=, >, >=, starts with, like_regex, and @?’s path) is the L and C keys that start with its names. The root (no names) is every document.
  • A filter (? (pred)) adds what pred requires, with @ at the chain so far: it passes only items pred is true for. When a chain’s filters narrow, the chain is not also required to yield an item: the keys under its names are a union that costs more than it narrows (§43.4).
  • &&, ||, !: the AND or OR of the parts, NOT pushed down to the leaves. An AND keeps the parts that narrow; an OR needs both. Under a NOT, exists and comparisons narrow nothing ($.a != 1 being false includes $.a being absent), and the operator flips: !(a && b) is an OR. GIN’s jsonb_ops and jsonb_path_ops decide AND or OR from the operator alone, so they take !(!($.a == 1) && $.a == 3) as needing $.a == 1 and miss every document where $.a is neither 1 nor 3, for which the jsonpath is true (PostgreSQL 18). Lion returns the sequential scan’s rows there.
  • is unknown, and anything else, narrows nothing.

43.2 Expanding the prefixes (lion_query_expand(), lion_multikey.c)

extractQuery returns the prefixes as keys with pmatch set, LION_SEARCH_MODE_LOSSY, and the tree in extra_data[0], built of LionKeyNodes. lion_extract_query_superset() reads that tree only when the column’s extractQuery is lion_jsonb_extract_query() (GIN’s own jsonb extractQuery leaves a tree of another type there), and takes an index: the executor’s callers pass theirs, and each prefix becomes the OR of the index’s keys that start with it, read off the ordered directory by lion_prefix_keys(), which serves tsquery’s foo:* too (§17, “Prefix lexemes”) and now bytea columns as well. A key a visible row has was in the directory before the walk, so the OR selects every row the prefix does. A prefix no key starts with selects no row; one with more keys than the query has left of its 1000 is every row; the tree is folded around both, and may come out NONE, LOSSY or ALL. The planner’s callers pass no index, and each prefix stands for its OR as one key.

A key longer than LION_MAX_KEY_SIZE is stored as its hash (§42.1), which keeps no prefix. So a prefix that could start a hashed key - every L/C prefix of a chain that must exist, and an L prefix with a value when its length plus five bytes per array-level count could pass the limit - is ORed with the prefix H, every hashed key.

In the count pushdown a jsonpath is always counted from the superset with a heap recheck (lion_locate_multikey() takes the superset branch for it), so the planner takes it only where a recheck is allowed: never under an OR. A literal one may come out ALL at run time as a parameter’s may, so it is no clause positive whatever its value (§17, “Prefix lexemes”, the cap).

43.3 Elsewhere

  • lion_extract_query_superset() takes the index; lion_keynode_leaf()/_op() are exported for lion_jsonb.c.
  • LION_STRAT_IS_MULTI() covers 15 and 16; LION_STRAT_IS_JSONPATH() names them.

43.4 Speed

On the §42.4 corpora (PostgreSQL 18, best of five, each index alone; every count the same as the other two indexes), counts over fifteen jsonpaths against GIN jsonb_ops and jsonb_path_ops:

jsonpath rows jsonb_ops jsonb_path_ops lion
@@ '$.status == "active"' 250619 224 ms (1.3x) 265 ms (1.6x) 167 ms
@@ '$.status == "draft" || $.status == "paused"' 374094 340 ms (1.3x) 355 ms (1.3x) 266 ms
@@ '$.tags[*] == "tag3"' 69551 125 ms (1.5x) 112 ms (1.4x) 82 ms
@@ '$.type == "type1" && $.flags.b == true' 11101 113 ms (8.6x) 30 ms (2.3x) 13 ms
@? '$.attrs ? (@.color == "c3" && @.size == "m")' 18315 60 ms (3.0x) 42 ms (2.1x) 20 ms
@@ '$.kind == "k3"' 52596 78 ms (1.8x) 57 ms (1.4x) 42 ms
@@ '$.kind == "k3" && $.user.country == "cc7"' 1073 25 ms (10x) 3.6 ms (1.5x) 2.5 ms
@@ '$.payload.amount == 500' 4 0.42 ms (3.2x) 0.08 ms (0.6x) 0.13 ms
@@ '$.customer == 123' 7 2.4 ms (22x) 0.05 ms (0.5x) 0.11 ms
@@ '$.customer > 99990' 107 1032 ms (1.5x) 1091 ms (1.6x) 669 ms
@? '$.gift' (on 1000 rows) 1000 943 ms (3400x) 690 ms (2500x) 0.28 ms
@@ '$.gift.wrap == true' 1000 0.50 ms (1.4x) 0.79 ms (2.1x) 0.37 ms
@? '$.items[*] ? (@.sku == "sku42" && @.qty == 3)' 76 13 ms (13x) 2.1 ms (2.1x) 0.97 ms
@@ '$.items[*].sku == "sku42"' 714 8.3 ms (4.5x) 1.1 ms (0.6x) 1.8 ms
@@ '$.shipping.method == "air"' 333711 382 ms (1.8x) 293 ms (1.4x) 214 ms

Ratios are the GIN time over lion’s. Every jsonpath count rechecks the heap, so the gap is smaller than for @> (§42.4). Neither GIN index narrows a bare existence path ($.gift, on a thousandth of the rows, cost them as much as a range over the whole table); lion reads it from its L/C prefixes. The range ($.customer > 99990) narrows only to rows with the path, so all three indexes read most of the table. A filter’s chain ($.attrs ? (...)) is not also required to exist when the filter narrows: that would be every key under its names, a union costing more than it narrows (counting this filter took 91 ms with it). Lion is slower than jsonb_path_ops, which hashes the path and value into one key, on point lookups of a few rows and on $.items[*].sku == "sku42", by under a millisecond.

43.5 Tests

jsonb_path.sql: random documents as in §42.5 against random jsonpaths (lax and strict; key, array, wildcard and method accessors; filters; comparisons, exists, starts with, like_regex, &&, ||, !, is unknown), a third of them a path and value read off a document, through bitmap scans, plain index scans and the count pushdown with the query a literal and a parameter, against a sequential scan, before and after updates and deletes; fixed cases (lax unwrapping, raw scalars, empty containers, numbers, the NOT case above); hashed keys; a path too common to expand; op ANY (array); the plans. Each of these makes it fail: deciding AND/OR from the operator alone, pinning the array levels to zero, and leaving out the hashed keys.

43.6 Not done

  • Ranges ($.price > 10) from the ordered L keys: needs an order-preserving number encoding in place of numeric_normalize()’s text. Today a range only requires its path to exist.
  • .** and .* with the names after them: the names before the wildcard are all a key can say.
  • Hashed keys keeping a prefix (H + the first bytes + the hash) would let a prefix find only its own hashed keys instead of all of them.

44. lionvec: nearest neighbours of pgvector’s vector (pg_lion_vector; lion_vec.c, lion_vec_scan.c, lion_vec_math.c; 2026-10-14)

The shape is

SELECT ... FROM t [WHERE <clauses lion indexes answer>] ORDER BY v <-> q LIMIT k

with <->, <#> or <=>, pgvector’s L2, negative inner product and cosine distance. pgvector’s own indexes answer it from a graph (HNSW), whose every hop is a random read and which is fast only while the graph - here 3.8 GB for 975,000 1536-dimension rows, as halfvec - is in memory, or from k-means lists (IVFFlat), which hold every row’s floats. Under a WHERE both filter AFTER the search, so a selective filter leaves a LIMIT short or sends the scan on through most of the index.

lionvec is IVF over compact codes. The build partitions the rows by k-means; a scan reads the partitions in the order of their centroids' distance to the query, and of each partition’s rows only a code of one bit a dimension - 224 bytes a 1536-dimension row, a twenty-seventh of its floats - whose error bound decides which few rows are read in full. It stops once the next centroid, less a margin, lies farther than the rows it has (§44.4), having read a few percent of the rows; among those it returns rows in the exact order of their distances, unless a bound of pg_lion.vector_epsilon standard deviations fails. Under a WHERE lion answers, it reads and scores only the rows lion’s set holds, and for a set of up to 2,000 rows, or of up to 1.5 a partition, computes every member’s distance outright (§44.6).

The codes are RaBitQ’s (Gao and Long, SIGMOD 2024): a fixed random rotation makes every coordinate of every vector look like an independent Gaussian, so that one bit a dimension - the coordinate’s sign - estimates an inner product with an error whose distribution is known whatever the data, and so a bound on it that holds with a chosen probability. What is lion’s here is the rest: the bound and the partitions decide what a scan reads next (§44.4), the further bit-planes and the heap are read only for the rows it cannot rule out, and a lion set restricts the whole (§44.6).

The first format had no partitions: every scan read every row’s code, in heap order, and the answer was exact (96 ms for the ten nearest of 975,000 rows, §44.9). At the scale this is for, a billion rows, that is 224 GB of codes a query. Format 2 (the same day) gave the exactness up for partitions; a simulation over the benchmark’s vectors chose their number and the margin (§44.9). Format 3 SPILLS: every row is in two partitions, its nearest and a second chosen by SOAR (§44.3), so that a row near a partition’s edge is found from either side, for a second CODE item a row.

44.1 Packaging

The access method lives in pg_lion’s library, which reads pgvector’s on-disk vector itself (a varlena of an int16 dimension count, two unused bytes and the floats: LionVecPgVector) and calls pgvector’s distance functions only through the operator class, so pg_lion loads, and its tests run, without pgvector. A separate extension, pg_lion_vector (requires = 'pg_lion, vector'), creates the AM and three operator classes for vector - vector_l2_ops (the default), vector_ip_ops and vector_cosine_ops - each ONE ORDER BY operator, whose strategy number (1, 2, 3) is the index’s metric; pgvector’s objects are named through @extschema:vector@. The rest of pg_lion sees four functions (lion_vec.h): lion_vec_index() and lion_vec_relation(), which tell a lionvec index by its own routines (amcostestimate, ambeginscan), not its name, so that an AM of that name another handler serves is never taken for one; lion_vec_restrict(); and lion_vec_scan_cost(). The changes to lion’s own code are _PG_init() calling lion_vec_init() (the settings, the 8-bit tables, the kernels) and LionOrdered’s (§44.6).

An operator class of one’s own is checked as far as lionvec reads through it. Its input type must be NAMED vector or halfvec, of length -1 (lionvec_type_ok()): the reader below takes every datum for a varlena of pgvector’s layout, and a class over int had it detoast an integer as a pointer, a crash (a review found it, 2026-10-15). The index’s open refuses any other type, and amvalidate reports the class. Its ORDER BY operator must return float8 or float4 (lionvec_distance_type_ok()), as amvalidate reports and a scan’s start checks; the scan reads each distance, and hands it back in xs_orderbyvals, in the operator’s own type, as core’s index_store_float8_orderby_distances() does. Until that review a float4 operator’s distances were read, and returned, as float8.

halfvec (2026-10-15; pgvector 0.7 and on: the same 8-byte header, then IEEE 754 binary16 halves) has the same three, halfvec_l2_ops (its default), halfvec_ip_ops and halfvec_cosine_ops, over pgvector’s l2_distance, halfvec_negative_inner_product and cosine_distance of halfvec. The format does not change: the codes, the 8-bit codes and the centroids are made from floats whatever the column holds, so a halfvec column’s index is the size of a vector column’s, while the column is half its size. An index knows its type from its operator class’s input type, rd_opcintype[0]: a halfvec when that type is NAMED halfvec in pg_type (by the syscache, for pgvector’s OIDs differ from database to database), a vector otherwise. It is looked up once an index open and kept in LionVecInfo.half, before rd_amcache is set, for the lookup may take a lock whose invalidation would free the cache; the build looks it up too, before there is a meta page. One reader turns either datum into floats, lionvec_datum_floats(): it detoasts it and checks its length (lionvec_datum_dim(): the header, a count not below zero and that many floats or halves, or ERRCODE_DATA_CORRUPTED), and returns a vector’s floats where they lie, or converts a halfvec’s into the caller’s array - the build’s and an insert’s, of the index’s dimensions - or, without one, a palloc’d one. A scan converts its query once a rescan; its exact distances are pgvector’s own functions of the datums (the ORDER BY operator’s), so a row read from the heap is never converted. The conversion is exact, every half being a float (lionvec_halves, a kernel like the others, §44.3): the portable one assembles the float’s bits - the exponent rebiased, a subnormal normalized at its leading one, infinities kept and a NaN made the quiet NaN of its payload - without a loop or a table, and F16C’s VCVTPH2PS converts eight halves at a time to the same bits. The errors name the type: different halfvec dimensions 3 and 2, and pgvector’s expected 3 dimensions, not 4 and halfvec must have at least 1 dimension. Measured on DBpedia (bench/results/2026-10-15-dbpedia-pg18/, “halfvec”): a halfvec column’s index probes the same partitions, reads the same pages and returns the same recall as the vector column’s; the exact path is 13 to 26% faster warm (4.6 against 5.5 us a member, half the TOAST chunks) and reads 9% fewer pages cold, not half, because a 3 KB halfvec still takes a TOAST page of its own for each of the scattered rows a narrow filter admits.

A column’s dimensions come from its type modifier, or, for a type without one, from the first vector the build or an insert meets (lionvec_set_dims(), under the meta page’s lock, so that two first inserts agree); a vector of other dimensions is pgvector’s own error, expected 3 dimensions, not 4, and a query of other dimensions different vector dimensions 3 and 2. At most LIONVEC_MAX_DIM dimensions, 8128 at 8K pages, as many whole 64-bit words as leave a REFINE item a page; pgvector’s HNSW and IVFFlat take 2000. Single-column, no INCLUDE, no search operators (amoptionalkey: a query with no WHERE at all is the point), forward only (amcanbackward = false: a SCROLL cursor gets a Material node), no bitmap scans. Scanned in parallel (amcanparallel, with amestimateparallelscan, aminitparallelscan and amparallelrescan: §44.4, “A parallel scan”) and built in parallel (§44.12).

44.2 Format (lion_vec.h)

Block 0 is the meta page: magic, format version 5, metric, flags (LIONVEC_META_SPILL: the build spilled, §44.3; LIONVEC_META_LOCATOR: it keeps a locator, §44.13), dims and the padded dimsp (a multiple of 64), the rotation’s seed, the MEAN chain’s head, the PART chain’s head and the partitions' number, the heads and tails of the two chains of rows in no partition, the REFINE tail, the partitions' SCALE (the rows' standard deviation about their centroids along one dimension, the unit of the margin, §44.4), the build’s row count, and the REGION chain’s head and the regions' number, appended in format 4 so that every earlier field keeps its offset (lionvec_check_meta() holds nregions > 0 to a valid head, and to no more than the partitions), and the locator’s fields, appended in format 5 (§44.13). Every page’s special space carries its next block, its type, LIONVEC_PAGE_ID and, on CODE pages, the least and greatest container key (§2) of the TIDs on it.

  • MEAN: the indexed rows' mean, rotated, as floats over as many pages as it takes; written by the build and never changed. Absent (mean_head invalid) when there were no non-NULL rows.
  • PART: one item a partition (LionVecPart), in order: its number, its CODE chain’s head and tail, the items the build put in it (its rows and, spilling, other rows' copies), the greatest |x| of those rows and their mean |x - c|^2 (an inner-product scan’s bound, §44.4, which must hold for every row the chain holds, copies included), and its centroid, coded as a row is
    • its plane 1 here, its 8-bit codes in a REFINE item whose TID is (partition, 0), which no heap row’s is (LionVecCentTid()). 248 bytes at 1536 dimensions, 32 a page: 62 pages for the benchmark’s 1,974 partitions. Written by the build after the CODE chains; an insert changes only a tail. In an index of regions (below) the partitions are numbered region by region, so that each region’s are one run of the chain.
  • REGION (format 4): in an index of LIONVEC_REGION_MIN (64) partitions or more, one item a region of the centroids (LionVecRegion, §44.3), in order: its number, its run of partitions - the first and how many - the block and offset of the run’s first PART item, and the region’s centroid coded as a row is, plane 1 alone (its TID and REFINE item invalid, its <x, mean> zero): 248 bytes at 1536 dimensions, 32 a page, two pages for DBpedia’s 53 regions and 16 for the 512 of LIONVEC_MAX_PARTS. A region k-means left without partitions is kept, with no run and no PART item, so that the count is always ceil(sqrt(C)) and the beam an insert takes of them the build’s. Written by the build after the PART chain, never changed. An insert reads the chain and then only the runs of the regions it picks; it checks that the regions number from 0, that their runs follow one another and cover the partitions, and that each PART item of a run is that run’s next partition, reading on along the PART chain from the item the region names (lionvec_insert_read_regions(), _read_run()), and any of it amiss is an ERRCODE_INDEX_CORRUPTED error, as every item’s length, place and line pointer are checked against its page (lionvec_item_ok()). An index of fewer partitions, one or none included, has no REGION chain (an invalid head, nregions 0), and an insert reads the whole PART chain. A scan reads no REGION page (§44.3, “Regions”).
  • CODE: one item a row, 32 bytes of header - TID, flags (LIONVEC_NULL, LIONVEC_ZERO: a cosine zero vector, LIONVEC_SPILL: a copy), |x|, alpha1, <x, mean>, the REFINE item’s block and offset, plane 1’s popcount - and plane 1, dimsp / 64 words: 224 bytes at 1536 dimensions, 35 a page. A NULL row or a cosine zero vector is the header alone. In an index that spills, every other row has a second item, its COPY, in its second partition’s chain: the primary’s every byte - the REFINE item’s place among them, so that there is one REFINE item a row, never two - but for LIONVEC_SPILL. A chain a partition, which the build writes in one run of blocks, in heap order (§44.5), so that a probe reads it sequentially and each page covers a narrow range of heap blocks; every partition has a page, even an empty one, for inserts to append to. Then two chains of rows in no partition: the REST chain, of every row of an index built without partitions (an empty table’s, §44.3), and the NULL chain, of the NULL rows and the cosine zero vectors, which no distance orders.
  • REFINE: one item a non-NULL row or a centroid, 16 bytes of header (TID, nloc, alpha8, |r|) and dimsp 8-bit codes: 1552 bytes at 1536 dimensions, five a page. Read only for the rows plane 1 cannot rule out and for the centroids a scan comes to. In an index that keeps a locator over vectors wide enough, every item ends with a slot for its row’s place in TOAST (§44.13), 20 bytes at 1536 dimensions, five a page still.
  • LOC and LOCDIR (format 5): in an index whose vectors may lie in TOAST, an entry a row whose vector does - its TID, its value’s id and its chunks' places - in heap order, and a directory of those pages (§44.13).

So an index of n rows of d dimensions is about n (d / 8 + 36) bytes of CODE pages, twice that when it spills, and n (d + 20) of REFINE pages - 1.7 GB for the benchmark’s 975,000 rows of 1536 without spill (225 MB of it CODE pages), about 2 GB with it, estimated - and C (d + 1.2 d / 8 + 70) of centroids. A scan reads the PART chain, 500 KB there, and the CODE pages of the partitions it probes, each twice as long when the index spills.

Format 4 had no locator and no REFINE slots. Format 3 had no REGION chain, and its partitions were numbered as k-means left them. This pg_lion refuses both as it refuses format 2 (lionvec_check_meta(): “has format version 4 … REINDEX it”), since the extension has no release to migrate from. Format 2 had no copies and no flags in its meta page.

Format 1 (§44 above) had one CODE chain, every row in heap order, and a DIR chain of each CODE page’s least and greatest TID, by which a filtered scan read only the pages that could hold a member. With partitions a filter’s members are spread over every partition, the build’s order is by partition first, and a filtered scan of more members than the exact path takes (§44.6) probes partitions as an unfiltered one does, so the directory went; the page’s container keys still skip a page that holds no member (§44.6).

44.3 Codes and partitions (lion_vec_math.c)

  • The rotation: three rounds, each flipping the signs of a pseudo-random half of the coordinates (pg_prng from the meta page’s seed: the same on every platform) and applying the normalized Walsh-Hadamard transform; when dimsp is not a power of two, two windows of the largest power of two below it, one at each end, overlapping, each after its own signs. Every step is orthogonal, so distances and inner products are unchanged. A vector is padded with zeros to dimsp, scaled to unit length first for cosine, and rotated; the rotation’s signs and the mean are kept per backend in rd_amcache (lionvec_get_info(), which hands out a copy, since a relcache invalidation - possible whenever a lock is taken, a TOAST fetch’s included - frees it).
  • Centring: the stored vector is x = rotated - mean. Any fixed mean is correct (inserts use the build’s); a near one makes |x| small and the bounds, which scale with it, tight.
  • Plane 1: the sign bits of x, and alpha1 = <sign(x) / sqrt(dimsp), x / |x|>, about 0.8.
  • The 8-bit codes: x_i sqrt(dimsp) / |x|, a standard Gaussian, in uniform steps of 1/32 over [-4, 4), clipped beyond (one coordinate in 16,000); r is the vector of the steps' middles and alpha8 = <r / |r|, x / |x|>. Uniform steps beat equal-probability steps by eight times in squared error, those being as wide in the tails as the tails are long.
  • The query (lionvec_query_init()): rotated and centred as the rows are, giving qc, and for plane 1 quantized to four bits a dimension, uniformly over its own range, held as four bit-planes interleaved with plane 1’s words, so that <sign(x), qc> is 2 sum_j 2^j popcount(bits & plane_j) rescaled, less sum(qc) (the kernels below).
  • The estimate and its error: <x, qc> is estimated as |x| <c, qc> / (|c| alpha), c the code’s vector. Its error is the part of x / |x| the code misses, sqrt(1 - alpha^2) long and, after the rotation, pointing nowhere in particular, times qc: a Gaussian of standard deviation |qc| sqrt(1 - alpha^2) / (alpha sqrt(dimsp - 1)), plus, for plane 1, the query’s own rounding, Delta^2 / 12 a dimension. A bound of epsilon of those either side holds with the Gaussian’s probability. The 8-bit codes' deviation is about 1/70 of plane 1’s, and their bound is 2 eps wide (LIONVEC_EPS8): still far the tighter, and its failures all but vanish (§44.10).
  • The key a scan orders by is the distance up to a constant of the query: |x|^2 - 2 <x, qc> for L2, -<x, mean> - <x, qc> for inner product and cosine (lionvec_key_bounds()); an exact distance d has key d^2 - |qc|^2 for L2, d + <mean, q> for IP and d + <mean, q> - 1 for cosine (lionvec_key_exact()). A bound that overflows to NaN is taken as no bound, (-inf, inf).
  • Partitions (lionvec_build_centroids(), lionvec_kmeans()). The build’s first heap scan takes a SAMPLE as it computes the mean: reservoir sampling (Vitter’s algorithm R) from a generator seeded alike every build, so that the same rows make the same partitions, of as many rows as half of maintenance_work_mem holds as 8-bit codes (rotated, not centred - the mean is not known yet - a percent off the floats, which k-means does not notice), at most 224 (LIONVEC_SAMPLE_MAX; 220 until the regions, below). The number of partitions C (lionvec_parts(), lion_vec_math.h) is the partitions reloption’s, or else 1 below 10,000 rows and 8 sqrt(n) above (LIONVEC_PARTS_PER_ROOT: 8,000 for a million rows, 2,715 for DBpedia’s 115,186), at most a sixteenth of the sample, never more than its rows, and 0 for an empty table. It was 2 sqrt(n) (1,974 for the benchmark’s 975,000) until DBpedia (§44.9): partitions a quarter the size read 1.55 to 1.78 times fewer rows for a recall in the numpy model, and on the index 1.5 times fewer CODE pages for 0.95 of the true ten, which is the direction a billion rows needs (§44.11), though at 115,186 rows no faster. The sixteenth means the default needs a sample of 128 sqrt(n) rows, 1,540 bytes each at 1536 dimensions, so maintenance_work_mem of 2 x 1,540 x 128 sqrt(n) bytes: 376 MB for a million rows. The default 64 MB holds 21,788 rows, a sixteenth of them 1,361 partitions, which a table of 28,986 such rows outgrows: a build whose sample has rows for fewer partitions than a sample of every row would make fails, at the first row that makes it so, naming the maintenance_work_mem that would hold enough, unless pg_lion.vector_build_sample is allow (§44.8). 1 GB holds 348,617, 44 a partition at a million rows, about the 42 the numpy model trained on with every row, and enough to 7,418,133 rows; every row of a million takes 2.9 GB. A sample row is dimsp + 4 bytes (its codes and its scale), so the cap of 224 rows is 25.8 GB at 1536 dimensions, reached at a maintenance_work_mem of 52 GB; it is 64 rows a partition at LIONVEC_MAX_PARTS (262,144, §44.8), and the sixteenth binds only past 220 partitions, so the default is 8 sqrt(n) up to that ceiling, which it reaches at 1.07 billion rows. A billion rows want 252,982 partitions, at least 4.05 million sample rows (6.2 GB, a maintenance_work_mem of 12.5 GB, 11,890 MB as the build’s error rounds it) to be given them, and the cap’s 224, 66 a partition, with 52 GB. (The cap was 220 rows, 3 GB, and the ceiling 65,536 partitions, which then held every table of more than 67 million rows - a billion’s 15,000 rows a partition.) k-means' share of the sample is at most 128 rows a partition (LIONVEC_SAMPLE_PER_PART_MAX) and never fewer than 65,536 rows (LIONVEC_SAMPLE_MIN; lionvec_kmeans_rows()): when the reservoir holds more than that, the partitions are still counted from all of it, and k-means is given a uniformly random 128 C of its rows, or 65,536, whichever is more (lionvec_subsample(): a partial Fisher-Yates shuffle of the slots from a seed of its own, LIONVEC_SEED_SUBSAMPLE, the slots kept sorted and the rows moved down in place, so that they keep the reservoir’s order, in which k-means' update sums them, and the build is the same every time), before k-means and before a parallel build copies the sample into shared memory, which then holds those rows alone (lionvec_build_subsample()). An iteration costs its rows, and past some rows a partition more of them move the centroids less than their iterations do (FAISS warns below 39 and cuts at 256). 64 was too few: on DBpedia in 679 partitions, where 64 a partition keep 43,456 of the 115,386 rows and 128 keep 86,912, the margin of 3 found 0.891, 0.900 and 0.906 of the true ten over three draws (seeds of the cut) at 64, against 0.909 with every row, and 0.916, 0.920 and 0.915 at 128; at 8 both were within the draws' spread of every row’s 0.984 (0.980 to 0.982 at 64, 0.981 to 0.985 at 128), and in 1,358 partitions 64 gave 0.918 and 0.979 against 0.917 and 0.982. At the default 8 sqrt(n) the cut is 1,024 sqrt(n) rows, past the floor from 4,096 rows and every row up to a million, so it falls only on a larger table whose sample holds more: none at a maintenance_work_mem of 1 GB (348,617 rows at 1536 dimensions), and at 52 GB’s 224, the rows of a table of between a million and 268 million rows past 1,024 sqrt(n); or on fewer partitions than the default, on a table of more than 65,536 rows. The floor is for the tables below it: k-means over 65,536 rows is seconds at most, so a cut saves little there, and such a table is then built from every row it sampled whatever its partitions, as it was before the cut, which the regression tests' expected outputs - the partitions a query probes, the pages it reads, for tables of 3,000 to 10,000 rows in 20 to 50 partitions - rest on (vector_filter.sql’s 400,000 rows in 20 partitions are cut, to 65,536). Lloyd’s k-means then runs over the sample, in the rotated space (which changes no distance), from C distinct sample rows the seed picks, for ten iterations or until no row moves; spherical for cosine, whose rows and so centroids are of unit length, so that nearest by L2 is nearest by angle too. A row’s nearest centroid is found among the candidates plane 1 leaves (lionvec_candidates(): each centroid coded as a row, a bound of LIONVEC_ASSIGN_EPS = 4 standard deviations on its distance, and the centroids whose lower bound is under the least upper bound), by their exact distance; about ten candidates of 200 at 320 dimensions and more, so that k-means costs a popcount scan of the centroids a row and a few float distances - of the centroids of the row’s regions alone at 64 partitions or more (below, “Regions”). A centroid left with no rows takes half of the largest’s (FAISS’s way). The centroids are then coded as rows are, against the mean. Every loop of it - the centroids' coding, the assignment, the means, the splits - checks for interrupts once an item (LIONVEC_CHECK_INTERRUPTS() in lionvec_kmeans_codes(), _assign() and _update(), the steps a parallel build’s leader and workers run too, §44.12; nothing in the unit tests' build), so that a cancel, statement_timeout or shutdown ends a build in k-means within one item’s work. Until a review found it (2026-10-15) none did, and a build was deaf to all three until k-means ended: days at the partitions reloption’s old ceiling of a million. Measured on 500,000 rows of 16 dimensions, maintenance_work_mem 1 GB, 60,000 partitions (the heap scans take 8 s): before, the build ran on past 150 s of a 60 s statement_timeout and a cancel, and took kill -9; after, it stops at 10.02 s of a 10 s one and 60.01 s of a 60 s one. The coding of the FINAL centroids after k-means, each one’s REFINE item put as it is coded (lionvec_build_centroids()), checks once a centroid as well since the 2026-10-18 review; it had no check, which at LIONVEC_MAX_PARTS = 262,144 partitions of many dimensions is seconds. The second heap scan, which finds each row’s partitions (below) and so costs as much as an iteration of k-means over every row, was deaf too, for another reason found measuring the first: the build kept the REFINE page it was filling locked from row to row, and a content lock holds off interrupts. It now keeps the page pinned and locks it only to add an item (lionvec_build_put_refine(), as a parallel build’s workers do, §44.12); no other backend reads the index yet. Measured on 200,000 rows of 256 dimensions, 1,800 partitions, maintenance_work_mem 64 MB (k-means about 8 minutes): a cancel early in the second scan took 78 s before, the scan’s whole length (the build 569 s), and 0.013 s after.
  • Assignment. The build’s second scan puts each row in the partition of the centroid nearest it by L2: of the candidates plane 1 leaves, by the centroids' 8-bit codes, whose error is about a seventieth of plane 1’s (lionvec_build_nearest()); an insert does the same from the PART chain and the candidates' REFINE items (lionvec_insert_nearest()). An insert keeps, as it reads the PART chain, only the partitions plane 1 cannot rule out so far (lionvec_cands_add(), the streaming form of lionvec_candidates_n(): the m least upper bounds met, and every partition whose lower bound is under the m-th), and drops those the bound has since passed whenever they fill an array that starts at 64 and doubles only when more than half stay: 40 bytes a candidate, where it kept every partition’s codes and bounds, about 250 bytes a partition at 1536 dimensions, 16 MB an insert at 65,536 partitions and 192 MB at the reloption’s old ceiling (a review found it, 2026-10-15). The candidates are the batch function’s, as the unit tests check; an insert’s memory at 1536 dimensions went from 539 KB to 224 KB at 1,250 partitions and from 1.54 MB to 359 KB at 5,000 (356 and 364 KB at 125). The root of the rows' mean squared distance from their centroids, over the dimensions, is the meta page’s scale; each partition’s own mean squared distance and its longest row go in its PART item (§44.4’s inner product). Both k-means and the assignment cost a plane-1 estimate of every centroid for every row, so the build grew as the rows times the partitions, n^1.5 at the default: 19 minutes for the benchmark’s 975,000 rows in 1,974 partitions, where format 1 took 98 seconds (§44.9, §44.11). DBpedia’s 115,186 rows took 159 s in 678 partitions and 347 s in 2,715, about 96 s of it the rest of the build and 0.09 s a partition; at 8 sqrt(n) the benchmark’s 975,000 rows, in 7,899, would take about four times its 19 minutes (not measured), and 86 minutes for a million rows in 8,000 were measured on another machine. Regions (below) make it (beam + 1) sqrt(C) estimates a row rather than C, and the second pass, a spilling build’s, (16 + 1) sqrt(C). Ten iterations are not fewer than four times the partitions want: numpy’s spherical k-means over DBpedia’s rows moved 0.8% of them in the tenth iteration at 2,715 centroids and 1.9% at 678, and its rows' mean cosine to their centroid was within 0.18% (0.36% at 678) of thirty iterations'. At low dimensions, where plane 1 leaves most centroids as candidates (61 of 200 at 64 dimensions), the cost is nearer the rows times the partitions' exact distances: vector_filter.sql’s 40,000 rows of 64 dimensions take 42 s in their default 1,600 partitions, 12 s in the 400 of 2 sqrt(n).
  • Regions (lionvec_regions_make(), lionvec_route(), format 4, 2026-10-16). C centroids, when there are LIONVEC_REGION_MIN (64) or more, are gathered into G = ceil(sqrt(C)) regions (lionvec_region_count(): 53 for DBpedia’s 2,715 or 2,717, 512 at LIONVEC_MAX_PARTS) by k-means over the centroids - their 8-bit codes the sample, as the rows' are, LIONVEC_REGION_ITERS (5) iterations from G of them the seed picks (LIONVEC_REGION_SEED, one past the rows'), spherical for cosine as the rows' k-means is - and then each centroid into the region whose centroid is nearest it after the last update, found as k-means finds a row’s. Its members are listed in ascending order (a counting sort, first[] and member[]); a region left with no members is kept, so that G is a function of C alone. A row is routed (lionvec_route()) by a plane-1 estimate of its squared distance from each region’s centroid, coded against the rows' mean as a row is (lionvec_region_key()); the beam nearest with members are taken, least key first, ties to the lower region, and then the next nearest until they hold the m centroids the row wants (lionvec_regions_pick()), sorted ascending, so that their members come in the centroids' order. Its candidates are then found among those members alone, by the rule over every centroid (lionvec_candidates_among(), lionvec_candidates_n()’s over a list of indices), and its nearest, and SOAR’s second, among them as before. A row costs G + beam C / G estimates, about (beam + 1) sqrt(C), where it cost C: some 310 rather than 2,717 in DBpedia’s k-means and 870 in its second pass, more where the regions are uneven (theirs held 16 to 141 centroids, whose squares summed to 65 a centroid where 53 even regions' would have to 51). The candidates include every centroid the full rule makes one that is in the beam’s regions, but not only those: the m-th least upper bound among fewer centroids may be larger, which admits more (the unit tests count it as a rate). A row whose nearest centroid lies in no region of its beam goes to the nearest that does; the partition’s spread and longest row, which the scan’s bounds come from, are of the rows it holds, so a scan finds it there all the same, as it would a row placed by a centroid that moved.
    • Where. In k-means, each iteration’s centroids are regrouped before its rows are assigned (lionvec_kmeans(), and a parallel build’s leader before it posts the iteration, §44.12), from the same seed each time, so that the same rows make the same centroids and regions; the update is unchanged. For the second pass the final centroids are regrouped once and the partitions RENUMBERED region by region (lionvec_build_renumber(): the centroids, their codes, floats and plane 1 permuted in place, the regions' members rewritten as runs), so that each region’s partitions are one run of the PART chain, which its REGION item names (§44.2); the renumbering changes no centroid, and the build routes by the renumbered regions, so a row’s candidates come in ascending partition order whether the build or an insert finds them. An insert reads the REGION chain, keys every region, takes the build’s beam (the same G and m) and reads those regions' runs alone (lionvec_insert_read_regions()); vector.sql checks that copies of rows inserted afterwards go to the originals' partitions and spill partitions. A scan reads no REGION page and ranks every centroid, as before: in the simulation that chose the regions, ranking partitions through their regions read a third more rows for a recall.
    • The beam is a constant, by what it is asked for: LIONVEC_BEAM (5) regions for a row’s nearest alone (k-means, and the second pass and inserts of an index that does not spill), LIONVEC_BEAM_SOAR (16) for its nine nearest (LIONVEC_SOAR_CANDS + 1: the second pass of an index that spills, and its inserts), never more than G (lionvec_region_beam()). A share of G, as the first design had it (ceil(G / 12), at least 4), would let a row’s cost grow as C, and the measurements did not ask for one: on DBpedia (§44.9, “Regions”) 5 found the partition exact assignment would have for 98.8% of the rows in 53 regions and 99.5% in 100, and 16 the copy’s for 99.5% in 27, 98.6% in 53 and 97.9% in 100, at a recall within the sample’s own spread of exact assignment’s. A row’s copy wants the larger beam: SOAR picks it from the nine nearest, which lie farther out, and 5 found it for 88% of the rows in 100 regions, 13 for 96.95%.
  • Spill (lionvec_soar_second(), the spill reloption, on by default). A row’s true neighbours lie mostly in its own partition, but a row near the edge of its partition has neighbours in the next one too, and a query near such a row may probe one and not the other. An index that spills puts each row in a second partition as well, by SOAR’s rule (Sun et al., “SOAR: Improved Indexing for Approximate Nearest Neighbor Search”, NeurIPS 2023): of the eight centroids next nearest the row after its own (LIONVEC_SOAR_CANDS), the one that minimizes |x - c|^2 + lambda <r, x - c>^2 / |r|^2, r = x - c1 the row’s residual from its own centroid c1, lambda = 1 (LIONVEC_SOAR_LAMBDA). The second term charges a partition whose residual x - c points the way r does: a scan passes c1’s partition by when r points at the query, the row lying much nearer the query than c1 does (§44.4’s margin), and a partition whose residual points the same way is passed by for the same reason. Eight candidates rather than every centroid cost no more than the assignment does already: they are those plane 1 cannot rule out of the nine nearest (lionvec_candidates_n(), the ninth least upper bound in place of the least; ten or so of 200 at 320 dimensions and more, as before, a hundred at 64), ranked by their 8-bit codes, the first being the row’s partition as before. Two partitions or more give two candidates or more - every bound having lo <= hi, as it does when the centroids' lengths are numbers no less than zero, which lionvec_part_at() checks, so that fewer is corruption - and so a row always has a second; one partition gives no copies, and the build records whether it spilled in the meta page’s flags. The build compares the row with the centroids' floats, kept centred alongside their codes; an insert, whose floats are gone, decodes the candidates' 8-bit codes (lionvec_decode8(): |x| alpha8 y / |y|, y the code’s vector, the projection of x on it, an error of |x| sqrt(1 - alpha8^2), about a percent of |x|) and chooses among those, reading each candidate’s REFINE item once, for its estimate, and keeping its codes. The floats, C dimsp of them, which the k-means held already, are kept to the end of the second scan: 12 MB for the benchmark’s 1,974 partitions of 1536 dimensions. Over clustered vectors at 64, 384 and 1536 dimensions (the unit tests, §44.10), SOAR chose a partition other than the second nearest for 25-29% of rows, the cosine squared of the copy’s residual with r fell from 0.34-0.38 to 0.26-0.32, and decoded centroids chose as the floats did for 97 to 98% of rows, the rest within a percent of the least cost. The scale counts each row once, from its own partition, so that the margin means what it meant; a partition’s spread and longest row are over everything its chain holds, copies included, since §44.4’s bound must hold for each of them.
  • Kernels (lionvec_kernels_init(), from lion_vec_init()). What a scan and a build spend their arithmetic on: plane 1’s popcounts against the query’s planes, for each of a query’s centroids and each row a probe reads, and in k-means for each centroid for each sample row each iteration; the 8-bit codes' sum <r, qc>, for each centroid and row refined; k-means' exact distances (lionvec_l2sq); the rotation; and a halfvec’s conversion (§44.1), once a row a build or an insert reads and once a query. Each has a portable kernel and, from x86-64 GCC 5 and clang 6, kernels compiled with __attribute__((target)), of which the best set the CPU runs is chosen at load, as lion_container.c chooses its own (§3): POPCNT, plane 1 a word at a time; AVX2, plane 1 a word’s four planes in one vector against the row’s word broadcast, each byte counted with VPSHUFB and the bytes summed by VPSADBW every 31 words, the sums four doubles at a time in four vectors, and the rotation’s signs and butterflies eight floats at a time, the strides under 8 by permutes inside a vector, and where the CPU has F16C, which its own __builtin_cpu_supports("f16c") checks (a name GCC knows from 11 and clang from 18; before them, as in the JIT’s bitcode an older clang compiles, the portable conversion stays), a halfvec’s halves eight at a time by VCVTPH2PS; and AVX-512, AVX2’s with plane 1 counted by VPOPCNTDQ (Ice Lake, Zen 4 and on; GCC and clang from 8), a word’s four planes in one instruction on a 256-bit vector. AVX-512 is taken for VPOPCNTDQ alone: AVX-512BW’s VPSHUFB counted plane 1 only a sixth faster than AVX2’s, and the 8-bit sum eight doubles at a time less than a tenth faster in cache, where a scan waits on the codes' cache lines more (§44.9, “Kernels”), so no 512-bit instruction runs, for which some cores lower their clock. make LION_NO_SIMD=1 builds the portable set alone. Every set returns what the portable one does, to the bit, so that on any CPU a scan refines the same rows and probes the same partitions, a build makes the same partitions and an insert picks the same one. Plane 1’s are integer counts. The sums are in double, of terms each kernel computes with the same operations - a code’s reconstruction -4 + (c + 1/2) / 32 is computed rather than looked up, which vectorizes, and is exact, a multiple of 1/64 under 4, as is its product with a float - into 16 partial sums (LIONVEC_LANES), the i-th term into partial i mod 16, added in a fixed tree (lionvec_lanes_sum()); the portable kernels add so too, which moved their rounding from the plain sum’s by an ulp or so: DBpedia’s index, rebuilt, gave the same recall, partitions, candidates and pages as before at every setting. lion_vec_math.c is compiled with -ffp-contract=off, for GCC and clang would otherwise fuse a multiply and an add wherever the target has fused multiply-adds (aarch64, x86-64-v3, -march=native) and round l2sq’s terms once where the other kernels round them twice. The rotation’s butterflies are the portable ones' additions and subtractions, of the same operands. The halves' conversion is exact either way; VCVTPH2PS, which MXCSR’s DAZ and FTZ do not touch, keeps a subnormal half and makes a NaN the quiet NaN of its payload, as the portable one does. The unit tests force each set (§44.10). lionvec_query_init() keeps its sums in locals and builds each word’s four planes in registers, without a branch: adding into the query’s own fields, which every store to qc and the planes might overwrite, and a branch on each bit of each code, which the CPU cannot predict, made it 45 us a row at 1536 dimensions, 16% of a build; it takes 12, and gives the same planes to the bit. 2026-10-26: the choice and plane 1 come from §50. lionvec no longer chooses its own set. Every kernel in lion_vec_math.c dispatches at each call on lion_simd_current(), the extension’s one choice (§50), which picks on every CPU the set lionvec_kernels_init() picked, by the same __builtin_cpu_supports() tests - AVX-512 (AVX2 with AVX-512F, VL and VPOPCNTDQ), else AVX2, else POPCNT, else portable - where function pointers set at load were; lionvec_kernels_init() now only resolves the choice and asks lion_simd_f16c() once, and the AVX2 and AVX-512 sets convert halves with F16C where it says the build knows it and the CPU has it. The build’s tests are lion_planes.h’s: LION_HAVE_X86_SIMD, LION_TARGET() and LION_HAVE_X86_F16C for LIONVEC_HAVE_X86, LIONVEC_TARGET() and LIONVEC_HAVE_F16C, which tested the same compilers; LIONVEC_HAVE_AVX512 went with the one AVX-512 kernel there was, and no macro of lionvec’s own is left. From §50: plane 1 is lion_planes_dot_interleaved(bits, planes, nwords, 4), that module’s pass of constant stride for four interleaved planes in each implementation, which lionvec_plane_dot(), now an inline function of lion_vec_math.h, calls; lionvec’s own portable, POPCNT, AVX2 and AVX-512 plane-1 kernels are deleted. Here still: the 8-bit sum, l2sq, the rotation and the halves, each portable and AVX2 (the halves F16C), the AVX-512 set running the AVX2 ones; and lionvec_query_init()’s planes, built in registers as above, which §50’s scatter reproduces to the bit but would need the codes in an array first. The unit tests force a set with lionvec_kernel_force() as before, its LIONVEC_KERNELS_* numbers now lion_simd_force() of the implementation of that number, which moves §50’s and the container library’s kernels with lionvec’s (lionvec_kernels_init() restores the best). vec_test passes with the same 119,295 checks (119,262 without the x86-64 kernels), every statistic it prints the same to the digit, and an index of 100,000 random unit vectors of 1536 dimensions, rebuilt under each build, is the same 28,020 pages to the byte but their LSNs. Measured on §50’s machine (Emerald Rapids, AVX-512 with VPOPCNTDQ, four cores shared with other builds; gcc 13.3, -O2, PostgreSQL 16), before / after, ns: lionvec_plane_dot() at 1536 dimensions in a harness linked as vec_test is, pinned to a core, the median of eight alternated runs of the median of 31 rounds: portable 136 / 133, POPCNT 30.4 / 30.1, AVX2 25.4 / 26.7, AVX-512 16.6 / 12.9; and lionvec_estimate1() around it 140 / 136, 46.1 / 43.8, 35.2 / 35.7, 21.3 / 18.9. That is after one fix in §50 (“Kernels”): with the portable pass inlined into lion_planes_dot_interleaved(), every call saved six registers and grew a frame before jumping to its kernel, and AVX2’s estimate took 39.2 against 36.5. A server whose flags have link-time optimization, as Ubuntu’s packages' do (-flto=auto, which PGXS builds the extension with), inlines the dispatch into lionvec_estimate1(): a load of the choice and a direct call, where the pointer was an indirect one. End to end, eight queries ORDER BY embedding <=> $1 LIMIT 10 on that table’s vector_cosine_ops index, warm, under EXPLAIN (ANALYZE, TIMING OFF), the two builds alternated over 20 sessions of seven rounds each, 140 runs of each query a build: the queries' medians 33.1 - 67.9 ms before and 32.7 - 66.3 after, the geometric mean of their ratios 0.983, within the spread between sessions (a tenth). Such a query probes 581 partitions, estimates some 40,000 candidates by plane 1 and refines 16,000, so plane 1 is about 2% of it. Its index’s build (REINDEX, whose k-means and assignment estimate their candidate centroids by plane 1), alternated, took 142.9 and 141.5 s before and 138.4 and 139.7 after. 2026-10-27: the 8-bit sum in integers (lionvec_dot8()). What §44.14 found worth doing instead of planes: lionvec_estimate8() no longer sums recon8[c_i] qc_i in double (lionvec_sum8(), 16 lanes, the AVX2 kernel four doubles at a time) but takes the codes against the query rounded to 16-bit integers once a scan, in lionvec_query_init(): `w_i = round((qc_i

    • mid) / delta)withmid = (max + min) / 2anddelta = (max - min) / 65534over qc's own range (plane 1's quantization is over the same range, in 15 steps), so that every w_i is within +-32767, kept withsum(w)and the rounding's variancee2 / dimsp(qw,qmid,qwdelta, qwsum,varq8;qw0below). The kernel computessum(c_i w_i)andsum(c_i)` exactly in integers: portably in sixteen 32-bit lanes over blocks of 64 runs of 16, widened to 64 bits a block (a product is under 223, so 64 fit a lane with room), and under AVX2 sixteen codes a step widened to 16 bits and multiplied in pairs by VPMADDWD into eight 32-bit lanes, widened every 64 steps, the bytes summed by VPSADBW; the POPCNT set runs the portable one and the AVX-512 set AVX2’s, as for the other kernels. Then

    for the rounded query q^ - the codes' reconstructions being (-4 + 1/64) + c / 32 - of which the first term is the query’s alone (qw0, once a query), and the estimate is what it was of that sum; its error adds the rounding’s variance under the root, sqrt(var1 (1 - alpha8^2) + varq8), as plane 1’s adds varq. The rounding is a 65,534th of the range a step, so varq8 is some delta^2 / 12, a hundred-thousandth of the codes' own variance at 1536 dimensions, and the coverage test’s calibration holds: its printed shares of rows outside 2 standard deviations moved by one in the fourth decimal (0.0404 to 0.0403 at 100 dimensions under L2, 0.0414 to 0.0415 at 1536), nothing outside 6. The integer sums are the same in every implementation by construction, whatever order each adds in, and the double arithmetic after them is the same code of the same inputs under -ffp-contract=off, so every set still estimates the same bytes (test_kernels_agree()). k-means' queries, one a sample row an iteration, are not rounded (qw NULL): they never reach the codes; a scan’s, the build’s assignment pass’s (once a row) and an insert’s are. lionvec_sum8() stays as the reference the unit test checks the estimate against (and vecplanes_eval’s comparison row); no scan or build runs it. Tested in test_kernels(), test_estimate8() and test_query_rounding() (§44.10): the kernel against a plain integer loop at every dimsp from 64 to 8128 from unaligned pointers, over codes and weights at and mixed between their extremes, every set the portable one’s; the rounding’s every property; and the estimate against the serial sum over the rounded query (to 1e-9), within the rounding’s reach of the floats' (lionvec_sum8()’s, 0.04 of that reach at most), its error as stated, every set the portable set’s bytes. vec_test passes 119,449 checks (119,395 without the x86-64 kernels; 119,295 and 119,262 before), under ASan and UBSan too, and mutants of the kernel (an oversized block, a dropped lane or half of the byte sum, floor for rint, 65,535 steps, the wrong constant, the variance over dimsp - 1 or left out of the error) each fail it. Measured on this tree’s machine - a 4-vCPU VM of a Xeon at 2.1 GHz with AVX-512 and VPOPCNTDQ, gcc 13.3, PostgreSQL 16.15 - in vecplanes_eval --costs (§44.14’s harness, at -O1 as the unit tests are built; ns a refine at 1536 dimensions, warm / from a 24 MB pool / the lines flushed, pinned to a core, one run of both rows), lionvec_sum8() 368 / 523 / 1,500 with AVX2, 365 / 511 / 1,500 with AVX-512 and 1,005 / 1,167 / 2,107 portably; lionvec_dot8() 78 / 241 / 1,136, 75 / 242 / 1,100 and 274 / 439 / 1,406 (271 / 438 / 1,384 with POPCNT): 4.7 times faster warm, 2.2 from the pool, 1.3 flushed. The prototype’s idot16 is 57 / 205 / 1,083: the byte sum’s VPSADBW shares the shuffle port with the widening, 20 ns warm; a rounding grid through zero would need no sum(c), for a step up to twice as wide, and is not done. In a scan (bench/results/2026-10-27-lionvec-dot8/): 100,000 random unit vectors of 1536 dimensions in TOAST, vector_cosine_ops at the defaults, the index built once by the old library (104 s) and read by both, 50 held-out queries ORDER BY embedding <=> q LIMIT 10, everything in 4 GB of shared buffers and prewarmed, the two libraries alternated over 4 sessions of 5 rounds each of every query: the sessions' medians of the queries' medians 42.8 - 47.0 ms before and 35.7 - 39.2 after (45.7 and 36.9 at their medians), and each query’s time under the new library over its time under the old 0.81 at the median of the 200 pairs, quartiles 0.78 and 0.85; each query probes 547.8 partitions and refines 14,957 rows on average under both (4 of the 50 queries' candidates or refines differ by a few: 16 rows more refined in 748,000, one partition more probed in 27,000, the bounds having moved by the rounding’s variance), and recall is 0.632 for every query under both, row for row. That is about the 11 to 13% §44.14 expected of random vectors, and a little more: the sums were a quarter of their query there, and this machine’s memory is slower against its cores. DBpedia was not on this machine; §44.14’s arithmetic put the kernel at 5% of its warm query, whose sums are a tenth of it. No format change: the codes are read as they are.

44.4 The scan (lion_vec_scan.c)

Two priority queues. The PARTITION queue holds every partition, by the key (§44.3) its centroid’s codes bound from below, less the MARGIN: how much nearer than its centroid a partition’s rows are taken to come. The ROW queue holds rows, by a lower bound on their keys. Whichever queue’s least entry is less is worked; at equal keys, the row’s:

  • a partition known from its centroid’s plane 1 (every partition starts so, lv_read_parts() having read the PART chain): read the centroid’s REFINE item and put it back by its 8-bit codes;
  • a partition known from its codes: PROBE it - read its CODE chain and put its rows in the row queue by plane 1’s bounds;
  • a row known from plane 1: read its REFINE item and put it back at the lower bound of its 8-bit codes (which replaces plane 1’s rather than tightening it: keeping the larger of the two would keep plane 1’s failures too);
  • a row known from its codes: fetch it from the heap, compute its distance with the operator’s own function on FormIndexDatum()’s value (so an expression index computes its expression), and put it back at its exact key;
  • a row known exactly: nothing left in the row queue, nor in a partition whose turn has not come, is taken to be nearer; return it, its distance in xs_orderbyvals and xs_recheckorderby = false;
  • REST: probe every partition probed so far again, for four times as many rows, skipping the rows returned already.

The REST chain is probed first, before either queue is worked: it holds every row of an index built without partitions, and nothing otherwise. At equal keys the better-known entry comes first (a row known exactly before REST or a bound at its key), then by distance and TID, so ties come out the same from run to run.

Probes. A probe computes the key bounds [lo, hi] of each of its rows. The scan keeps the L smallest upper bounds of the rows probed so far in a max-heap; their largest, tau, is a key at least L rows lie within, so a row whose lo exceeds tau cannot be among the next L, and only the rows with lo <= tau go into the row queue. One REST entry, at the least lo of those left out, stands for them. L is pg_lion.vector_batch (100), or the LIMIT the scan is told when that is more (“A known LIMIT” below), until the first REST, and four times as many after each. A LIMIT k query thus probes the partitions whose centroids, less the margin, lie nearer than its k-th row, refines about the rows plane 1 cannot tell from the k-th, and fetches about k and a few.

The margin (lv_margin()) is pg_lion.vector_margin (3) standard deviations of the rows about their centroids along one dimension, the meta page’s scale: a row is nearer the query than its centroid by the part of its offset from the centroid that points at the query, and that part’s spread, |x - c| / sqrt(d) for a random direction, shrinks with the dimensions as a partition’s rows come to lie ever more nearly at one distance from any query. For L2, whose key is the squared distance less a constant, the margin comes off the distance (lv_part_key()); for cosine, off the key; for inner product, see below. It is one number for every partition: a constant margin did as well as any scaled by a partition’s own radius in the simulation (§44.9), as the rows farthest toward a query lie about as far out in a small partition as in a large one when one k-means made them. A row whose partition’s turn does not come before the LIMIT is met is missed; that is the scan’s approximation, and the margin trades it against the partitions read.

The pause. The margin alone leaves a tail: unfiltered at a margin of 2, the partitions a query probed had a median of 74, a mean of 143 and a maximum of 1,430 of the 1,974 (§44.9). A query whose ten nearest lie far from every centroid finds many centroids within its tenth row’s key less the margin, and probes them all, though few hold a row nearer than those it has. So the scan counts the rows it scores since one last entered the least upper bounds of the rows it is after, and once those are in hand and the count reaches pg_lion.vector_patience (30) units - a unit the rows an average partition would hold were there 2 sqrt(n) of them, n the rows of the build (the meta page’s count): sqrt(n) / 2 rows, or under a restriction twice that share of the members (lv_patience_unit()) - it PAUSES (lv_probe_part()). The rows it is after are a TIER: besides the L least upper bounds it keeps the K least for K = 10, 20, 40 and so on below L (LvTier), and the next row’s tier is the least K past the rows the pass has returned, or L. Paused on a tier, it takes those K as found and probes no partition until it has returned them or the next row lies past the K-th upper bound; then the next tier, whose count has run all along, decides the next pause. A REST, which starts a batch of four times the rows, ends a pause too. Paused, the scan still returns rows in order; a row nearer than those a pause returned, in a partition not probed yet, is skipped when it is found, as one the margin leaves till too late is. The pause trades those rows against the tail: in the numpy model (vtoy’s patience.py), at 975,000 rows in 1,950 partitions, a patience of 20 partitions reached 0.98 recall reading a mean of 28,000 rows a query (95th percentile 75,000), where the margin alone needed 57,000 (227,000).

How a pause ends was found by its failures. Ended when the row queue was empty, it returned every row the probes had kept below tau, and tau, the L-th least UPPER bound, lies past many more than L rows: a restricted scan pausing at once (vector_filter.sql’s, a patience of one partition’s members) returned 1,255 rows of a LIMIT 2000 over 9,000 members, the other 7,745 skipped as out of order. Ended only once the next row lay past tau, it filled the LIMIT but still returned rows far past the L nearest, and a LIMIT 1000 on the benchmark got 0.955 of its true rows; ended after L rows too, 0.983, against 0.980 for a LIMIT 10. Counting a restricted scan’s patience in members rather than rows lets a filter pause as an unfiltered scan does; counted in rows, a 1% filter never paused.

The tiers came from the local million-row benchmark (2026-10-16), where a query’s 95th percentile took three to four times its median. Counted against L alone (pg_lion.vector_batch, 100), a LIMIT 10 went on probing while rows entered the 100 nearest, which it would never return. On DBpedia’s 115,186 rows, LIMIT 10 at a margin of 8: 200 partitions at the median and 401 at the 95th percentile for 0.986 of the ten nearest (4.8 and 9.2 ms); by tiers, 159 and 254 for 0.985 (3.8 and 5.7 ms). A smaller L instead (vector_batch = 10) did as well for a LIMIT 10 and cost a LIMIT 100 2.5 times as much, its RESTs reading every chain again; by tiers a LIMIT 100 pauses at its own 100, as it did, but returns its first ten at the first tier’s pause and loses a row it finds later that belonged among them: 0.9908 at a margin of 8 for 336 partitions, against 0.9900 for 273. A filter’s members come a few to a partition, and a tier of ten fills and stalls sooner than one of 100: with the same unit, DBpedia’s 1% filter, probed (vector_exact_rows = 0), fell from 0.969 of the ten nearest at the default margin to 0.939; with twice the unit, 0.961, probing 368 partitions at the median against 530. At each recall a filter now probes fewer partitions than it did: 0.9925 at 1% for 820 against 908, 0.9935 at 10% for 360 against 448.

A known LIMIT (2026-10-16). A pause on a tier ends once it has returned that tier’s K rows, and probing resumes for the rows the query still wants; a member it then finds, in a partition not probed yet, nearer than rows already returned is skipped. A LIMIT 10 is stopped by its Limit at the first pause and loses nothing so; a LIMIT 100 paused at 10, 20, 40, 80 and L’s 100 in turn, each pause ending by count, and lost what it found after each (the ten “returned at the first tier’s pause” above). Through LionOrdered on DBpedia’s 115,186 rows (vector_exact_rows = 0, so that the 1% filter probes too), at a patience of 30, a LIMIT 100 found 0.9884 of its hundred nearest under a 3% filter at a margin of 8, against 0.9997 never pausing, and 0.9845 against 0.9999 under 1%; and at that margin it returned the plain scan’s rows (an Index Scan with a Filter) for 24 to 58% of the queries. A smaller LIMIT was to pause where it does, not a larger one at every tier on its way. So LionOrdered hands the scan the rows its query level’s LIMIT takes, OFFSET included, when the planner knows them (§44.6, lion_vec_scan_limit()), and the scan keeps no tier below them (lv_reset_rows()): a LIMIT 100 at the default L has none and pauses on L alone, whose rows the LIMIT takes whole, as before the tiers; a LIMIT 50 first pauses at 80; a LIMIT of 1 to 10 keeps every tier, LV_TIER_MIN being the least. On the same 200 queries (bench/results/2026-10-16-lionvec-limit) a LIMIT 100 at a patience of 30 now pauses once a query rather than five times, skips 0.01 rows a query where it skipped up to 0.97, and finds 0.9942, 0.9940 and 0.9941 of its hundred nearest at a margin of 8 under 10%, 3% and 1% (0.9913, 0.9884 and 0.9845 before), 0.9982 to 0.9988 at 16 (0.9963 to 0.9974) and 0.9679 to 0.9823 at 3 (0.9677 to 0.9727), probing the same partitions and reading the same pages to within three, in the same time. With the pause off nothing changes, and a LIMIT 10 is the same row for row and count for count at every filter, margin and patience measured. What is left of the gap to the pause off is the members in partitions the pause leaves unprobed, the pause’s trade; at 3% and a margin of 8 LionOrdered now returns the plain scan’s rows for 57% of the queries (39% before), the plain scan probing twice the partitions for 0.9987. A LIMIT the scan is not told keeps every tier: none, a generic plan’s Param, one under grouping, aggregates, DISTINCT or a window, and every LIMIT of the plain scan, which core’s Index Scan runs and no plan data reaches - a filter lion cannot index, or none, whose LIMIT 100 is the 0.9908 above.

A LIMIT past L paused on L by count all the same, then found the rest in further passes, each REST probing every chain probed so far again for four times the rows: at a patience of 30 a LIMIT 1000 under 10% and 3% filters made 2.5 and 2.75 passes a query, read 11,800 to 21,800 CODE pages and found 0.9840 to 0.9998 of its thousand nearest at margins of 3 to 16 (0.9838 to 0.9997 with every tier). So a scan told its LIMIT makes it L when it is more (lv_start()), up to an eighth of the rows work_mem holds, where REST stops growing L too (lv_cap(); about 10,900 rows at the default 4 MB, past which it RESTs as before): the LIMIT 1000 makes one pass and pauses at most on its thousand, reads 5,300 to 7,900 pages and finds 0.9863 to 1.0000 - at a margin of 8, 0.9997 under 10% and 0.9999 under 3% (0.9987 and 0.9983) from 6,816 and 7,793 pages (19,251 and 21,729), in 75 and 67 ms (142 and 137). With the pause off it returns the same rows, every query, from 2.2 to 2.8 times fewer pages. LIMITs of 10 and 100 are untouched, L being 100 already; and L at most an eighth of work_mem’s rows keeps the queue’s bound work_mem’s (lv_cap() is never under four L’s).

The patience grows with the margin past twice its default (lv_patience_scale(): by margin / 6 at the default). A larger margin asks for more of the nearest rows, and a patience that stayed put held the recall to what the scan had found by then: 0.9935 of the ten nearest at any margin from 12 (0.9865 with the tiers). In step, the margin alone reaches 0.9975 at 24 (0.999 of a LIMIT 100’s at 32); up to twice the default the margin stops a scan first, and the patience only its tail.

The unit was the average partition’s own rows, the rows or members over the partitions, until the default went from 2 sqrt(n) partitions to 8 sqrt(n) (§44.3). Then a patience of 30 paused after a quarter of the rows, and on DBpedia held the unfiltered recall to 0.966 at any margin in 2,715 partitions, where 678 reached 0.988 (§44.9). Counted in the rows of 2 sqrt(n) partitions, whatever their number, the pause is where it was at the old default, and the partitions reloption no longer moves it: 30 in 2,715 partitions now pauses where 120 did. The unit takes n from the meta page’s rows as of the build rather than from the partitions, and is at least one row. In an index that spills (§44.3), each row is in two partitions and the probes count items, so the unit is doubled for the copies.

Inner product takes one more step. An L2 row nearer the query than its centroid by its offset’s projection is so only by that projection less |x - c|^2 / 2 (|q - x|^2 = |q - c|^2 - 2 <q - c, x - c> + |x - c|^2), and a cosine row likewise on the sphere; the margin is calibrated against that. An inner product’s row gains all of <q, x - c>, so a partition of rows of mixed lengths keeps its answers - the rows that stick out farthest along q - beyond any such margin: vector_filter.sql’s ten clusters, of rows 2.4 to 3.3 long, got 0.61 of their ten nearest at margins 0 and 2 alike and all of them only at 4. So lionvec_lift_key() lifts each partition’s rows by one more coordinate, sqrt(M^2 - |x|^2), M the partition’s greatest |x| (its PART item’s), which puts them at M from the origin, where the nearest to the unit query (q / |q|, 0) are those of the greatest <q, x> (Bachrach and others, RecSys 2014). Their centroid then lies inside that sphere by the rows' spread, their mean |x - c|^2 (the PART item’s too), at squared distance 1 + M^2 - spread - 2 <q / |q|, c>, and the margin comes off that distance as off L2’s. On unit vectors this is L2’s ranking for the unit query; on the test’s clusters it finds all ten from a margin of 1. Each partition’s own M, rather than the table’s, keeps one long row from widening every partition’s margin. An insert longer than its partition’s M is lifted to no height, and its bound is the looser for it until REINDEX. The unit tests check the algebra: at a margin of the lifted rows' greatest distance from their lifted centroid, the bound holds for every row and query (the triangle inequality there).

Strict order. When a bound fails, a row turns up after rows farther than it were returned. It is skipped (nskipped in the DEBUG1 line), so the scan’s output is always in order: a failure costs recall, never order, and the executor never needs to recheck.

Memory. The row queue is capped at work_mem (sizeof(LvItem) each, never under four batches' worth). When a probe reaches the cap it drops the rows tau has since passed by, and if that is not enough, CUTS: the half with the least lower bounds stay, the cut is the least bound of those that went, and the probes keep only rows below it until the next REST; a REST at the cut stands for the rest. Rows AT the cut stay - REST comes out after every row at its key - so a scan of a million equal vectors returns cap / 2 of them rather than none. After a REST, L is min(4 L, max(L, cap / 8)).

Progress. A hash keyed by TID keeps what the scan has learnt of each row across RESTs: its codes' bounds once refined, its exact key once fetched, and GONE once returned or found missing. A probe starts a row from that, not from plane 1 again. Without it a pass cut short by work_mem whose kept rows all refined past the cut was made again, the same, for ever - the rows came back from plane 1 below the cut, refined past it, and the cut’s REST came out first (found by vector.sql’s work_mem = 64kB, vector_batch = 1 case, 2026-10-14). With it, every REST moves at least one row up a level or out.

Copies. In an index that spills a row has two CODE items, in two partitions' chains, and a scan may probe both. Each must be met once a PASS - from the start or a REST to the next REST - or a row would be refined, fetched or returned twice, and would stand twice among the L least upper bounds, making tau too small and the pause too early. So the progress hash also records each row a pass ENTERS - whose lower bound is within tau and below the cut, so that it goes into the L least upper bounds and the row queue - and the pass’s number (lv_enter()); the row’s other item, met later in the same pass, is skipped and counted (ncopies, the DEBUG1 line’s last field, N copies skipped). The first met wins: the other has the same bounds, being the same bytes, and tau and the cut only fall within a pass, so it could do no more than the first did. A row left out of the pass (lo > tau) is not entered, and its other item is left out too, for the same reason; the REST that stands for them starts a new pass, in which each row is entered once again. A row returned is GONE in every pass. A row that leaves the pass unworked - dropped by lv_shrink() or a cut, left out on the second look at tau after its upper bound went in, or still queued at a REST - is forgotten if the hash knows nothing more of it (lv_forget()): its other item is then left out as it was, tau and the cut only falling within a pass. So the entries that say only that a row entered are the row queue’s own rows, within its work_mem cap, and the rest are the rows refined, fetched or returned, as without spill. Kept, they would grow to every row within tau of every pass, orders of magnitude past work_mem in a deep scan under a small one (a review found it). An assert build checks at each REST that none is left. A probe of an index that spills looks every row up, even in a pass whose hash was empty when it began, so that a TID a corrupted chain holds twice is skipped as a copy rather than returned twice. The patience counts items, which the probes see, so a partition’s worth is twice the rows over the partitions; restricted, it is twice the members over the partitions, the members themselves counted once, from the filter. A walk (NULL or cosine zero query, or no ORDER BY) reads each partition’s chain and skips its copies, so that it returns each row once.

The hash’s bound (2026-10-15, after a review found the hash unbounded). The GONE entries stay for the scan’s life, so the hash grew with every row returned: a scan with no LIMIT over a hundred million rows would hold gigabytes, past any work_mem, and fail in palloc where a sort would spill. It is now held to hash_mem, as a hash join’s table is: as many entries as the largest of simplehash’s power-of-two arrays within it holds below its fill of 0.9 (lv_seen_most()), 48 bytes each; the PLANE1 entries, the row queue’s own, come with its cap and are counted apart. When an entry would not fit, lv_seen_evict() walks the hash, in three tiers:

  • the entries a probe no longer needs: those of rows whose PLANE-1 upper bound, kept in the entry (hi1, carried in the queue’s items from the probe), lies below the key of the row returned last. The scan’s order is strict, so no such row can be returned again, and from that walk on a probe drops a row whose upper bound lies below that key, as the hash would have - and a refine or a fetch one whose codes or distance put it there, which the hash then keeps. Nothing is lost: the probe tells those rows by plane 1 alone, as it first read them. (A row whose plane-1 bound fails so, not yet returned, is lost, as one the margin leaves till too late is.) This is the review’s own suggestion made exact: the rows returned are dropped as the scan moves past them, as far as plane 1 can tell them from the rest.
  • if that freed less than a quarter of the hash, the rows behind the last that plane 1 cannot tell: returned at a lesser distance, or known by their codes or distance to lie nearer. A probe meets them again and refines (or fetches) them once more, to find them behind again;
  • and if that is still less than a quarter, what it knows of the rows ahead (CODES8, EXACT).

The second and third lose knowledge, and bring back the livelock that knowledge prevents (above): so after four passes that lost knowledge since the scan was last done with a row, it stops (lv_may_probe()); passes that lose nothing in between do not reset that count, since they may only win back what the last one lost (vector.sql’s scan at 64 kB made 65,000 passes for 20 rows when they did). A walk is allowed once a quarter of what the hash may hold has come in since the last, which pays for it; when none is, or a walk frees no room, a row’s codes or distance are not kept, and a row returned must be, so the scan stops after it. In an index that spills, an entry lost also lets the row’s other item enter its pass, so that the queue may hold the row twice; whichever comes out second finds the row returned, or behind, and is dropped. A scan that cannot go on fails the query by default, lionvec scan of "i" outgrew hash_mem after N rows, an ERROR whose HINT names work_mem, hash_mem_multiplier, a sort, a LIMIT and pg_lion.vector_overflow (§44.8). Set to stop, that ends the scan instead: it returns no more rows and raises a NOTICE, lionvec scan of "i" stopped after N rows, what it knew of its rows having outgrown hash_mem; what it returned is in order, each row once, the nearest it found. The ERROR is the default (chosen 2026-10-16) so that a query that asked for every row never gets fewer without failing; pgvector’s iterative scans end without a word. The DEBUG1 line counts the walks and those that lost knowledge. How far a scan gets depends on how well plane 1 tells rows apart, so on the dimensions and the data. Measured with stop, no LIMIT, work_mem 4 MB and so hash_mem 8 MB (117,964 entries) unless said, one row of the table as the query:

  • 3,006 rows of 40 dimensions (vector.sql’s), at 128 kB and hash_mem_multiplier 1 (1,843 entries): every row, in order, the first tier alone making room.
  • 2,000,000 rows of 16 dimensions, where plane 1 tells rows apart worst: before, every row, the scan’s memory 165 MB, 81 s; after, it stops after 31,250 rows, 30 MB, 2.4 s; at 16 MB after 88,212; at 64 MB every row, 241 MB (its queue’s cap and the hash both grow with it), 33 s.
  • 200,000 rows of 256 dimensions in 100 clusters of 2,000, uniform noise about each centre, so that the other 198,000 rows lie at distances within a few percent of each other, inside plane 1’s bound: before, 199,994 rows (6 out of order), 25 MB, 1.9 s, each row refined once; after, it stops after 14,912 rows, 18 MB, 2.0 s, having refined 604,286 (five walks, each losing knowledge); at 8 MB every row as before, 36 MB, 2.1 s, no walk. The rows a scan must know of at once to return the next is the table here, and the hash holds 59% of it.

A LIMIT of 10 or 1,000 over the last, 200 queries, costs what it did before (2.8 and 5.3 ms a query either way, within the noise of runs). pgvector’s iterative scans bound theirs the same way, and tighter: HNSW’s ends, without a word, at hnsw.max_scan_tuples (20,000) rows or work_mem times hnsw.scan_mem_multiplier (1). The bound is read once, as the scan starts (seenmost), so that lv_remember() costs no more than it did: a new entry with room is one insert, and only a full hash looks a row up first.

NULLs and zeros. NULL rows come last, with NULL distances, from the NULL chain; before them, in a cosine index, the zero vectors, whose distance is NaN (float8 sorts NaN after every number, as the sequential scan’s sort does), from the same chain. A NULL query returns every row of every chain with a NULL distance; a cosine zero query every non-NULL row with NaN, then the NULLs; an index with no vector yet only its NULLs. A walk copies a page’s TIDs out under the page’s lock, so a VACUUM between two calls cannot make one come back twice. A scan without an ORDER BY, which the planner never makes (its cost is infinite, and from PostgreSQL 18, which weighs disabled nodes before costs, it carries two disabled nodes, as pgvector’s do: without them it beat a sequential scan disabled by enable_seqscan = off, and a count(*) failed as an index-only scan, found by the insert spec of §44.10), walks every row.

Prefetch (2026-10-16). A cold scan waits on its reads one at a time: a probe reads a partition’s CODE chain a page after another, a refine a REFINE page, an exact distance a heap page and its vector’s TOAST pages, and none starts before the one before it is in. While pg_lion.vector_prefetch is fadvise (§44.8: the default on 16, and what its on was until 2026-10-17; from 17 the scan reads through streams instead, “Read streams” below) the scan asks for the pages it will read next before it reads them, through PrefetchBuffer(): as many ahead as the relation’s tablespace’s effective_io_concurrency, to LIONVEC_PREFETCH_MAX (64), so->ipf pages of the index and so->hpf of the heap, 0 asking for none (lv_start_prefetch()). PrefetchBuffer() pins nothing and is posix_fadvise() under it on 16 to 19 alike, whatever 18’s io_method: the kernel reads the pages into its cache meanwhile, and the scan’s ReadBuffer() finds them there. It is a hint, never a read, so the rows a scan returns and their order are the same with it on or off, and so is every count of the DEBUG1 line but the two it adds, N pages prefetched (M not in shared buffers), M the pages PrefetchBuffer() did not find in shared buffers, which the kernel may still hold. It asks for:

  • the rest of a chain the build wrote. The build writes each partition’s CODE chain in blocks one after another, and its PART item counts its items, so its first lionvec_chain_pages() pages at lionvec_page_items() items a page (so->pnrows, kept by lv_read_parts() beside the heads) lie from its head on: reading block b of that run a probe asks for those to b + dist (lionvec_prefetch_end()), and once a page’s next is not b + 1 for no more of the chain, the pages inserts appended lying a page here and there. The PART chain, which every scan reads whole, likewise, its pages counted from the partitions', and a walk’s chains for as long as each runs on block after block; not a probe’s of the REST chain, whose length nothing records.
  • the queues' next pages: the REFINE pages of the centroids at the top of the partition queue (LV_CENT1) and of the rows at the top of the row queue (LV_PLANE1), and the heap blocks of the rows whose exact distances come next (LV_CODES8). Of a queue’s first LV_PF_TOP(dist) = 4 dist
    • 1 entries, which hold its least (a binary heap’s first levels), the dist least at the level (lionvec_least()), after lv_read_parts(), after each probe, and again once half as many entries at that level as the last look chose have come out, so that what was asked for stays about dist ahead (lv_prefetch_due()). Of the row queue only the rows the scan works before its next probe, keyed below the next partition’s key or the end of a pause if later (lv_prefetch_bound()): a look at the least 64 rows after every probe asked for 2,941 REFINE pages of a DBpedia scan that refined 97 rows. An entry asks once at a level (pf, cleared when it moves on), however many looks find it near the top, and a table of the 128 blocks of each relation last asked for, by hash, keeps a page that several entries share - five centroids' REFINE items to a page at 1536 dimensions - from being asked for again.
  • the exact path’s members (lv_exact()), in TID order, each one’s heap block dist members before its fetch, a block’s members asking once.

The heap is asked for only when it is the heap access method’s, whose TIDs name its blocks, and its vectors lie in its pages (lv_vector_inline()): stored PLAIN or MAIN, or small enough, within the larger of TOAST_TUPLE_THRESHOLD and the table’s toast_tuple_target (both 2,032 bytes at 8K pages unless the table sets the target: 506 dimensions of vector, 1,012 of halfvec; under toast_tuple_target = 8160, every vector an index takes). The heap moves a row’s values to TOAST only once the row is past the threshold, and only until it is within the target. Until 2026-10-17 the threshold alone decided, so a table loaded under a raised target - the DBpedia benchmark’s, KNN_INLINE - had its vectors in its pages and none of them asked for. Past both pgvector’s EXTERNAL storage puts a vector in TOAST, which a fetch reaches through the TOAST index, the heap page the least of what it reads; asked for alone, on DBpedia’s 1536 dimensions, the heap pages bought nothing, the exact path cold taking 113 ms a query under the 1% filter against 109 without. Reading a toasted vector’s chunks ahead needs its TOAST pointer, from the heap tuple, first, or the chunks' places kept in the index, which an index of vectors that may lie in TOAST keeps (§44.13): its exact path streams them, and its rerank asks for a row’s first chunk page with its heap page.

Measured (§44.9, “Prefetch and parallel scans”): cold, each query alone after a restart with the page cache dropped, a serial scan at a margin of 3 took 17.4 ms against 21.2 without (its wait on reads, by track_io_timing, 11.4 ms against 17.7, for 410 pages from the kernel), at a margin of 8 32.4 against 49.4 (20.6 against 41.9, for 972 pages), and the exact path over vectors kept in the heap, under a 1% filter, 33.5 against 70.6 (11.0 against 64.8, for 1,213 pages). Warm, every page in shared buffers, an ask is a lookup of a page that is there already, 4 to 9% of a scan when every page was asked for (2.39 ms against 2.29 at a margin of 3, 5.51 against 5.06 at 8). So the scan watches what its asks of each relation find (LionAsk, lion_readahead.h): after LION_ASK_WARM (32) asks in a row that found their page in shared buffers it lets the next LION_ASK_DORMANT (256) asks pass unasked, then asks again. A cold relation is asked for all along, a warm one a ninth as often (a margin-8 scan asks for 65 pages rather than 880), and one warm in parts loses at most 256 asks at a time; the PART chain, which every scan reads and so is warm whatever else is, does not count. Warm it then costs about 2%: 2.33 ms against 2.29 at a margin of 3 and 4.99 against 4.91 at 8. While a relation’s asks rest, the looks into the queues for them rest too (2026-10-27), as they do while a stream rests (“Read streams” below): on 16 a look read up to 257 entries of its queue and chose its asks only for lion_ask() to let them pass, 5.6% of a warm DBpedia query by perf and 6% of a random vector’s. So a look whose every ask would pass - the index’s, or the heap’s and, of a row placed in TOAST, the TOAST relation’s too - is let pass and chooses nothing, which has the next entry out at its level bring the next; each such entry counts one against the rest, the ask the look would have made for it, so the rest ends after 256 of them as it did after 256 asks, and the first look after it asks for all it chooses, a resting look having marked nothing. It asks a little more that way, 93 pages a warm DBpedia query against 80 and 2,346 against 1,876 on random vectors, and looks 450 ns less a refine (“The refine’s reads” below).

Read streams (2026-10-17; lion_readahead.c, lion_readahead_ring.c). From PostgreSQL 17 a scan reads the pages it expects through core’s read stream (read_stream_begin_relation()) instead of asking the kernel for them: pg_lion.vector_prefetch = stream, the default there (§44.8). The stream reads them into shared buffers itself - adjacent blocks in one read of up to io_combine_limit, on 18 by an I/O worker or io_uring as io_method says, under io_method = sync and on 17 by preadv() after posix_fadvise() advice - and hands the scan each buffer pinned, so a page read ahead lands in shared buffers rather than the kernel’s cache, and is looked up once. Core’s stream, as 17’s and 18’s read_stream.c have it, reads only the blocks its callback names, in the order named, and calls the callback only from read_stream_next_buffer(), as its consumer pulls: it looks ahead to a distance that starts at one block (a full I/O’s, io_combine_limit, under READ_STREAM_FULL), doubles each time the consumer waits on a read and falls by one at each block found in shared buffers; once the callback names no block the stream gives nothing more until read_stream_reset(), which starts it again from one block ahead. It cannot be paused, and it starts no read before its consumer’s first pull. A scan’s reads fit that only in part. It knows the run of the chain it is reading, and in a REST pass the chains it will probe; past those it predicts (the REFINE pages at the top of its queues, the next partition’s chain) and reads out of the order it predicted, some pages never, some it never predicted, as its queues reorder under the rows and partitions coming in.

lion_readahead.c sits between, through a ring of the blocks the reader expects (lion_readahead_ring.c, frontend-safe, unit-tested against a model of core’s stream in test/unit/readahead_test.c). The reader pushes a block, urgent - the rest of the run it is reading - or predicted; the stream’s callback takes urgent blocks first, then predicted ones, then what the reader’s refill pushes when it is asked for a block and none is pending, and with nothing names none, which ends the stream. A read takes its block from wherever it is: held, pulled from the stream already by a read of a later block, its pin handed over; given to the stream, which is pulled up to it, the blocks before it held for their own reads; pending, moved to the front, and the stream started again if it had ended - once it holds nothing, which a restart releases; tracked nowhere, read apart (ReadBuffer()), unless the stream is idle with blocks pending, which the read then sets going. Held and given blocks are pins, at most the ring’s limit: a block given when the ring is full pushes out the oldest held one, released unread, and with every pin a given one the callback names none. A stream that ended is begun anew with READ_STREAM_FULL, a full I/O ahead, if it read anything in since it last started, and reset, which costs less, if it found every block in shared buffers (lion_readahead_op_restart()). And a stream rests as the asks do, seeing its reads rather than guessing from PrefetchBuffer(): after LION_ASK_WARM (32) reads in a row that read nothing in, it pushes nothing and refills nothing for the next LION_ASK_DORMANT (256) reads, which are ReadBuffer()’s, or until one reads a block in (lion_readahead_dormant()), and the scan skips its looks into the queues meanwhile. A reader that knows its whole list in order needs none of that: lion_readahead_begin_list() is a plain stream over it, from a full I/O ahead, holding as many pins as core’s own limits allow (GetPinLimit()), as a bitmap heap scan’s does. It rests too, blind as the asks are: after 32 blocks in a row it read nothing in for, its callback names none, the stream ends once it has handed out what it was named - however many rests that lasts, a stream far ahead holding hundreds - and the reads left of the rest are the reader’s own (lion_readahead_next() returns those blocks unpinned), after which the stream is reset and goes on from the list’s next block - a warm block it hands out being a lookup and a pin that the reader’s own read repeats. Where a stream is not wanted, or on 16, the same calls ask the kernel (lion_ask()) and read with ReadBuffer(), or only read: the fallback for lion’s readers as for lionvec’s. Every pin is the current resource owner’s, so an error releases them with its transaction, subtransaction or portal.

A lionvec scan has three streams of the index, in its own context, kept from one rescan to the next and reset - every pin released, every block forgotten - at each rescan, at each change of phase (the walks after the queues) and when it stops (lv_readahead_reset()): so->chains, the PART chain, each probe’s run of its chain, pushed urgent so->ipf pages ahead, its first page too (lv_run_start()), REST’s chains and the walks'; so->crefs, the centroids' REFINE pages; and so->rrefs, the rows'. Each holds up to twice so->ipf pins, at least LION_READAHEAD_MIN_PINS (4), and no more than an LV_RA_SHAREth (an eighth) of the backend’s fair share of shared buffers (lion_pin_fair_share(): GetPinLimit() on 18, the same arithmetic on 17), reading half its pins ahead; a pool that leaves a stream fewer than 4 has the scan ask the kernel instead. The queues' looks (lv_prefetch_queue()) push their pages in the order of their keys, which is the order a stream reads them in, and a stream that wants a page with none pending looks into its queue at once (lv_crefs_refill(), lv_rrefs_refill()), rather than once half the last look is read, so that it does not end. Between REST passes the chains' refill predicts the partitions to probe next (lv_chains_refill()): of the partition queue’s top, the least centroid known by its 8-bit codes whose chain was not pushed yet, a chain at a time to so->ipf pages ahead, one another process has claimed passed over. A chain it predicts and the scan does not probe - its LIMIT met, a centroid refined after coming first - is read for nothing and released unread; without the prediction each probe’s stream ended with its run and started again from one block, and cold scans took 1.3 to 2.3 times as long (§44.9).

The heap’s pages are another matter. The rows' heap blocks for their exact distances (LV_CODES8) are asked of the kernel in every mode (lv_prefetch_heap()): a stream’s pins on them would stay from one amgettuple() call to the next, between which the executor runs anything, and the fetch prunes a page only when its own pin is the backend’s only one (heap_page_prune_opt(), ConditionalLockBufferForCleanup()), which a stream’s would not be. The exact path reads its heap blocks inside one call (lv_exact()): there a list stream over the members' blocks, each block taken from it at its first member and its pin dropped before the fetch pins it again, and the stream ended before the call returns. The index’s streams do hold pins between calls - the pages they read ahead - which nothing waits on: lionvec’s VACUUM takes no cleanup lock and never recycles a page (§44.5), its WAL is generic WAL, whose redo on a standby takes none either, and the scan holds no lock between calls. A parallel scan’s participants each have their own. The DEBUG1 line names what the streams did: N pages streamed (M read in), A read apart, W streamed unread, R restarts - the reads they served and the pages of those they read into shared buffers, the reads no stream was given (unpredicted, or made while resting), the pages they read ahead and released unread, and the times one ended and started again - and the heap’s asks as H heap pages prefetched (X not in shared buffers). On 16, stream is accepted and is fadvise.

The refine’s reads (2026-10-27). A refine reads one REFINE item - a row’s (lv_refine()) or a centroid’s (lv_refine_part()): lion_readahead_read(), which on 16 is ReadBuffer(), the share lock, lionvec_check_page(), lionvec_refine_at() and lionvec_estimate8() over its codes - and warm, every page in shared buffers, a row’s took 2.6 us on DBpedia at 1536 dimensions, 1.0 of them in the estimate (the timing of 2026-10-26 found 3,125 ns, 1,810 of them from the read through the item’s checks). A shared-buffer hit does not cost that; a chain of cache misses does, each waiting for the one before it. Timing each step twice over, the second time on the same inputs a moment later (temporary rdtsc instrumentation, each timestamp behind lfence, so that a step waits for its own loads), gives the instructions' cost as the repeat and the misses' as the difference. On 4 KB pages, first and repeat, ns a row: ReadBuffer() 871 and 92 - the buffer table is a chained hash, and its segment, its element and then the buffer’s descriptor are read one after another, three misses, two thirds of the step in hash_search_with_hash_value() and a quarter in PinBuffer() by perf; the lock 40 and 41; lionvec_check_page() 789 and 28 - the page header, then the special space where its pd_special points, on the page’s other 4 KB half, two more misses in a row, each a TLB miss as well; lionvec_refine_at() 180 and 34 - the line pointer lies in the header’s line, the item’s first line is the miss; and lionvec_estimate8() 986 and 396 - 24 lines of codes read in order, which the hardware prefetcher catches only after the first few. A centroid’s refine (245, 168, 98 and 681) is cheaper only because its 543 pages come up query after query and stay in the cache. The rest of the list it was not: a query’s rows share no REFINE page - 85 refines read 85 pages, none the one before it, and even a random-vector query’s 13,400 refines read 10,180 pages, 0.45 of them the one before - so refining a page’s rows together, or keeping its pin from one refine to the next, would save nothing (centroids do share, 118 refines on 62 pages, 1.7 of them consecutive: not worth a pin cache); and the checks cost 28 ns once their lines are there. Huge pages take the TLB misses out of all of it, the buffer table’s and the descriptors' lines included: a row’s refine 3,480 to 3,110 ns serialized, a DBpedia query 6% faster and a random-vector one 11%, so huge_pages = on is worth recommending for a warm index - a setting, not a change.

What the scan can change is the order of the misses: it asks for the lines before it needs them (lion_prefetch(), lion_compat.h: __builtin_prefetch, which reads nothing, waits for nothing and never faults), so that they run together. Once the buffer is pinned and before it is locked, both refines prefetch the page header and the special space, whose place on a lionvec page is fixed (BLCKSZ - MAXALIGN(sizeof(LionVecPageOpaqueData))), nothing of the page read before the lock; and lionvec_refine_at(), once the line pointer has given it the item and its length is checked, prefetches every line of the item before it reads the header (lion_prefetch_range(), a macro: GCC 13 deletes a function that only prefetches), so that the header’s miss and the codes' overlap and the estimate finds its codes on the way. The chain after the pin is then the header and the special space together, then the item, in place of four misses and the prefetcher’s run-up: serialized, a row’s refine on DBpedia 3,590 to 2,950 ns, the check 843 to 100, the estimate 911 to 488 and the lock 40 to 472, where the two lines' misses now land (a locked instruction waits for them). The buffer table’s three misses stay: the lookup is core’s, an extension cannot prefetch into it, and the pages being distinct no refine shares one. The look into the queues before a refine (lv_prefetch_due()) was the other cost, 450 to 540 ns of a row’s refine on DBpedia and 5.6% of a warm query by perf, mostly to choose asks that then rested: it rests with the asks now (“Prefetch” above), 47 ns. Measured (§44.9, “The refine’s reads”): warm, the same rows in the same order, a DBpedia query 1.151 to 1.071 ms and a random-vector query 50.7 to 39.7 ms on 4 KB pages, 1.083 to 0.976 and 45.2 to 33.9 on huge pages; cold, the same reads and the same time.

A parallel scan (2026-10-16). amcanparallel: with pg_lion.vector_parallel_scan on (off by default since 2026-10-17, the measurements below finding it slower everywhere; off, lo_knn_partial() drops a lionvec order’s partial paths), an unfiltered ORDER BY of the distance may be a Parallel Index Scan under a Gather Merge (§44.7 prices it; a filtered one stays serial, below). Each participant - every worker, and the leader unless parallel_leader_participation is off - runs the whole scan above on its own: it reads the meta page and the PART chain, ranks every centroid, and keeps its own queues, tau, tiers, pause and progress hash, returning its rows in the order of their distances, each with its distance in xs_orderbyvals, which Gather Merge merges into one order. What the participants share is whose each chain is. LvShared, in the area core gives the access method after the ParallelIndexScanDesc (at ps_offset, ps_offset_am from 18), holds a claim bit a chain - partition p’s bit p, then the REST chain’s and the NULL chain’s - and a participant probes a partition only if its pg_atomic_fetch_or_u32() found the bit clear (lv_claim(), in lv_probe_part()), and drops one another has claimed as it comes out of its queue (lv_claimed()), before reading the centroid’s REFINE item, counting it neither among its probes nor in so->probed. The bits are LIONVEC_MAX_PARTS + 2, 32 KB, the most check_meta() allows, as before 18 the estimate cannot see the index. So each partition is probed by one participant and each of its rows returned by that one, in that one’s order, and the merged order is in distance as the serial scan’s is: a row leaves its participant when nothing in that participant’s queues, nor in a partition it may still claim, is nearer by the bounds, and the other participants' rows reach Gather Merge in their own order. The REST chain is the first participant’s to reach lv_start(), and a REST pass probes again the chains its participant probed, for four times the rows. The walks claim chain by chain the same way (lv_walk_claim()), so a walk through the index returns every row once; the NULL chain is claimed once a scan, by the first walk to reach it, and that participant walks both its zero vectors and its NULLs, whose NaN and NULL distances sort after every other participant’s rows. amparallelrescan, which core calls with no worker running, clears the area. A restricted scan (LionOrdered’s, §44.6) is never parallel, which lv_start() asserts: LionOrdered’s paths are not partial, and under pg_lion.vector_postfilter exact or error a partial path that a Filter would follow is removed as the serial one is (§44.7).

In an index that spills (§44.3) a row’s two items lie in two partitions, which two participants may claim, each then finding the row. So LvShared holds the rows returned as well, an open-addressed set of 32,768 TIDs (256 KB), each slot set once by a compare-and-swap from zero: the participant that records a row first returns it and the other drops it (lv_take_row(); rows another returned in its DEBUG1 line, 3 to 7 a query on DBpedia). A row that finds none of its 32 slots free nor its own is returned through its own item and not its copy - one partition holds a row’s own item, and one participant that partition - so it never comes out twice, whatever the set holds, and is missed only where no participant probes its own partition.

The stop rules stay each participant’s own, over the partitions it claimed and the rows it found. A row’s place in its participant’s order needs nothing else, and a bound shared between the participants - the least tau of all, so that one stops where another’s rows already lie nearer than its next partition, as the serial scan stops on all of its rows - would put a heap of upper bounds that every probe updates between them, and make one participant’s pause hang on the others' rows. The hope was that each participant, scoring about a share of the rows and finding a share of the nearest, would pause where the serial scan does in its own count, so that together they probe about what it probes. They do not: its rows are its own partitions' and so lie farther, which carries the margin farther down the order, and the patience is a count of rows, which a participant scores at the serial scan’s rate. On DBpedia at a margin of 3 each of two participants probed 29 partitions and each of four 33, where the serial scan probed 36, so together 1.6 and 3.7 times its partitions - and they found more of the nearest rows for it, 0.935 and 0.9515 of the ten nearest against 0.919. To that recall the serial scan, at a wider margin, probes 1.24 times fewer partitions than two participants (at a margin of about 3.7) and 2.0 times fewer than four (4.6), and as many times fewer CODE pages; at a margin of 8 two participants' 0.994 cost 1.03 times the serial scan’s partitions for it (at about 15), and four’s 0.996 lies past the serial scan’s best measured (0.9945 at 16, from 382 partitions to their 791). That is the ratio a shared bound would win back, which §44.11 leaves to it.

The executor runs the workers ahead of the leader. A worker does not wait to be asked: it runs its scan and puts each row in its tuple queue until the queue is full or the leader ends the query, while Gather Merge waits first for a row from every participant and then takes the least of their heads. So under a LIMIT 10 the participants together returned 17.3 rows a query with two participants and 36.8 with four, against the serial scan’s 11 (the query’s own row is filtered out), and computed 24.3 and 54.2 exact distances against 11.8: the executor’s model, not the scan’s, which cannot know how many rows the leader will take. And the first row waits for the slowest participant’s first, each having ranked every centroid and probed to its own rules, and for the workers' start, about 10 ms a query here: warm, under EXPLAIN (ANALYZE, VERBOSE), the Gather Merge’s first row came at a median of 12.1 ms with one worker and 14.7 with three, where each worker’s own first row came 1.6 and 2.1 ms into its scan and the serial scan’s 0.6 ms into its (an assert-enabled build, pg_lion preloaded, pgvector loaded by each worker). The parallel scans took 5 and 8 times as long as the serial one (§44.9).

44.5 Concurrency, WAL and VACUUM

  • Insert (lionvec_append()): under the meta page’s exclusive lock, which serializes inserts, the row’s partition is found (§44.3), its REFINE item goes on the REFINE tail and its CODE item on its chain’s tail - its partition’s, the REST chain’s in an index without partitions, or the NULL chain’s - each extended with a new page when full; in an index that spills, its copy then goes on its second partition’s tail (§44.3). A generic WAL record a step, as one takes at most four buffers: the REFINE item with the meta page, whose REFINE tail it may move; then the CODE item, its page, any new one, and the PART item or the meta page that holds the chain’s tail when the tail moves; then the copy, as the CODE item. A crash after the first leaves a REFINE item nothing points at, which costs its space and nothing else. A crash after the second leaves a row without its copy: it is still in its nearest partition, and only a scan that would have found it through the copy alone misses it - recall, for that row only, until REINDEX; no scan returns a wrong row, nor one twice. Scans and VACUUM lock one page at a time and never the meta page with another, so the order meta, REFINE, CODE, PART, then the copy’s CODE and PART, each record’s pages released before the next record’s are locked, cannot deadlock against them. A probe follows the chain to its end as it then stands, so it may meet rows inserted since its snapshot; the heap fetch drops them. Every row a scan’s snapshot sees had its items appended before that snapshot was taken, so none is missed. An insert reads the whole PART chain, 60 pages for 1,975 partitions, and the REFINE items of its candidates.
  • Build: two heap scans. The first computes the mean (its rows normalized for cosine) and takes the sample, from which k-means finds the centroids (§44.3), whose REFINE items are the first written; it fails the build at the first row with which the rows so far want more partitions than the sample has rows for (pg_lion.vector_build_sample, §44.8). The second writes each row’s REFINE item in place and puts its CODE item into a tuplesort (maintenance_work_mem) keyed by its group - its partition, then the REST and NULL chains - and its TID, from which the CODE chains are written in one run of blocks, each group starting a page, and each partition’s count, spread and longest row summed in that order; then the PART chain, then the meta page. In an index that spills the sort holds two items a row, the copy under its second partition’s group, so it is twice the size and spills to disk at half the rows; the meta page’s row count counts each row once, the PART items' counts every item. The whole range is WAL-logged at the end (log_newpage_range()). Neither scan joins a synchronized scan (allow_sync = false), which could start part-way through the heap: the sample must be the same from build to build, and the second scan’s REFINE items then lie in heap order, as each partition’s CODE items do whatever order the rows came in, being sorted by TID. Built in parallel from PostgreSQL 16 (§44.12). Its phases are reported in pg_stat_progress_create_index (lionvecbuildphasename()), numbered on from core’s initializing as nbtree’s are: sampling table, the first scan, and assigning rows to partitions, the second, with the blocks the heap AM reports, and the second with its rows done, every 1024 (LIONVEC_PROGRESS_ROWS), of the rows the first met, which differ only by a row pruned between the scans or one a CONCURRENTLY build’s snapshots disagree on; between them computing centroids, k-means' iterations done of LIONVEC_KMEANS_ITERS, reported after each that moved the centroids (through LionVecKmeansWork’s progress serially, by the leader in parallel), fewer than all of them when k-means stops early; then sorting rows, the CODE items' sort, writing partitions, the rows written to the CODE chains, each by its own item and not its copy, and WAL-logging index. Only a serial build or a parallel one’s leader reports, as in core’s builds; in the second scan each process adds its rows to a shared atomic count every 1024, before it says it is done, and the leader reports the count. CREATE INDEX CONCURRENTLY and REINDEX work as for any AM. An unlogged index’s init fork is the meta page and empty REST and NULL chains.
  • VACUUM (lionvecbulkdelete()): walks every CODE chain (lionvec_chain_heads(): the partitions' from the PART chain, then REST and NULL), removing the dead rows' items (PageIndexMultiDelete()), and collects their REFINE locations, in maintenance_work_mem, which it then removes a REFINE page at a time in block order, each checked by TID first, with PageIndexTupleDeleteNoCompact(): the line pointers stay, so the other rows' CODE items still point right. A crash between the two steps leaves REFINE items nothing points at, which the next VACUUM does not find; they cost space and nothing else. In an index that spills a dead row’s two items go as their chains are walked, and its REFINE location is taken from its primary alone, so that it is removed once; the row is counted once, by its primary, in the rows removed and the index’s rows. When the locations fill maintenance_work_mem mid-walk they are removed then, and a copy in a chain not walked yet may point a while at a REFINE item gone or reused; a scan that meets it finds no item with its TID and takes the row as gone, which it is. A crash part-way leaves copies of rows whose primaries went; the heap’s line pointers stay dead until a VACUUM completes, so the next one removes them. The bulk-delete callback is asked of a copy’s TID as of its primary’s: twice the lookups in lazy VACUUM’s dead-TID store, and twice the TIDs in validate_index()’s sort under CREATE INDEX CONCURRENTLY, whose merge with the heap skips a TID it has passed, so the duplicate costs a sort entry and nothing else. Pages are never recycled; the centroids' REFINE items, which no CODE item names, are never removed.
  • A scan against VACUUM. A scan copies what it needs of a CODE item (the REFINE location among it) into its queue and holds no lock between rows; through read streams (§44.4, “Read streams”) it holds pins of the index pages they read ahead, which VACUUM, taking no cleanup lock, does not wait for, and never one of the heap’s. VACUUM may remove the REFINE item before the scan reads it, and an insert may reuse the slot: lionvec_refine_at() returns the item only if its TID is the row’s, and the scan treats a missing one as a row gone, which it is - VACUUM removes only rows dead to every snapshot. A reused slot with the same TID (the heap reused the TID too) holds a row inserted after the scan’s snapshot, which the heap fetch drops. A row the scan returns it has just found visible to its snapshot, so VACUUM cannot have removed it, and the executor’s own fetch finds it. A centroid’s REFINE item missing is corruption.
  • Serializable: ampredlocks = false, so a reader takes a predicate lock on the whole index (index_beginscan()), and an insert checks it.
  • old_snapshot_threshold (PostgreSQL 16 only; 17 removed it): with it set VACUUM may remove rows a snapshot older than the threshold still sees - their CODE and REFINE items with them - so “VACUUM removes only rows dead to every snapshot” above does not hold. Core’s access methods turn that into “snapshot too old” by testing every page they read (TestForOldSnapshot()); lionvec tests none, and the heap fetch it returns rows through (table_index_fetch_tuple(), 16’s heap_hot_search_buffer()) none either, so such a snapshot’s nearest neighbours would come back short without a word: on a 16 server at a threshold of one minute, a REPEATABLE READ transaction counted 2,000 rows through a lionvec scan, another deleted 500 of them and VACUUMed a minute and a half later (tuples: 500 removed), and the same scan in the same transaction counted 1,500, where a sequential scan raised “snapshot too old”. As lion is (§9), lionvec is not read while the threshold is set: lionveccostestimate() adds disable_cost to every order, so a sort of the rows wins; lion_ordered_set_rel_pathlist() raises no pg_lion.vector_postfilter = error for a filtered order, which no lionvec scan answers then; and what reaches one anyway, a plan with every other path disabled, is ERRCODE_FEATURE_NOT_SUPPORTED in lv_start(), at the scan’s first amgettuple (the snapshot is not known in ambeginscan) and before it reads a page, in each process of a parallel scan. Inserts, VACUUM and builds are not refused.
  • Format checks: every page’s type and id before anything on it is read; every CODE item’s length against its flags and dimsp, its flags against each other (a copy is neither NULL nor zero), and a copy anywhere but a partition’s chain of an index that spills; the meta page’s flags (none unknown, and spill only with two partitions or more); every REFINE item’s length, and a centroid’s alpha8 and code length finite; every PART item’s length, flags and blocks, its centroid’s length, its longest row and its spread finite and not negative and its alpha1 finite (a negative length makes a bound whose lower end lies past its upper, and the ranking of an insert’s candidates come up short, which is checked too), and the PART chain’s numbering against the meta page’s count; chain cycles (lionvec_chain_step()); the meta page’s fields. Each is ERRCODE_INDEX_CORRUPTED, never a crash. Every page’s header is within the bounds PageAddItemExtended() PANICs outside of (lionvec_check_page(), §3, 2026-10-18). And an insert, which holds the meta page from its first page to its last, refuses a block it is about to read that names the meta page - a chain end or refine_tail on the meta page, a PART item’s tail or centroid, a REGION item’s run, a chain’s next link - as it refuses a LOC tail that is also the LOCDIR tail and a partition’s tail that is its PART page, BEFORE the lock (lionvec_insert_readbuf()): a buffer content lock is not reentrant, and the second one waited for this very backend with interrupts held off, which only a restart of the server ended (test/sql/vector_corrupt_records.sql). Every REFINE item lies inside its page’s data, as a PART or REGION item must (lionvec_item_ok()): lionvec_refine_at() checks its line pointer’s offset against pd_upper and its end against pd_special after its length and before the item’s first prefetch or read, where an offset past the page - up to 32,767 - had a scan read the item from whatever memory follows it (the same file, 2026-10-27).

44.6 Filtered nearest neighbours (LionOrdered, lion_ordered.c)

§30’s node walks an ordered index path filtered by lion’s TID set. A lionvec index’s order by distance qualifies as an ordered path when it has one ORDER BY clause, no index clauses, a forward direction and is not hypothetical (lo_knn_path()), and is paired with each lion access of the relation as a B-tree’s is. The plan differs from §30’s in four places:

  • The set is built at once (LO_FLAG_BUILD), never lazily: the scan needs every member before its first row. The ORDER BY clause goes in a sixth custom_exprs list (LO_EXPR_ORDERBYS), and the executor builds its scan key with ExecIndexBuildScanKeys(isorderby = true), its run-time key (a Param query vector, a correlated subquery’s) among the node’s others, re-evaluated on rescan. EXPLAIN shows it as Order By.
  • The set is handed to the scan after every index_rescan() (lion_vec_restrict(): its container keys, ascending, and its containers), and with it the rows the query takes (lion_vec_scan_limit()): root->limit_tuples as the plan recorded it (LO_PRIV_LIMIT), the LIMIT of the node’s query level and its OFFSET, or -1 where the planner did not know them - no LIMIT, a generic plan’s Param, grouping, aggregates, DISTINCT or a window above it. The scan keeps no tier of its pause below them, and makes them L when they are more, to an eighth of work_mem’s rows (§44.4, “A known LIMIT”). They say where to pause and what to gather, not how many rows to return: a residual filter or a join that takes fewer rows than the scan returns gets more, from probing resumed, and REST. The scan counts the members and then does one of two things:
    • No more than pg_lion.vector_exact_rows (2,000) members, or than pg_lion.vector_exact_per_partition (1.5) a partition if that is more (lionvec_exact_limit()): it fetches every member from the heap, computes its distance with the operator’s function, sorts them (NaN last, then by TID) and returns them in order (lv_exact()), reading no CODE page. A member not visible, or whose vector is NULL, is left for the NULL walk, a cosine zero vector (NaN) for the zero walk. This is the exact answer, at a heap fetch a member; it replaces probing where the members are so thin over the partitions that most partitions hold none or one and the nearest member lies past many - in the simulation a 0.1% filter of 975,000 rows needed a fifth of the index probed for 0.95 recall.
    • More: it probes as an unfiltered scan does (§44.4), at the same margin, its patience counted in members. The k-th nearest member lies farther out, past more centroids within the margin, and the pause ends those probes. Before the pause counted members the margin was narrowed instead, by 1.3 standard deviations for each tenfold the members thinned the rows by (in the simulation the margin a recall needed fell about that much a tenfold) and then 1.8; with the pause, narrowing cost recall and saved no time - on the benchmark, through LionOrdered, narrowed by 1.8, 0.975, 0.974, 0.970 and 0.952 of the ten nearest under filters of 10%, 3%, 1% and 0.3%, and not narrowed 0.981, 0.980, 0.981 and 0.965, in the same time at each (27 ms at 1%). A probe skips a CODE page whose container keys hold no member, and every row not a member before it is estimated; a refine, a fetch and a return happen only for members.

    A set past hash_mem is in the node’s arena (§30.4, “Spilling”), a mapped temporary file, and the scan probes it there as it probes one in memory: the containers are the same, at addresses that do not move. (Until 2026-10-19 such a set had degraded to NULL containers, meaning every TID of that key’s blocks, which the scan took whole and the node rechecked.)

  • No fetch-and-sort switch: the pathkeys are a distance, not columns a sort could take over.
  • No values from the index: heap mode always.

Which path (lionvec_exact_limit(), 2026-10-16). The exact path costs a heap fetch and a distance a member, as many as the members whatever the index. Probing costs partitions and CODE pages: the k-th nearest member lies past more centroids the thinner the members, and a probe reads a partition’s CODE pages - skipping those whose container keys hold no member - for the members among them, so where they are about one a partition each probe reads a page or two for one member, and the pause, counted in members, comes late. What decides is the members a partition, which a count of members does not follow from one index to another. With a flat 2,000, a million rows' 1% filter (10,063 members in 7,999 partitions, 1.26 a partition) probed for 0.956 to 0.98 of the ten nearest in 13 to 20 ms, where every member’s distance took 11 ms for all ten; raised to the 12,000 that takes it, the count would have made DBpedia’s 10% filter exact too (11,377 members in 2,715 partitions, 4.2 a partition: 26 ms, against 8.5 to 13 ms probed for 0.956 to 0.994). Warm, the two crossed at two to three members a partition on both data sets; cold, the exact path is a random heap read a member, and on DBpedia’s 115,186 rows crossed at 1,200 to 1,800 members, under one a partition (§44.9). So the scan computes every member’s distance when the members are no more than pg_lion.vector_exact_rows (2,000: near where the cold costs crossed, whatever the partitions) or pg_lion.vector_exact_per_partition (1.5, below the warm crossing) times the partitions, whichever is more, and the cost (§44.7) asks the same of the same meta page, so that the planner prices the path the scan will take.

The node’s early stop - every member met - ends the scan without the probes that would find nothing. Since the restricted scan returns members only, a LIMIT is never left short by the filter.

Cost (lo_cost_knn()): the lion lookups and the set as lo_cost() prices them, plus lion_vec_scan_cost() for the members (§44.7) as start-up; then per member, at worst, a random page (none where the exact path read the row’s heap page before its first row, §44.7, “Exact path pricing”), an index tuple, a probe and a tuple, the recheck where the set leaves quals to it (its members are then the candidates, §40.8), the quals of a member the recheck passes, and the target. On 20,000 rows of 200 dimensions (format 1) the planner chose a sort of the lion bitmap scan’s rows for a filter keeping 1% of them, the node from 3% to 30%, and the plain lionvec scan with a Filter from 60%, where most rows pass and the set costs more than the rows it saves.

Filter at the fetch: tried and removed (2026-10-19). For a day (2026-10-18) LionOrdered had a second flavour over a lionvec order, for wide filters, where the set is most of the query (845 ms of 850 to 1,050 at 10% on the 50M-row benchmark, §44.7, “The set’s pieces”): it built no set and handed the scan the lion clauses instead, which the scan tested on the heap tuple of each row it fetched for its exact distance, dropping a row that failed; the planner offered both flavours and kept the cheaper. Measured by the maintainer at 50M rows, it was removed, and the set is the one way the node restricts a lionvec scan: - its results depended on the plan. Knowing a row’s membership only at its heap fetch, the filter’s scan counted its patience, its tiers and its refine threshold over every row, scaled by the planner’s estimate of the filter’s share, where the set’s counts members. The two agreed at pg_lion.vector_patience = 0 only: with the pause on, a 3% filter at a margin of 12 had a recall of 0.997 through the filter and 0.980 through the set, and a 1% filter at a margin of 6 0.967 and 0.943, so a query’s recall moved with the flavour its costs chose. - its tail grew sharply with the margin. Every row it fetched to test was a random read of a 130 GB heap: at a margin of 12 its slowest queries took 14 to 37 s, the set’s 1.4 s at most.

The invariant: for a query’s rows not to depend on its plan, the scan must know which rows are members where it walks, before it scores them - as the set tells it, by container key - and a heap row, read after the walk has counted it, cannot.

Post-filtering (2026-10-16). A lionvec scan under a Filter is another search than the restricted scan, not the same one with the rejected rows left out. The scan ends a pause, and moves the frontier behind which it skips a late row, on the rows it returns (lionvecgettuple(): pauseleft and passret count every row returned, the tiers step on passret, and lastdist is the last row’s distance); the executor applies the Filter after amgettuple, and nothing of it comes back to the AM. The restricted scan returns members only. So under a selective filter the plain scan’s pauses end by count on rows the Filter throws away and its probes resume, as if it hardly paused, while its frontier runs past members it has not found yet. On the 1M-row DBpedia set (200 queries, LIMIT 10), at 1%, a margin of 8 and the default patience of 30, the plain scan returned 0.993 of the true ten for 29,075 pages and 17.6 pauses a query, LionOrdered 0.980 for 3,483 pages and 0.98 pauses, the same rows for 85.5% of the queries; with the pause off and a margin of 3, where both probe the same partitions, the plain plan is the lower, 0.960 against 0.966, from the members it skipped behind a frontier that non-members had moved (21 a query against 0.1). Same settings, other rows: the plan decided the answer, which it must not.

So a lionvec order a Filter would follow is not a plan (pg_lion.vector_postfilter, §44.8, at error, its default since 2026-10-17, or exact). LionOrdered’s set_rel_pathlist_hook takes it out of the relation’s path list before core’s set_cheapest() (lo_knn_take()) whenever the relation has a clause the scan would leave to its Filter: a restriction clause that is not pseudoconstant (a gating Result tests those once), for a partial index not implied by its predicate - such an index, whose predicate implies every clause, stays the filter’s order - and not the index’s own (lo_knn_own(), 2026-10-17): a clause of the columns the index’s key reads alone, the vector’s (v IS NOT NULL, a distance bound v <-> q < 0.5), rejects the NULL vectors the scan returns last, or rows by the distance it returns them in, and so leaves the rows before them as they were; it filters the scan’s rows under every setting. Core’s sequential scan, which a cheaper order may have made add_path() discard, is offered again. The node still walks the order, and over a lion side only if that side answers every clause but the index’s own: the node’s own Filter would reject members the scan had returned just as the plain scan’s does. Core’s lion accesses are a bitmap scan’s, which leaves an index whose bitmap costs more than the rows it saves to the heap scan’s Filter (choose_bitmap_and(): tag < 50 beside cat IN (1, 2)), so when none of them answers every clause the node ANDs the access answering most of the clauses left, among core’s and each lion index’s alone, until one does (lo_knn_cover()), and the price decides as for any lion side. A filter lion indexes answer thus always goes through LionOrdered (unless pg_lion.enable_ordered_scan is off, or the set would not fit hash_mem); any other fails at plan time under error, EXPLAIN included, naming the clauses no lion index answers, and under exact is planned exactly, by a sort of the filter’s rows over a sequential, bitmap or B-tree scan (WHERE id = 5 ORDER BY v <-> q LIMIT 1 sorts the one row the primary key finds). The approximate answer is then asked for in so many words:

SELECT ... FROM (SELECT ..., embedding <=> $1 AS d FROM t [WHERE <clauses lion answers>]
                 ORDER BY d LIMIT 1000) s
 WHERE <the other clauses> ORDER BY d LIMIT 10;

The planner pushes no clause below a LIMIT, so the inner scan is unfiltered or restricted through LionOrdered, and how far it reads, 1,000 rows here, is the query’s to say. allow plans as before: the plain scan with a Filter and LionOrdered with one compete on cost, which is what the planner chose from 60% above and at 10% and 3% in §44.7 and §44.9. What the rule does not reach: a set with a lossy lion clause has its rows rechecked by the node, which filters members the scan returned; a join’s clause filters an ordered scan of one side (a nested loop over it) and is no restriction of that relation; and a distance bound few rows pass leaves a LIMIT unfilled, so the scan reads on to its end, as pgvector’s do.

Row-level security (2026-10-17). lionvec refuses it. A policy is a restriction no lion index answers, on a table the node does not scan (§30.1), so under error every lionvec query of a role a policy filters would fail, with a WHERE or without, and so would the subquery, whose scan the policy filters too. lionvec_refuse_rls() builds no lionvec index on a table with row-level security enabled, or on a partition of one, and fails at plan time, under every pg_lion.vector_postfilter and for the owner too, a query that a lionvec index of such a table could order; any other query of the table is planned as ever. A table that inherits from one with row-level security (INHERITS, not a partition) is not looked at: under error the parent’s policies fail a lionvec query of the child through the parent, as any filter LionOrdered cannot take does.

The default, error (2026-10-17). exact was the default for a day; from 2026-10-17 it is error, so that a query whose filter lion cannot answer says which answer it wants: the subquery’s approximate one, or exact’s sort, which reads every row that passes the filter. What the default fails, by lo_knn_take()’s rule unchanged, is every lionvec order under a restriction clause no lion index answers and no partial lionvec index’s predicate implies, however cheap the exact plan: a key’s id = 5 (or id = $1 in a generic plan); a volatile clause (random() < 0.1); a correlated outer column’s clause in a LATERAL subquery (note = q.note); a plain view’s WHERE; in a partition, the parent’s clause on the partition key, which every row of the partition passes (core keeps it as the partition’s Filter); and on PostgreSQL 16, IS NOT NULL on a NOT NULL column other than the vector, which 17 and later drop when planning. A partial lionvec index whose predicate implies one clause is no way round another that lion answers: LionOrdered’s set must answer every clause, the predicate’s too, so WHERE note = 'n1' AND tag < 50 over an index WHERE note = 'n1' and a lion index on tag fails, the error naming note = 'n1'. What it does not fail: no WHERE; a pseudoconstant one (WHERE $1, WHERE current_user = 'app'); a WHERE proven false; the vector column’s own clauses (v IS NOT NULL, a distance bound), alone, through the plain scan’s Filter, or beside a filter lion answers, through LionOrdered’s (2026-10-17: they failed for a few hours before); a filter lion indexes answer (through LionOrdered, which a generic plan’s cat = $1 and a LATERAL subquery’s cat = q.cat reach too); a join’s clause; a filter of a CTE or of a security-barrier view’s subquery (the order is then above it, a sort); and a lionvec order an HNSW index’s path has already beaten in add_path(), which is then the plan, with its Filter, as pgvector’s.

44.7 Costs (lionveccostestimate())

Start-up is the cost of the first row, which is most of a scan’s. A restriction the exact path takes (§44.6: lionvec_exact_limit() of the meta page’s partitions, as the scan asks it) costs a member an index tuple, a row’s estimate and the ORDER BY’s function, and the pages the scan will read, as core prices the sort of the lion bitmap scan’s rows it competes with (“Exact path pricing”, below; until 2026-10-18 a random page a member). Otherwise: every centroid’s estimate (LIONVEC_ROW_OPS, 2 cpu_operator_cost: the popcounts and a square root); the share of the rows a scan probes; a random page a probed partition and the CODE pages' share of the index (by the recorded dimensions, or 1536 before any) read sequentially; 0.1 cpu_index_tuple_cost a row walked and LIONVEC_ROW_OPS a member scored; and a batch’s refines and exact distances, a random page and an index tuple each. The share probed is LIONVEC_PROBE_PAUSED (0.46) C^(-1/3) and, under a filter keeping a share s, s^(-LIONVEC_PROBE_SEL_PAUSED) (0.27) times as much, at most all: the power of C the simulation’s, the rest the benchmark’s at the default margin and patience, where scans probed 3.9% of the 1,974 partitions unfiltered and 6.3%, 8.9%, 12% and 18% through LionOrdered under filters of 10%, 3%, 1% and 0.3% (the fit gives 3.6%, 6.8%, 9.3%, 12.5% and 17%). With pg_lion.vector_patience = 0 it is LIONVEC_PROBE_SHARE (0.92) C^(-1/3) and s^(-LIONVEC_PROBE_SEL) (0.2), the benchmark’s at a margin of 2 before the pause, when a filter narrowed the margin (§44.6); the same margin unnarrowed probes more. On DBpedia’s text-embedding-3-large rows (§44.9) the share fell from 678 to 2,715 partitions as C^(-1/3) says, within 25% of the model’s level under filters and at under half of it unfiltered, where a margin of 3 stops early on that data; the constants were left as they are. In an index that spills, by the meta page’s flags (§44.3), the CODE pages' share is of two CODE items a row to one REFINE item, a partition’s rows walked are twice its rows, and the share probed is LIONVEC_SPILL_PROBES (0.93) times as much: at the same margin such a scan probed 91 to 94% of the partitions a scan without did (§44.9). That is one data set, and the costs do not see what spill buys, the recall.

Each further row adds a refine and a fetch at worst and a share of the start-up over pg_lion.vector_batch: the farther rows lie past more partitions, and the planner prices a LIMIT under a Filter as LIMIT over the filter’s selectivity rows of the scan. Without pauses the share is LIONVEC_DEEPER (1), a scan of the benchmark’s index having probed 7.3% of the partitions for 10 rows, 23% for 100 and 61% for 1,000 at a margin of 2. With them it is LIONVEC_DEEPER_PAUSED (0.06): the same scans probed 4.1%, 4.6% and 50%, the first batch coming from the rows a scan holds when it pauses and the deeper ones much as without, which one share a batch cannot follow; 0.06 prices the first batches right. The charge is what keeps a plain scan, walking past every row a 1% filter rejects, from looking as cheap as LionOrdered reading the members alone: charged a refine and a fetch a row and no more, the two tied at 1% and the planner’s pick ran 7 times slower (202 ms to 30). Before the pause, with the full charge, the planner picked the plain scan at 30% and 10% (15 and 30 ms, LionOrdered 196 and 82) and LionOrdered at 3% and 1% (40 and 26 ms). With pauses it picks the plain scan down to 3% (9 ms at 30% against LionOrdered’s 206, 15 at 10% against 81, 32 at 3% against 41) and LionOrdered at 1% (24 ms against the plain scan’s 256, whose 1,000 rows 0.06 prices low); at 0.3% and below it sorts the bitmap scan’s rows (27 ms, LionOrdered 27). A path with no ORDER BY costs infinity. Selectivity 1, correlation 0.

Until 2026-10-18 (“Exact path pricing”, below) the exact path was priced at a random page a member, the price of its cold run, a random heap read a member (§44.9), and four to six times the probes' at the filters pg_lion.vector_exact_per_partition newly sends down it (3.6 and 5.7 times below). The plain scan with a Filter, priced low under a selective filter (LIMIT / selectivity rows at 0.06 of a batch’s start-up each, above; the million-row follow-up’s problem 1), then comes out cheapest: on DBpedia’s 115,186 rows in 2,715 partitions (§44.9), at 2% and 3% the planner’s estimate was 5,248 and 3,954 for the plain scan, 9,183 and 13,829 for LionOrdered’s exact path, 8,735 and 12,800 for the sort of the bitmap scan’s rows, where LionOrdered probing had been 2,552 and 2,426 - so its plan there went from LionOrdered probing (warm, 0.964 of the ten nearest in 8.7 and 9.0 ms at the default margin) to the plain scan (0.960 in 16.9 ms, 0.957 in 10.8 ms), where LionOrdered, forced, is exact in 4.2 and 7.8 ms and the sort exact in 2.9 and 4.6. The cost follows the scan, as it must for the planner to compare what will run; what is wrong is the plain scan’s price, which the follow-up’s problem 1 is to fix. Until then pg_lion.enable_plain_scan = off with enable_sort = off and pg_lion.pushdown_margin = 1000 takes LionOrdered, and pg_lion.enable_plain_scan = off alone the sort, both exact there (enable_indexscan = off disables LionOrdered with the plain scan, §39). 2026-10-16: under a restriction the plain scan is no longer a candidate (§44.6, “Post-filtering”), so the comparison of it with LionOrdered that this charge decides arises only under pg_lion.vector_postfilter = allow. On the 1M-row DBpedia set, at random_page_cost 1.1, it priced the plain scan under the 1% filter at 5,030 against LionOrdered’s 5,915 and the planner took it, 32 ms at a margin of 2 against LionOrdered’s 14 (FOLLOWUP.md’s Problem 1: a scan that a Filter un-pauses by count, 17.6 times a query there, pays about a whole start-up’s probes for each further hundred rows, measured at 115,000 rows, not 0.06 of one); there the filter is now LionOrdered’s or a sort’s with no constant changed. LIONVEC_DEEPER_PAUSED still prices an unfiltered scan’s deeper batches, and the plain scan under allow.

The set’s pieces (2026-10-18). Until then lo_cost_knn() priced the set as lo_cost() does for a B-tree’s walk (§30.3): the lion access’s own cost - a summary entry a bucket - and cpu_operator_cost a container key, nothing a member, so the 50M-row benchmark’s 10% set, 1,220 summary entries and 260,417 keys built in 845 ms, cost 650. Both now add the pieces the set gathers (lo_set_pieces()): each leaf’s source hands the set a container for each entry it reads at each container key that entry’s rows reach, so they cost

min(members, entries x keys) x LO_PIECE_TUPLES x cpu_tuple_cost

with entries from lion_cost_path_entries() (lion_amcost.c, summed over a BitmapAnd’s or BitmapOr’s leaves): for a key column’s range, lion_cost_ranges_entries()’s - a summary a bucket it covers whole, and the keys at its ends, or without summaries every key between its bounds; for an equality list longer than a batch (lion_scan_list_batch()), a LIST source, its elements; else one; a leaf’s most of any column. A 10% equality on a column the heap is ordered by brings a piece a key; a range over a unique column a piece a member. A piece was first LO_PROBE_TUPLES (2) cpu_tuple_costs, the walk’s 170 ns an entry, which the 50M build’s 5M pieces came to (845 ms); at 5M rows the same filter’s 500,000 members took 44 to 48 ms, 1.36 members a piece, 90 ns a member. Since the pieces are gathered by key (2026-10-19, §30.4) the 2M emulation’s take 105 ns each where they took 278, so a piece is LO_PIECE_TUPLES (1) cpu_tuple_cost, 85 ns on the walk’s scale: a little under the measurement, where a tie should fall, the set’s answer being the one that does not move with the plan. To be measured again at 50M rows.

Exact path pricing (2026-10-18). Under the default pg_lion.vector_postfilter the exact path’s one competitor is core’s sort of the lion bitmap scan’s rows under the LIMIT, and it was still priced against the plain scan, above: a random page, an index tuple and LIONVEC_ROW_OPS cpu_operator_cost a member, and in lo_cost_knn()’s run another random page a row returned. Core prices the sort’s input as cost_bitmap_heap_scan() does: the heap pages the rows lie on, each read once in block order - Mackert and Lohman’s 2Tn / (2T + n) of a table of T pages for n rows, rounded up and at most T (compute_bitmap_pages(), one loop) - each at random_page_cost less (random_page_cost - seq_page_cost) x sqrt(pages / T), a single page at random_page_cost; a tuple and the ORDER BY’s expression a row (pgvector’s distance at COST 100, §44.11: 0.25); the sort; and never the detoasting of a vector, its TOAST pages read one at a time through the TOAST index after the row’s heap page. With the vectors in TOAST the exact path reads those same TOAST pages by their places (§44.13), through a stream, and no heap page but those of the rows it returns, yet was charged the most: on 200,000 rows of 1536 dimensions in TOAST (1,728 heap pages) at random_page_cost 1.1, a 1% filter (1,944 members) was planned as the sort, at 1,852, over LionOrdered at 2,207, and ran in 37 ms warm against LionOrdered’s 8 (less the query vector’s 10 ms InitPlan). The exact path’s start-up for n members (lionvec_exact_cost()) is now

ORDER BY start-up + n x (cpu_index_tuple_cost + LIONVEC_ROW_OPS x cpu_operator_cost
                         + ORDER BY per row) + pages

the ORDER BY’s costs cost_qual_eval()’s of the path’s indexorderbys, which the sort’s input pays a row the same, and its pages as the scan will read them, which lionvec_reads_placed() tells the planner as it tells lv_start_toast() (a heap table, the index keeping a locator over the heap’s TOAST relation, pg_lion.vector_toast_direct on, a vector of at most LIONVEC_LOC_MAXRUNS chunks; the scan’s MVCC snapshot is the one thing the planner cannot ask, and under 16’s old_snapshot_threshold no lionvec order is read at all, §9). With P(n, T) the bitmap heap scan’s page price above (lionvec_pages_in_order()):

  • by place: P(n, L) + D x seq_page_cost, at the index’s tablespace’s costs, with L the LOC pages (the index’s rows over 291, a page’s one-run entries, LIONVEC_LOC_SIZE(1), rounded up) and D the LOCDIR’s (L over 407), and no page of the heap or of TOAST a member. Both plans read a member’s TOAST pages and core prices them for neither: charged to the exact path alone, even at seq_page_cost a member, they would keep the sort the plan at 0.3% to 3% below (476.61 + 286 = 762.61 against the sort’s 679.31, 675.19 + 966 = 1,641.19 against 1,186.02, 1,238.59 + 2,885 = 4,123.59 against 1,754.46), where LionOrdered runs 2.0 to 2.7 times faster warm. A row returned still costs a random heap page in lo_cost_knn()’s run, its check (lv_check_placed()).
  • from the heap (the vectors inline, or the locator not used): P(n, the heap’s pages), at the heap’s tablespace’s costs, the pages the bitmap heap scan reads for the same rows, and nothing more a row returned, whose page the scan read before its first (lion_vec_scan_cost()’s *fetch, 0 here, which lo_cost_knn() charges a row in place of the random page it charged every row).

P is compute_bitmap_pages()’s count and not index_pages_fetched()’s, whose re-reads past the table’s share of effective_cache_size are an index scan’s, its rows in the index’s order: the exact path reads its pages once each in block order, as the bitmap scan does. The probing branches are unchanged. On vecbench’s 100,000 rows of 1536 dimensions (a uniform u under a lion index; cosine distance at COST 100; random_page_cost 4; a lionvec index of 2,530 partitions, so an exact limit of 3,795 members; in TOAST 834 heap pages, 100,000 TOAST pages and 344 LOC pages, inline 100,000 heap pages), LIMIT 10, the Limit’s estimate before and after, the plan picked, and the two plans' times - the scans themselves unchanged - in ms, medians of 5 warm and of 3 cold (the server restarted, the OS’s page cache kept):

vectors filter members LionOrdered before after sort plan before after warm, node / sort cold, node / sort
TOAST 0.3% 286 1,217.87 476.61 679.31 sort LionOrdered 1.03 / 2.05 5.48 / 7.07
TOAST 1% 966 3,952.69 675.19 1,186.02 sort LionOrdered 2.43 / 5.11 12.87 / 19.38
TOAST 3% 2,885 11,712.34 1,238.59 1,754.46 sort LionOrdered 5.62 / 15.38 27.32 / 45.07
TOAST 10% (probing) 9,990 2,139.91 2,139.91 3,975.46 LionOrdered LionOrdered 10.82 / 49.42 37.12 / 116.44
inline 0.3% 310 1,298.50 1,284.22 1,288.27 LionOrdered sort, then LionOrdered leaned 0.65 / 0.58 3.00 / 2.63
inline 1% 1,034 4,228.78 4,114.14 4,133.83 sort sort, then LionOrdered leaned 1.75 / 1.83 8.34 / 11.19
inline 3% 3,020 12,267.29 11,266.11 11,332.36 sort sort, then LionOrdered leaned 3.98 / 5.37 23.96 / 24.83
inline 10% (probing) 10,143 2,157.84 2,157.84 32,814.98 LionOrdered LionOrdered 13.00 / 17.83 52.30 / 90.16

In TOAST the exact path is now the plan wherever it is taken, 2.0 to 2.7 times faster than the sort warm and 1.3 to 1.65 times cold. Inline the two plans read the same heap pages and are priced within 1% of each other, near enough for the planner’s 1% fuzz to decide (the sort, as before at 1% and 3%, and at 0.3%, running in 0.88 to 1.35 times the node’s time) until the lean below. At random_page_cost 1.1 the node is the plan at all six (in TOAST 305.83, 628.79 and 1,163.19 against the sort’s 346.50, 914.17 and 1,708.06; inline 429.48, 1,415.34 and 4,072.35 against 433.53, 1,435.04 and 4,138.60). The benchmark’s vectors are one vector repeated (its generator’s InitPlan): the exact path and the sort compute every member’s distance whatever they are, so their rows stand; the probing rows show only that their prices did not move. On vector_toast.sql’s 3,000 rows of 1536 dimensions with a kilobyte of text each (§44.10), a 1% filter (30 members) costs the node 66.65 against the sort’s 105.94 in TOAST, where it cost 167.15 (0.15 ms against 0.16), and 125.71 against 123.55 inline, where it cost 167.21: the sort is the plan there and runs in 0.06 ms against the node’s 0.16, a tenth of a millisecond of the node’s own start that no price follows. vector_filter.sql’s 1,200 members on some 850 of vfb’s 1,482 inline pages cost the node 1,781.98 against the sort’s 1,807.57, where they cost 4,761.08, the two running alike.

Leaned (2026-10-19). The exact path’s start-up is then divided by LIONVEC_EXACT_LEAN, 1.2: it is planned wherever its price is within 1.2 times the sort’s, the maintainer’s preference for the exact path at the margins over a sort that runs about as fast - no sort, no hash_mem, the rows in order from the first - where the near-ties above went to the sort. On the 100,000 rows above the node is now the plan at all six exact-path cases: in TOAST 406.33, 575.83 and 1,056.07 against the sort’s 679.31, 1,186.02 and 1,754.46; inline 1,072.75, 3,435.52 and 9,407.94 against 1,288.27, 4,133.83 and 11,332.36, where it runs in 1.12, 0.96 and 0.74 times the sort’s time warm. On vector_toast.sql’s 3,000 rows the inline node costs 104.76 against the sort’s 123.55 and is the plan, 0.16 ms against the sort’s 0.06, the tenth of a millisecond above; vfb’s 1,200 members cost the node 1,484.98 against the sort’s 1,807.57. The probing branches are not leaned: the sort’s competitor there is priced on its own terms (above), and the exact path alone was the near-tie.

A parallel scan (2026-10-16; §44.4). cost_index() asks lionveccostestimate() for the index’s cost before it counts the path’s workers, and of the result divides only its own CPU part by the parallel divisor, so a partial path came out at about the serial scan’s price. lion_ordered_set_rel_pathlist(), the set_rel_pathlist_hook, which runs before core makes its Gather paths, takes each partial lionvec path out of the relation’s partial paths and offers it again (lo_knn_partial(), so that add_partial_path() orders and prunes them by the cost they now have) at lion_vec_parallel_cost()’s: the ranking of every centroid, nparts x LIONVEC_ROW_OPS x cpu_operator_cost, which every participant pays, in full; the rest of the start-up and every row after it over the divisor - costsize.c’s get_parallel_divisor(), which is static there and so copied (lionvec_parallel_divisor()): the workers, and while the leader takes part 1 less 0.3 a worker, none from four workers on. On DBpedia’s 2,717 partitions at random_page_cost 1.1 the serial start-up is 584.79, of which the ranking is 2,717 x 2 x 0.0025 = 13.585: with two workers (a divisor of 2.4) a partial path starts at 13.585 + 571.205 / 2.4 = 251.59, with three (3.1) at 197.84, as EXPLAIN shows. Gather Merge adds parallel_setup_cost (1,000) and parallel_tuple_cost (0.1) a row, so at the default costs no parallel scan of this index is chosen, its setup alone costing more than the serial scan’s start-up - and rightly, the parallel scans having run 4 to 8 times slower (§44.9). The workers are core’s, compute_parallel_worker()’s: from *indexPages, which is the CODE pages a scan reads, against min_parallel_index_scan_size (512 kB), and from the heap pages the rows' fetches would read, against min_parallel_table_scan_size (8 MB) - DBpedia’s heap is 768 pages, its vectors in TOAST, and gets none without lowering it - unless the table’s parallel_workers says how many, and at most max_parallel_workers_per_gather. Under pg_lion.vector_postfilter exact or error lo_knn_take() removes a partial path that a Filter would follow as it removes the serial one (§44.6, “Post-filtering”: a Gather Merge over a Parallel Index Scan with a Filter is the same post-filter), with nothing to bring back for it, a partial sequential scan having no pathkeys it could have lost to; LionOrdered’s paths are serial.

44.8 Settings

  • pg_lion.vector_margin (real, -10 to 1000, default 3): how much nearer than its centroid a partition’s rows are taken to come, in standard deviations of the rows about their centroids along one dimension (§44.4). Larger probes more partitions and misses fewer rows; 100 probes, on any data a test has used, every partition that could hold a nearer row, and the regression tests set it so, with the pause off, to compare with a sort. Below zero probes a partition only once rows nearer than its centroid by that much are in hand. A filter no longer narrows it (§44.6). The default was 2 until the pause; with the pause, 3 took the benchmark’s recall from 0.964 to 0.979 for 1.3 ms (§44.9).
  • pg_lion.vector_patience (int, 0 to 1,000,000, default 30): how many units of rows a scan scores without one entering the nearest rows it is after (its tier, §44.4) before it pauses, a unit sqrt(n) / 2 rows, n the build’s, the rows an average partition would hold in 2 sqrt(n) partitions - under a lion filter twice that share of the members - whatever the partitions' number, and margin / 6 times as many past a margin of 6; 0 never pauses, and then a filter’s scan probes far more than the costs assume (§44.7). Smaller pauses sooner, probing fewer partitions and missing more rows. 30 is where the 975,000-row benchmark’s tail ended, in its 2 sqrt(n) partitions, where a unit is a partition’s rows: at a margin of 3, 30 and 40 gave the same recall (0.979), 40 a 95th percentile a third longer.
  • pg_lion.vector_exact_rows (int, 0 to 100,000,000, default 2,000): how many members a lion filter may admit for the scan to compute every member’s distance from the heap, exactly, instead of probing (§44.6), whatever the partitions; 0 leaves the choice to pg_lion.vector_exact_per_partition. The exact path costs a heap fetch a member - at 1536 dimensions a 6 KB vector, a page of its own in TOAST or inline, so 2,000 members are 12 MB - where a probe reads a few hundred to a thousand CODE pages (§44.9) but, under a filter that leaves most partitions empty, many for nothing: in the simulation a 0.1% filter of 975,000 rows probed a fifth of the index for 0.95. It is the floor under the ratio below, which matters in an index of fewer than 1,334 partitions, where 1.5 a partition is under 2,000: a probe’s fixed costs (every centroid ranked, a thousand pages or more however thin the filter) are most of a scan there, and on DBpedia’s 115,186 rows the probes took 7.4 ms for 0.953 of a 0.3% filter’s ten nearest (345 members, 0.13 a partition, 1,584 pages), the exact path 0.5 ms for all ten (§44.9). The default sat where the two cost about the same on the 975,000-row benchmark, and stays where they cost about the same cold, whatever the partitions: on DBpedia, a random heap read a member at 50 to 60 us each, 1,204 members took 75 ms exact against 99 ms probed for 0.98, and 3,472 183 against 67. Raise it where storage is fast or exactness is wanted, lower it where the vectors are wide and the disk slow. The planner prices both (§44.7), so it may sort the filter’s rows itself instead, as it did at 1% on DBpedia (§44.9), which is exact too; since 2026-10-18 the exact path is priced as that sort is (§44.7, “Exact path pricing”), and with DBpedia’s vectors in TOAST, read by place, it should be the plan there (not measured).
  • pg_lion.vector_exact_per_partition (real, 0 to 1,000,000, default 1.5): the members a lion filter may admit for each of the index’s partitions, on average, for the scan to compute every member’s distance instead of probing, however many members that is; the larger of it times the partitions and pg_lion.vector_exact_rows is the bound (lionvec_exact_limit(), which the scan and its cost both ask). 0 leaves the choice to pg_lion.vector_exact_rows. What decides between the two paths is how thinly the members spread (§44.6): at one or two a partition a probe finds about one member for the CODE pages it reads, and the pause, counted in members, comes late, while the exact path’s heap fetches are as many as the members whatever the partitions. A flat count misfires across sizes: 10,063 members at a million rows, 1.26 a partition of 7,999, are faster exact; 11,377 at 115,186 rows, 4.2 a partition of 2,715, are faster probed. 1.5 sits below where the two crossed warm on both data sets, at two to three a partition (§44.9), for the exact path’s cost cold, a random heap read a member where the probes read CODE chains a partition at a time: at 1.28 a partition on DBpedia (its 3% filter) the exact path was 1.3 times faster warm than the probes for 0.98, and 2.7 times slower cold. It takes up to 12,000 members in a million rows' 7,999 partitions (1% is 10,063) and 4,072 in the 115,186 rows' 2,715 (3% is 3,472), and leaves 3% of the million and 5% of the 115,186 to the probes. Raise it where the heap is in memory and exact answers are worth a few milliseconds more; lower it, or set it to 0, where cold reads dominate.
  • pg_lion.vector_overflow (enum, error or stop, default error): what a scan does when what it knows of its rows outgrows hash_mem (§44.4, “The hash’s bound”): fail the query, or end the scan at the rows it returned, with a NOTICE. Only a scan with no LIMIT, or a large one, over rows plane 1 cannot tell apart gets there.
  • pg_lion.vector_postfilter (enum, error, exact or allow, default error; 2026-10-16, the default exact until 2026-10-17): whether a plan may apply a filter to the rows a lionvec scan returns (§44.6, “Post-filtering”). error: no plan does; a filter lion indexes answer goes through LionOrdered, over a set that answers every clause, and a query whose lionvec order would be filtered and that no LionOrdered path answers fails at plan time, EXPLAIN too, naming the clauses no lion index answers, and hinting at the subquery that filters an unfiltered or lion-filtered scan’s first rows and at exact and allow. exact: the same, but such a query is planned exactly, a sort of the filter’s rows. allow: the plain scan with a Filter, and LionOrdered with one, are plans chosen by cost, as before; for comparing plans and for §44.9’s earlier measurements. pg_lion.enable_ordered_scan = off takes LionOrdered away, not the rule: under error or exact a filtered query then gets the error, or an exact plan. A pseudoconstant clause, tested once by a gating Result, one a partial lionvec index’s predicate implies, and the index’s own, of the vector column alone (v IS NOT NULL, a distance bound; 2026-10-17), are no filter here. A table with row-level security lionvec refuses under every value (§44.6, “Row-level security”).
  • pg_lion.vector_build_sample (enum, error or allow, default error; 2026-10-19): what a build does when its sample has rows for fewer partitions than the table’s rows want - fewer than a sample of every row would make (lionvec_parts_short(), lion_vec_math.h). The build chooses its partitions from a reservoir of what half of maintenance_work_mem holds, and makes no more than the reservoir has rows for: 16 a partition it chooses by itself, one a partition the partitions option asks for (§44.3). Were that all, the partitions would follow a memory setting, unseen: at the default 64 MB a table of more than 28,985 rows of 1536 dimensions would get 1,361 partitions whatever its size - a billion rows, which want 252,982, 734,000 rows each - and a partitions of 30,000 over 40,000 rows 21,788, with only the DEBUG1 line to say so; and the partitions decide which rows a scan reads at a margin (§44.4), so the answers would follow it too. error: the build fails in its first pass, at the first row with which the rows counted so far want more partitions than the full reservoir has rows for. With the reservoir’s size fixed, a count of rows that is short stays short as the rows grow - the partitions made stop at what the reservoir allows, and those wanted only grow - which the unit tests check over every count to 200,000 and every 0.05% from there to 4 billion, so the table’s rows would be short too and the error is never wrong; and it comes before the rest of the first pass, k-means and the second pass, the build’s other 98% (§44.12): at 64 MB a billion rows of 1536 dimensions fail at the 28,986th. A parallel build’s first pass is the leader’s alone, so it fails before its parallel context is set up. The message names the partitions wanted, those the sample has rows for, and the maintenance_work_mem whose reservoir would make them all - the rows lionvec_parts_sample() says, at lionvec_sample_mem(), rounded up to a megabyte; a kilobyte less than the unrounded figure is short (vector_build_sample.sql). For a pass that stops before the table’s end those are for the rows it reckons the table holds (lionvec_build_rows_about(): the rows counted times the heap’s blocks over those read, which it reads in order from the first, “About” in the message), so that the setting named is for the table and not for the rows read: on 40,000 rows of 1536 dimensions at 64 MB the build failed in 1.0 s at the 28,986th row, reckoning 39,954 and naming 76 MB, where building the 1,361 partitions took 10.2 s and the 1,600 at 76 MB 11.7 s (75 MB failed, its sample having rows for 1,595). A table whose rows are not spread evenly over its blocks may be reckoned low, and then fails again at the setting named, later in the pass and reckoned better. allow: the partitions the sample has rows for, with a NOTICE that counts the rows and names the same maintenance_work_mem. Under neither does a build fail for what no memory changes: a table of fewer than 10,000 rows has one partition however few of them the sample holds, and a partitions above the table’s rows wants as many as the rows, and is held to those alone. Nor for the reservoir’s cap: 224 rows have 16 each for 220 partitions, four times LIONVEC_MAX_PARTS (a static assertion), so past 1.07 billion rows the build wants 262,144, the format’s ceiling, whatever the memory.
  • pg_lion.vector_prefetch (enum, off, fadvise or stream, default stream from 17 and fadvise on 16; 2026-10-17, a boolean from 2026-10-16, whose on, the default, and off it still takes): how a scan reads ahead the pages it will read next (§44.4, “Prefetch” and “Read streams”), as many ahead as the tablespace’s effective_io_concurrency (a query’s setting; maintenance_io_concurrency is not read), to 64; an effective_io_concurrency of 0 reads none ahead whatever it says. stream reads the index’s pages, and the exact path’s heap pages, through read streams into shared buffers; fadvise asks the kernel for them (PrefetchBuffer()), as on did; off reads each when it comes. On 16, which has no read streams, stream is fadvise, and SHOW says so. The rows' heap pages for their exact distances are asked of the kernel under fadvise and stream alike. The rows a scan returns, and their order, are the same under all three. On DBpedia (§44.9, “Read streams”), each query alone with its pages on the disk, under 18’s io_method = worker, stream took a margin-8 scan to 43.2 ms against fadvise’s 47.6 and off’s 67.2, a 1% filter’s probing to 62.0 against 82.8 and 100.9, and the exact path under it to 53.3 against 58.9 and 108.3; a margin-3 scan, 346 pages, about even. With its pages in the kernel’s cache, though, a stream’s read is an I/O worker’s or, under io_uring, a kernel thread’s, where ReadBuffer() copies the page itself: stream cost 0 to 40% more than fadvise on the index’s pages under worker and io_uring and 2 to 14% under sync, and on the exact path half the time under worker’s default 3 workers (1.17 times under 16) and one and a half to twice under io_uring. Warm, every page in shared buffers, stream costs 2 to 11% over off and fadvise within 5%. So stream where the index is read from the disk; fadvise where it lives in the kernel’s cache and not in shared buffers, under io_uring above all; off where it lives in shared buffers.
  • pg_lion.vector_toast_direct (boolean, default on; 2026-10-17): whether a scan of an index that keeps a locator reads vectors in TOAST by their chunks' places (§44.13) - the exact path’s members through the locator, the rerank’s rows through their REFINE items - every chunk checked, rather than through the heap tuple and the TOAST index. The rows returned, their order and their distances are the same either way, which the tests compare; inserts keep the places up whatever it says. Off is the way back should a place ever be wrong in a way the checks miss.
  • pg_lion.vector_parallel_scan (boolean, default off, 2026-10-17): whether a lionvec order gets partial paths at all (§44.4, “A parallel scan”); off because the parallel scan was slower than the serial one on every query measured.
  • core’s max_parallel_workers_per_gather, parallel_setup_cost, parallel_tuple_cost, min_parallel_index_scan_size, min_parallel_table_scan_size and the table’s parallel_workers: with pg_lion.vector_parallel_scan on, whether an unfiltered scan runs in parallel, and in how many processes (§44.7, “A parallel scan”). At the defaults it does not: the setup costs more than a serial scan’s start-up.
  • partitions (reloption, 0 to 262,144, default 0; 1,000,000 until a review found the build at a million partitions days long and uncancellable, then 65,536 until the regions, §44.3): the partitions the build makes; 0 chooses by the rows' count, 8 sqrt(n) above 10,000 rows, at most a sixteenth of a sample of every row (§44.3; 2 sqrt(n) until DBpedia, §44.9); a build whose sample, what maintenance_work_mem holds, has too few rows for either fails (pg_lion.vector_build_sample, above). 262,144 (LIONVEC_MAX_PARTS, 218) is what 8 sqrt(n) gives at 1.07 billion rows. A partition number is an int everywhere, never 16 bits, and the sort’s group (LIONVEC_SORT_GROUP, an int4) holds it with room for the two groups after the partitions. The build’s arrays of the partitions (lionvec_build_arrays()) are outside maintenance_work_mem, in the parallel context’s shared memory for a parallel build: the centroids' floats, C dimsp x 4 bytes, the 8-bit codes and plane 1 in C (dimsp + dimsp / 8) bytes, a few words a partition more, the sample’s assignment, 4 bytes a row, and the regions' arrays: 2.1 GB at 262,144 partitions of 1536 dimensions, 1.6 GB of it the floats (403 MB at 65,536); the leader’s region k-means works in another C dimsp bytes of codes, 403 MB there, and an insert in its candidates alone (§44.3). Read by the build alone, so a change takes a REINDEX. It does not move the pause: pg_lion.vector_patience counts the rows of 2 sqrt(n) partitions, not of these (§44.4). When it counted these, on DBpedia a patience of 30 in 2,715 partitions held the unfiltered recall to 0.966 at any margin, where 120, the rows of 30 in 678 partitions, reached 0.993.
  • spill (reloption, boolean, default on): whether the build puts each row in a second partition as well (§44.3), recorded in the meta page, which inserts then follow; with fewer than two partitions it does nothing. Read by the build alone, so a change takes a REINDEX. Off saves the copies' CODE pages - an index of 1536 dimensions a ninth smaller - and an insert’s third WAL record, for less recall at each margin: on 115,000 embeddings, 0.84 of the ten nearest at the default against 0.92 (§44.9).
  • pg_lion.vector_epsilon (real, 1 to 10, default 4): the bounds' width in standard deviations. Each plane-1 bound fails with the Gaussian’s two-sided probability, 0.27% at 3, 0.006% at 4; a failure matters only for a row that belongs in the answer and is found too late, so the rate of queries with a miss is far lower. Wider bounds refine and fetch more rows. The default was 3 until the format-1 benchmark (§44.9): there one query in 100 under the 0.1% filter lost a row at 3, in each of three runs, and none of 400 at 4, which refines twice the rows for a few percent of the time.
  • pg_lion.vector_batch (int, 1 to 100000, default 100): rows the probes gather candidates for before REST; each REST gathers four times as many, probing every partition probed so far again. It no longer sets where a LIMIT 10 pauses (the tiers do, §44.4), and a LIMIT past it that LionOrdered knows is L in its place, to an eighth of work_mem’s rows (§44.4, “A known LIMIT”), so it need not follow the LIMIT: it is L for the plain scan, a LIMIT the planner does not know, and one below it.
  • maintenance_work_mem: half of it holds the build’s sample (§44.3), and the CODE items' sort, two items a row when the index spills; a parallel build divides the sort’s share among its processes, as core’s builds do, and holds the sample once, in shared memory (§44.12). A sample row takes dimsp + 4 bytes, 1,540 at 1536 dimensions, so the default partitions want 2 x 1,540 x 16 8 sqrt(n) bytes, 376 MB at a million rows and 12.5 GB at a billion, a build given less failing (pg_lion.vector_build_sample, above), and k-means is given at most 128 a partition of it, never fewer than 65,536 rows (§44.3), 1.6 GB of sample at a million rows' 8,000 partitions (so 3.2 GB is the most a million rows can use); the reservoir stops at 224 rows, 25.8 GB, which 52 GB reaches. A parallel build’s shared memory holds the sample after the cut, and the arrays above.
  • max_parallel_maintenance_workers (core’s, default 2) and the table’s parallel_workers: the workers a build asks for, whatever the heap’s size or maintenance_work_mem (§44.12).
  • work_mem bounds a scan’s row queue (§44.4).

44.9 Measured

Data. Qdrant’s dbpedia-openai-1M-1536-angular set: 975,000 OpenAI ada-002 embeddings of DBpedia entities, 1536 dimensions, unit length, in items(id int, u float8, embedding vector(1536)), and the first 100 of its test queries; u is uniform in [0, 1) (seed 42) and unrelated to the vectors, so u < s keeps a share s of the rows, with a lion index on it. The ground truth is the exact top 10 by cosine, computed in numpy for each query and filter. Against pgvector 0.8.6’s HNSW on embedding::halfvec(1536) (m = 16, ef_construction = 64, as pgvector’s documentation builds it for 1536 dimensions) - lionvec indexes the vector itself. PostgreSQL 16, 4 vCPUs (Xeon, 2.8 GHz; AVX-512 but no VPOPCNTDQ, so the POPCNT kernel), 16 GB, shared_buffers 5 GB, work_mem 256 MB, maintenance_work_mem 1 GB, no parallel query, and pgvector’s three vector distance functions at COST 100, as lionvec’s documentation asks (§44.11). bench/vector/knn.py; the results are in bench/results/2026-10-14-lionvec-partitions/.

Size and build. lionvec 1,752 MB (225 MB of it the 28,810 CODE pages, 0.5 MB the PART chain), HNSW 3,809 MB, the table 63 MB and its TOAST 7.6 GB. The default 1,974 partitions, k-means over a sample of 348,617 rows (half of maintenance_work_mem): REINDEX took 18 min 58 s with the table in the page cache, where the unpartitioned format took 98 s. Nearly all of it is the O(n C) of §44.3 - ten k-means iterations over the sample and the assignment of every row, each a plane-1 estimate against every centroid and the exact distances of the few it cannot rule out.

Warm - every index, the table and its TOAST in memory (HNSW and the table in shared buffers, the rest in the page cache, pg_prewarm; each method’s queries run once untimed first); the median of 100 queries, LIMIT 10, and the 95th percentile. Recall is the share of the true ten returned. The plans are the planner’s own: the plain lionvec scan with a Filter at 10%, LionOrdered at 1%, and at 0.1% a sort of the lion bitmap scan’s 953 rows, for either index. lionvec at its defaults (margin 3, patience 30), at margins 2 and 4, and with the pause off:

Filter lionvec, defaults margin 2 margin 4 no pause HNSW, ef_search 40 100
none 7.9 ms (p95 16), 0.979 6.6 (15), 0.964 9.1 (18), 0.983 14 (68), 0.984 4.7 (6.5), 0.900 8.1 (11), 0.954
10% 13.5 (29), 0.978 15 (29), 0.974 15 (28), 0.979 60 (123), 0.996 8.7 (14), 0.931 8.0 (15), 0.939
1% 24 (40), 0.981 27 (41), 0.981 27 (38), 0.981 109 (136), 1.000 37 (62), 0.947 41 (66), 0.950
0.1% 9.9 (11), 1.000 13 (18), 0.999

HNSW under a filter is its iterative scan (hnsw.iterative_scan = relaxed_order); without it, at ef_search 40, it returned 3.9 of the ten rows at 10% (recall 0.393) and 0.4 at 1% (0.038). Times vary between runs: HNSW at ef_search 100 took 6.2, 7.0 and 8.1 ms unfiltered in three, the defaults 8.6, 8.4 and 7.9.

What the scans did, averaged over the 100 queries. Unfiltered, at margins 2, 3 and 4: 65, 77 and 86 partitions probed (3.3%, 3.9% and 4.3% of 1,974), 993, 1,171 and 1,295 CODE pages read, 0.5 to 0.8 pauses, about 120 rows refined and 10.5 fetched; with the pause off, 254 partitions and 3,856 pages. Under the 10% filter, through the plain scan, 99 to 104 partitions, 955 rows refined and 104 fetched (the filter keeps one row in ten of those returned); with the pause off, 725 partitions. Under the 1% filter, through LionOrdered, 244 partitions at every margin - every query paused, before the margin would have stopped it - 65 refined and 10.5 fetched; with the pause off, 1,511 of the 1,974, why a filter’s margin was narrowed before the pause (§44.6). No bound failed; 0 to 0.6 rows a query were skipped out of order.

The pause is what ends the tail. The partitions a query probed unfiltered: with it, a median of 68, a 95th percentile of 175 and at most 256; without it, 150, 837 and 1,664, a query’s time following them (correlation 0.84, at a margin of 2 before the pause). For that it gives up half a point of recall unfiltered (0.984 to 0.979) and two under the 1% filter (1.000 to 0.981). A patience of 40 gave the same recall as 30 at a margin of 3 (0.979) with a 95th percentile a third longer (17.9 ms against 13.5, in an earlier run). By LIMIT, unfiltered over 40 queries: 4.1%, 4.6% and 50% of the partitions probed for 10, 100 and 1,000 rows, recall 0.980, 0.975 and 0.983, in 10.5, 17 and 212 ms; with the pause off, 15%, 39% and 74%, 0.985, 0.997 and 0.999, in 17, 60 and 219 ms.

Before the pause, at a margin of 2 narrowed by 1.3 a tenfold under a filter (before-pause/), lionvec took 8.5 ms (p95 37) for 0.968 unfiltered, 32 ms (96) for 0.989 at 10% and 31 ms (59) for 0.979 at 1%; margins 1 and 3 took 5.3 ms for 0.939 and 14 ms for 0.984 unfiltered. The unpartitioned format, which read every CODE page of the rows it scored and returned the exact ten (no bound failed in 400 queries at ε = 4), took 96 ms unfiltered, 138 ms at 10%, 34 ms at 1% and 6.6 ms at 0.1% (reading only the 965 CODE pages that held a member, by a page directory this format no longer has). So partitions are 12 times faster unfiltered, 10 times at 10% and 1.4 times at 1%, and slower at 0.1%, where the planner now sorts the members' rows fetched from the heap.

Cold - PostgreSQL restarted and the page cache dropped before each query; the median of 5:

Filter lionvec HNSW, ef_search 40 (iterative under a filter) unpartitioned
none 85 ms (49 to 155) 118 ms 402 ms
1% 196 ms 1,254 ms 196 ms
0.1% 210 ms (planner: Sort of the filter’s rows) 251 ms (the same) 112 ms

Unfiltered, a cold query reads the PART chain and the CODE chains of the partitions it probes, about 9 MB, where the unpartitioned format read all 218 MB; its time varies with the partitions a query needs. Cold reads are noisy here: the first run of these five queries, straight after the warm benchmark, took 779 ms unfiltered (one query 3.6 s) and HNSW’s 1% 1,739 ms, and the next 76 ms and 1,244 ms. Interleaved, each query after its own restart, twelve queries took a median of 89 ms at the defaults and 173 ms at the old ones (a margin of 2, no pause), reading 28 and 67 MB. Under the 1% filter it reads lion’s set and the probed partitions' CODE chains, where HNSW’s iterative search fetches row after row that fails the filter. At 0.1% the sort fetches 953 TOASTed vectors.

Memory-capped - a stream of queries with PostgreSQL in a memory cgroup of 1 or 2 GB (its processes, shared_buffers of 128 MB, and the page cache its reads fill), which holds lionvec’s CODE pages but not HNSW’s 3.8 GB. Each method after a restart with the page cache dropped, run with rows of the table as queries until the cgroup is full and for 100 more, then timed on the 100 test queries; the median, and the pages a query read from disk (the backend’s read_bytes):

Memory Filter lionvec HNSW, ef_search 40 (iterative under a filter) unpartitioned
1 GB none 19 ms, 100 pages 51 ms, 767 pages 157 ms, 100 pages
1 GB 1% 28 ms, 0 pages 691 ms, 9,229 pages 31 ms, 0 pages
2 GB none 13 ms, 41 pages 41 ms, 672 pages 161 ms, 25 pages
2 GB 1% 30 ms, 0 pages 465 ms, 7,853 pages 29 ms, 0 pages

What lionvec reads from disk are the CODE chains of partitions the stream had not yet probed, and the REFINE items and vectors of the rows it refines and fetches; under the 1% filter the stream never filled the cgroup (5,000 rows of the table as queries, and the members' working set still fit). The pause cut the 95th percentile by a third here: 41 ms with 1 GB, 65 ms before it at a margin of 2. The unpartitioned format read as few pages but scored every row, and most of its time was most likely the CODE pages being copied into 128 MB of shared buffers on every pass, which a scan of 4% of them mostly escapes. HNSW reads 650 to 800 random pages a query unfiltered, and 8,000 to 9,000 under the filter, of graph and of rows that fail it.

Spill (format 3, §44.3). The benchmark above is format 2’s, which did not spill; spilling, its index would have some 28,800 CODE pages more, about 1,980 MB. Measured since on a smaller set: the first three of the 26 parquet shards of DBpedia entities embedded by OpenAI’s text-embedding-3-large at 1536 dimensions (text-embedding-3-large-1536-embedding), 115,386 rows, 200 of them held out at random (seed 0) as the queries, the ground truth their exact ten nearest by cosine among the other 115,186. One cosine index in the default 678 partitions, built with spill on and then off; PostgreSQL 16, 4 vCPUs (Xeon, 2.1 GHz, the POPCNT kernel), shared_buffers 2 GB, everything in memory, no parallel query; LIMIT 10, the median of the 200 queries, and the DEBUG1 line’s partitions probed and pages read, averaged. Queries all but never paused: a patience of 0 gave the same figures at margins 0 to 4.

Spill No spill
Build 175 s 157 s
Size 235 MB, 6,912 CODE pages 210 MB, 3,624 CODE pages
Margin Spill: recall, partitions, pages, ms No spill: recall, partitions, pages, ms
0 0.793, 4.5, 53, 2.0 0.710, 4.9, 30, 2.1
1 0.835, 6.4, 76, 2.0 0.757, 7.0, 42, 1.9
2 0.886, 9.4, 110, 2.4 0.808, 10.2, 60, 2.2
3 (default) 0.919, 14.2, 164, 2.7 0.843, 15.3, 88, 2.1
4 0.946, 20.6, 237, 3.0 0.881, 21.9, 124, 2.3
5 0.962, 30.1, 342, 3.7 0.912, 32.0, 180, 3.0
6 0.975, 41.6, 468, 3.9 0.934, 45.4, 252, 3.2
8 0.963, 81.2, 444, 4.2

At each margin spill found 0.065 to 0.083 more of the ten nearest, probing 91 to 94% as many partitions (LIONVEC_SPILL_PROBES, §44.7) and reading 1.7 to 1.9 times the CODE pages, for up to 0.7 ms more. At each recall it probed less than half the partitions and read fewer pages: 0.919 at a margin of 3 from 14 partitions and 164 pages in 2.7 ms, where without spill a margin of 5 gave 0.912 from 32 partitions and 180 pages in 3.0 ms; 0.962 at a margin of 5 from 30 partitions and 342 pages in 3.7 ms, where without it a margin of 8 gave 0.963 from 81 and 444 in 4.2 ms. A spilling scan skipped 122 copies a query at a margin of 0 and 553 at 3, and fewer rows out of order (0.38 a query at 3, against 0.65): a row near its partition’s edge comes in time from the partition beyond. Recall at each margin is lower here than on the benchmark’s set (0.84 without spill at the default, against 0.98): other embeddings, and an eighth of the rows. What this measures is the one setting against the other; the benchmark’s million rows spilling are still to be measured.

Regions (format 4, §44.3; 2026-10-16). The same 115,386 rows, every one of them indexed (so the default is 2,717 partitions, in 53 regions), the same 200 query rows each kept out of its own answer; cosine, spilling; PostgreSQL 18 (an assert build), 4 cores shared with other work, maintenance_work_mem 2 GB, the table cached; recall of the true ten, the median and 95th percentile of the 200 queries' times, after a pass to warm. Main, the build before the change, ran on one cluster, the rest on a second of the same build and settings, where the change with the regions and the cut switched off for the measurement (“every centroid”) built main’s index again: the same recall, 0.913 and 0.9865, in 287.9 s against 291.6.

2,717 partitions Main Regions
Build, serial 291.6 s 108.2 s
Build, 3 workers and the leader 84.9 s 35.9 s
Size 247.1 MB 246.9 MB
Margin 3: recall, median, p95 0.913, 3.2 ms, 6.2 ms 0.919, 3.3 ms, 6.5 ms
Margin 8 0.9865, 6.6 ms, 10.7 ms 0.9845, 7.1 ms, 11.4 ms

The parallel build’s index answered as the serial one’s (the same recall at both margins). In 679 partitions (27 regions), where the cut to 128 a partition keeps 86,912 of the rows, and in 10,000 (100 regions, no cut):

Every centroid, every row Regions, and the cut
679: build serial, 3 workers 108.0 s, - 62.7 s, 23.2 s
679: margin 3, margin 8 0.909, 0.984 0.913, 0.980
10,000: build serial, 3 workers -, 248.7 s -, 65.0 s
10,000: margin 3, margin 8 0.9285, 0.992 0.932, 0.989

The recall moves by as much from one draw of the sample to another: without regions, at 679 partitions, three draws of the cut to 128 gave 0.916, 0.920 and 0.915 at a margin of 3 and 0.981, 0.982 and 0.985 at 8, and the regions' k-means makes other centroids as another draw does. The assignment’s MATCH, the share of the rows whose partition (and, spilling, whose copy’s partition) the routed second pass chose as exact assignment over the same final centroids would, measures the routing alone. With one beam for both passes, in 53 regions:

Beam Partition Copy’s partition Margin 3 Margin 8
2 94.8% 67.9% 0.8965 0.974
3 97.1% 78.2% 0.908 0.980
4 98.3% 84.3% 0.9115 0.983
6 99.1% 90.5% 0.911 0.984
8 99.5% 93.7% 0.9165 0.9825
exact 100% 100% 0.913 0.9865

A spilling row’s copy wants more: SOAR picks it from the nine nearest, which lie farther out. With k-means' beam at 5, the second pass’s at 10, 13, 16 and 20 matched 99.78, 99.89, 99.94 and 99.97% of the partitions and 96.2, 97.8, 98.6 and 99.2% of the copies' in 53 regions (recall 0.9205 to 0.919 at 3, 0.985 to 0.9845 at 8: k-means' centroids decide it); 13 and 16 matched 96.95 and 97.9% of the copies' in 100 regions, and 7 and 10 93.1 and 96.6% in 27. Sixteen is the least that held 97% at all three. With k-means routed at 5 and the second pass exact, the recall was 0.9195 and 0.9845, and with k-means exact and the second pass at 5, 0.9095 and 0.984: neither pass’s routing moves it past the draws' spread. K-means' own beam, the second pass’s at 16: 3 and 4 gave 0.914 and 0.9145 at a margin of 3 and 0.9815 at 8, half a hundredth under exact assignment’s 0.9865, and 5 gave 0.919 and 0.9845 (4 gave 0.914 and 0.9825 in 679 partitions, against 0.909 and 0.984), so 5 is the least within 0.003 of it at both margins. The regions' own k-means: 2, 5 and 10 iterations matched 98.6, 98.8 and 98.9% of the partitions at a beam of 5, and 86.8, 88.1 and 88.4% of the copies' partitions, for 0.9145, 0.916 and 0.9135 at a margin of 3 and 0.981, 0.985 and 0.982 at 8; the regions held 16 to 141 centroids of 2,717 (6 to 58 of 679, 4 to 287 of 10,000), none empty. Build times with the match measured include an exact assignment of every row besides, and are not given.

Simulation. Before the partitions were written, a numpy model of the scan chose their number and the margin (vtoy’s ivf.py, not in the tree): the benchmark’s 975,000 vectors, k-means at 256, 1,024 and 4,096 partitions, the first 100 test queries, partitions taken nearest first by their centroids less a margin, and the rows read until the ten nearest (under each filter) were in hand. The rows of the partitions read, for 0.95 recall:

Partitions none 10% 1% 0.1%
256 50,800 (5.2%) 51,700 90,200 272,000
1,024 27,800 (2.9%) 31,900 67,300 207,000
4,096 16,600 (1.7%) 20,300 46,500 160,000

The unfiltered share falls about as C^(-1/3) (0.26 C^(-1/3) is 4.1%, 2.6% and 1.6%), which is LIONVEC_PROBE_SHARE; a filter keeping a share s reads about s^(-0.3) times as many, since its nearest matching rows lie farther out. Ordering partitions by centroid less a margin scaled by each partition’s own radius (its largest or its rms distance to its centroid) read as many rows or more at every recall, which is why the margin is one number. Spilling each row to its second-nearest partition as well (SOAR’s spill, chosen by the residual’s angle) read 17% fewer rows at 4,096 partitions for twice the CODE items; SPANN’s (a second partition within 10% or 25% of the nearest’s distance) read more. The margin a recall needed, in cosine units at 1,024 partitions: 0.010 for 0.95, 0.025 for 0.98, 0.035 for 0.99 unfiltered, about 0.016 less for each tenfold a filter thins the rows, and about the same at 256 and 4,096 partitions; the rows' standard deviation about their centroids along one dimension, σ, was 0.0122, so a margin of 2σ, the default before the pause, is about 0.98 recall.

Growth. How the rows a scan reads grow with the table, in the same model (vtoy’s scale.py): random subsets of 60,938 and 243,750 of the 975,000 rows and all of them, k-means at 2 sqrt(n) partitions and at one per 500 rows, the 100 queries' true ten within each subset, and the margin that reaches each recall chosen afresh at each size. The rows read, the mean over the queries, and in brackets their median and 95th percentile:

Rows Partitions 0.95 recall 0.98 recall
60,938 493 (2 sqrt(n)) 2,146 (1,262; 6,495) 5,907
243,750 987 6,403 (3,518; 18,285) 16,157
975,000 1,974 27,086 (13,122; 104,816) 73,589
60,938 121 (n / 500) 5,252 (3,480; 15,156) 10,322
243,750 487 8,584 (4,588; 30,776) 23,240
975,000 1,950 23,085 (10,934; 74,955) 56,781

At 2 sqrt(n) the partitions grow as the rows' root and the rows read grow almost as the rows (n0.91 at 0.95 recall): the probed share hardly falls. At 500 rows a partition they grow as n0.53 (0.95) and n0.61 (0.98), the partitions probed as n0.53 too (10, 17 and 46 at 0.95). Taken that far, a billion rows in two million partitions would read about 900,000 rows a query for 0.95 and four million for 0.98 - 230 MB and 1 GB of CODE pages at the benchmark index’s 242 bytes a row, from some 1,900 and 8,000 partitions, against 5.6 and 14 MB at 975,000 rows; three sizes over a sixteenfold range do not pin a power down a thousandfold further, so this is a direction, not a forecast.

The mean is the tail’s: the median query reads less than half the mean and the 95th percentile three to four times it, as in the benchmark. Capping the partitions probed cuts the tail for little recall. At 975,000 rows in 1,950 partitions, with the margin for 0.98: no cap, 0.981 and a mean of 56,781 rows (95th percentile 226,626); at most 10% of the partitions, 0.979 and 40,729 (102,971); at most 5%, 0.971 and 28,612 (55,643) - more recall than the margin for 0.95 with no cap (about 0.956, 23,085 and 74,955) at a 95th percentile a quarter lower. At the smaller sizes a cap of 5% or 10% did the same.

A pause does better than a cap, and is what the scan does (§44.4). In the same model (vtoy’s patience100.py), the scan stopping once P partitions in a row have put nothing into its 100 nearest: at 975,000 rows in 1,950 partitions, with the margin for 0.98, P = 20 read a mean of 28,062 rows (95th percentile 74,986) and P = 40 38,740 (100,481), against 56,781 (226,626) with neither, the cap of 5%’s rows at 0.98 recall rather than 0.971; at 243,750 rows in 487 partitions, P = 20 read 18,142 (48,213) against 23,240 (68,741). Between those two sizes the rows read for 0.98 grew as n0.32 with P = 20 and n0.64 without - two sizes, so a direction only, but the one that matters at a billion rows (§44.11).

DBpedia. The method again on a public set of another model’s embeddings (bench/vector/knn_dbpedia.py, bench/results/2026-10-14-lionvec-dbpedia/): shards 0-2 of Qdrant’s dbpedia-entities-openai3-text-embedding-3-large-1536-1M, 115,386 text-embedding-3-large rows of 1536 dimensions, 200 of them held out as queries (the numpy model’s default_rng(0)), the true ten by cosine among the other 115,186, filters keeping 11,377, 3,472 and 1,204 rows (10%, 3%, 1%). PostgreSQL 16, pgvector 0.8.7, 4 vCPUs, every relation in 2 GB of shared buffers, maintenance_work_mem 1 GB (the sample is every row); the server’s median over 200 warm queries. lionvec is 210 MB in 678 partitions (2 sqrt(n)) and 221 MB in 2,715 (8 sqrt(n)), built in 159 s and 347 s; HNSW 450 MB, 187 s.

The margin does not carry over. At the defaults lionvec returned 0.843 of the true ten unfiltered, in 1.1 ms, probing 15 of 678 partitions (88 CODE pages), where ada-002’s got 0.979; 0.95 took a margin of about 7 (2.4 ms) and 0.98 one of 11 (3.2 ms). HNSW took 1.9 ms for 0.951 at ef_search 40 and 4.1 ms for 0.979 at 100. The margin a recall needs, in the rows' spread about their centroids, is the embeddings' own.

In 2,715 partitions, at the same margin and patience, a scan probed 2.5 times the partitions and read three quarters of the CODE pages (37 and 68 at a margin of 3) for 0.835, in 1.4 ms. Along the margins, 0.95 took 234 CODE pages against 358 in 678 partitions - 1.5 times fewer, and about 5,700 rows scored against 11,000, 1.9 times fewer (the numpy model’s 1.78) - but 2.9 ms against 2.4: each query ranks four times the centroids, and a partition of 42 rows spans two 35-row pages, so a probe reads 1.75 pages for 1.2 pages of rows. At a patience of 30 the finer partitions never reached 0.98 (0.966 at margins 12 and 16), as the patience then counted partitions' worth of rows, a quarter as many there; at 120, the rows of 30 in 678 partitions, they did, in 566 pages against 578 and 4.5 ms against 3.2. Counted since in the rows of 2 sqrt(n) partitions (§44.4), a patience of 30 on the same index gave the 120’s recall in every row, and its partitions and pages to within 0.07 a query: 0.98 in 566 pages, 327 partitions and 4.1 ms. At this size the finer partitions buy pages and rows, not time; the default changed for the rows a scan reads at a billion (§44.11), where the centroids need an index of their own anyway. Under 10% the planner took the plain scan with a Filter at both sizes (0.879 and 0.872 at a margin of 3, 5 to 8 ms); under 3% LionOrdered in 678 partitions (0.917, 5.0 ms) but a sort of the lion bitmap scan’s 3,472 rows in 2,715 (exact, 20 ms, where LionOrdered took 5.8 ms for 0.908), its estimate 2,034 against LionOrdered’s 2,213, which charges a random page for each partition probed, a number that grows as C2/3; under 1% a sort of the 1,204 rows (5.3 ms) and LionOrdered’s exact path (5.6 ms) alike. HNSW’s iterative scan took 10.7 ms at 3% (0.953) and 26.7 ms at 1% (0.959); the planner sorted for it there too.

Both (lionvec-p0-spill.json: the defaults since, 2,715 partitions spilling, format 3, a patience of 30 in the rows of 2 sqrt(n) partitions; 356 s, 247 MB). At a margin of 3 the scan returned 0.917 of the ten nearest from 35 partitions and 114 CODE pages in 1.6 ms, where the old default gave 0.843 from 15 and 88 in 1.1 ms; 0.976 at 6 (111 partitions, 351 pages, 3.2 ms), 0.986 at 8 and 0.993 at 12 and 16, the finer partitions' ceiling. Along the margins, 0.95 took 203 CODE pages, 63 partitions and 2.2 ms against the old default’s 358, 65 and 2.4 and the finer partitions alone’s 310, 177 and 2.7; 0.98 464 pages, 148 partitions and 4.1 ms against 578, 106 and 3.2 and 566, 327 and 4.1. So the two together read 1.8 times fewer pages at 0.95 and 1.25 times fewer at 0.98 than the old default (about 7,100 and 16,200 items walked against 11,000 and 18,000 rows), in the same time at 0.95 and 1.3 times longer at 0.98: the ranking of four times the centroids is a fixed cost each query, and a partition’s 84 items span 2.6 pages. The numpy model’s 2.7 and 2.5 times fewer rows counted rows, not pages, and no copies. Under the filters, at 0.95 and 0.98, the plain scan at 10% read 455 and 791 pages (7.0 and 8.9 ms) against 589 and none reached (0.969) in 678 partitions; LionOrdered at 3% 671 and 1,155 (6.0 and 8.3 ms) against 771 and 1,210 (5.6 and 7.1). The planner’s plans are the finer partitions' (above): the plain scan with a Filter at 10% (0.948 at a margin of 3), a sort of the lion bitmap scan’s rows at 3% (exact, 15 to 21 ms, where LionOrdered took 7.2 ms for 0.967) and at 1%. HNSW: 0.951 in 1.9 ms at ef_search 40, 0.979 in 4.1 ms at 100.

Cold (cold.json; the server restarted and the page cache dropped before each query, the median of 7, the index above and HNSW’s). Unfiltered, lionvec at a margin of 3 (0.917) read 478 pages in 22 ms, at 7 (0.95) 1,314 in 50 ms and at 12 (0.993) 1,378 in 61 ms; HNSW at ef_search 40 (0.951) 926 in 50 ms and at 100 (0.979) 1,918 in 106 ms. Under the 3% filter LionOrdered at a margin of 3 (0.967) read 1,607 pages in 64 ms, where HNSW’s iterative scan (0.953) read 5,365 in 293 ms and the planner’s sort of the bitmap scan’s rows (exact, for either index) 5,620 in 209 to 218 ms, every matching row’s vector from TOAST; under 1% lionvec’s exact path read 2,721 in 90 ms, HNSW’s iterative scan (0.959) 11,870 in 539 ms. So cold, lionvec reads what HNSW reads unfiltered and three to four times less under a filter; on the million-row set it was 85 ms against 118 unfiltered and 196 against 1,254 at 1% (above), not re-measured with spill.

The share probed at a margin of 3, in 678 and 2,715 partitions: 2.2% and 1.4% unfiltered (§44.7’s 0.46 C^(-1/3): 5.2% and 3.3%), 8.1% and 4.8% through the plain scan at 10% (9.7% and 6.1%), 14.9% and 10.3% through LionOrdered at 3% (13.5% and 8.5%). Between the sizes it fell by 1.4 to 1.7 times, the model’s C^(1/3) 1.59; unfiltered it is under half the model’s level because a margin of 3 stops this data’s scans early, at 0.84 recall. So the constants stay.

Kernels (2026-10-15; the index above, on the same machine, whose Xeon has AVX-512 with VPOPCNTDQ, so the AVX-512 set, §44.3). Timed by section with rdtsc (its own cost taken off), a warm query at a margin of 3 spent 35% of its time in the arithmetic before them: plane 1’s estimates of the 2,715 centroids and 2,914 rows 15%, the 8-bit sums of 123 centroids and 88 rows 18% (1.8 us a call), the rotation and the query’s planes 4%; the rest was the walk of 36 partitions' CODE pages around those estimates (28%), the PART chain’s ranking (13%), the 11 heap fetches with their exact distances (6%), the executor (7%) and the scan’s queues. At a margin of 7 the arithmetic was 32% (the walk 47%); under LionOrdered at 3% 19% at either margin, most of it the 8-bit sums of 800 and 1,600 centroids, the executor’s filter a quarter at a margin of 3. The build spent 59% of k-means in the candidates (plane 1 against every centroid), 24% in their exact distances and 16% in the query’s planes. Per call at 1536 dimensions, the library in a harness (gcc -O2, the median of six), ns:

call before POPCNT AVX2 AVX-512
lionvec_estimate1() (plane 1) 60 58 38 27
lionvec_estimate8() (the 8-bit sum) 1,170 1,055 386 AVX2’s
lionvec_l2sq 1,100 955 270 AVX2’s
lionvec_rotate(), us 35.0 34.1 6.2 AVX2’s

“Before” is the POPCNT kernel and the plain sums (its l2sq, a static function, timed as a copy); the portable set is POPCNT’s with plane 1 counted without the instruction (164 ns); the query’s planes took 46 us before and take 12 with any set. End to end, the median execution time of the 200 queries, the median of eight runs alternating the builds, before and after: unfiltered 1.52 -> 1.18 ms at a margin of 3 (0.917) and 3.73 -> 2.98 at 7 (0.980); the plain scan at 10% 6.46 -> 5.29 and 9.38 -> 7.31; LionOrdered at 10% 7.74 -> 7.12 and 11.04 -> 10.01, at 3% 7.03 -> 6.21 (0.967) and 11.97 -> 10.54 (0.994); the exact path at 1% and the planner’s sorts as before. Recall, partitions, candidates and pages were the same at every setting. The arithmetic is now 21% of the query at a margin of 3 and 10 to 11% under LionOrdered at 3%: in a scan a centroid’s or row’s 8-bit estimate takes 0.6 to 1.3 us, against 0.39 in cache, waiting on its codes' 24 cache lines - prefetching them took a third off a sum out of cache in a harness (1,083 -> 675 ns), not done - and plane 1’s estimates are 7% inside a walk of 42 to 66%. The build took 148 s against 348 (the medians of three and two), the same index.

The exact path or the probes, by members a partition (2026-10-16; knn_dbpedia.py exact with KNN_EXACT_SELS and KNN_EXACT_MARGINS, on main’s build at 1c85107). DBpedia’s 115,186 rows as above, the vectors inline (KNN_INLINE: a row a heap page), the index of 2,715 partitions spilling, every relation in 2 GB of shared buffers; 200 queries, LIMIT 10, through LionOrdered (enable_sort off and the lionvec index’s own scan out of the way, as knn_dbpedia.py’s lo settings put it - §39, “Core’s switches”), its exact path with pg_lion.vector_exact_rows past every filter, its probes with it at 0 and the default patience. The server’s median time, the median of three runs' (two other builds shared the 4 cores), and the 95th percentile; recall and pages, the same in every run. Probed at the default margin, at the margin nearest 0.98 of the ten nearest and the one nearest 0.99:

Filter Members A partition Exact path Probed, margin 3 … about 0.98 … about 0.99
0.3% 345 0.13 0.5 ms (p95 0.7) 0.9530, 1,584 pages, 7.4 ms (12.4) 0.9825 (12), 2,597, 10.9 (18.0) 0.9890 (16), 3,219, 13.7 (21.1)
1% 1,204 0.44 2.0 (2.7) 0.9610, 1,172, 8.6 (13.2) 0.9795 (8), 1,511, 10.0 (15.5) 0.9925 (16), 2,644, 15.2 (23.6)
2% 2,288 0.84 4.2 (5.9) 0.9640, 955, 8.2 (12.1) 0.9795 (6), 1,100, 9.3 (14.3) 0.9940 (12), 1,918, 12.9 (19.3)
3% 3,472 1.28 7.8 (9.4) 0.9640, 790, 9.1 (14.0) 0.9770 (4), 944, 9.9 (13.9) 0.9895 (8), 1,319, 12.4 (18.4)
5% 5,760 2.12 8.9 (13.1) 0.9610, 621, 7.5 (11.5) 0.9750 (4), 803, 8.9 (12.9) 0.9915 (8), 1,258, 11.0 (16.8)
7% 8,001 2.95 19.2 (24.4) 0.9575, 521, 8.9 (12.6) 0.9765 (4), 691, 10.8 (15.9) 0.9880 (6), 926, 11.1 (16.7)
10% 11,377 4.19 26.4 (33.4) 0.9560, 433, 8.5 (12.6) 0.9740 (4), 593, 10.2 (14.7) 0.9870 (6), 848, 11.5 (16.3)

The exact path’s time follows the members, 1.5 to 2.4 us each (a heap page in shared buffers and a distance); the probes' hardly moves with the filter, 9 to 11 ms for 0.98, because the thinner the members the more partitions they probe and pages they read for them - 1,584 pages at 0.3% against 433 at 10% for about 0.955. The two crossed warm at about two members a partition (5%: 8.9 ms exact against 8.9 for 0.975), and at three (7%) the probes took 10.8 ms to the exact path’s 19.2. The planner’s own exact plan, a sort of the lion bitmap scan’s rows by distance, took 0.4, 1.5, 2.9, 4.6, 6.9, 17.5 and 24.7 ms: the same rows, read a page at a time by the bitmap heap scan, without LionOrdered’s set and a fetch a member through the index’s scan.

Cold - the server restarted and the page cache dropped before each query, the median of seven, the pages read from the kernel (EXPLAIN (ANALYZE, BUFFERS)): the exact path read a heap page a member, in 23 ms at 0.3% (346 pages), 75 ms at 1% (1,209), 183 ms at 3% (3,492) and 692 ms at 10% (11,402, 89 MB), 50 to 60 us a page; the probes, at the margins nearest 0.98 above, 99 ms at 1% (1,907 pages, margin 8), 67 ms at 3% (1,724, margin 4) and 71 ms at 10% (1,361, margin 4). Cold the two crossed at 1,200 to 1,800 members, whatever the partitions.

At a million rows (the maintainer’s measurement, 2026-10-15/16, bench/results/followup-1m/ FOLLOWUP.md, problem 4: 999,800 rows of the same embeddings in 7,999 partitions, vectors inline, PostgreSQL 18 on a Ryzen 7 5700X3D): under 1%, 10,063 members, 1.26 a partition, a sort of the lion index scan’s rows - every member’s distance, what the exact path computes - took about 11 ms warm for all ten, where LionOrdered probed for 0.956 in about 13 ms and 0.98 in 19.6; under 3%, about 30,000 members, 3.75 a partition, LionOrdered took 21 and 28 ms for 0.95 and 0.98, where the exact path, at the 1% filter’s rate a member, would take about 33. Cold, LionOrdered took 331 ms at 1% (a margin of 3); the exact path’s 10,063 random heap reads (80 MB) were not measured, and their cost was accepted.

Hence pg_lion.vector_exact_per_partition = 1.5 (§44.8): below the warm crossing of two to three a partition at both sizes, so that 3% of the 115,186 rows (1.28) and 1% of the million (1.26) are exact - for all ten, 1.3 and 1.8 times faster warm than the probes for 0.98, at the cold price above - and 5% of the 115,186 (2.12) and 3% of the million (3.75) are probed. And pg_lion.vector_exact_rows stays 2,000, near the cold crossing, for an index whose partitions are too few for the ratio to say. Through the planner it does not show yet at 2% and 3% here, where the exact path’s price sends it to the plain scan with a Filter (§44.7).

Prefetch and parallel scans (2026-10-16; §44.4, “Prefetch” and “A parallel scan”). DBpedia’s 115,386 rows as above in dbp(id, v vector(1536)), the cosine index of 2,717 partitions spilling (format 4, 247 MB; its heap 768 pages, the vectors in TOAST); the 200 queries are rows of the table, each kept out of its own answer by WHERE id <> q under pg_lion.vector_postfilter = allow, a Filter over the scan, the truth the numpy ten nearest of the others. PostgreSQL 18.6 built with assertions, pgvector 0.8.7, 4 vCPUs, 15 GB, shared_buffers 3 GB, work_mem 64 MB, effective_io_concurrency 200 (so 64 pages ahead), io_method worker, random_page_cost 1.1, max_parallel_workers 4; the disk a virtio device, about 55 us a page read cold. LIMIT 10; participants 1 (serial), 2 (a worker and the leader) and 4 (three and the leader), the parallel plans forced by parallel_setup_cost, parallel_tuple_cost, min_parallel_index_scan_size and min_parallel_table_scan_size at 0. Warm: pg_prewarm of the index and the table, a first pass under EXPLAIN (ANALYZE, BUFFERS) (the pages, shared hit and read, workers' included), a second timed by the client, the median and 95th percentile of the 200, with the scans' DEBUG1 lines on (the partitions probed, CODE pages read, rows returned and fetched and pages prefetched, summed over a query’s participants):

Margin Participants Recall Prefetch off, ms (p95) on Partitions CODE pages Pages Returned, fetched Prefetched
3 1 0.919 2.18 (4.37) 2.39 (4.89) 35.7 116 519 11.0, 11.8 325
3 2 0.935 11.07 (15.12) 11.62 (17.03) 58.3 188 1,072 17.3, 24.3 666
3 4 0.9515 17.09 (25.00) 16.21 (22.01) 131.7 418 2,756 36.8, 54.2 1,712
8 1 0.9845 4.94 (7.70) 5.20 (8.56) 163.4 510 1,291 11.0, 11.5 880
8 2 0.994 20.12 (30.52) 21.52 (36.48) 373.4 1,141 3,110 18.2, 23.3 2,143
8 4 0.996 24.67 (35.46) 27.28 (40.08) 790.6 2,365 8,299 37.9, 48.6 5,738

The counts are prefetch off’s; on, the serial scans' are the same to the row, and the parallel ones' within 3% (0.9535 and 0.9955 of the ten nearest at four), which participant claims a partition first changing from run to run. None of the pages prefetched warm was missing from shared buffers. The serial scan at other margins, warm, for the recall a parallel scan returns:

Margin 4 5 6 10 12 16
Recall 0.9425 0.958 0.973 0.990 0.992 0.9945
Partitions 52.4 74.6 98.2 231.4 289.0 382.2
CODE pages 169 239 311 717 891 1,171
Pages 623 757 900 1,681 2,003 2,505
ms 2.76 2.58 3.61 6.25 7.02 8.26

Interpolated to the recall of each parallel scan at a margin of 3, the serial scan probes 47.1 partitions for two participants' 0.935 and 65.3 for four’s 0.9515 (margins of about 3.7 and 4.6), so the parallel scans probe 1.24 and 2.0 times as many and read as many times the CODE pages, and 1.8 and 3.9 times the pages in all, each participant reading the PART chain and its centroids' REFINE pages for itself; at a margin of 8 two participants' 0.994 is 1.03 times the serial scan’s partitions at about 15 (1.29 times its pages), and four’s 0.996 lies past the serial scan’s best measured, 0.9945 at 16 from 382 partitions to their 791. In time the parallel scans lose everywhere: 4.3 and 6.4 times the serial scan’s at the same recall at a margin of 3, 2.5 times two participants' at 8. Each participant ranks all 2,717 centroids and probes to its own stop, Gather Merge waits for the slowest one’s first row, and the workers are started for each query, about 10 ms before their first row (§44.4).

The planner, at its default costs with max_parallel_workers_per_gather 3: the serial scan, 584.79..599.57 for the LIMIT, also with min_parallel_table_scan_size 0 (without which the 768-page heap gets no workers); with parallel_setup_cost 0 as well, a Gather Merge of two workers, 251.61, and of three with the tuple cost and min_parallel_index_scan_size 0 too, 197.88 (§44.7’s arithmetic). So at the default costs it never picks the parallel scan, which is what the times say.

Cold, one pass over the 200 queries after a restart with the page cache dropped (the server’s time under EXPLAIN (ANALYZE, BUFFERS) with track_io_timing, its mean, and the pages read from the kernel): nearly every query after the first few found the PART chain and most centroids in shared buffers, so this is a warm scan’s time plus its own partitions' reads. Serially at a margin of 3, 115 pages read, 8.54 ms against 7.27 with prefetch (its wait 6.4 ms against 4.4); at 8, 124 pages, 9.78 against 9.42 (5.23 against 4.41). In parallel, 151 to 204 pages, 20.2 against 18.8 (two at 3), 20.2 against 20.0 (four at 3), 30.0 against 27.7 and 32.2 against 31.6 (two and four at 8). Each query alone cold, the server restarted and the page cache dropped before each (12 queries, prefetch off and then on for one query and on and then off for the next, paired; the server’s mean, the wait on reads by track_io_timing, and the paired difference with its standard error):

Each query alone, cold Pages read Prefetch off, ms on Wait, off on Off less on On faster
serial, margin 3 410 21.2 17.4 17.7 11.4 3.8 +- 1.2 9 of 12
serial, margin 8 972 49.4 32.4 41.9 20.6 17.0 +- 4.1 10 of 12
two participants, margin 3 591 37.2 28.3 28.4 15.5 8.8 +- 2.0 11 of 12
LionOrdered, 10% (below) 922 44.5 37.0 33.3 20.6 7.5 +- 2.2 10 of 12

Seven queries each, unpaired, gave the rest: four participants at a margin of 3, 41.5 ms either way; two and four at 8, 74.0 against 69.1 and 86.1 against 74.6. Cold too the parallel scans are slower than the serial one, reading 1.4 to 3.1 times its pages from the kernel.

Under filters, through LionOrdered, on a copy of the rows (dbpf: the 115,186 other than the queries, u uniform in [0, 1) by default_rng(42), a lion index on it, its own lionvec index of 2,715 partitions, 247 MB; the truth by numpy among each filter’s rows), serially, at the default margin. At 10% (11,377 rows) the planner’s plan is LionOrdered, probing: 0.954 of the ten nearest, 141.6 partitions, 447 CODE pages, warm 9.58 ms against 8.80 with prefetch (732 pages asked for); each query alone cold, 44.5 against 37.0 (above). At 3% and 1% (3,472 and 1,204 rows) the planner sorts the lion bitmap scan’s rows, exact, 27.5 and 9.0 ms warm; LionOrdered, forced, takes the exact path, 27.1 and 8.4 ms warm, and asks for nothing, the vectors being in TOAST - its cold times, 263 against 256 ms at 3% and 97 against 111 at 1%, are the noise of seven cold queries. The same copy with its vectors in the heap (dbpi, STORAGE PLAIN, a row a page, 900 MB and its own index), each query alone cold, ten queries paired as above:

LionOrdered Pages read Prefetch off, ms on Wait on reads, off on
exact, 1% 1,213 70.6 33.5 64.8 11.0
exact, 3% 3,493 208.3 103.0 194.9 36.9
probed, 10% 844 44.9 36.1 30.8 17.7
unfiltered (index scan) 344 15.1 12.2 12.6 7.2

The same rows either way. The exact path asks for each member’s heap page 64 members ahead, and its wait on reads fell by five sixths, its time by half: the rest is the reads' own CPU and the fadvise calls, one a page. Warm there it took 2.62 against 2.80 ms at 1%, 8.20 against 8.89 at 3%, 11.52 against 10.89 probing at 10% and 1.28 against 1.28 unfiltered (the server’s medians). Warm on dbp, paired (six rounds of the 200 queries, off and on in turn, the median of the rounds' client medians): with every page asked for, a serial scan at a margin of 3 took 2.29 ms against 2.39 with prefetch, at 8 5.06 against 5.51, two participants at 3 11.98 against 11.71: 4 and 9% serially, the lookups in shared buffers of pages already there and the looks into the queues, and in parallel less than the noise. Asking no more once the asks keep finding their pages there (§44.4, “Prefetch”: dormant for 256 asks after 32 found), the same rounds gave 2.29 against 2.33 at a margin of 3, 4.91 against 4.99 at 8 and 10.67 against 10.81 with two participants: about 2%, a margin-8 scan asking for 65 pages rather than 880.

Read streams (2026-10-17; §44.4, “Read streams”; bench/results/2026-10-17-lionvec-read-streams/, knn_readahead.py). DBpedia’s 115,186 rows as knn_dbpedia.py load makes them, each vector in its row (toast_tuple_target = 8160, a row a heap page, 900 MB), the cosine index of 2,715 partitions spilling (247 MB), the first 10 of the 200 queries; PostgreSQL 18.6 built without assertions, 4 vCPUs, 15 GB, shared_buffers 2 GB, effective_io_concurrency 200 (64 pages ahead), io_combine_limit 16, the disk a virtio device. Each query under pg_lion.vector_prefetch off, fadvise and stream, and under main at 64e7443 (on, which is fadvise but asked for no heap page of this table, its raised toast_tuple_target not counted until a9a27c5), in three states: device, the server restarted and the page cache dropped before each query, the configurations taking turns query by query; os, the server restarted before each query with every relation in the page cache; warm, every relation in shared buffers, the configurations of one build taking turns in one session. The kinds: unfiltered at margins 3 and 8; LionOrdered probing under filters of 10, 3 and 1% (the exact path off); its exact path under 1 and 3% (1,204 and 3,472 members). Every query returned the same rows in the same order in every configuration, state and io_method. The server’s median ms over the 10 queries, and in brackets each query’s stream time over its fadvise time, the median of those ratios (off is io_method = worker’s; it reads nothing ahead, and plain ReadBuffer() reads in the backend under every method):

Device off worker (3): fadvise, stream sync: fadvise, stream io_uring: fadvise, stream
margin 3 25.3 20.9, 21.9 (1.12) 20.8, 18.8 (0.94) 19.1, 20.3 (1.04)
margin 8 67.2 47.6, 43.2 (0.93) 48.1, 41.0 (0.79) 46.4, 39.8 (0.86)
10%, probed 58.0 58.1, 46.8 (0.94) 51.5, 41.7 (0.87) 51.6, 47.9 (0.85)
3%, probed 95.9 72.9, 57.9 (0.79) 76.9, 61.6 (0.79) 76.0, 68.1 (0.87)
1%, probed 100.9 82.8, 62.0 (0.80) 93.0, 72.2 (0.81) 88.2, 72.9 (0.85)
1%, exact 108.3 58.9, 53.3 (0.90) 56.1, 67.2 (1.17) 64.9, 43.8 (0.66)
3%, exact 318.3 172.8, 151.6 (0.88) 168.1, 158.2 (0.93) 168.3, 111.8 (0.65)

From the disk the streams are the faster on 18 of the 21, by 6 to 35% query by query; about even at a margin of 3 (0.94 to 1.12), whose scan probes 29 partitions and reads 346 pages, too few for reading ahead to pay much either way; and slower on the exact path at 1% under sync, where a stream’s reads are the backend’s own preadv() after posix_fadvise() advice, much as fadvise’s are (67.2 ms against 56.1). The exact path, a list known whole, gains most, its wait on reads all but gone (under worker 3.5 ms against 19.3 at 1% and 5.6 against 54.4 at 3%): under io_uring its 3,472 heap pages took 112 ms against 168 asking the kernel and 318 with neither. Main’s on asked for the index’s pages as fadvise does and for none of the heap’s, so its exact path took off’s time (115.5 ms at 1%). The streams read more pages: 402 against 346 at a margin of 3, 996 against 861 at 8 and 1,708 against 1,500 at 1% - chains the refill predicted that the scan did not probe, its LIMIT met or a centroid coming first, released unread (64, 228 and 677 a query) - and the disk read 4 to 19% more on the index’s kinds (11.1 MB against 9.7 at a margin of 8 under worker). A scan’s streams ended and started again 35 to 72 times a query, each time a stream had nothing left to read and the scan then read a page it had pushed.

With every page in the kernel’s cache and none in shared buffers, a read is a copy of 8 kB, and handing it to another thread costs more than it: under io_method = worker an I/O worker reads every page a stream reads, and under io_uring PostgreSQL submits a buffered read with IOSQE_ASYNC once more than four are in flight (method_io_uring.c), so the kernel’s io-wq threads copy it; ReadBuffer() copies it in the backend under every method.

OS cache off worker (3): fadvise, stream sync: fadvise, stream io_uring: fadvise, stream
margin 3 6.3 6.4, 11.5 (1.40) 7.3, 7.1 (1.02) 8.2, 8.9 (1.31)
margin 8 18.2 17.9, 21.7 (1.18) 19.8, 18.5 (1.11) 16.9, 18.9 (1.22)
10%, probed 20.2 23.8, 23.6 (1.00) 19.5, 21.5 (1.08) 19.8, 30.4 (1.23)
3%, probed 28.0 26.1, 29.8 (1.10) 28.3, 30.8 (1.14) 27.0, 31.1 (1.19)
1%, probed 29.2 28.4, 26.5 (1.05) 31.0, 31.1 (1.02) 27.6, 36.4 (1.10)
1%, exact 18.4 19.1, 8.6 (0.44) 19.8, 21.6 (1.10) 22.3, 34.0 (1.54)
3%, exact 52.5 55.1, 23.2 (0.45) 56.6, 61.4 (1.07) 55.8, 107.9 (1.96)

So from the kernel’s cache the streams cost: 0 to 40% on the index’s pages, whose reads a scan mostly waits on one at a time, and on the exact path one and a half to twice the time under io_uring, every copy a thread’s hop, where under worker its many reads in flight, copied by three workers at once, took half the time. Sixteen workers (io_workers = 16, off, fadvise and stream only) read the disk faster still - the exact path 37.2 ms at 1% against 61.9 asking the kernel, 99.5 at 3% against 170.0, a margin of 8 37.1 against 45.4 - and the kernel’s cache slower, sixteen processes on four CPUs: 10.9 ms against 6.2 at a margin of 3 (1.72), the exact path 1.17 times fadvise’s.

Warm, every page in shared buffers, each query against itself under the same build’s off in one session (warm-rest3.json, 20 queries, 5 runs each): fadvise 0.98 to 1.05, stream 1.02 to 1.11 (1.11 at a margin of 3, 1.07 at 8, 1.02 to 1.05 filtered and on the exact path); and a build whose streams did not rest (r3, the ring’s stream ever pushed and refilled, the list ever streamed), 1.06 to 1.15 against its own off (1.15 at a margin of 3, 1.13 at 8, 1.11 and 1.15 at 3 and 1%). The rest roughly halves what a warm stream costs, as dormancy did the asks' (§44.4, “Prefetch”), so it stays, for the streams too: of a warm margin-8 scan’s 1,098 reads, 341 come from its streams and 758 are read apart, and of the exact path’s 1,204 blocks at 1%, 165 come from its list’s stream. The matrices' warm runs, a session for each build and the configurations taking turns, agree: stream over fadvise 0.93 to 1.04 under each method.

What the parts are worth, from smaller runs (5 queries, io_method = worker, device): without predicting the next partitions' chains (nopred), each probe’s stream ended with its run and started again from one block, and a scan took 68.4 ms at a margin of 8 against 42.8, 146.9 at 1% against 62.8, 65.5 at 10% against 49.6 and 31.7 at a margin of 3 against 19.3 (ab-restart.json); a stream that ended reset rather than begun anew from a full I/O ahead (r2) took 62.7 ms at 10% and 25.4 at a margin of 3 against 49.6 and 19.3, and the same at 8 and at 1%. The exact path through the ring, before the list stream, took 65.9 ms at 1% (its wait 45.4 ms), through a list stream 50.9 (3.2) and asking the kernel 57.8 (18.1); at 3% 163.8, 132.2 and 166.0 (ab-list.json); a list stream without READ_STREAM_FULL, which starts one block ahead, 53.7 and 146.4.

The vectors in TOAST (toast-worker-3.json): pgvector keeps a vector of more than 506 dimensions there unless told otherwise, so most tables of real embeddings do. The same rows loaded with KNN_INLINE=0 into another database - the heap 7.6 MB, its TOAST 900 MB - with an index of their own (2,715 partitions, 247 MB), under io_method = worker, from the disk and warm. Nothing reads a toasted vector ahead (§44.4, “Prefetch”; §44.11). The index’s streams gained what they gained inline: a margin-8 scan took 43.0 ms streaming, 47.7 asking the kernel and 57.9 reading as it went, a 3% filter’s probing 67.4, 81.2 and 97.4, a margin of 3 about even (29.5, 26.3 and 28.7). The exact path at 1% read its 2,717 pages - the members' heap pages, the TOAST index’s and the chunks - one at a time under every setting, 157 to 168 ms, 135 to 146 of them waiting on reads, where the inline table’s, streamed, took 53; warm it took 11.2 ms against 4.6, detoasting 1,204 vectors.

Vectors in TOAST read by place (2026-10-17; §44.13, “Measured”; bench/results/2026-10-17-lionvec-toast-direct/): the same dbpedia_toast, its index REINDEXed as format 5, under pg_lion.vector_toast_direct off and on. The exact path from the disk took 68.3 ms at 1% against 179.7, reading 1,631 pages against 2,717 and waiting 12.7 ms against 152.7, and 158.6 at 3% against 457.2; warm 7.9 against 14.1 and 16.4 against 31.8. The probing kinds were 6 to 7% faster from the disk and the same warm, and the inline table’s queries took main’s time.

The refine’s reads (2026-10-27; §44.4, “The refine’s reads”). The 115,186 DBpedia rows as knn_dbpedia.py load makes them (each vector in its row, toast_tuple_target = 8160, 900 MB) and the cosine index the build chooses (247 MB), the 200 held-out queries, LIMIT 10 at the default margin 3 and patience 30; and rnd, 100,000 random unit vectors of 1536 dimensions (a row a page, 782 MB), a cosine index of 2,529 partitions in 51 regions (216 MB), 20 random unit queries - a query of random vectors probes 477 partitions and refines 13,400 rows and 2,400 centroids, where DBpedia’s probes 35 and refines 85 rows and 118 centroids. PostgreSQL 16.15 (apt.postgresql.org’s), pgvector 0.8.7, a VM of 4 vCPUs of a Xeon at 2.1 GHz with AVX-512 and 15 GB, shared_buffers 2 GB, huge_pages off unless said, effective_io_concurrency 200 (64 pages ahead, asked of the kernel: 16 has no streams), every relation prewarmed, each query run once untimed and then timed under EXPLAIN (ANALYZE, BUFFERS, TIMING OFF), the median of the 200 (the 20), the builds taking turns round by round in one server, each build’s median of 3 rounds (2 on huge pages). The steps of a row’s refine, ns a refine, serialized (each timestamp behind lfence), the first time and, in brackets, a repeat with its inputs cached, on 4 KB pages; the builds: main, the prefetches (“prefetched”), and with the look resting (“resting”):

Rows' refine, ns due read lock check at est after all
DBpedia, main 454 1,055 (92) 40 (41) 843 (28) 175 (34) 911 (396) 112 3,590
DBpedia, prefetched 486 1,010 472 100 281 488 117 2,954
DBpedia, resting 47 990 468 108 273 483 116 2,485
random, main 220 670 (82) 38 (37) 560 (30) 188 (35) 986 (380) 162 2,824
random, prefetched 222 636 219 80 261 445 150 2,012
random, resting 83 638 212 77 271 454 156 1,892

A centroid’s refine on DBpedia, huge pages: 1,796 ns on main, 1,291 prefetched, 1,186 resting. The timing itself costs - with it a DBpedia query took 1.45 ms against 1.15 - so the shares are the instrumented scan’s: the rows' refines 18% of it on DBpedia and 55% on the random vectors on main, 14% and 46% resting. Without it, the server’s median ms, and the PrefetchBuffer() calls a query:

Warm DBpedia, ms asks random, ms asks
main 1.151 80 50.7 1,876
prefetched 1.129 80 40.5 1,876
resting 1.071 93 39.7 2,346
main, huge pages 1.083 80 45.2 1,876
prefetched, huge pages 1.039 80 37.5 1,876
resting, huge pages 0.976 93 33.9 2,346

By perf on the resting build, 4 KB pages, the look (lv_prefetch_queue()) is gone from a DBpedia query’s profile (5.6% on main), lionvec_check_page() is 3.1% of it against 5.7 and lionvec_sum8_avx2() 4.9 against 8.0; on the random vectors 5.9 against 12.9 and 14.7 against 25.2, and lionvec_refine_at(), which now issues the item’s prefetches and meets its header’s miss, 7.2 against 1.3. The buffer table’s hash_search_with_hash_value() keeps its 10 and 15%. Cold, each query alone after a restart with the page cache dropped, every build read the same pages (344 a DBpedia query, 11,757 a random one) and the times are the disk’s noise: main 13.1 ms on DBpedia against 11.6 resting, 8.8 and 7.2 of them waiting on reads (20 queries each, two passes), and the random vectors 264 against 259 (12 each); the prefetches alone 10.2 and 240 (10 and 6). Every build returned the same rows in the same order for every query. The prefetches cost nothing cold: a page that is not in shared buffers is read before they are issued.

44.10 Tests

  • test/unit/vec_test.c (make unit, and under ASan and UBSan with SANITIZE=1), against the arithmetic alone, with exact-size heap buffers: the 8-bit tables (evenly spaced and symmetric, each code the inverse of its step, extremes, infinities and NaN); the rotation at 64 to 8128 dimensions (norms and inner products kept to 1e-5; the same signs from the same seed and others from another; one window and two; a basis vector spread out; rotated coordinates with the variance, kurtosis and tails of a Gaussian); lionvec_prepare() (padding, cosine scaling, zero vectors); the encoding (bits, popcount, both alphas, a row at the mean); the query’s quantization and planes; every set of kernels the build and CPU have, forced in turn (§44.3, “Kernels”): plane 1 against a bit-at-a-time count at 1 to 127 words, the 8-bit sum and l2sq against their sixteen partial sums added in the header’s order and the rotation against the portable one’s, all to the bit, at every dimsp from 64 to 8128, from unaligned pointers, over inputs of every scale, denormals and signed zeros, and end to end the same rotations, estimates and k-means as the portable set’s (2026-10-27: and lionvec_dot8() against a plain integer loop, both its sums, at every dimsp from 64 to 8128 from unaligned pointers, over codes and 16-bit weights at and mixed between their extremes, every set the portable one’s results; the query’s rounding - every coordinate within half a step, the extremes at +-32767, the sums and the variance as stated, a constant query’s zeros, and the same query but for the rounding with qw NULL; and the estimate: the affine formula against the serial sum over the rounded query to 1e-9, within the rounding’s reach of the floats' estimate, its error as stated, and every set the portable set’s bytes) (test/unit/vec_test_nosimd is the same program built with -DLION_NO_SIMD); the halves' conversion, the portable one against a reference computed in double from each half’s fields, over all 65,536 halves (2,046 NaNs, each quiet with its payload), and each set’s against the portable one’s to the bit, over all of them and at every length 0 to 40 from offsets 0 to 7, with nothing written past the end; lionvec_datum_dim()’s bounds for both types (shorter than the header, a header alone, a byte short of the count’s elements, negative counts); the key algebra against true distances; and the bounds' COVERAGE on clustered data at 100, 384 and 1536 dimensions for each metric: plane 1 outside 1 sd 30 to 32% of the time (the Gaussian’s 31.7%), outside 3 sd 0.16 to 0.29% (0.27%); the codes outside 2 sd about 4% (4.6%) and never outside 6 sd, in 24,000 row-query pairs each. A heavy tail would fail it. The partitions: lionvec_candidates() at 64, 320 and 1536 dimensions - at a wide bound the nearest centroid is always a candidate and every key within its bounds, at the build’s bound it is missed at most once in 200, and the candidates are few (10 of 200 at 320 dimensions and more); no centroids, and identical ones; lionvec_lift_key() over partitions of 300 rows of 24 dimensions, of mixed lengths and of unit length - at a margin of the lifted rows' greatest distance from their lifted centroid it bounds every row’s key for every query, on unit rows it is the L2 bound of the unit query, and it falls as the margin grows; and lionvec_kmeans(), plain and spherical - it converges, every row ends with its nearest centroid, every centroid is its rows' (normalized) mean to 1e-4, the same seed gives the same centroids, a zero mean gives the same as none, one centroid is the mean, as many as rows one row each, with eight centroids a cluster none mixes two clusters, and identical rows finish; and its steps (§44.12), each iteration’s rows assigned in chunks of 1, 7, 32 and all, the chunks taken last first and alternating between two scratch areas, giving its centroids, assignment and iterations to the bit, for as many centroids as clusters and eight times as many. Spill: lionvec_rank() against a sort, with ties, NaN and infinities; lionvec_candidates_n() for the nine nearest at 64, 320 and 1536 dimensions - for one the same as lionvec_candidates(), at a wide bound every one of the nine a candidate, at the build’s bound one missed at most once in 200, and still few (10 of 200 at 320 dimensions and more, 107 at 64), and the same from the streaming form an insert keeps (lionvec_cands_add(), with the insert’s pruning) for one and for nine; lionvec_decode8() at 64 to 8128 dimensions - its error |x|^2 (1 - alpha8^2) to 1e-3, at most 3% of |x| and 1.2% on average, orthogonal to the decoded vector; lionvec_soar_second()’s algebra on a constructed case whose answer flips where the arithmetic says (a candidate along the residual against one across it, at distances either side of sqrt 2), against a reference at lambda 1 and 0.5, and on one candidate, none, identical ones, a row at its centroid and NaN; and over clustered rows at 64, 384 and 1536 dimensions in 120 partitions, that it takes the least cost, that lambda 0 is the plain second nearest, that SOAR differs from it for between a tenth of the rows and all (25-29%), that the copies' residuals point less along r than the plain second’s, and that decoded centroids choose as the floats do for at least 90% of rows (97%), the rest within 1% of the cost. The cut and the regions (§44.3): lionvec_subsample() from 0 to 100,003 slots, m distinct and ascending, every slot when m is all of them, the same from the same seed and another from another, and each slot as often as the rest over 8,000 seeds; lionvec_candidates_among() the same as lionvec_candidates_n() over the listed centroids taken alone, and over all of them; and over 40 clusters at 128 dimensions in 600 centroids, 25 regions - the count (none below 64), the beam never more than the regions and never less for nine than for one, lionvec_regions_pick() against a reference over 2,000 sets of keys with empty regions, ties, NaN and infinities, the members a partition of the centroids, ascending in each region, at least 99% of the centroids in the region nearest them, the same regions from the same seed and, after renumbering, the same centroids routed in the same order; a row’s routed candidates the beam’s members' and among them every exact candidate there, exact candidates for at least 97% of rows (100%), its routed nearest the exact one for at least 97% (100%), and no empty region picked; and k-means routed, by chunks of 1, 32 and all against lionvec_kmeans() with regions, to the bit, regions included, the same again from the same seed, without regions the exact k-means, and its cost within 2% of the exact one’s (1.0000). Prefetch (§44.4): lionvec_items_per_page() and lionvec_chain_pages() at their edges (no room, no items, a page’s worth and one more, 232 - 1 items), lionvec_prefetch_end()’s window - dist pages after the block, short of the run’s end, never before the next block, and no block number wrapping at 232 - and lionvec_least() against lionvec_rank()’s set over 2,000 sets of keys with ties, NaN and infinities, each key kept before every key left out, and lionvec_least_order() putting it in lionvec_rank()’s order, the order a read stream is given the pages in (§44.4, “Read streams”). The exact path’s page price (§44.7, “Exact path pricing”; 2026-10-18): lionvec_pages_in_order() at EXPLAIN’s bitmap heap scan price, 875.23 for 966 rows of 834 pages, every page at seq_page_cost from twice as many rows as pages, a single page at random_page_cost, and over 5,000 cases never more pages than rows or than the relation’s, at a price between the two.
  • test/unit/readahead_test.c (make unit, and under ASan and UBSan with SANITIZE=1; 2026-10-17): the ring between a reader’s predictions and core’s read stream (lion_readahead_ring.c, §44.4, “Read streams”), driven through its ops against a model of core’s stream - the blocks its callback named, pinned in that order and handed out one a pull, a distance that doubles when a read waits and shrinks when it does not, the callback asked only during a pull, a block now and then taken back unstarted (read_stream_unget_block()), and an end once the callback names no block, until a reset - with every buffer a pin the model counts, so that a pin taken twice, released twice or left behind shows: every read a pin of its block, held, given, pending or not tracked; a list read in its order, and one fed by refill, through the stream alone; urgent blocks before predicted ones; pins never past the limit, a full ring releasing its oldest held block, and a stream with every pin given ending and starting again at a later read, only once it holds nothing; a clear releasing every pin; a stream handing back the wrong block, or none, reported; and random pushes, reads, refills and clears at several limits, lion_ring_check() after every step and every block given read or released unread.
  • vector.sql: every ORDER BY run through the index and through a sequential scan and sort, compared distance by distance and row by row, with pg_lion.vector_margin = 100 and the pause off, for each metric, three queries and LIMITs 1 to past the table, over 3,000 rows in 20 partitions; no LIMIT, OFFSET, NULL and cosine zero queries and rows; a scan cut by work_mem of 64kB with a batch of 1 (the livelock of §44.4); inserts of a far cluster into the partitions nearest it, UPDATE, DELETE and VACUUM, a table emptied by DELETE, REINDEX and CREATE INDEX CONCURRENTLY; an expression index and a partial one; a SCROLL cursor fetching backward; an untyped column (dimensions from the first row, pgvector’s errors); an unlogged table; an index built on an empty table, whose inserts go to the REST chain, then rebuilt with more partitions than rows; the default margin’s recall (at least 0.9 of the ten nearest for each metric, over 20 queries, pausing) and a negative margin’s (less than all of a hundred between clusters); a scan pausing after every partition (a patience of 1) still returning a LIMIT 300 whole and in order; and the refusals (another type, too many dimensions, an array, two columns, the partitions and settings' ranges, an operator class over int - amvalidate false, CREATE INDEX an error - and one whose operator returns integer, which amvalidate reports and a scan refuses); an operator class whose operator returns float4, valid, its scans' distances float4 and the operator’s own; the progress hash held to hash_mem (§44.4): a scan with no LIMIT at 128 kB, whose hash holds 1,843 entries for 3,001 rows, returning every row as a sort does, and at a work_mem of 64 kB failing with its ERROR, or, with pg_lion.vector_overflow set to stop, stopping with its NOTICE, what it returned the sort’s first rows; the setting refuses any other value. Spill, its indexes' CODE items read off their pages with pageinspect: after the build, inserts, UPDATE, DELETE and VACUUM and REINDEX, every row with a vector has one primary and one copy, in another partition’s chain (each block placed by following the chains from the PART items' heads), naming the same REFINE item; NULL rows and cosine zero vectors one item; no item of a deleted row left after VACUUM, in either chain, and one REFINE item a row with a vector, by TID, before VACUUM and after it, whose dead rows' items it removed; the comparisons with a sort above, which spill doubles - each row once, past the table’s rows, through a walk, and cut by work_mem; 1,200 rows of 30 distinct vectors in 12 partitions, so that every distance is a forty-way tie, at LIMITs that end a tie and with no LIMIT; spill = off and a REINDEX, after which no row has a copy and the scans agree, and inserts into it; no copies with no partitions, with one, nor in the REST chain; two and five partitions, fewer than the nine SOAR ranks, at the build and at inserts; the recall over twin tables, spilling and not, as many of the ten nearest at the defaults and more at a margin below zero, for each metric; and the reloption’s refusals. Regions: every index’s meta page and REGION chain read off its pages (vt_regions(): format 4, the regions' count ceil(sqrt(C)), numbered from 0, their runs covering the partitions in order, each naming its run’s first PART item), none below 64 partitions; 6,400 rows in 100 partitions for each metric, then 1,000 copies of rows inserted, every copy in its original’s partition and at least 90% in its copy’s partition too (vt_placed(); SOAR’s choice from decoded centroids is not the build’s from floats for every row, §44.3), and the comparisons with a sort over them.
  • vector_filter.sql: LionOrdered over each metric against the ordinary plan, with pg_lion.vector_margin = 100 and the pause off, for eight filters (equality, range, AND, OR, <>, a correlated column, a boolean, no rows) and four LIMITs, over 10,000 rows in 50 partitions - the filters of up to pg_lion.vector_exact_rows members by their exact distances, the others probing; far, zero and NULL queries; a prepared statement’s generic plan; a correlated subquery rescanning the node; the plan of an OR of two lion indexes' clauses, its set built from both; a spilled set (400,000 rows of 3 dimensions in 20 partitions, hash_mem 64kB), exact, the same ten rows, and planned at 64kB, where the spill is priced and the node is the plan; a filter of 200 members reading less than a quarter of the index blocks one of 4,000 does, by pg_statio_user_indexes, and each threshold moving it, by the DEBUG1 line and the blocks: pg_lion.vector_exact_per_partition alone (pg_lion.vector_exact_rows 0) has the 200, half a member a partition of 400, exact at its default and probing at 0, and the 4,000, ten a partition, exact at 10 and probing at 9.99, in the same order either way; LionOrdered’s start-up priced under it as under pg_lion.vector_exact_rows, not as the probes, and (2026-10-18) from the heap within a tenth of 1 / 1.2 of the sort’s price (the lean of 2026-10-19), its members' heap pages priced as the bitmap scan’s (§44.7, “Exact path pricing”); the setting’s range; inserts, deletes and VACUUM under the filter; the default margin’s recall for each metric under a filter of 9,000 members (rows of mixed lengths, which an inner product ranked unlifted got 0.6 of, §44.4); a restricted scan pausing at once (a patience of 1) still filling a LIMIT 2000 of its 9,000 members, in order, through the node (the short LIMIT of §44.4); the node switched off, the query failing under the default pg_lion.vector_postfilter, the filter’s rows sorted under exact, and under allow the vector scan with a Filter. Spill: LionOrdered returns each member once and the same rows as the ordinary plan (the vector scan with a Filter, under allow), every member of two filters for each metric, at the margin of 100 and at the defaults, before and after DELETE and VACUUM under the filter. The ordinary plan is the node off under exact (since 2026-10-17, when the default error would fail it), a sort of the filter’s rows; so it is in vector_half.sql and vector_limit.sql.
  • vector_limit.sql (2026-10-16): the restricted scan told its query’s LIMIT (§44.4, “A known LIMIT”), over vector_filter.sql’s 10,000 rows in 50 partitions at the margin of 100, where every partition comes before every row, and a patience of one unit, so that each tier pauses as soon as it stalls; each query through the node, with the scan’s DEBUG1 line, and against a sort. A LIMIT 100 under filters of 9,000 and 3,000 members pauses once, on L, skips nothing and returns the hundred nearest, where the tiers paused five times and lost 27 and 26 of them; so do LIMIT 10 OFFSET 90 and a custom plan’s LIMIT $1; a LIMIT 50 pauses once, at 80, and returns its fifty nearest (23 with the tiers); LIMITs 10 and 5 are as they were, a pause at ten each; a generic plan’s LIMIT $1, which the planner does not know, keeps every tier and loses the 27 as before; a subquery’s own LIMIT under an aggregate; and a LIMIT 300, L in its place, which pauses once on its 300 having probed 33 partitions (with L at 100, twice, having probed all 50), and at a work_mem of 64 kB, where L is 170, overflows the queue and RESTs as it did.
  • vector_postfilter.sql (2026-10-16): pg_lion.vector_postfilter (§44.6, “Post-filtering”), each plan shown with the sequential and bitmap scans and the sort disabled, which leaves the vector scan with a Filter the cheapest where it is a plan, and each query’s rows compared with a sort’s, at the margin of 100 and no pause, over 4,000 rows in 20 partitions. SHOW, the default error (2026-10-17). At exact: a lion filter through LionOrdered, two lion columns over one set where a bitmap scan would filter one; a filter no lion index answers, alone, beside a lion one, and a primary key’s, sorted; the vector column’s own clauses (2026-10-17), v IS NOT NULL and a distance bound, the vector scan with a Filter, beside a lion filter LionOrdered with one, and beside a filter no lion index answers, sorted; no filter, and a pseudoconstant one, the vector scan; LionOrdered off, sorted; all with the sort’s rows, distance bounds that fill the LIMIT and that do not among them. At the default, error: the errors, EXPLAIN’s too, with the hint naming the subquery and the settings, for a filter lion does not answer, a mixed one, a lion one with LionOrdered off, a primary key’s, and v IS NOT NULL beside a filter lion does not answer; the same with error set; a lion filter’s LionOrdered; no filter, a pseudoconstant one, a generic plan’s WHERE $1, and the vector column’s own clauses, alone, beside a lion filter and pushed into a subquery as d < 3, with the sort’s rows; the suggested subquery, restricted and not, with the same rows as a sort; a partial lionvec index whose predicate implies the filter, its scan, and with a clause more, which lion answers, the error naming the predicate’s clause. Row-level security (2026-10-17): with it enabled, the owner’s query fails under error, exact and allow, EXPLAIN’s too, a query lionvec cannot order still runs, and REINDEX and CREATE INDEX fail; with it disabled, the vector scan again; a partitioned table’s index built, then the parent’s row-level security enabled, a query of the parent and one of the partition failing, and a new index on the partition refused. At allow: the vector scan with a Filter, LionOrdered with one, and their rows the sort’s.
  • vector_build_sample.sql (2026-10-19): pg_lion.vector_build_sample (§44.8), at 8 dimensions and a maintenance_work_mem of 1MB, PostgreSQL 16’s least, which holds 7,710 sample rows, 16 each for 481 partitions; the partitions are read off the meta page (pageinspect), and no setting is below 1MB, so every major version runs the same. 10,000 rows want 625: the build fails, at its last row, with the DETAIL and the HINT’s 2MB; at 2MB, and at the unrounded 1,329kB, it makes 625, and at 1,328kB a REINDEX fails, its sample having rows for 624, and leaves the 625. 20,000 rows want 1,131: the build fails at the 10,000th row, the first that makes the sample short, which an index expression counting its calls through a sequence shows, reckoning the 20,000 as 19,894; under allow it makes 481, with the NOTICE of the 20,000 counted, and at error again a REINDEX fails and leaves them, the index still answering. The same with the table’s parallel_workers at 2: the error, and under allow 481. The partitions option of 300 over 400 rows of 2,000 dimensions, whose 1MB holds 255 (stored in line, a row a heap page, so the 400 are reckoned exactly), fails at the 256th row, makes 255 under allow and 300 at the hint’s 2MB - 2,000 dimensions so that k-means over a sample that small is quick, where at 8 an option over 10,000 rows took 5.6 s for its 7,710 partitions under allow and 5.4 s for 8,000 at 2MB; an option of 2,000 over 1,000 rows, from a sample of every row, makes 1,000 and does not fail, nor does a table of 9,999 rows, of one partition, at 1MB; and the setting refuses any other value. In test/unit/vec_test.c: lionvec_parts() at the rule’s figures, the reservoir’s at §44.3’s (21,788 rows at 64 MB and 1536 dimensions, 348,617 at 1 GB, the cap from 52 GB) and a billion rows' (252,982 partitions, 4,047,712 sample rows, 12,174,759 kB); and for options of 0 to 262,144, reservoirs of 1 to 224 rows and every count of rows to 200,000 and every 0.05% from there to 4 billion, that a sample of every row is never short, that the cap takes no partition away, that short stays short as the rows grow, and that the maintenance_work_mem the error names makes the partitions wanted, at 64 to 8,128 dimensions, where a kilobyte less does not.
  • test/isolation/lionvec_insert_window.spec (an injection point build, PostgreSQL 17 and on; 2026-10-15, after a review found lionvec’s concurrency untested): an insert parked after its REFINE record (lionvec-insert-refined), and after its CODE record before its copy’s (lionvec-insert-coded), against a cursor opened before it, which drains while it is parked and returns exactly a sort’s rows under its snapshot, each once; a scan started while it is parked, which waits for the meta page ((*), a buffer lock) and then sees none of its rows; and a scan of the whole index afterwards, every row once. lionvec_vacuum_scan.spec: 2,500 of 3,000 rows deleted, VACUUM in two flushes of its REFINE locations (maintenance_work_mem 64 kB), then 2,500 inserts into the REFINE slots and heap TIDs it freed, against a cursor parked after five rows with REFINE locations in its queue and against a scan parked between two pages of a chain (lionvec-probe-page); each returns exactly a sort’s rows under its snapshot, each once, and the whole index afterwards every row once.
  • vector_half.sql: the three halfvec operator classes, and amvalidate; each ORDER BY through the index and through a sort, as vector.sql’s, with the margin of 100 and the pause off, for each metric over 3,000 halfvec(40) rows in 20 partitions, three queries and LIMITs 1 to past the table, generic plans whose query is a Param, NULL and cosine zero queries and rows, and a query of other dimensions; inserts of a far cluster after the build; LionOrdered against the ordinary plan under lion indexes' filters for each metric, probing and exact (pg_lion.vector_exact_rows at 5,000 and 0, pg_lion.vector_exact_per_partition 0, by the DEBUG1 line); an expression index casting a vector column to halfvec; an untyped column (dimensions from the first row, pgvector’s errors); and the refusals (too many dimensions, an array).
  • vector_parallel.sql: the parallel build against the serial one (§44.12), each index built both ways and read off its pages with pageinspect into digests - the meta page’s counts, scale, flags, regions and mean, the PART items but for their chains' blocks, the REGION items but for their PART items' blocks, each CODE chain’s items in order but for their REFINE locations, and the set of REFINE items - which must match, and the nearest rows the same through either, for each metric, spilling and not, one partition, a halfvec expression and 100 partitions in 10 regions (L2 and cosine; no sample here is cut, every table being below LIONVEC_SAMPLE_MIN rows), over 6,000 rows with NULLs, a cosine zero, deleted rows and HOT chains, with the table’s parallel_workers at 2; CREATE INDEX CONCURRENTLY in parallel against a serial build; and the count of workers by the DEBUG1 lines: 10,000 rows in parallel without parallel_workers, at the default maintenance_work_mem and at 1 MB, where every process’s sort spills to disk (and the index is still a serial build’s), an expression index and a parallel-safe partial one in parallel, and serially at 9,999 rows, in a temporary table, over NULL rows alone and under a parallel-unsafe predicate; and the cut (§44.3): 70,000 rows in 20 partitions, whose k-means is given 65,536 of them (the DEBUG1 line says so) before a parallel build shares the sample, the meta page and PART items - the centroids, counts, spreads and longest rows - a serial build’s. On PostgreSQL 16, whose core says “serially” of every build but btree’s, the expected output is vector_parallel_1.out.
  • vector_scan_parallel.sql (2026-10-16): the parallel scan (§44.4, “A parallel scan”), forced (no setup or tuple cost, no least sizes, the table’s parallel_workers at 2) and shown with the sequential and bitmap scans and the sort disabled, over 3,000 rows of 24 dimensions in ten clusters, NULL vectors and a zero vector among them, in 20 partitions for each metric, spilling: the plans, a Gather Merge over a Parallel Index Scan of the operator’s index; at the margin of 100 with the pause off, each metric’s distances in order and rows against a sort’s (ps_cmp(), run whole by EXECUTE, as a PL/pgSQL loop over a query never runs in parallel), near a cluster and far from all, at LIMITs 1, 10 and 300 and none, with the leader taking part and not; a NULL query’s walk and a cosine zero query’s, every row once, and the order’s tail - the farthest rows, the zero vector, the NULLs; a Gather Merge on a nested loop’s inner side, run twice (amparallelrescan), the nearest three each time; an index that does not spill; halfvec; filters at the default pg_lion.vector_postfilter, error, the one no lion index answers failing though a parallel scan is a path, and under exact, through LionOrdered or a sort and never a Parallel Index Scan with a Filter, and under allow that plan, each with the sort’s rows (ps_cmp()’s sort under exact); reading ahead off, asking the kernel and through streams, and effective_io_concurrency 0, the same rows in parallel, and serially at the defaults through pauses, passes and LionOrdered’s restricted scans, exact and probed; and at the defaults in parallel, each participant stopping by its own rules, the share of the ten nearest over 20 queries for each metric at least 0.9 and within 0.1 of the serial scan’s. How many workers start is the server’s, and nothing shown depends on it. vector_filter.sql, vector_half.sql and vector_limit.sql turn the prefetch off around their DEBUG1 lines, whose prefetch counts depend on what shared buffers hold.
  • vector_stream.sql (2026-10-17): the three ways of reading ahead (§44.4, “Prefetch” and “Read streams”) against each other, the rows and their order, over 6,000 rows of 24 dimensions in 8 partitions - a fifth in each category, one in a hundred under each tag, NULL and zero vectors among them - an L2 index and a spilling cosine one, and a table whose rows are all in its REST chain. Each query runs under off, fadvise and stream, first with the tables' and indexes' pages evicted from shared buffers (pg_buffercache_evict(), 17 and later; on 16 they stay) and then warm, where the asks fall dormant and the streams rest: probes and refines, REST passes (a batch of 100, and of 20), the NULL and zero walks, LionOrdered’s exact path and, with it off, its probes, the REST chain, a nested loop’s rescans, at effective_io_concurrency 8, 1 and 0; cursors fetching a row at a time and scrolling; a cursor whose index grows by 3,000 rows between fetches; a parallel scan (pg_lion.vector_parallel_scan on); chains inserts appended to; half of each relation evicted, the upper and then the lower, so that within a scan the streams rest and wake and the asks fall dormant and ask again; an exact path over 900 members a heap page each, the lower 480 pages evicted, at every even effective_io_concurrency from 14 to 80, so that the list’s stream rests with more blocks in it than a rest lasts and the last of them comes out at many points of a rest and of the reads between two (where, at 16 and 34 on 18 and 70 to 76 on 17, the scan once took the end of a backlog that the rest had ended for the end of the list, and failed); scans that fail part-way, in a subtransaction, at the top level and in a cursor, and the same rows after; the setting’s names and the boolean’s spellings, and its default; and the DEBUG1 line of a warm exact path each way, the list stream resting by turns. On 16 the expected output is vector_stream_1.out, stream there being fadvise.
  • vector_toast.sql, vector_toast_corrupt.sql, test/isolation/lionvec_toast_scan.spec and test/recovery’s phase 5 (2026-10-17): vectors in TOAST read by their chunks' places, and every history and damage that can make a place wrong (§44.13, “Tests”); and (2026-10-18) the plan a 1% filter gets by the exact path’s price, in TOAST and inline (§44.7, “Exact path pricing”).
  • vector_corrupt_records.sql (2026-10-18): an insert into an index whose meta page names itself as the NULL chain’s end and as refine_tail, which waited for its own lock for ever, and into a CODE tail whose pd_lower is inside the page header, which PANICked: each an ERROR now (§44.5, “Format checks”); and (2026-10-27) a scan that refines a row whose REFINE item’s line pointer puts it past the page, its length right, which read the item from the memory after the page: an ERROR before a byte of it is prefetched or read. On 16 too: the file is read back through ALTER INDEX ... SET TABLESPACE (§3, 2026-10-18).
  • test/modules/lion_hooktest/sql/oldsnapshot_vector.sql (2026-10-18; make hookcheck on 16 with pgvector, next to oldsnapshot.sql on its server started with old_snapshot_threshold = 60, §44.5): a plain order and a filtered one (no LionOrdered, and no error under error) planned as a sort, with a sort’s rows; with every other path disabled, a lionvec scan priced with disable_cost and stopping at its ERROR - inline, under a Filter (allow), and over vectors in TOAST, whose index keeps a locator; and inserts, updates, deletes, VACUUM (the index’s row count the table’s after it) and REINDEX not refused. Against the code before, the filtered order failed at plan time and every forced scan ran. A parallel scan cannot be forced there: its path and the serial one both carry disable_cost, which puts them within add_path()’s fuzz of each other, and the serial one is kept.

44.11 Not done

  • A billion rows. Partitions are what make one possible, and three things stood in the way. The build’s k-means and assignment, and a scan’s ranking, cost a plane-1 estimate of every centroid for every row or query: O(n C) and O©; and the sample’s cap of 220 rows, at 16 a partition, with the reloption’s ceiling, held the default to 65,536 partitions (15,000 rows each at a billion). The build’s are the regions' now (§44.3): O(n sqrt©), each row routed through about sqrt© region centroids and the members of 5 regions in k-means, 16 in a spilling second pass; and the cap is 224 rows and the ceiling 262,144 partitions, so that a billion rows get the default 8 sqrt(n), 252,982 partitions of 3,950 rows, given the maintenance_work_mem (12.5 GB at 1536 dimensions for the sixteen a partition the default needs, without which the build fails, 52 GB for the cap’s 66, §44.8). At 503 regions of some 503 centroids that build is, by arithmetic (not measured), 16.8 million sample rows x 10 iterations x about 3,000 estimates for k-means, 5 x 1011, and a billion rows x about 8,550 for the second pass, 8.6 x 1012, where every centroid for every row was 4.2 x 1013 and 2.5 x 1014. DBpedia’s 53 regions save less (2,717 estimates a row, against some 310 and 870), and its build went from 292 s to 108 s serially (§44.9). The old default, 2 sqrt(n), kept the partitions few but let the rows read grow almost as the rows (§44.9’s growth: n0.91). 8 sqrt(n) (§44.3) is four times finer, 125 rows a partition at a million rows, but still grows as the root: 4,000 rows a partition at a billion. The scan’s step is still O©: it ranks every centroid, a 65 MB PART chain a query at 262,144 partitions, where routing it through the regions read a third more rows for a recall in the simulation (§44.3). A large table wants partitions of a fixed few hundred rows - two million of them for a billion rows, which makes the scan’s O© the whole cost, a 500 MB PART chain a query, and a build of 2.4 x 1013 estimates even through 1,415 regions, unless an index over the centroids that a scan can use as well (regions with a better order than their centroids', or a small graph) ranks them by reading a few thousand. And even then, by the margin alone, the rows read grew as n0.53 to n0.61 at a fixed partition size: one to four million rows a query at a billion, 230 MB to 1 GB of CODE pages. With the pause (§44.4) they grew as n0.32 between the model’s two largest sizes, which taken that far is about 260,000 rows, 60 MB of CODE pages, a parallel scan’s tens of milliseconds from memory (arithmetic on two sizes, not measured). Spill (§44.3) is the other lever on the rows read, measured so far at 115,000 rows (§44.9); the index over the centroids is still to be built.
  • The sample’s rows (§44.8, pg_lion.vector_build_sample). The build fails rather than make fewer partitions than its rows want, but which rows the reservoir draws, and so the centroids, still follow maintenance_work_mem wherever it holds fewer rows than the table has: another size of reservoir keeps other rows, and k-means is given 16 to 128 of them a partition as it holds more (§44.3). The answers at a margin follow the centroids. A reservoir of one size whatever the setting - k-means' 128 a partition, or every row - spilled to a temporary file past maintenance_work_mem as LionOrdered’s set is (§30.4), would make the same centroids at every setting; at a billion rows of 1536 dimensions it is 32 million rows, 50 GB, and k-means reads it ten times.
  • What the parallel scan leaves (§44.4, “A parallel scan”; 2026-10-16). LionOrdered stays serial: its lion set is one process’s, and a parallel restricted scan would need the set shared or built in each participant, and a partial path of the node. The stop rules are each participant’s own: on DBpedia two and four participants together probed 1.6 and 3.7 times the serial scan’s partitions at a margin of 3, 1.24 and 2.0 times what the serial scan probes for the recall they returned, and two 1.03 times at a margin of 8 (§44.9). A bound shared through the parallel area - the participants' least tau, or their tiers' - would win that back, as might a participant’s patience counted in its share of the rows; it is worth its shared state only where the probes, not the workers' start and Gather Merge’s wait for every participant’s first row, are what a parallel scan pays, which on DBpedia they were not. (The read stream this item once proposed in place of PrefetchBuffer() is §44.4’s “Read streams”, 2026-10-17; what it leaves is the next item.)
  • What reading ahead leaves (§44.4, “Prefetch” and “Read streams”; 2026-10-17). A vector in TOAST is not read ahead at all, by the asks or the streams: its chunks are found through the TOAST index from the heap tuple’s pointer, which reading them ahead would have to read first, a heap page and an index descent a row before its chunks could be named; pgvector stores a vector of more than 506 dimensions there unless its column says otherwise, so most tables of real embeddings read their exact distances' chunks one after another: on DBpedia so stored the exact path at 1% took 157 to 168 ms from the disk under every setting, against 53 with the vectors in the heap’s pages and streamed (§44.9), while the index’s streams gained what they gain inline. Reading the chunks ahead wants the heap tuples first - a stream over the members' heap blocks whose reader, a block ahead, reads their TOAST pointers and names the chunks' TOAST index pages and then their pages to a second stream - or the chunks' TIDs kept beside the index’s own items, which §44.13 built (2026-10-17): the locator for the exact path, a REFINE item’s place for the rerank. The rows' heap pages for their exact distances are asked of the kernel under stream too, a stream’s pins being unwelcome between amgettuple() calls (§44.4); a scan asked for 0 to 26 of them a query on DBpedia, fetching 10 or 11 rows for their distances, too few for a stream to be worth its pins. The executor’s fetch of a returned row is a buffer hit, the scan having fetched it for its distance just before, so nothing is read ahead for it. From the kernel’s cache a stream costs more than ReadBuffer() under 18’s worker and io_uring (§44.8), core handing each read to an I/O worker or a kernel thread; a default by io_method, or a stream that stays away while its relation’s reads are quick, is left. And lion’s own readers do not stream: lion_count’s heap recheck reads its sorted blocks a ReadBuffer() at a time where it does not stream already, and a lion set’s leaf walk asks for one right-link ahead. A list stream (lion_readahead_begin_list()) fits the first, a ring (lion_readahead_begin()) the second; the exact path’s list stream, from the disk, took its 3,472 heap pages in 112 to 152 ms against 168 to 173 asking the kernel (§44.9).
  • sparsevec, bit. The codes do not care where the floats came from; each needs an operator class and a reader for its format, as halfvec has (§44.1).
  • The double heap fetch. Each returned row is fetched by the scan for its distance and again by the executor; the second is a buffer hit. An index-only distance would need the REFINE item to be exact, which it is not.
  • REFINE space. VACUUM leaves REFINE line pointers and frees the space only for the tail page’s inserts; a table with much churn grows until REINDEX. A drifting distribution does not make the index wrong - any mean is correct, and an insert goes to the nearest centroid there is - but loosens the bounds and unbalances the partitions until REINDEX recomputes both.
  • A small work_mem with loose bounds (few dimensions, a large LIMIT) makes many RESTs, each probing again every partition probed so far; the DEBUG1 line shows them.
  • The planner underprices a sort by distance, at pgvector’s own costs. Its distance functions cost 1 (procost), and a 1536-dimension vector is stored out of line, so the planner sees an 18-byte column and a cheap operator: sorting a filter’s rows by distance looks cheaper than the lionvec scan, and was chosen at 548 ms over the scan’s 136 ms for 97,000 rows, at 82 ms over 31 for 9,700, and cold at 1,209 ms over 199 (format 1). Repricing core’s Sort paths whose key is a lionvec distance would be new territory for lion’s planner hooks, and is not planned. Instead lionvec’s documentation has the functions its operator classes use (l2_distance, vector_negative_inner_product and cosine_distance of vector, and l2_distance, halfvec_negative_inner_product and cosine_distance of halfvec for a halfvec column) set to COST 100 with the extension, and the benchmarks assume it. pg_dump does not keep the setting (an extension member’s), and a pgvector update may reset it. enable_sort = off does the same for one query.
  • Choosing between lionvec and HNSW. With both on one column, the planner takes HNSW for nearly every query, as pgvector prices an HNSW scan at almost nothing: on 200,000 random 256-dimension rows, about 13 against lionvec’s 3,100 unfiltered (format 1).
  • Progressive refinement from planes 2 to 8 (evaluated 2026-10-26, §44.14): the 8-bit codes kept as seven bit-planes, a refine reading them until its bound decides. On DBpedia it would stop at 3.2 planes a row and 4.0 a centroid, but the sums are a tenth of a warm query, reading the REFINE item costs a refine more than its sum, and an integer kernel over today’s bytes takes as much off: not worth a format. That kernel, the codes against the query rounded to 16-bit integers (60 ns against 326 warm, 373 against 682 out of L2), is done: lionvec_dot8(), 2026-10-27 (§44.3, “Kernels”, with its measurement in a scan).

44.12 Parallel build (lion_vec.c, lion_vec_math.c; 2026-10-15)

A build’s time is arithmetic, not reading: DBpedia’s 115,186 rows of 1536 dimensions, in 2,715 partitions, took 134.2 s serially (PostgreSQL 18, 4 cores, the table cached) - the first pass 2.8 s, k-means 108.0 s, the second pass 23.1 s, the sort and the writing 0.4 s; another run took 124.8 s, the machine being shared. Both k-means' iterations and the second pass find, for each row on its own, the nearest of the centroids by a plane-1 estimate of each (§44.3), so both are shared out among processes. With 3 workers and the leader the same build took 37.8 s: the first pass 3.0 s, k-means 28.3 s, the second pass 6.2 s, the writing 0.4 s (38.2 s in another run); the shared steps ran 3.8 and 3.7 times as fast on four processes. It runs on PostgreSQL 16 as well, counting its own workers rather than waiting for core’s (below).

The regions (§44.3, 2026-10-16) cut the arithmetic of both shared steps, and the parallel build keeps its share of what is left: on a 4-core machine shared with other work, where the build before them took 291.6 s serially and 84.9 s with 3 workers (115,386 rows in 2,717 partitions), it took 108.2 s and 35.9 s, 3.0 times as fast with the workers (§44.9, “Regions”). What the leader does alone each iteration is the regions' k-means: C centroids into sqrt(C) regions, 5 iterations at C sqrt(C) estimates each, some tens of milliseconds here and, by arithmetic, 7 x 108 estimates at LIONVEC_MAX_PARTS, where an iteration over the 224 rows the sample can hold is 5 x 1010 shared among the processes.

  • The phases. The first pass, the mean and the sample, stays serial, and with it the check that fails a build whose sample is too small for its partitions (§44.8), before the parallel context is set up: reservoir sampling draws the same rows only when it meets them in the same order, and it is a fiftieth of the time. With the partitions known, the leader sets up a parallel context (lionvec_begin_parallel(), the workers' entry lionvec_parallel_build_main()): a shared struct with a parallel heap scan from the first block (as a serial build’s scans, allow_sync = false, §44.5), the sample, copied into it and the leader’s own copy freed, so that it is held once, the arrays k-means and the assignment work in (LionVecBuildArrays: the centroids, their plane 1 and 8-bit codes, their floats, the sample’s assignment and the mean, laid out by one function in the leader and in every worker, and kept in local memory by a serial build), a shared tuplesort, and the workers' WAL and buffer usage, which the leader adds to its own. It then posts the workers work, under a spinlock and a condition variable, and takes part in each itself:
    • KMEANS, each iteration: the leader codes the centroids (C rows, a moment), regroups them into their regions when there are 64 or more (lionvec_regions_make(), into the region arrays LionVecBuildArrays holds in shared memory beside the centroids', §44.3, “Regions”) and posts the iteration; every process takes the sample’s rows 32 at a time (LIONVEC_KMEANS_CHUNK) from an atomic counter, assigns each its nearest centroid in the shared assignment and counts the rows that moved; the leader waits for the last row, then moves the centroids to their rows' means, alone. The counter’s high 32 bits are the iteration’s, so that a process still finishing the last iteration takes none of this one’s rows, and its compare-and-swap never runs past the end.
    • SCAN, once: the second pass over the parallel heap scan. Each process puts its rows' CODE items into its own run of the shared sort, with maintenance_work_mem divided among them as core’s builds divide it, and their REFINE items on REFINE pages it extends the index by for itself; the leader merges the runs and writes the CODE chains, the PART chain, the REGION chain and the meta page, as a serial build does, ends the workers, and WAL-logs the index. Before it posts the pass the leader has made the final centroids' regions and renumbered the partitions region by region in the shared arrays, so that every process routes each row through the same regions, by the same numbers, as a serial build’s second pass does.
  • A serial build’s index. lionvec_kmeans() is its steps run in a loop (lionvec_kmeans_start(), _codes(), with regions lionvec_regions_make(), _assign() over a range of rows, _update()): a row’s assignment depends on the centroids, their regions and the row alone, the regions are made by one process (the leader, as a serial build makes them: the same k-means of the centroids from the same seed), and the update sums the rows in the sample’s order in one process, so the centroids come out of a parallel build to the bit as out of a serial one, however the rows were shared out (the unit tests, §44.10, which route by chunks in two “processes” with scratch of their own; vector_parallel.sql’s 100-partition indexes, whose meta page, PART and REGION items match a serial build’s). A row’s partitions, its own and its copy’s, depend on the centroids and their regions alone. The CODE items' sort was keyed by the group and a count in scan order; it is keyed by the group and the TID now, so the merged runs come out in the order a serial build’s scan made, and the partitions' counts, spreads and longest rows, and the meta page’s scale, which the scan callback summed, are summed as the CODE chains are written, in the sort’s order, so that the floating sums do not depend on which process met which row. What differs is where the rows' REFINE items lie: each process fills pages of its own, so their blocks, the REFINE locations in the CODE items, and the chains' blocks, laid after the REFINE pages, are not a serial build’s. A scan reads the same codes in the same order either way, and gives the same answers: DBpedia’s 200 queries returned the same ten rows through either at margins of 3 and 8.
  • How many workers (lionvec_build_workers()): max_parallel_maintenance_workers, or the table’s parallel_workers when it sets it, at most max_parallel_maintenance_workers, as for core’s builds. Core’s count (plan_create_index_workers()) would also size a build by the heap’s main fork and by maintenance_work_mem. pgvector stores a vector of more than about 500 dimensions out of line, in the TOAST table: DBpedia’s main fork is 7.6 MB, under min_parallel_table_scan_size, and core gives its 115,186 rows none, though the build’s cost is arithmetic over the rows, whatever their bytes. And core keeps 32 MB of maintenance_work_mem a process, which at the default 64 MB allows one worker, for its sorts; a lionvec build’s processes share the sample, and each holds only its share of the sort, which spills (at 1 MB, vector_parallel.sql). So lionvec counts its own, and core’s goes unused. (pgvector’s HNSW build asks for max_parallel_maintenance_workers whatever the size too, but only once core’s count is more than none.) None below 10,000 rows with a vector unless parallel_workers is set (LIONVEC_MIN_AUTO_ROWS, below which a build is one partition by default and quick); none for an index whose expressions or predicate are not parallel safe (lionvec_build_parallel_safe(), is_parallel_safe() as core asks it), for a temporary table, which workers cannot read, or with no partitions. 17’s amcanbuildparallel is set, as the build is parallel; core then counts workers for its DEBUG1 line alone, and on 16, which counts only for btree, the build is parallel all the same. Either way core’s line may say “serially” of a build lionvec’s next line says it “built in parallel”. If no worker starts, the leader undoes the parallel setup and builds serially.
  • Shared memory. The sample, up to half of maintenance_work_mem (§44.3; 177 MB for DBpedia), and the arrays (the centroids' floats and codes, 22 MB there) are in the parallel context’s DSM segment, sized before the workers start. Where /dev/shm is small - Docker’s 64 MB unless --shm-size raises it - creating the segment fails, and so does the build, as pgvector’s parallel HNSW build does, whose documentation says the same; the guide says it, and that max_parallel_maintenance_workers = 0 builds serially. The sort’s runs are in temporary files, as core’s are.
  • Locks and cancel. A worker opens the heap and index with the locks index_build() took for the leader, ShareLock and AccessExclusiveLock, or for CREATE INDEX CONCURRENTLY ShareUpdateExclusiveLock and RowExclusiveLock, and scans with the leader’s snapshot - an MVCC one under CONCURRENTLY, else SnapshotAny, as table_index_build_scan() wants. The second pass used to hold its REFINE page’s content lock from one row to the next; an LWLock holds off interrupts, so a cancel waited for the page to fill, and a worker’s for the leader, whose condition-variable wait is where a worker’s error reaches it. Each row’s REFINE item now locks the page for itself, which costs nothing measurable, and a serial build’s cancel is as prompt.

44.13 Vectors in TOAST: reading them by their chunks' places (the locator; format 5, 2026-10-17)

pgvector stores a vector of more than 506 dimensions (a halfvec of more than 1,012) in TOAST unless the column or the table says otherwise (§44.4, “Prefetch”), so most tables of real embeddings keep theirs there. A fetch of such a vector reads the row’s heap page for its TOAST pointer, descends the TOAST index on (chunk_id, chunk_seq) for its chunks, and reads their pages: three random reads where an inline vector costs one, none of which can be named before the one before it is in, so neither the asks nor the streams read them ahead (§44.11, “What reading ahead leaves”). The exact path (§44.6) pays that a member: on DBpedia’s 115,186 rows so stored (dbpedia_toast, §44.9) the 1% filter’s 1,204 members cost 158 ms from the disk (157 to 168 under every setting, §44.11), 2,717 pages read one at a time - 708 heap pages, about 800 of the TOAST index’s 1,273 and 1,204 of chunks - and 11.4 ms warm, against 53 and 4.6 to 4.9 with the vectors in the heap’s pages and streamed. A probing scan fetches a dozen rows or so, each the same three reads.

So an index whose vectors may lie in TOAST keeps where each row’s chunks lie - the TOAST relation’s block and line pointer of each, as RUNS (below) - in two places: in a map from heap TID to those places, the LOCATOR, which the exact path reads in heap order beside its members, and then each member’s chunks directly, a page a vector in the usual case, through a read stream over blocks it knows whole; and, where the vectors leave room, in each row’s REFINE item, which the probing path’s rerank has just read when it comes to fetch the row, and which then names the row’s chunks' page beside its heap page. A place is only ever a hint: every chunk read there is checked to be the chunk it should be, anything else sends that row down the heap path, and a row read so is returned only once its heap tuple, fetched under the scan’s snapshot, shows the same value.

What the prototype was worth (before it was built: a throwaway C function outside the repository, on the benchmark cluster - PostgreSQL 18.6 release, io_method = worker, 3 workers, §44.9’s settings; the 10 queries of knn_readahead.py’s x1 and, for 5 of them, x3; the medians). From the disk (the server restarted and the page cache dropped before each query) and warm (every relation in shared buffers, 3 runs a query): x1 as it ran, under EXPLAIN (ANALYZE, BUFFERS); and the members' vectors read by their chunks' places (from pg_column_toast_chunk_id() and the TOAST table’s ctids, computed beforehand), each chunk checked as heap_fetch_toast_slice() checks it, the distances computed with pgvector’s cosine_distance(), after a 400-page relation read in order standing for the locator, and the ten nearest rows' heap tuples fetched and their TOAST pointers compared with the values read. Every configuration returned the ten rows x1 did. p the pages read, w the wait on them (track_io_timing):

1% (1,204 members), disk warm 3% (3,472 members), disk warm
the exact path then 158.3 ms, 2,717 p, 137.6 w 11.41 427.2 ms, 5,620 p, 370.7 w 29.09
by place, read one at a time 124.1 ms, 1,614 p, 118.3 w 3.98 336.4 ms, 3,882 p, 321.6 w 11.58
by place, asked of the kernel 64 ahead 80.3 ms, 1,614 p, 28.9 w 4.15
by place, through read streams 55.2 ms, 1,614 p, 10.1 w 3.97 141.5 ms, 3,882 p, 13.4 w 11.83
the same function, heap fetch and detoast 189.0 ms, 2,712 p, 158.2 w 11.17 461.6 ms, 5,600 p, 379.2 w 37.03
… the heap blocks streamed 184.2 ms, 2,712 p, 146.4 w 11.27
inline, streamed (mx-worker-3.json) 53.3 ms 4.9 151.6 ms 12.6

Streaming the members' heap blocks alone bought nothing (184 against 189 ms), as §44.4 found asking for them; reading by place, streamed, made the toasted table’s exact path as fast as the inline table’s. What the built scan does is under “Measured”, below.

Where a value’s chunks lie. A chunk holds TOAST_MAX_CHUNK_SIZE (1,996) bytes at 8K pages and a page holds four, so a 1536-dimension vector (6,148 bytes) is three chunks and 160 bytes, four tuples, which toast_save_datum() inserts one after another, each where the last went if it fits. In dbpedia_toast, bulk loaded, every one of the 115,186 values has its four chunks on one page of its own, at line pointers 1 to 4, and the TOAST blocks follow the heap’s TIDs (correlation 0.99999), so the members' chunks come in ascending blocks. Elsewhere (a scratch cluster, 2,000 to 6,000 random rows each): 768 dimensions and halfvec(1536), two chunks, all on one page; 1024 dimensions, three chunks (4,100 bytes), two values in three on one page and the third across two; 3072 dimensions, seven chunks, every value across two pages or three; 1536 dimensions from four concurrent inserters, 70 to 74% on one page in one run of line pointers, 25 to 30% across two pages, under 1% in three runs; and after half of 6,000 rows deleted at random and a VACUUM, the 3,000 values inserted into the holes all on one page each. So a value’s place is a list of RUNS - a block, a first line pointer, and how many chunks follow at consecutive line pointers - one in the common case, two for a value across pages (LionVecLocRun, lionvec_loc_runs(), lion_vec_math.h, which merges the consecutive line pointers of a block and nothing else).

What is stored where

  • The locator: the exact path’s. Its members come from a lion set, as heap TIDs in heap order, never from the index’s items, so their places must be found by heap TID. One entry a row whose vector lies in TOAST, on pages of their own (below), read in heap order beside the sorted members - streamed, and only the pages whose heap blocks hold a member.
  • The REFINE item’s place: the rerank’s. A row a probe reranks has had its REFINE item read just before, so a place there costs the rerank no read, and its TOAST page can be asked for with its heap page (§44.4, “Prefetch”). Every REFINE item of an index that keeps them ends with a slot (LionVecRefineLoc): the value’s id and refloc_runs runs, min(n, ceil(n / 4) + 1) for a value of n chunks - 2 at 768, 1024 and 1536 dimensions and for halfvec(1536), 3 at 3072 - of which the item’s nloc (the header’s former pad) are in use: 20 or 28 bytes, which leave every one of those widths its REFINE items a page (10, 7, 5 and 2). A centroid’s item, and a row’s whose vector has no place or whose place takes more runs than the slot holds (a 1536-dimension value across three pages), has nloc 0, and is reranked from the heap as before; every REFINE item of the index is then one size. Built in the same format as the locator, so that one REINDEX brings both.
  • Not in the CODE items. Every probe reads every CODE item of a partition for a dozen fetches: 12 to 20 bytes more an item would be 5 to 9% more CODE pages read by every scan, for a saving on a few rows.
  • Not the REFINE item’s place in the locator, yet. An exact path that bounded its members by their 8-bit codes and fetched only those that could make the top k would want each member’s REFINE item, and the locator could carry its place (6 bytes). But a REFINE page holds five items and a 1% filter’s members are scattered one to a page, the same count as their TOAST pages, so it pays only for wider vectors or with codes in the locator itself. An entry carries a flags byte, 0 now and checked so, so that a later format can add the place under a flag.

The format (format 5)

  • LOC pages (LIONVEC_PAGE_LOC, 6): one item an entry (LionVecLoc): the heap TID; the number of runs (1 to the value’s chunks); flags (0); the value’s va_valueid; then the runs (BlockNumber blk, OffsetNumber off, uint16 n). 12 bytes and 8 a run: 24 with its MAXALIGN and 28 with its line pointer for one run, 291 a page; 36 for two runs, 226 a page. The value’s length is not kept: an entry is made only for an uncompressed value of exactly the length the index’s dimensions give (8 + 4 dims bytes for a vector, 8 + 2 dims for a halfvec, va_extsize four less), so the chunks' number and lengths follow from dims, as core’s checks have them (lionvec_toast_extsize(), _nchunks(), _chunk_size()). The special space’s minkey and maxkey are the least and greatest heap BLOCK of the page’s entries, not container keys as on CODE pages.
  • LOCDIR pages (LIONVEC_PAGE_LOCDIR, 7): one item a LOC page (LionVecLocDir: its block and the least and greatest heap block it has held an entry for), 12 bytes, 407 a page: one page for DBpedia, 9 for a million rows. The build’s items come first, in heap order with disjoint ranges, filling loc_bpages pages from loc_head - consecutive blocks, which the build writes one after another - the STATIC LEVEL, loc_built items. On the same chain after them (on the static level’s last page, and the pages after it) come the item of the LOC page the build leaves empty for inserts to append to (lo invalid and hi 0: no range yet), and those of the pages inserts add. Each LOCDIR page’s special space holds the least and greatest heap block of its items' ranges. VACUUM narrows no range (below), so a range holds every entry the page has, and perhaps fewer.
  • REFINE items end with the slot above in an index whose meta page gives refloc_runs, and their length is checked against it (lionvec_refine_at(): the item exactly 16 + dimsp + 4 + 8 refloc_runs bytes, nloc at most refloc_runs).
  • The meta page appends, after nregions (format 4’s last field, ending at byte 104): loc_head and loc_dtail, the LOCDIR chain’s head and tail; loc_tail, the LOC page inserts append to; toastrelid, the heap’s TOAST relation as the build found it; loc_built and loc_bpages, the static level’s items and pages; loc_root, a level over the static one, InvalidBlockNumber and not read (below); refloc_runs, 0 when the REFINE items have no slot. pd_lower is 136. A flag, LIONVEC_META_LOCATOR (2), says the index keeps a locator, and lionvec_check_meta() holds the fields to it: all of the blocks valid with it, none without.

A second level. This version reads the LOCDIR whole on every exact-path query: one page for DBpedia, 9 for a million rows (72 KB), 85 for ten million - and for a billion rows of 1536 dimensions in TOAST, 3.44 million LOC pages (27 GB) under 8,450 LOCDIR pages, 66 MB read for a query whose members' entries lie on a few hundred LOC pages. Past some tens of millions of rows a query should read only the LOCDIR pages whose ranges hold its members' blocks, and the static level is laid out so that a later version can, with no change of format: its items are in heap order with disjoint ranges, on loc_bpages consecutive blocks from loc_head, each page’s special space holding its items' least and greatest heap block, and loc_built says where it ends. So a later version may

  • search it: a binary search of the static level’s pages by their special spaces, about log2(loc_bpages) reads (13 at a billion rows) for the first member’s block, then onward through the members' sorted blocks, the pages they need and those a search passes; or
  • write a level over it at build time: an item a static page (its block and its range), 21 pages of 407 for a billion rows' 8,450, named by loc_root.

Either reads, of the static level, only the pages whose ranges hold a member’s block, and then the items after loc_built - the tail’s and the inserts' - whole, as this version does (they are bounded by the inserts since the build, and by the append area’s re-sort if it is built, below). Neither changes a byte this version writes or a rule by which it reads: an index this version built is searched as it is, loc_root left invalid; one a later version built with a root this version reads whole, as it reads its own, since it never reads loc_root (nor checks it), the root’s pages lie on no chain it walks (scans and VACUUM walk the LOCDIR chain from loc_head, and the LOC pages its items name), and inserts append past the static level, which neither a root nor a search ever has to follow. The static level stays right as long as the index does: inserts never touch its items, and VACUUM empties its pages' entries but narrows no range.

Which rows have an entry: those whose stored vector is on disk in TOAST (VARATT_IS_EXTERNAL_ONDISK), uncompressed (VARATT_EXTERNAL_IS_COMPRESSED false, so EXTENDED storage’s pglz or lz4 values have none), of exactly the dimensions' length, in the heap’s own TOAST relation (va_toastrelid), and whose chunks the TOAST index finds as heap_fetch_toast_slice() would - chunk_seq 0 to n - 1 once each, each TOAST_MAX_CHUNK_SIZE but the last - in at most n runs (lionvec_toast_value(), _ptr_ok(), _locate(), lion_vec_loc.c). A NULL, a vector inline, compressed, or anything else has no entry, and is read as it always was.

Which indexes keep a locator (lionvec_build_locator_plan()): an index of a plain column (not an expression, whose values the build computes and no TOAST relation holds) of a heap-AM table that has a TOAST relation, whose vectors may lie there - its storage EXTERNAL or EXTENDED, and its vector past the larger of TOAST_TUPLE_THRESHOLD and the table’s toast_tuple_target (lionvec_vector_inline(), §44.4’s rule, moved to lion_vec.c to share) - or whose build met a row whose vector did all the same (its REFINE items then without a slot). Of those, the REFINE items carry their rows' places when the dimensions are known before the build’s second pass and an item with its slot fits a page. An index built over no rows (an empty table’s, and ambuildempty(), an unlogged table’s init fork) decides by the plan alone, dims 0 taken to be past any threshold, and writes an empty LOC page and LOCDIR page; its REFINE items have no slot, the dimensions being unknown. Decided by the build, recorded in the meta page’s flag, and kept until REINDEX. An index whose vectors lie inline - the DBpedia benchmark’s own table, any table under toast_tuple_target = 8160, any column PLAIN or MAIN, any vector of 506 dimensions or fewer - keeps none: no page, no work at insert, build or VACUUM, and the scan tests one flag. A table that later toasts some of its vectors (other wide columns can push even a short vector out) reads those rows as before.

The exact path (lv_exact(), lv_exact_places())

When the scan starts (lv_start_toast()) it reads by place if the index keeps a locator, pg_lion.vector_toast_direct is on (§44.8; off reads every row as before, which the tests compare against), the heap is heap-AM and its TOAST relation the meta page’s toastrelid, the dimensions are known, and the snapshot is an MVCC one (on 16 under old_snapshot_threshold no scan gets this far, below). It opens the TOAST relation once, with AccessShareLock as a detoast takes it - held to the end of the transaction, which keeps the relation from being rewritten (VACUUM FULL) or truncated under the scan - and counts its blocks then: a place at or past that count is never read. Then:

  1. The members, from the lion set, in TID order, as before.
  2. The LOCDIR chain, whole (lionvec_locdir_read()), and of its items those whose range holds a member’s heap block (lionvec_loc_select(): each item’s range against the members' sorted, distinct blocks by a binary search); those LOC pages, in block order, through a list stream over the index (§44.4, “Read streams”; asked of the kernel or read plainly under fadvise and off, and on 16). A page of the static level only, its entries in TID order, is searched: each member between its first entry’s TID and its last’s found by a binary search of its entries (lv_exact_page_sorted()), about 9 of 291 read; a page an insert’s item names, its entries in the order they came, is read whole, each entry looked up among the members by TID. An entry read is checked as every item is (lionvec_loc_at(): its length against its runs, its runs against the value’s chunks and a page’s line pointers, its TID and value id valid, its flags 0; ERRCODE_INDEX_CORRUPTED if not). A member met by two entries that differ takes neither. An entry out of order on a static page, which only damage leaves, can hide a member’s from the search; that member is read from the heap. At 1% of DBpedia the 1,204 members' entries lie on all 394 LOC pages, 3 or so a page: reading each of the 115,186 entries and looking it up among the members cost a warm query 4 ms more than the search, 11.4 against 7.2 in one session.
  3. The injection point lionvec-exact-located, where the isolation spec parks a scan.
  4. The members with an entry, by their first chunk’s place: one list stream over the TOAST relation (as far ahead as its tablespace’s effective_io_concurrency, to 64) naming each run’s block in turn (a block repeated in a row once), each member’s chunks read from their pages under a share lock and checked (lionvec_place_run(), below), the vector assembled in one buffer, its header checked (lv_value_ok(): lionvec_datum_dim() the index’s dimensions, its unused field zero), and its distance computed with the operator’s function on that datum - the same bytes the heap fetch would have handed it. A member whose chunks fail any check joins the rest; a cosine zero vector, its distance NaN, is left to the zero walk, as the heap path leaves it.
  5. The rest - no entry, a place past the end, or a failed check - as before: the heap fetch, its visibility, the detoast, their heap blocks streamed.
  6. Every distance sorted as before (NaN last, then TID), and returned in order. A member read by place has had no heap fetch, so before it is returned the scan fetches its heap tuple under the scan’s snapshot (the RETURN-TIME CHECK, lv_check_placed()): not visible, it is passed over, as the heap fetch passes it over; visible, its vector’s TOAST pointer must name the value read (va_valueid, va_toastrelid; lv_check_pointer()), and if it does not the index has named another value for a live row, which no history below allows: ERRCODE_INDEX_CORRUPTED, never a wrong answer. The executor’s own fetch of the row is then a buffer hit; the rows of a LIMIT 10 cost ten heap reads, which the executor would make anyway.

The streams end before lv_exact() returns, so no TOAST or LOC page is pinned between amgettuple() calls. Once more than half of a scan’s reads by place have failed, after 64 of them (a TOAST relation rewritten on its own, below), the scan reads by place no more (lv_place_ok()): such a scan pays for 64 reads at most. The DEBUG1 line adds N read by place (L locator pages, F not), C checked. Memory: an entry a member, the LOCDIR’s items (12 bytes a LOC page), one vector.

The checks of a chunk (lionvec_place_run()) are heap_fetch_toast_slice()’s, each a fallback where core raises an error, and those a place read directly needs besides: the page a heap page (initialized, no special space); the line pointer within it and LP_NORMAL; its tuple visible to the TOAST snapshot (HeapTupleSatisfiesVisibility() under get_toast_snapshot() from 18, init_toast_snapshot() on 16 and 17, through lion_compat.h’s lion_toast_snapshot()), as the TOAST index’s own fetch of it would require - a TOAST tuple is never updated, so the tuple at the line pointer is the chunk, with no HOT chain to follow; a TOAST tuple’s three columns, within the tuple’s own bytes; its chunk_id the entry’s value id; its chunk_seq the one the run’s place in the entry says; chunk_data not NULL, not compressed or external (a plain or short varlena); and its length TOAST_MAX_CHUNK_SIZE, or the remainder for the last. HeapTupleSatisfiesVisibility() is heapam’s, as core’s own TOAST fetch is; master’s move of index fetches into the table AM leaves it.

The probing path’s rerank (lv_rerank_placed())

lv_refine() reads a row’s REFINE item for its 8-bit codes; in an index whose items have a slot, a row whose slot is in use is marked (tp) with the first block of its place (in the item’s union with plane 1’s <x, mean>, which is done with by then). When the row comes near the top of the row queue for its exact distance, lv_prefetch_queue() asks for its heap block, as before, and for that TOAST block, both of the kernel (PrefetchBuffer()), under every vector_prefetch but off, as many ahead as the larger of the two relations' tablespaces' effective_io_concurrency: a stream’s pins between amgettuple() calls are unwelcome, as for the heap pages (§44.4). The DEBUG1 line adds N TOAST pages prefetched (M not in shared buffers). Then, at the rerank: the REFINE item read again for its slot (its page read a moment before; lionvec_refine_at() checks the TID, so an item VACUUM removed and an insert reused gives none); the heap tuple fetched under the snapshot - not visible, the row is done, and costs no TOAST read; visible, lv_check_pointer() holds its value to the slot’s, ERRCODE_INDEX_CORRUPTED otherwise; then the value read by its runs a block at a time (lionvec_place_read(), every block within the count), its chunks checked as above and its header by lv_value_ok(), and its distance computed. A read that fails detoasts the fetched row’s vector, as the heap path does, and counts toward lv_place_ok(). The heap fetch comes first, unlike the exact path’s, because every row the rerank reads needs its heap tuple anyway - its exact distance decides whether it is returned, and most of a probe’s dozen are - so the check costs nothing more there, and what reading by place saves is the TOAST index’s descent, a random read that no ask could start before the heap page was in.

Why a place that checks is the right chunk

Two facts of core’s, and one of the index’s:

  • A visible row’s value is present, where it was put. A row’s chunks are deleted only by the transaction that deletes the row or changes its vector, and removed only once that transaction is older than every snapshot’s xmin; a scan that sees the row holds an xmin before it. A standby replays the removal only after cancelling, or waiting out, every query whose snapshot could see the row (ResolveRecoveryConflictWithSnapshot()), as it does for any heap tuple. A chunk tuple is never updated, and while it is there nothing moves it from its line pointer or gives the line pointer to another: pruning and defragmenting keep line pointers, and VACUUM frees one only once its tuple is gone. Only a rewrite of the TOAST relation on its own moves chunks (below), which the scan’s lock holds off while it runs. So for a row the scan sees, the places its entry was made with hold its value’s chunks, which pass every check.
  • A value id names one value among the chunks that can pass. toast_save_datum() takes a new value’s id from GetNewOidWithIndex(), which refuses an OID for which the TOAST index finds any tuple under SnapshotAny - dead, uncommitted or aborted alike (“SnapshotToast considers dead rows as active indefinitely”, as catalog.c says) - and a value’s chunks are in the TOAST index from their inserts until VACUUM has removed them, so no two values whose inserters committed or are running share an id. Core’s own detoast depends on this, taking every tuple the index finds for the id under SnapshotToast, which passes every tuple whose inserter is not known to have aborted (HeapTupleSatisfiesToast()). So after a rewrite, which copies only those values' chunks, a chunk that checks is the one core’s detoast of that value would return, wherever it now lies. A chunk an aborted inserter left behind passes SnapshotToast until its abort is hinted, and its id can be given out again after the OIDs wrap; but its row is not visible to any snapshot, which the return-time check finds, and no entry made later can take a present value’s id.
  • An entry names its row’s value. It is written from the row’s own stored datum - the build’s from the tuple it scans, an insert’s from the tuple the heap stored at its TID (below) - and that value is never changed in place: TOAST values are immutable; a HOT update cannot touch an indexed column and carries the same pointer (toast_tuple_init() keeps an unchanged external value’s reference, and heap_attr_equals() compares the raw datums, so a HOT update’s vector is the same pointer, SET embedding = embedding included); a non-HOT update makes a new TID with an entry of its own. The entry leaves with its TID: VACUUM removes it in the same ambulkdelete() as the row’s CODE items, before the heap marks the TID unused, so a TID reused for another row never finds the old row’s entry (a crash part-way leaves the TID LP_DEAD until a VACUUM completes, as for every index).

So a row the scan sees, read by place, has exactly the distance its heap fetch would give it, or falls back. A row it does not see may read another value’s bytes at a place that checks - its value removed and its id given to a new one, which then happens to lie there - and is passed over by the return-time check before it is returned; its distance only placed it in the sort. The checks of the vector’s header keep such bytes from reaching the operator’s function as a vector of other dimensions, which pgvector’s functions would refuse with an error; they raise none for two vectors of the same dimensions, whatever their floats. Through each history (each tested where named, in vector_toast.sql unless said):

  • An UPDATE of another column, HOT (the root TID keeps its entry, and every version the same pointer) or not (a new TID, a new entry, the same value); SET embedding = embedding, the same pointer, HOT; an UPDATE that changes the vector, a new value at a new TID, the old one’s chunks deleted by it; a DELETE and the rows inserted again; an INSERT … SELECT from the table itself, which the heap stores anew under new value ids. Each followed by the exact path and the probing path, direct reads on and off, the same rows, order and distances, no error.
  • An insert rolled back: its rows are in the index and the locator and their chunks in TOAST, passing SnapshotToast until their inserter’s abort is hinted; read by place, each is passed over by the return-time check, and once hinted, each read fails and falls back to a heap row that is not there (both in vector_toast.sql).
  • Pruning and defragmenting a TOAST page (heap_page_prune_opt(), PageRepairFragmentation()) move tuples' bytes within the page and keep their line pointers; the place still names the tuple. A pruned chunk is LP_DEAD and fails the check. Both need a cleanup lock, which the reader’s pin and share lock hold off while it copies a chunk.
  • VACUUM of the TOAST relation reusing line pointers (VACUUM (PROCESS_MAIN false), and autovacuum, which processes a TOAST table on its own: the common way an entry goes stale): a value whose row is dead is removed, its line pointers made unused and given to the next chunks inserted on that page, while the heap row and its index entries stay. The new tuple’s chunk_id differs - or, its id given out again since, is that value’s, whose row is not the scan’s - and the row is dead either way. Tested with rows inserted into the space after, and in the isolation spec with a scan parked between its entries and its chunks.
  • VACUUM FULL or CLUSTER of the table, and every ALTER TABLE that rewrites it (a column’s type, SET ACCESS METHOD, SET LOGGED/UNLOGGED, a volatile default) write a new heap and TOAST relation and rebuild every index (reindex_relation()), the locator with it. ALTER TABLE ... SET TABLESPACE copies both block for block, every place kept (not tested). SET STORAGE, SET COMPRESSION and SET (toast_tuple_target) move no stored value; the rows they store otherwise later go as before (SET STORAGE PLAIN is tested).
  • VACUUM FULL of the TOAST relation alone (VACUUM FULL pg_toast.pg_toast_N, which a superuser may run) rewrites its chunks to new places and keeps their value ids and the heap: every place may be stale, and each read by place either checks - the same value’s chunk, by the second fact - or falls back. The scan stops trying after its first 64 (above); a REINDEX makes the locator whole. The meta page’s toastrelid does not see it (the relation keeps its OID), nor would its relfilenode, which SET TABLESPACE changes too, with every place kept. Tested, and on a standby (test/recovery, phase 5).
  • REINDEX builds the locator anew; REINDEX of the TOAST index moves no chunk.
  • pg_upgrade keeps relfilenodes (15 on), TOAST relations' OIDs, the value ids in the pointers, and every TID, and refuses a cluster of another TOAST_MAX_CHUNK_SIZE (pg_control): the locator stays right, as the index’s other pages do.
  • Standbys replay the locator’s generic WAL to the same pages, and the conflict rule above keeps a query from seeing a removal its snapshot could see; a stale place on a standby is stale as on the primary (test/recovery, phase 5).
  • OID wraparound of value ids: the second fact; an id repeats only for a value no longer present, or for an aborted inserter’s chunk whose row no snapshot sees.
  • A speculative insertion that lost (INSERT ... ON CONFLICT) leaves an index entry, a locator entry among them, and chunks whose xmin is made invalid, which SnapshotToast refuses: the read falls back, and the heap fetch finds the row dead.
  • CREATE INDEX CONCURRENTLY: the build as below, and validate_index()’s inserts, whose TID is a HOT chain’s root that a later version may follow with another value: such a row has no entry (lionvec_stored_pointer() takes only a tuple that ends its chain, as an insert’s does).
  • old_snapshot_threshold (16 only; 17 removed it): a TOAST value a snapshot older than the threshold sees may be removed, and core’s fetch then fails with “snapshot too old” on the TOAST index’s pages, which a read by place does not read. With it set no lionvec index is read at all (§9, “old_snapshot_threshold”; §44.5): the scan refuses before it starts, so there is nothing to read by place and no case of its own here. (Until 2026-10-18 the scan ran and only read every vector through the table instead - which the threshold made no more right, VACUUM’s early horizon removing the CODE items of rows such a snapshot sees.)
  • Serializable transactions: a member read by place and not returned has no heap fetch, and so no SIREAD lock on its tuple, where the heap fetch takes one. lionvec takes a predicate lock on the whole index (ampredlocks = false, §44.5), which every insert into it checks, and a change of a member’s vector is a non-HOT update and so an insert into the index; a delete of a member not returned does not change what the query returned. A row returned is fetched, and locked, by the return-time check, and a row the rerank reads by place is fetched before it is read. No isolation spec shows it (an open question below).

Build (and parallel build)

The build’s callback has the row’s stored datum (FormIndexDatum() over the tuple the scan read), a TOAST pointer for a toasted vector. For a row that qualifies, lionvec_toast_locate() scans the TOAST index for its value id as heap_fetch_toast_slice() does (toast_open_indexes(), the valid index, systable_beginscan_ordered() under the TOAST snapshot), takes each chunk tuple’s t_self with its checks, and makes the runs; the entry goes into the build’s sort as one more group, after the NULL chain’s (LIONVEC_SORT_LOC_GROUP, nparts + 2), keyed by its TID, and its place into the row’s REFINE item’s slot when it fits. The rows' vectors are detoasted as before (one more descent of the TOAST index a row, its pages just read). After the CODE chains the build writes the group’s entries to LOC pages in heap order, one run of blocks, each page’s range in its special space (lionvec_loc_writer_*()); then the empty LOC page inserts append to; then the LOCDIR, an item a page, the static level first; the meta page records them, loc_built, loc_bpages, toastrelid, refloc_runs and the flag. log_newpage_range() covers them with the rest. A parallel build’s workers do the same into the shared sort (each opening the TOAST relation and its indexes with AccessShareLock, as a detoast does), and the leader’s merge writes them in heap order: the locator comes out of a parallel build as out of a serial one, page for page (tested). Rows the build passes over as dead have no entry; RECENTLY_DEAD and in-progress ones do, their values being present. DEBUG1 says a locator of N rows in TOAST, its refine items with their places (or without places).

Inserts

An executor’s insert hands aminsert() the vector before TOAST took it (heap_insert() toasts a copy, and the slot keeps the original), and an INSERT ... SELECT of another row’s vector hands it that row’s pointer while the heap detoasts it and stores it anew under another id. So an index that keeps a locator reads the row’s tuple at ht_ctid - just written by this transaction, its page in shared buffers, under a share lock - for the pointer the heap stored (lionvec_stored_pointer(); the tuple must be LP_NORMAL and the end of its chain), finds its chunks through the TOAST index as the build does, and compares their bytes with the value it was handed, detoasted: the same, or no entry (lionvec_insert_locate()). All of that before the meta page is locked. Under the meta page’s lock, the REFINE item (with the place in its slot), the CODE item and its copy go in as before, then the injection point lionvec-insert-unlocated, then the entry (lionvec_loc_append()): on the tail LOC page, the tail’s LOCDIR item - the last of the chain - widened to its heap block, one generic WAL record of the two pages; when the tail is full, a new LOC page with the entry, its LOCDIR item on the chain’s last page, and the meta page naming it the tail, in one record; and when that LOCDIR page is full too, a new one after it, in the same record: four pages at most, which a generic WAL record takes. A crash before the entry’s record leaves a row without an entry, which reads as before; the entry is the last thing written, so it never names a row whose items are not in (test/recovery, phase 5, crashes an insert parked there).

The append area. Inserts append in arrival order, not heap order, so a page they fill covers the heap blocks they landed in: a narrow range for a table that grows at its end, a wide one under updates that move rows (a non-HOT update is an insert at a new TID, its vector the same value) or inserts into space VACUUM freed. A query reads every LOC page whose range holds a member’s block, so a wide range is read by most queries. The cost is bounded by the inserts since the build: an update of 10% of DBpedia’s rows would be 40 LOC pages more a query, a millisecond or two from the disk. A re-sort by VACUUM - the pages inserts added, but the tail, gathered once they are more than a few and their ranges overlap, sorted by TID and written back over the same pages in order, their LOCDIR items narrowed - is not built (an open question below).

VACUUM

lionvecbulkdelete() walks the LOC pages after the CODE chains (lionvec_vacuum_locs(): the LOCDIR items sorted by block, each LOC page under an exclusive lock), removing the entries of dead TIDs (PageIndexMultiDelete(), a generic WAL record a page that loses any), the bulk-delete callback asked once a row more. It narrows no range: a reader passing a page over by a range that is too wide loses nothing, but one too narrow would miss entries an insert adds while it runs, and nothing gains by the work. Pages are never recycled (§44.5): a page emptied stays in the LOCDIR, and an insert reuses no free space but the tail’s, as CODE and REFINE pages do until REINDEX (§44.11, “REFINE space”). A crash part-way leaves entries of TIDs still LP_DEAD, which the next VACUUM removes. A REFINE item, and its slot, goes when the CODE items that point at it do, as before, so a crash part-way can leave a REFINE item no CODE item points at; nothing reads it (§44.5).

WAL

Generic WAL, as every write of lionvec’s, in both of lion’s modes: lionvec writes no record of the custom resource manager’s, which installcheck-rmgr runs it under all the same. The recovery harness’s phase 5 (test/recovery/README.md) builds a table of 768-dimension vectors in TOAST, runs a write load of inserts, updates HOT and not, of the vector and of other columns, deletes, rollbacks and INSERT ... SELECT, beside VACUUMs of the table and of its TOAST relation alone, under wal_consistency_checking = 'generic'; kills the server three times in the middle of it, the last with a VACUUM in flight, and once with an insert parked at lionvec-insert-unlocated; and after each recovery, and on a hot standby under the same load and after a VACUUM FULL of the TOAST relation alone, reads the locator off the pages with pageinspect and checks it against the heap and the TOAST relation (no entry names a value other than its TID’s tuple stores; every row the snapshot sees whose vector is in TOAST has an entry of its value, where its chunks are; a REFINE item’s place is its row’s entry’s), and the exact and probing paths' rows, order and distances with direct reads on and off and a sort’s.

Format 5

The meta page’s fields are appended, every earlier one at its offset, the page types are new, and the REFINE item’s slot is past its codes, its nloc in the former pad. A format-4 index is refused with the REINDEX hint, as format 3 is: there is no release to migrate from (vector_toast_corrupt.sql makes a meta page format 4’s, and a REINDEX brings it back).

Measured

bench/results/2026-10-17-lionvec-toast-direct/: knn_readahead.py on dbpedia_toast (above; the heap 7.6 MB, its TOAST 900 MB), its index REINDEXed by this version - format 5, a locator and REFINE places - under pg_lion.vector_toast_direct off and on, on the prototype’s cluster and settings (§44.9; io_method = worker, 3 workers); the 10 queries of each kind from the disk (the server restarted and the page cache dropped before each query) and warm (every relation in shared buffers, 5 runs a query). The median ms, off / on, and in brackets the median of each query’s on over its off:

Query Disk Warm
unfiltered, margin 3 30.5 / 27.2 (0.94) 2.6 / 2.6 (0.96)
unfiltered, margin 8 53.7 / 44.8 (0.93) 6.2 / 6.3 (0.99)
3%, probed 70.8 / 63.9 (0.93) 10.2 / 10.0 (0.99)
1%, exact 179.7 / 68.3 (0.38) 14.1 / 7.9 (0.55)
3%, exact 457.2 / 158.6 (0.34) 31.8 / 16.4 (0.52)

The exact path at 1% read 1,631 pages where it had read 2,717 - the LOCDIR page, 394 LOC pages (the members' entries lie on all but two of the 396), the 1,204 vectors' pages, the 10 rows' heap pages and 22 of the TOAST index’s (below) - and waited 12.7 ms on them, not 152.7; at 3% 3,915 pages, not 5,620, and 12.2 ms of waiting, not 395.9: within 12 ms of the inline table’s exact path at 1% (56.2 ms the same morning, its rows' heap pages streamed) and level with it at 3% (158.3), and within 13 and 17 ms of the prototype’s (55.2 and 141.5, a 400-page stand-in for the locator, no entry checked). Warm, the exact path took a little over half its time (7.9 ms against 14.1, and 16.4 against 31.8; inline 4.3 and 11.5, the prototype 4.0 and 11.6, its members given it): the TOAST index’s descents saved, a page a vector, every chunk still checked.

The probing kinds read each row they rerank by its REFINE item’s place - 10 a query, none failing, the return-time check finding each row’s value its own - asking the kernel for the chunks' page beside the row’s heap page where they ask for the heap’s (1 and 16 asks at margins 3 and 8): 6 to 7% faster from the disk, query by query, but no fewer pages read (429 against 430 at a margin of 3). The query’s own projection detoasts each row returned, through the TOAST index, after the scan, whichever way the scan read it: the plan’s index scan computes embedding <=> q for the ORDER BY’s column, which no index scan can hand it (below, “Open questions”). That costs the exact path too: the 10 rows' TOAST index descents, 22 of its 1,631 pages. Warm, the probing kinds took the same time (0.96 to 0.99). Every query returned the same rows in the same order under both settings, with the recall the inline table’s.

On the inline table (dbpedia, whose index keeps no locator, read by this version and by main at 1be037c) the queries took main’s time: from the disk each query’s over main’s 0.90 to 1.07 by kind (the medians; inline-worker-3.json), and warm, in eight sessions taking turns between the two libraries, 0.97 to 1.00 (quartiles 0.95 to 1.06; inline-warm-alternating.json). Runs of one session a library had made it look 0.79 to 1.59, by which library ran first.

Costs

  • Size: an entry of 28 bytes a row with its vector in TOAST (36 across two pages), about 0.45% of a 1536-dimension vector’s 6.2 KB of TOAST; the LOCDIR 0.25% of that; the REFINE slot no page at the widths above. On DBpedia, REINDEXed: 396 LOC pages of entries, the inserts' empty tail and one LOCDIR page, 3,260,416 bytes - 1.26% of the 258,949,120 the index took as format 4 - and the REFINE pages the same 23,581, five items a page with their 20-byte slots as without.
  • Build: a TOAST index lookup a row, its pages just read by the detoast; the sort holds a tuple more a row, of 20 to 30 bytes, beside its CODE item’s 224 at 1536 dimensions. DBpedia’s serial REINDEX took 131.0 s, main’s 137.1 (one run each): the lookups are lost in the build.
  • Insert: a heap page, a TOAST index descent and the chunks' pages, all in shared buffers, the value compared, and one generic WAL record of the LOC tail’s and the LOCDIR page’s changes. Measured (knn_toast_insert.py: 5,000 rows of DBpedia’s by INSERT ... SELECT into 20,000 under a fresh index, its vectors in TOAST): 121 bytes more WAL a row, the REFINE slot’s 20 among them (50.79 MB against main’s 50.19, +1.2%), and 2,275 ms against 2,211 (medians of 8, taking turns; each pair’s ratio 0.83 to 1.13, their median 1.00), the spread of the runs wider than the difference. pg_lion.vector_toast_direct does not change what an insert does: an index keeps its locator up whatever the setting.
  • VACUUM: the LOC pages walked and the callback asked once a row more.
  • The exact path: the LOCDIR, the LOC pages with members, and a TOAST page a vector, streamed, and a heap page a row returned; memory as above.
  • The rerank: the REFINE item read again (a buffer hit), and the TOAST index descent saved.
  • Inline tables: nothing - no locator, and one flag tested by the scan; measured, main’s time (above).

The cost model (§44.7) prices the exact path at a random page a member, which a vector read by place is; it priced the toasted table’s two to three pages a member as one before. Nothing in it changes, and pg_lion.vector_exact_rows - set where the exact path and probing cost the same cold, on the inline table - may be worth raising for toasted tables. 2026-10-18: read by place, a member now costs no page but its share of the LOC pages, the sort it competes with paying for the same TOAST pages nothing either (§44.7, “Exact path pricing”).

Tests

  • Regression, vector_toast.sql: a table of 1536-dimension vectors in TOAST, some NULL - the build’s locator checked entry by entry against the heap and the TOAST relation (tt_check(), reading LOC and REFINE pages with pageinspect), what the scans read with every page in shared buffers (the DEBUG1 counts), and every vector_prefetch the same rows; then each of the histories above, each followed by the exact path and the probing path with direct reads on and off, compared row for row, order and distance (tt_cmp()), none raising an error; the TOAST relation vacuumed alone and rewritten alone, the fallbacks counted; VACUUM, and REINDEX making every place right again. Then 768, 1024 and 3072 dimensions and a halfvec(1536), values in one run, across two pages and across three, a REFINE slot of three runs; rows of every kind under cosine (NULLs, zero vectors, compressed, inline, in TOAST) and an expression index (no locator); a parallel build’s locator the same as a serial one’s, page for page; indexes that keep none (short vectors, toast_tuple_target), and an unlogged table’s init fork. Then (2026-10-18) 3,000 rows with a kilobyte of text each under a 1% filter: LionOrdered’s exact path, reading by place, the plan in TOAST (§44.7, “Exact path pricing”), and priced within a tenth of 1 / 1.2 of the sort (the lean of 2026-10-19) reading from the heap (pg_lion.vector_toast_direct off) and inline, where it is the plan too.
  • Regression, vector_toast_corrupt.sql (17 on, pg_buffercache_evict()): pages damaged on disk - a LOC entry’s run past the TOAST relation’s end, on another value’s page, at a line pointer the page lacks, naming a value no row stores: each falls back; a LOC entry naming another row’s value and place: the return-time check’s ERRCODE_INDEX_CORRUPTED; an entry that is not one (flags, runs miscounted): ERRCODE_INDEX_CORRUPTED; a REFINE item’s place wrong: the rerank falls back; naming another value: the error; and a format-4 index, refused until a REINDEX.
  • Isolation, lionvec_toast_scan.spec: an exact-path scan parked at lionvec-exact-located while rows are deleted, the TOAST relation vacuumed alone and new rows take the freed places, its rows compared with a sort under the same snapshot and its lock on the TOAST relation shown held to its transaction’s end; and a probing cursor parked with places queued while the table is vacuumed and its REFINE items, heap TIDs and TOAST space are taken again.
  • Recovery: phase 5, above.
  • Unit (test/unit/vec_test.c): the chunks a vector takes at each width and their sizes, the runs a REFINE slot holds, runs made from chunk places against a model, a run list’s checks, and the LOC pages chosen for a scan’s members against a brute-force model.

Open questions

  • The append area’s re-sort (above): now, or once an update-heavy measurement asks for it?
  • The second level (above): a binary search of the static level or a level over it, and from how many rows - neither needs a format change.
  • Serializable: a member read by place and not returned takes no tuple SIREAD lock; the index-wide predicate lock covers a change of its vector (an insert into the index), and a delete changes nothing the query returned. Argued, not shown: a spec of a serializable pair would show the read-write conflict still found.
  • An option not to keep a locator (a reloption read by the build), for a table whose vectors lie in TOAST but whose queries never take the exact path - the locator then costs its pages and inserts their entries for nothing but the rerank.
  • The rows returned are detoasted once more by the query’s projection: the index scan’s target list computes embedding <=> q for the ORDER BY, which the executor never takes from xs_orderbyvals (only a recheck reads them). Ten rows a query, a TOAST index descent each and its chunk page, after the scan - 22 of the exact path’s 1,631 pages from the disk, whose chunk pages it has just read. The scan has every one of the values in hand; handing them to the projection would need the planner’s help (a target list naming the scan’s distance), not the index’s.

44.14 Progressive refinement from planes 2 to 8 (evaluated 2026-10-26; not done)

The idea: keep a row’s 8-bit codes (§44.3) as seven more bit-planes, planes 2 to 8 of each code, most significant first - plane 1 being the code’s top bit, the sign - so that a refine reads plane 2, bounds the row by the codes' top two bits, and stops as soon as that bound puts the row past the k-th; else reads plane 3; and so on, plane 8 being the codes as a refine reads them today (§44.4), 1,536 bytes at 1536 dimensions. It was evaluated before the format was touched, with a frontend prototype on lionvec’s own encoder (test/unit/vecplanes_eval.c, make vecplanes-eval; not in make unit), on DBpedia’s rows and on vec_test’s kinds of data, and with the scan’s refines timed on a server. The format is not changed: on DBpedia a refine does stop early, but the 8-bit sums are a tenth of a warm query, the larger part of a refine is reading its REFINE item, which planes do not shorten, and a kernel over today’s bytes takes about as much off as the planes would, with no new format. That kernel is built (2026-10-27, lionvec_dot8(); §44.3, “Kernels”), and the sums it replaced are kept as the unit test’s reference, nothing else.

The estimator. The top m bits of a code c leave 28-m steps; their middle is

r_m(c) = -4 + (c_m + 2^(7-m)) / 32 = (-4 + 2^(2-m)) + sum over p <= m of 2^(3-p) b_p(c)

c_m the code with its 8 - m low bits cleared and b_p its plane-p bit, so that r_8 is lionvec_recon8[] and r_1 is 2 sign©, and r_m is linear in the planes: for any v,

<r_m, v> = (-4 + 2^(2-m)) sum(v) + sum over p <= m of 2^(3-p) S_p(v)

S_p(v) the sum of v over the dimensions plane p has set - one more S_p a plane. The estimate is §44.3’s with r_m for the code’s vector, |x| <r_m, qc> / (|r_m| alpha_m) with alpha_m = <r_m / |r_m|, x / |x|>, and its error, by the same argument (the part of x / |x| that r_m misses, sqrt(1 - alpha_m2) long, points nowhere in particular after the rotation), a Gaussian of standard deviation

|x| / alpha_m * sqrt(|qc|^2 (1 - alpha_m^2) / (dimsp - 1) + varq)

varq the query’s own rounding a dimension, e^2 / dimsp, as plane 1’s (§44.3), when the S_p come from a quantized query, and 0 when they come from its floats; a bound of epsilon of those either side holds with the Gaussian’s probability, whatever the data. Each prefix needs its own alpha_m and |r_m| a row, both computed by the build from x: six pairs more than the REFINE item holds (alpha8 and |r| are m = 8’s), 48 bytes. For a standard Gaussian coordinate (numerical integration) alpha_m is 0.7979, 0.8681, 0.9607, 0.9897, 0.9974, 0.99935, 0.99983, 0.99996 for m = 1 .. 8, the relative error sqrt(1 - alpha_m^2) / alpha_m 0.756, 0.572, 0.289, 0.144, 0.072, 0.036, 0.018, 0.0094: near plane 1’s at two planes, a fifth of it at four, an eightieth at eight, halving a plane from three on. The rows' own, measured on 2,000 clustered rows of 1536 dimensions, are those to the third digit (0.79763, 0.86831, …, 0.99996). Over every pair of those rows and 12 queries the bound was outside 1, 2 and 3 standard deviations for 0.295 - 0.308, 0.037 - 0.045 and 0.0016 - 0.0028 of them at every m from 2 to 8, floats or quantized, against the Gaussian’s 0.317, 0.046 and 0.0027 (the same at 384 dimensions); eight planes' estimate from the S_p was lionvec_estimate8()’s to the bit, and the codes' top bit was the sign bit in all 3,072,000 dimensions. (A zero coordinate, or a negative one that rounds into code 128, would differ; a format would make the encoder give the top bit the sign’s.)

Its sums: popcounts, not masked floats. S_p from the floats is a masked sum of dimsp floats a plane, in double in 16 lanes as lionvec_sum8()’s; the prototype’s portable, AVX2 (a mask, compare and AND) and AVX-512VL (a masked add, 256 bits) versions agree to the bit, and a plane takes 245 ns with AVX-512, 432 with AVX2 and 1,820 portably at 1536 dimensions warm, against the whole 8-bit sum’s 326, 406 and 890: a plane costs most of what all eight do, and the planes cost more than the bytes they save. From the query quantized to 8 bits a dimension over its own range, as 8 interleaved planes (plane 1’s quantization, four bits wider), S_p is qlo popcount(P_p) + qdelta lion_planes_dot_interleaved(P_p, Q, dimsp / 64, 8): integer counts in every implementation, and the rest a few operations in double, as deterministic as lionvec_estimate1() under -ffp-contract=off. 32 ns a plane with AVX-512, 56 with AVX2. Its rounding adds about three quarters of the codes' own variance at plane 8, a sixth at plane 7 and next to nothing below, and the bound, which counts it, stays as calibrated (the ranges above are the quantized sums' as well as the floats'). No new kernel: the weighted sum of k = 8 interleaved planes is lion_planes.c’s general pass.

The planes a refine needs. A best-first scan refines a row, or a centroid less the margin, whenever its lower bound lies below the key the scan stops at, about the k-th row’s; so a progressive refine reads planes 2 to m until the prefix’s lower bound passes that key, or all seven, and a row whose plane-1 lower bound passes it is never refined. The prototype keys every row against the query exactly, takes the tenth least key as the stopping key, and for each row and centroid that plane 1 leaves below it records the first plane whose bound passes it: the bound LIONVEC_EPS8(eps) wide at every plane, as the codes' is (“2 eps”), or eps wide at planes 2 to 7, as plane 1’s (“eps”), eps the default 4; the S_p quantized. Partitions are k-means' (three iterations over a strided sample of at most 32,768 rows, exact distances, spherical for cosine) at the default 8 sqrt(n), no spill, and the centroids' bound comes off the margin of 3 as the scan’s does (lv_part_key()). DBpedia is §44.9’s text-embedding-3-large set, its 115,186 rows and 200 held-out queries as knn_dbpedia.py holds them out, cosine; vec_test’s kinds are 20,000 rows and 40 queries of 1536 dimensions, cosine: “clusters” make_clusters()’s 12 centres, “random” isotropic Gaussian rows (random unit vectors), “tight” 200 centres with each row 0.4 of a centre’s length from its own. “Refined” is the mean a query, “past today” the share the 8-bit codes put past the stopping key, then the share past by planes 2 - 3, 2 - 4, 2 - 5 and 2 - 6, and the planes and bytes a refine reads on average, at 2 eps and at eps:

data refined past today by 3 / 4 / 5 / 6 (2 eps) planes, bytes (2 eps) planes, bytes (eps)
DBpedia, every row 87.9 88.0% 48.9 / 76.3 / 83.8 / 86.4% 3.17, 609 2.32, 446
DBpedia, rows of the 27.1 partitions probed 63.1 86.2% 45.1 / 73.5 / 81.4 / 84.3% 3.30, 634 2.47, 474
DBpedia, centroids (2,715) 98.9 72.6% 31.3 / 56.7 / 65.9 / 69.8% 4.05, 777 3.34, 641
clusters, rows 1,667 99.0% 0.1 / 26.4 / 81.6 / 95.5% 3.98, 765 2.97, 570
clusters, centroids (1,131) 94.0 0.2% 0 / 0 / 0 / 0% 7.00, 1,344 7.00, 1,343
random, rows 8,687 99.9% 62.4 / 96.0 / 99.2 / 99.7% 2.43, 466 1.50, 288
random, centroids 1,128 33.9% 0.4 / 6.7 / 17.8 / 26.1% 6.18, 1,187 5.83, 1,119
tight, rows 101 80.0% 0 / 0 / 2.0 / 33.4% 6.00, 1,152 5.17, 992
tight, centroids 5.8 0% 0 / 0 / 0 / 0% 7.00, 1,344 7.00, 1,344

So on DBpedia a refine does stop early: half the rows at plane 3, three quarters by plane 4, a centroid at four planes on average, 40 to 51% of the codes' bytes (the 12% of rows that are fetched, and the 27% of centroids probed, read all seven). The model is near what the scan does: on the server below it refined 85 rows and 119 centroids a query and probed 35 partitions, so that 70% of the centroids it refined were passed by (§44.9’s 2026-10-15 count: 88 rows, 123 centroids, 36 partitions); the model refines 88 rows - 63 in the 27 partitions a margin alone probes, with no spill - and 99 centroids, 73% passed by. The floats' S_p give the same shares to a tenth of a point. Synthetic data brackets it: random vectors, whose distances plane 1 hardly tells apart, stop at plane 3; tight clusters, whose refined rows are the cluster’s own and nearly as near as the tenth, at 6; and where the margin is wide against the centroids' spread (clusters, tight) a refined centroid is a probed one and reads everything. At 384 dimensions the shares are within two points of these; L2 on the clusters, 3.98 planes, the same.

What a refine’s sums cost, ns at 1536 dimensions, warm (one row again and again), from a pool of 16,384 rows (24 MB of each layout: past a core’s 2 MB of L2, within the L3) one at a time in a random order, and with the lines it reads flushed first (clflush), the median of 9 rounds, on the Emerald Rapids of §50 pinned to a core; “bytes, 16-bit query” is the comparison below:

AVX-512 AVX2 POPCNT portable
lionvec_sum8() 326 / 682 / 1,399 406 / 767 / 1,535 888 / 1,312 / 1,867 890 / 1,142 / 1,803
planes 2 - 3, quantized 58 / 230 / 609 109 / 332 / 652 184 / 349 / 675 552 / 757 / 955
planes 2 - 4 86 / 328 / 698 161 / 441 / 774 276 / 477 / 1,102 828 / 1,078 / 1,317
planes 2 - 8 227 / 572 / 1,028 386 / 729 / 1,420 657 / 921 / 1,506 1,965 / 2,216 / 2,554
bytes, 16-bit query 60 / 373 / 747 60 / 323 / 912 442 / 673 / 1,561 509 / 718 / 1,691

Weighted by DBpedia’s planes, a row’s sums would take 95 / 313 / 737 ns with AVX-512 where they take 326 / 682 / 1,399, a centroid’s 124 / 370 / 819; with AVX2 172 / 439 / 842 and 220 / 510 / 983 against 406 / 767 / 1,535.

How much of a scan that is. Timed inside the scan (a build of this tree with clock_gettime() around lv_refine(), lv_refine_part() and their parts, not kept), PostgreSQL 16, 2 GB of shared buffers, every relation prewarmed, AVX-512, LIMIT 10 at the defaults (margin 3, patience 30, spill), two or three rounds after a warming one. DBpedia’s 115,186 rows, 2,715 partitions: a query took 1.44 ms at the median and refined 85.2 rows and 118.5 centroids (35.1 partitions probed, 10.8 rows fetched). The rows' refines were 15.8% of its time and their sums 5.0%; the centroids' 11.9% and 6.1%. A row’s refine took 3,125 ns: 108 looking into the queue for what to read ahead, 1,810 reading its REFINE item (the read, the lock, the page’s and the item’s checks), 988 the sum and 66 after; a centroid’s 1,697, its sum 866. On 100,000 random unit vectors of 1536 dimensions (50 queries) a query took 57 to 67 ms, refined 14,082 rows and 2,413 centroids, and the rows' refines were 63 to 65% of it, their sums 22 to 25%: 2,982 ns a refine, of which 1,474 reading the item and 1,039 the sum.

So progressive planes would take 5 to 5.5% off a warm DBpedia query (about half of the sums' 11.1%, at the pool’s ratios or the flushed ones) and 13 to 16% off the random vectors' - before anything a real scan would add: the scan does not know the stopping key until it stops, so a refine must stop against what it knows (the next key in its queues, or the tier’s k-th least upper bound, §44.4), and a row put back with planes unread costs another 1.5 to 1.8 us reading its item if it comes back, which the model does not charge; nor its partial sum and planes read, which a queue item, held to 48 bytes, has no room for. Cold, nothing: a REFINE page is read whole whatever part of an item a refine looks at.

Why not.

  • A kernel over the codes as they are takes nearly as much. Against the query rounded to 16-bit integers, sum c_i w_i is exact in integers in every implementation - the same to the bit by construction, as plane 1’s counts are - and
  • What a refine costs most is not its sum. Reading the REFINE item is 1.8 us of a row’s 3.1; fewer bytes of the item leave it as it is.
  • The format would cost: 48 bytes a row of alpha_m and |r_m|; a second quantization of the query (8 planes, 1.5 KB at 1536 dimensions); plane 1 against that query as well - the CODE item’s plane 1 is gone by the time a row is refined, so either a pass over it at every probe for the rows a probe keeps, or plane 1 again in the REFINE item, which takes the most dimensions from 8128 to 8064; a row’s planes read and partial sum in its queue item and its progress entry; a refine stopping against a key it does not know (above); and the build, inserts (SOAR’s decoded centroids, §44.3), VACUUM’s checks, lion_index_verify() and the SQL diagnostics reading planes where they read bytes - for a few percent, and for data like DBpedia’s.

When to look again: if reading a REFINE item became cheap enough that the sums were most of a refine (refines batched by page, say, or the codes nearer the queue), the bytes a refine reads would be what it waits on, and planes stop at 40 to 51% of them on DBpedia; or for vectors whose distances plane 1 tells apart as badly as random ones', where rows' refines are two thirds of a query and planes stop at 30% of the bytes. The 16-bit integer kernel for lionvec_sum8() was the change worth making now: made on 2026-10-27, as lionvec_dot8(), and measured in a scan (§44.3, “Kernels”).

The prototype (test/unit/vecplanes_eval.c): part 1 the prefixes' alphas and the bounds' calibration on clustered rows, part 2 the planes needed on vec_test’s three kinds at 384 and 1536 dimensions (and L2 at 1536), part 3 the costs (--costs alone); --file ROWS QUERIES DIMS runs part 2 on raw float4 vectors, cosine - DBpedia’s here, written from the shards as knn_dbpedia.py reads them, the 200 queries of np.random.default_rng(0).choice(115386, 200, replace=False) apart. It checks that eight planes are the floats' estimate (lionvec_sum8()’s, to 1e-6 of its error; 2026-10-27: and lionvec_estimate8()’s but for its query’s rounding, within that rounding’s reach, 5e-3 of the error at most), that its masked sums and integer dot agree with their portable forms to the bit, and that the bounds are calibrated (at most 0.40 outside 1 standard deviation and 0.006 outside 3), and exits 1 if not; it runs in a few minutes, DBpedia’s file in five (its k-means most of it). Its kernels are the evaluation’s own, not lionvec’s; the flushed column needs the x86-64 build (clflush).

45. Partial indexes in the count pushdown (2026-10-19)

Every lookup the count planner made skipped partial lion indexes: lion_find_roaring_index() and lion_match_index() took the first valid, NON-PARTIAL index on a column. A workload whose queries all carry the same few dense filters - three boolean or low-cardinality columns that keep most of the table, beside the filters and the GROUP BY that change from query to query - gained nothing from a partial index on those filters, CREATE INDEX ... USING lion (g, c1, c2) WHERE f1 AND f2 = 'a' AND f3 = 'n', whose sets hold only the rows the defaults keep: the count read the whole index and intersected the defaults' sets, each nearly the table, into every count. Now a count whose restrictions imply a partial lion index’s predicate may read that index and leave out the restrictions the predicate implies, as core leaves them out of a partial index scan’s indrestrictinfo - wherever every row the count can reach is one the predicate holds for. lion_plan_partial.c holds the rule; the planner’s lookups, the cost and EXPLAIN take the index as they take any.

45.1 Held to the predicate’s rows

Let Ω be the rows a partial index P holds: the table’s rows its predicate is TRUE for. A set read from P for a clause c holds the rows of Ω that c selects - it agrees with c inside Ω and is empty outside it - and a restriction left out because the predicate implies it is TRUE inside Ω. Inside Ω the count therefore computes the query exactly. It goes wrong only by counting a row OUTSIDE Ω, where nothing it read from P says anything, and that happens wherever the rows a count adds up do not come from one of P’s sets:

  • the restrictions are left out and the rest is read from a whole index: WHERE f1 AND f2 = 'a' AND f3 = 'n' AND y = 2, P on the three without y: y = 2 from the whole index counts every row with y = 2;
  • a set of P’s is subtracted, not intersected: a = 1 AND x <> 3 with P WHERE x IS NOT NULL reading x. <> is the rows of the other sources less x = 3 and less the NULL entry (§35); P has no NULL entry, so the rows with a = 1 and x NULL - for which x <> 3 is NULL - are counted;
  • the count of every row (§14, §35) or a complement (§35, “Complements in an OR”) taken from a whole index;
  • a multi-key query known only at run time (§17), or a literal one with a prefix too common to expand, which may turn out to narrow nothing, and a count with nothing that narrows is the heap scan of lion_count_scan_filtered(): every row.

So the rule, checked per relation counted - the table, or each leaf partition with its own partial index - and per RUN of the count (the WHERE’s, run 0, and each filtered count’s, §10 “Filtered counts”): a run that reads P, or any run at all when restrictions were left out, must be HELD to Ω by one of

  • P’s entries driving it: a driving column read from P - a GROUP BY column’s walk (§10, §20), the decoded walk’s columns (§34: a row outside Ω has no value in P’s column and is in no combination), the column a count(DISTINCT) walks (§26), the column a sum over every row walks (§14, §35), the FK-side join’s fact group (§27). Every count of the run is ANDed with one of P’s entries, which hold Ω’s rows only;
  • a positive clause of the run read from P, outside any OR: an equality (the FK-side join’s lookup of a key among them, §27), an IN list, IS NULL (P’s NULL entry holds Ω’s NULL rows), a range taken as a source (§32) - each a set ANDed into every count of the run;
  • an OR with no complement leaf, each of whose arms the partition keeps has such a leaf.

Never a multi-key clause (above), a negated one - <>, IS NOT NULL, IS DISTINCT FROM, a NOT of a multi-key query - which subtracts, or an OR with a complement arm, which counts rows outside a set. A clause the partition’s bounds leave out (§16) reads no index there and holds nothing. And a sum over every row drives a filtered count’s run only when the run has no source of its own: one that has (an equality, a list, IS NULL, an OR of no range and no complement arm) is counted as an intersection, not by the walk (lion_count_runs(), lion_run_has_plain_source()), and needs a hold of its own.

lion_partial_anchored() checks it once the targets are collected (lion_count_path_targets(), lion_fkjoin_setup()); false declines the attempt, and the plan made with whole indexes stands. So does a relation counted that the scope holds no index for, when restrictions were left out: they are left out only where every relation of the scope has one, and that relation’s rows outside the predicates would be counted. An attempt that reads no partial index and leaves nothing out is declined too: it is that plan again. The rule is a planning rule only: the executor counts what the plan names, as it always did.

45.2 Planning: one attempt per candidate

  • Candidates (lion_partial_find()): for each relation counted, its valid partial lion indexes (not hypothetical: they have no entries) whose predicate its restrictions imply - predicate_implied_by(indpred, baserestrictinfo, false), core’s own proof (check_index_predicates()), with, for a leaf partition, its partition constraint, which every row of it satisfies. Join clauses prove nothing about the rows a count reads. The smallest first, by the index’s rows.
  • The attempts (lion_try_count_path()): the count path is planned with whole indexes, as before, and then once for each rank up to LION_PARTIAL_TRIES (3): each relation reads its rank'th candidate, or whole indexes alone where it has none at that rank. add_path() keeps the cheapest; on an exact tie the whole-index plan, made first, stays.
  • The scope (lion_partial_enter(), lion_partial_leave(), left in a PG_FINALLY as the expression columns' scope is, §41): lion_find_roaring_index() and lion_match_index() try the relation’s scope index first and then whole indexes; no other partial index is read. lion_vcol_collect() takes the scope index’s expressions too, so a partial expression index answers its expression (exprindex.sql).
  • Leaving restrictions out (lion_partial_drops(), in lion_count_path_where() before a clause is numbered, so nothing downstream sees it): a restriction is left out when every relation counted has a scope index whose predicate implies it (mapped onto each partition’s columns), it holds no mutable function and no subquery (core keeps those in indrestrictinfo too), and it names no column the query prints outside its aggregates - the target list and HAVING - other than a GROUP BY column: a column a restriction pins is printed from the key the index stored (§10), which needs the restriction. A restriction left out is not a source, is not priced, and EXPLAIN lists it. Only the WHERE’s restrictions: a FILTER condition is never left out.
  • count(col) (lion_count_path_pred_nonnull()): a column the predicate proves non-NULL (col IS NOT NULL implied) counts every row, as a NOT NULL column does (§10): the query implies the predicate, so every row the count reaches has the column.
  • The sum over every row (§14, §35): lion_count_all_driver() prefers a column of the scope index, so that the walk holds the count; among those, as before, the fewest entries.

45.3 What the change leaves as it was (the audit)

  • Every “every row”: the count of every row and the sum over all of a column’s entries (§14, §35) hold the count only when P drives them; the NULL entry and IS NULL read from P hold it (Ω’s NULL rows); negated sources and complements (§35), <> among them, never hold it, whatever index they read; a count(DISTINCT)’s existence tests (§26) are driven by its column’s entries, and held when that column is P’s; residual clauses (§10) are tested on the heap rows the other sources select, which are Ω’s once the run is held. Each case is in countpartial.sql.
  • Uniqueness proofs stay non-partial: count(DISTINCT pk) as count(*) (§26) and the FK-side join’s dimension key (§27) still need a whole unique btree index.
  • The visibility interlock (§9): a partial lion index’s entries are written and removed by the same paths as a whole one’s, and a TID from it is judged visible the same way. A HOT update cannot move a row into or out of Ω behind the index’s back: lion is not a summarizing access method, so the columns of its predicate block HOT (RelationGetIndexAttrBitmap()), and a visible version reached through a P’s TID satisfies the predicate.
  • Top k (§36): an entry’s TID count still bounds its group’s count: the rows counted are in the entry.
  • reltuples (§18): the one fix. ambuild reported the postings it wrote - one a key column, one an element - as index_tuples, so a partial index’s reltuples, which the planner sizes it by (estimate_rel_size()), was a multiple of its rows until the first VACUUM or ANALYZE. The build now reports the rows it added (lion_spool_nrows()), serial and parallel, as VACUUM does.
  • Estimates: n_distinct from the directory (§33), the selectivity hooks and the facet counts (§7) still read whole indexes only: a partial index’s keys are not the column’s.
  • EXECUTE (§10, “Privileges”): lion_replaced_functions() walks every restriction, left out or not, so a function the query calls is still checked.
  • Generic plans: a Param proves no predicate that needs a constant, so a generic plan of st = $1 reads the whole index; a strict clause on a Param does prove col IS NOT NULL, for every value of the Param, and x > $1 may read a partial index WHERE x IS NOT NULL.

45.4 Cost

The model prices a partial index by its own sizes (lion_plan_cost.c):

  • lion_index_rows(): the rows the index holds - the table’s for a whole index; for a partial one idx->tuples (its reltuples, rows since 45.3), or the table’s rows times the predicate’s selectivity where it has none yet, as core prices a partial index scan;
  • a clause’s set (lion_cost_set_clause()): its members are the index’s rows times the clause’s share of them, sel(pred AND clause) / sel(pred) (lion_index_clause_share()), and its containers and pages follow from those;
  • a group’s rows in the walk, the pairs of two GROUP BY columns and the decoded walk’s columns: the index’s rows over the column’s entries.

What the model gains is the work the plan no longer does: the restrictions left out are not sources, and the sets left are smaller. There is no bias toward the partial index beyond that.

45.5 The plan and EXPLAIN

custom_private member 22, LION_PRIV_PARTIAL (shape 27): NIL, or a list of two - the OIDs of the partial indexes the count reads, and the restrictions left out, deparsed, as String nodes. TLKINDS moves to 23, the members are 24, the magic is 0x5242491b; the codec checks the list’s shape and its OIDs. Only EXPLAIN reads it: the targets name their indexes, partial or not, as before.

EXPLAIN names a partial index among the Lion Indexes as it names any (pc_p1.a (a = 3)), prints the restrictions left out under Implied by Index Predicate, and for a partitioned table, whose Lion Indexes line names no index, the leaves' partial indexes under Partial Indexes:

 Custom Scan (LionCount)
   Lion Indexes: pc_p1.g (g), pc_p1.b (b = 3)
   Implied by Index Predicate: act, (st = 'on'::text), (kind = 'k0'::text)
   Group Key: g

pg_lion.enable_count_partial (on) turns the attempts off: the count reads whole indexes only.

45.6 Measured

4,000,000 rows of a synthetic table (f1 boolean, true in 93% of rows; f2, 'a' in 90%; f3, 'n' in 92%; together 77%), g of 50 values, c1 of 10, c2 of 1,000, c3 of 5; a whole index (f1, f2, f3, g, c1, c2, c3) and a partial one (g, c1, c2, c3) WHERE f1 AND f2 = 'a' AND f3 = 'n' (51 and 36 MB), VACUUM (FREEZE, ANALYZE), no parallel workers, the release build, warm cache; the best of seven EXPLAIN ANALYZE runs, every query with the three defaults in its WHERE:

query (beside the defaults) whole index: containers, ms partial index: containers, ms chosen
count(*) ... AND c1 = 3 1,840, 0.47 460, 0.20 partial
count(*) ... AND c1 = 3 AND c3 = 2 2,300, 0.57 920, 0.49 partial
count(*) ... AND c2 IN (5, 50, 500) 2,760, 0.48 1,380, 0.07 partial
count(*) ... AND c2 < 100 183,922, 27.9 45,935, 2.4 partial
g, count(*) ... GROUP BY g 25,300, 5.8 23,000, 3.0 partial
g, count(*) ... AND c1 = 3 GROUP BY g 25,760, 7.0 24,380, 5.7 partial
c1, c3, count(*) ... GROUP BY c1, c3 8,740, 28.9 6,900, 22.3 partial
count(DISTINCT c2) ... AND c1 = 3 9,272, 3.0 4,732, 1.7 partial
three counts, two FILTERs 5,060, 1.3 3,220, 0.85 partial
count(*) (the defaults alone) 1,380, 0.33 2,300, 0.54 partial
count(*) ... AND c1 <> 3 1,840, 0.44 4,600, 2.4 partial

With a third of the heap’s pages not all-visible (10,000 rows updated), where both plans recheck the same rows in the heap: c1 = 3 7.4 against 7.1 ms, the IN list 1.7 against 1.5, GROUP BY g 110 against 89, the range 79 against 43.

The last two rows were the model’s, not the index’s: the count of every row of P is a sum over the fewest-entry column’s entries (c3’s five, 2,300 containers), which the model priced at a third of intersecting the three dense sets (1,380) though it took 1.6 times as long; and the subtraction of a <> from each of those entries was not priced at all - nor over a whole index, where count(*) WHERE c1 <> 3 cost what count(*) costs (§35) - so the partial plan, which subtracts per entry, was taken where the whole index’s one merge is five times faster. Both are priced now (§10, “Dense sources”: the sum’s pages by the bytes its sets take, the AND of dense sets by their words, negated sources where the merges subtract them), and the whole index is taken for both. Measured again on a table of the same shape (4M rows on 25,478 pages, 399 container keys; the two indexes 42 and 35 MB), medians of nine runs, the plans chosen by the library before that change and after it; the times are the plans' as they are now:

query (beside the defaults) whole index, ms partial index, ms chosen before chosen now
count(*) ... AND c1 = 3 0.42 0.16 partial partial
count(*) ... AND c1 = 3 AND c3 = 2 0.47 0.33 partial partial
count(*) ... AND c2 IN (5, 50, 500) 0.41 0.06 partial partial
count(*) ... AND c2 < 100 22.6 2.1 partial partial
g, count(*) ... GROUP BY g 5.5 2.9 partial partial
g, count(*) ... AND c1 = 3 GROUP BY g 6.4 4.9 partial partial
c1, c3, count(*) ... GROUP BY c1, c3 11.1 15.3 partial partial
count(DISTINCT c2) ... AND c1 = 3 3.0 1.6 partial partial
three counts, two FILTERs 0.96 0.66 partial partial
count(*) (the defaults alone) 0.30 0.47 partial whole
count(*) ... AND c1 <> 3 0.40 1.60 partial whole
count(*) ... AND c2 <> 7 AND c1 <> 3 0.49 1.90 partial whole

The GROUP BY of two columns keeps the partial index, which is now the slower of the two: the whole index’s plan of it took 30.4 ms before and takes 11.1, the partial index’s 24.1 and 15.3 (45.8).

With two parallel workers allowed, the whole index’s GROUP BY with c1 = 3 runs in parallel (§10, “A GROUP BY in parallel”) and the partial index’s does not (45.8), and the partial plan is still chosen: 6.0 ms against 5.1 serially on the second table, and as much with c1 = 3 AND c3 = 2. GROUP BY g alone, whose whole-index plan went parallel and took 10.7 ms against 3.1 on the first, stays serial over both indexes now that its serial price no longer overstates it (§10, “Dense sources”): 5.9 ms against 3.0.

45.7 Tests

test/sql/countpartial.sql compares every query, as a multiset both ways round, with a sequential scan with the pushdown off, on random data with deleted rows and rows moved into and out of the predicates on pages the visibility map no longer vouches for, and prints the custom scans, the indexes read and the restrictions left out: a predicate the restrictions match, one stronger restrictions imply (x > 20 under WHERE x > 10) and one they do not; a predicate on indexed columns and on a column no index has; a predicate that is a <>, implied by an IN list; NULLs, <>, IS DISTINCT FROM, NOT IN, complement ORs, ORs across the partial and the whole index, a multi-key clause; GROUP BY one, two and three columns (the decoded walk mixing the partial and the whole index), by the predicate’s own column, by a column a restriction pins (printed and not), HAVING, the top k, coalesce, SELECT DISTINCT; filtered counts beside a sum over every row; count(col) the predicate proves non-NULL; count(DISTINCT); residual clauses; generic and custom plans of prepared statements; the FK-side join, inner, outer, semi and anti, with the partial index on the fact side and a partial lion index on the dimension; a partitioned table with partial indexes on the parent and one on a single partition; EXPLAIN, and the setting off; and, with the whole index dropped, the partial index’s sums over every row and its subtractions from every entry, which the model leaves to the whole index’s merge where there is one (45.6). exprindex.sql: a partial expression index whose predicate the query implies answers, one whose predicate it does not imply declines.

45.8 Not done

  • One partial index a relation an attempt, three attempts at most: two partial indexes are never read in one count (one for the GROUP BY column, another for a clause), nor a partial index beyond the third smallest a relation has.
  • FILTER conditions are never left out, only the WHERE’s restrictions.
  • The FK-side join’s dimension is an ordinary plan (§27), whose scans read a partial index as core plans them (LionOrdered over pcd_region in the test); the rule above applies to the fact side.
  • Uniqueness proofs (45.3).
  • A GROUP BY whose WHERE is left with a single source after the restrictions go is priced as a merge per group rather than one collection of the WHERE (§10, “WHERE Sets Collected”); on 30,000 rows that prices the whole index’s four-source WHERE, collected once, below the partial index’s one set, which on four million rows it no longer does.
  • The GROUP BY of two columns (45.6) over the partial index is priced at 1,438 units for 15.3 ms, 94 a millisecond, and the whole index’s at 3,687 for 11.1: a GROUP BY’s own work is priced below its time (§10, “Dense sources”), the pairs' most; and c1’s 774 rows a container key lie just past the 768 above which a build writes a NARROW of three words a block rather than an ARRAY, so 1,700 of its 3,990 containers are ARRAYs, which the model takes for NARROWs.
  • A parallel GROUP BY (§10) needs a WHERE that is collected - two sources or more, or a union
    • and a positive one: a partial index that leaves one source or none after the restrictions go is counted serially, against the whole index’s parallel plan (45.6).
  • Estimates still come from the table’s statistics (45.3).

46. The dimension by probe (LionProbeJoin, lion_probejoin.c; 2026-10-23)

The shape is

SELECT CASE WHEN d.region IN ('north', 'east') THEN 'ne'
            WHEN d.tier = 1 THEN 't1' ELSE 'other' END AS k,
       f.channel, count(*)
  FROM fact f JOIN dim d ON f.dim_id = d.id
 WHERE d.active AND d.tier < 4
   AND NOT EXISTS (SELECT 1 FROM blocked b WHERE b.dim_id = f.dim_id)
 GROUP BY 1, 2

  • a GROUP BY over a join of a fact table to a dimension by the dimension’s primary key, its keys CASE or boolean expressions over the dimension’s lion-indexed columns, perhaps beside the fact’s own columns; count(*); a WHERE on the dimension its lion indexes answer; perhaps a NOT EXISTS over a small table keyed by the dimension’s key; and no lion index on the fact. Nothing of lion’s so far took it: the FK-side join (§27) reads the fact’s rows from a lion index on its key; a count by cells (§10, “Counts by cells”) counts one table; the nested loop’s probe (§40.10) needs a B-tree that covers what the query reads of the dimension, and the group keys read the lion-indexed columns themselves. Core’s plan is a nested loop over the fact’s rows into the dimension’s primary key - an Index Scan a fact row, the dimension row fetched from the heap for the Filter and for the group keys' columns - and on a dimension far larger than memory that is millions of random heap pages, minutes cold. A hash join reads the dimension’s WHERE whole, which is most of that heap.

    HashAggregate Group Key: CASE WHEN ((d.region = ANY (‘{north,east}’::text[]))) THEN ‘ne’::text WHEN ((d.tier = 1)) THEN ‘t1’::text ELSE ‘other’::text END, f.channel -> Custom Scan (LionProbeJoin) on dim d Probe Index: dim_pkey Probe Cond: (d.id = f.dim_id) Lion Cond: (d.active AND (d.tier < 4)) Lion Indexes: dim_active, dim_region_tier Cell Predicates: (d.region = ANY (‘{north,east}’::text[])), (d.tier = 1) Cell Indexes: dim_region_tier Anti Join Cond: (b.dim_id = f.dim_id) -> Seq Scan on fact f -> Seq Scan on blocked b

LionProbeJoin takes the join and the group keys' predicates; core’s Agg does the rest. For each fact row, read from core’s own path of the fact, it looks the key up in the dimension’s primary key, tests the TID found against lion sets of the dimension built once before the first row - the WHERE’s, and the LABELS that give each column the predicates read its outcome - asks the visibility map, and emits the fact’s columns and each predicate’s truth. The heap is read for a page the map does not call all-visible, and for every row found when the WHERE’s set is inexact, to recheck the WHERE on the heap tuple (Rows Removed by Lion Recheck). The group keys are computed above the node from those truths, setrefs.c matching each predicate inside them to the node’s column for it (LION_TL_CELLPRED, as §10’s cells are matched).

  • What qualifies (lion_fkjoin_recognize_probe(), pj_try_shape()). At the GROUP BY of a join rel (lion_create_upper_paths(), UPPERREL_GROUP_AGG, no grouping sets, no set-returning target): an inner join of exactly two tables, either way round, joined by exactly one clause, a strict equality of plain Vars whose dimension side is the one column of a unique B-tree (lion_column_unique_index()) and whose operator is an equality of that B-tree’s family (the probe’s operator is the family’s member for the fact column’s type); no placeholder, no pseudoconstant clause, no LATERAL reference. The dimension is a plain heap table, not a parent, not the query’s result relation, with no security quals or barrier clauses and no TABLESAMPLE, and every one of its restriction clauses is answered by a lion access with nothing left to a filter (lion_ordered_where_access(), below), or it has none. The grouping input target - what the node emits - holds expressions each of the fact’s columns alone or of the dimension’s alone, none volatile, with no SubPlan and no placeholder; the dimension’s are functions of predicates §10’s analysis decides (lion_cells_analyze(): equalities, IN lists, null tests, booleans, <>, ranges of a scalar column - “Ranges”, below - and @>, && or @@ of a multi-key column, with constants, each of a column with a non-partial lion index). A dimension column outside a predicate - grouped by itself, under a CASE’s THEN, an aggregate’s argument - keeps the node out: its value is not in any set. The fact’s path returns plain Vars, the join key among them. There must be a set to build: a WHERE on the dimension or a predicate, or core’s nested loop already is the plan. The fact needs no lion index; where it has one the FK-side join is offered too, and the price chooses.
  • The NOT EXISTS (lion_probe_anti()). At most one anti join, from a sublink, whose syntactic lefthand is the two tables and whose righthand is the rest of the query: a small inner side, one table or a join of several, each passing §27’s member checks, unparameterized, joined by exactly one clause - a strict equality, of the join’s collation, between a Var of the inner side’s target and the fact’s key or the dimension’s (b.dim_id = f.dim_id or = d.id) - whose operator is an equality of the probe B-tree’s family for the inner column’s type. Its other clauses are the inner side’s own, in its path.
  • The WHERE’s set (lion_ordered_where_access(), lion_ordered.c). The cheapest of the lion accesses core builds for the dimension’s restriction clauses over its lion indexes alone (§30.2, lo_lion_accesses()) that leaves nothing to a filter (lo_residual()); and, where none does, an AND of one access per clause (lo_where_and()): core’s own AND (choose_bitmap_and()) leaves out a leaf that selects too little to pay for its lookups in a bitmap scan - active of two rows in three - which the probe’s set must answer all the same, or read the heap for it. Each clause the leaves before it do not answer takes the cheapest access core builds for that clause alone. The set is built as §30.4 builds a walk’s (lo_build_node()), exact or not: an inexact one - a condition lion can only bound - is rechecked on the heap tuple, fetched for that, with the original qual (Rows Removed by Lion Recheck). An access with a lossy index clause is not a candidate, in either path (lo_where_exact()): the range core derives from a LIKE prefix over a column of “C” collation (code LIKE 'k1%7' as code >= 'k1' AND code < 'k2') still names the original clause, so lo_residual() counts it answered, while the range’s set holds rows the pattern rejects and calls itself exact - nothing would recheck them, and the node counted them (2026-10-23 review). Every LIKE is lossy in core, an exact pattern too, so a LIKE on the dimension leaves the node unoffered; the range written out is the clause itself, and the node takes it.
  • The labels (pj_labels()). §10’s analysis gives each column the predicates read its OUTCOMES - a scalar column’s values grouped by the truths they give every predicate, outcome 0 any other value’s, and NULL’s; a multi-key column’s leaves true of a row as bits, and NULL past them - and each predicate its truth in every outcome. A scalar column gets a set for each outcome but 0: its values' equality, = ANY of an array where it has several (an OR of equalities where their type has none or they are of two types), ORed with IS NULL where NULL’s outcome is it; a row in none of them has outcome 0. A multi-key column gets a set for each leaf - the leaf’s own clause - and one of IS NULL; its outcome is the mask of the leaves whose sets hold the row, or NULL’s. Each is an IndexPath made for the clause (create_index_path() over a made-up IndexClause), priced as lion prices a set (lion_ordered_set_price()), and built as the WHERE’s is. A label must be exact - §10 takes only predicates a posting set decides - and the node stops with an error if one is not. The truths are §10’s (lion_cells_pred_truth(), the one function for both): true, false or NULL, by SQL’s rules.
  • Ranges (cells_ranged_order(), cells_ranged_outcomes(), lion_plan_cells.c). The node asks §10’s analysis for ranges (lion_cells_analyze(..., ranges)), which a count does not: c < v, <=, >= and > - a BETWEEN is two of them, and a constant on the left is read commuted - of a scalar column other than a boolean, v a constant, the operator a range strategy of the family of the column’s scalar lion index for one type on both sides, whose equality of that type tells the column’s values apart (cells_range_eqop()), and every range of the column one that index answers as a range (lion_match_index(), LION_CLAUSE_RANGE: the strategy, the family’s comparison, the collation). The column is a RANGED dimension. Its PIECES are the m values its predicates name - a bound’s and an equality’s alike - in the family’s order (LION_CMP_PROC), the m + 1 gaps around them (below the first, between neighbours, above the last) and NULL: 2m + 2 of them (cells_npieces()). The values are told apart by the equality (cells_value_no()), so numeric’s 1.0 and 1.00 are one; two the order still calls equal decline the node. Every predicate’s truth is one across a piece: a bound’s by the places - a value at place r is below the bound at place k for r < k, and the values of gap g for g <= k, since they are below the value at place g; an equality’s, an IN list’s, <>’s by the value, a gap naming none; NULL’s by SQL’s rules, a range of NULL being NULL (cells_eval()). The pieces of one signature are an OUTCOME, as a scalar column’s values are, and outcome 0 is the one of the most rows by the planner’s estimates - a value’s by its equality’s selectivity, a gap’s by its two bounds' together (clauselist_selectivity()), never NULL’s alone - so that the labels hold the fewest: for score > 0.6 of scores from 0 to 1, outcome 0 is the rows at or below 0.6, and the label the rows above. Each other outcome’s label is its RUNS of adjacent pieces, ORed: a value alone is its equality, the outcome’s single values gathered into one = ANY; a longer run is the interval from its first piece to its last - >= the value it begins at or > the value below the gap it begins at, <= or < at its end likewise, an end at the column’s first or last gap none - one IndexPath of both bounds, its set built as lion builds a range’s (§28, exact); and IS NULL where NULL’s outcome is it. An empty gap, between 2 and 3 of an int, is an empty set. The order is the index’s, so every value of the column is in exactly one piece and a row in one label at most: numeric’s NaN, which the order puts above every number, is in the last gap, where score > 0.6 is true of it, as the operator says. A bound that is no Const - a Param, CURRENT_DATE, now() - keeps the node out: the outcomes are places in the order of the values, and a value known only at run time has none when the plan is made (“Not done”).
  • A fact row (pj_next()). Its key is read from the fact’s slot; a NULL finds nothing, as the strict equality finds nothing. Otherwise the B-tree is descended to it (index_rescan() of one scan opened for the whole run, the probe’s scan key the join operator’s procedure) and each TID found, in order, is tested: in the WHERE’s set, not in the anti join’s, and visible - by the visibility map, tested under the leaf’s pin (the scan asks for index tuples, so nbtree keeps it, §40.5), or by the heap, whose fetch follows the HOT chain from its root to the version the snapshot sees (index_fetch_heap(); on 20 table_fetch_tid() and table_tuple_fetch_row_version()). The first TID that passes is the row’s dimension row: the key is unique, so no other entry of it holds a version the snapshot sees. The fact’s columns are copied into the scan tuple, each column’s outcome found from its labels, and each predicate’s truth from its outcome. A row taken without a heap visit takes its heap page’s predicate lock, as an Index Only Scan does.
  • Correctness. §40.5’s argument, per probe: the B-tree’s entries and the sets' members are both HOT chains' root TIDs; a version reached from a root has the indexed values of every version of its chain, a HOT update changing none; a TID’s entries are removed from every index before VACUUM lets its line pointer be reused, so an entry and a member of one TID describe the same chain; a page the map calls all-visible holds only versions every snapshot sees; and the sets are built after the query’s snapshot, from entries that version already had. The anti join’s set is every TID the B-tree holds for the inner side’s keys, visible or not: a fact row’s probe meets a TID t there only where an inner key k has the entry (k, t) beside the fact row’s (fk, t), so that k = fk by the family’s equality and the row has a match; and where the dimension row has an inner row with an equal key, that key’s lookup found its TID. Inner keys that are NULL find nothing, and a fact row with no dimension row is not joined anyway.
  • The anti join’s set (pj_build_anti()). Built after the other sets, before the first fact row: the inner side’s child read whole, each non-NULL key looked up in the same B-tree under the query’s snapshot, and every TID found sorted (a datum tuplesort of the TIDs' codes, within work_mem and spilling past it) into a set (lion_probe_sets_put_codes()), a container a container key. Anti Join Keys counts the inner rows and the TIDs.
  • The sets' memory. lion_ordered.c’s: a LionProbeSets lends the sets a LionOrderedState’s allocator, charged against hash_mem and spilling past it to the arena’s temporary file (§30.4, “Spilling”) - never refused for its size. Lion Sets counts them, their members and containers, the bytes spilled, and the builds.
  • Cost (pj_make_path()). The children are core’s, at core’s price: the fact’s cheapest total path (or its cheapest partial path), and the anti join’s inner side’s. In the competitor’s units (§39), with lion_units_cold()’s discount: the sets' builds, each priced as lion prices a set - its lookups less the excess of §40.10’s container price, a cpu_operator_cost a container key, the copy of its pieces - and their spill (lion_ordered_set_price(), lion_ordered_spill_price()), started before the first row; the anti join’s keys, each a lookup (lion_ordered_lookup_price(): the probe IndexPath’s indextotalcost and an entry’s probe) and a sort of their TIDs; for each fact row a scan of the probe as §40.10 prices it (lion_ordered_probe_price(): core’s parameterized IndexPath of the B-tree, its descent and pages amortized over the loops, an entry’s probe of the WHERE’s set, a member’s visibility-map test, the heap pages of the members on pages not all-visible through Mackert and Lohman, and of every member where the WHERE’s set is inexact, all-visible or not, as lo_cost()’s heap recheck fetches each for the recheck), with loops the fact’s rows; for each member - the share of the dimension’s rows the WHERE keeps - a cpu_operator_cost for the anti join’s test, each label’s and each predicate’s truth; and the emitted rows' cpu_tuple_cost and target. The cold discount weighs the pages the node reads - a B-tree leaf a probe (index_pages_fetched() over the loops), the members' heap pages not all-visible (every member’s where the WHERE’s set is inexact, as the probe’s price fetches them; lion_ordered_set_exact()), and the lion indexes' pages the builds read - against the competitor’s, which is a nested loop’s where core’s cheapest plan is one (§39, “A nested loop’s cold reads”), the loop into the dimension below a NOT EXISTS’s anti join over it and a Memoize’s misses where one sits over its probe (§39, “The join that reads the dimension”): the node’s reason to be is the heap page each of the loop’s probes reads. Each path weighs its own (pj_units()): the serial node its loops' reads over its price, the parallel form a participant’s - its share of the probes, every build - and both against the loop’s cost less its outer side where that side reads the fact, which the node reads through core’s path at core’s price. It divides the price by at most pg_lion.cold_max_discount. Never a bitmap heap scan’s cold reads (§39, “A bitmap heap scan’s cold reads”): the node reads the fact through core’s own path, as the competitor’s scans of it do, and the dimension a descent at a time, synchronous reads the stream’s price would under-charge. No knob beyond the switch.
  • Above the node (pj_add_aggs()). The Agg core would build over the join: hashed where the GROUP BY can be hashed (GROUPING_CAN_USE_HASH, no ordered aggregate), a sort and a sorted Agg otherwise, a plain one without a GROUP BY (sorted for a GROUP BY of constants alone), the HAVING its qual. The node’s rows are the join’s estimate, input_rel->rows.
  • Parallel (pj_add_aggs()). Where the fact has a partial path, the aggregates can be split and every rel may be read in parallel: Finalize HashAggregate (or a plain one), Gather, Partial HashAggregate over the node over the fact’s partial path. The node is parallel-safe, never parallel-aware: each participant reads its share of the fact’s rows and builds the sets for itself - core gives a CustomScan that is not parallel-aware no shared memory, so §40.9’s shared set has no place to live - and reads the anti join’s inner side whole, through its cheapest parallel-safe path (never the Gather the serial node may read). The price divides the rows (lion_parallel_divisor()), not the builds; a participant’s probes are priced at its own loops, its share of the fact’s rows, and the cold discount is the partial path’s own (above). A sorted GROUP BY stays serial. EXPLAIN ANALYZE’s counters are the leader’s.
  • Params and rescans (pj_rescan()). The sets' leaves' quals are custom_exprs, their keys a CaseTestExpr that lo_fix_qual() replaces with the index column, so setrefs.c fixes their Params; a generic plan’s $1 or a correlated subquery’s outer reference in the WHERE is a run-time key. A rescan reads the fact’s child again, handing it the changed Params core hands only to a plan’s outer and inner children, and keeps the sets unless a Param of their quals, or of the anti join’s inner side, changed (N builds). A predicate takes constants only (§10).
  • The plan (pj_plan_path()). A CustomScan whose scanrelid is the dimension, so that core opens it and EXPLAIN names it, with a custom_scan_tlist of the fact’s columns, the predicates and three resjunk entries EXPLAIN deparses - the probe’s clause key first, the WHERE’s qual, the anti join’s clause - never filled, there so that setrefs.c moves their Vars with the range table. Its children are the fact’s plan and the anti join’s inner plan (custom_plans). The recheck qual, the dimension’s Vars as INDEX_VAR, is in custom_private and initialised with no parent, so that its Vars are read from the heap tuple’s slot it is given, deformed as that slot’s type is, rather than from the virtual scan slot whose type the parent fixes; the planner took no WHERE with a SubPlan, which would need one.
  • EXPLAIN (pj_explain()). Probe Index, Probe Cond, Lion Cond and Lion Indexes (the WHERE’s), Cell Predicates and Cell Indexes (the labels'), Anti Join Cond; under ANALYZE Probes, Null Keys, Rows Removed by Lion Cond, Rows Removed by Anti Join, Heap TIDs Skipped via VM, Heap TIDs Rechecked (the heap fetches), Rows Removed by Visibility, Rows Removed by Lion Recheck (an inexact WHERE), Lion Sets: S, M members in C containers[, rechecked][, spilled N kB][, N builds] and Anti Join Keys: R rows, T TIDs. Group keys show the matched predicates with an extra pair of parentheses, as §10’s do.
  • Settings. pg_lion.enable_probe_join (on) offers the node; pg_lion.enable_count_pushdown = off, which keeps lion’s upper-path hook out, takes it away too. enable_indexscan and enable_indexonlyscan disable it as they disable core’s index-only scans (lion_path_core_switches(), LION_CORE_INDEXONLY): it answers the dimension from its indexes and has no form that reads the rows. The join methods' settings do not touch it - it is an upper path, not a join path - so the regression tests disable them to have it be the plan where core’s would win. btreescan_parallel.sql’s FK-side section and fkjoin_semi.sql switch it off, as they switch the other joins off: the plans they pin are other nodes', and on their tables the node is priced the cheaper. On the first - 6,070 of 60,000 fact rows under a 12,000-row dimension - its estimate is 2,600 against the FK-side join’s 3,228 (1,305 against 1,540 in parallel), yet it runs slower warm, 5.6 ms against 4.3 (7.7 against 6.2 in parallel; PG16, the median of seven runs): a mischoice of about a millisecond, 5,913 probes into a 12,000-row dimension. On the second, a fact filter with no entry (x = 77), it is 16.7 against 138, and both run in hundredths of a millisecond.

Privileges (§9, “EXECUTE on what a count replaces”; 2026-10-28). The node initialises only the WHERE’s quals (for its set) and the heap recheck; the join’s clause, which core’s nested loop initialises as its Index Scan’s indexqualorig, the anti join’s clause, and the predicates of the GROUP BY’s expressions, which setrefs.c replaces with the node’s columns in the Agg above, nobody would. So the planner records their functions with the dimension’s WHERE in custom_private member PJ_PRIV_EXECUTE (pj_execute_functions()), and pj_begin() checks them for the current user before anything starts, plain EXPLAIN included. node_contracts.sql runs the node through each.

Declined, and why.

  • A join path (set_join_pathlist_hook) emitting the dimension’s columns: those are what the node never reads. Its rows are useful only to the expressions over its predicates, which only the grouping target holds; a join path’s target is the join rel’s, its Vars.
  • A dimension column outside a predicate from the probe B-tree’s INCLUDE columns: §40.10’s probe does that, as a join’s inner side, where the B-tree covers the query; here the columns are the lion-indexed ones, which the primary key does not hold.
  • Sets shared among parallel participants, for the reason §40.10 gives.
  • Refusing a set past hash_mem: a set spills, as every lion set does, and its price says so.

Not done.

  • An outer or semi join (the inner join’s semantics are the node’s: no dimension row, no row), a dimension that is a join, a partitioned dimension, more than one dimension.
  • A predicate with a Param or a stable bound (since >= CURRENT_DATE - 365): §10’s outcomes are of constants. Such a predicate could be a bit of its own, as a multi-key column’s leaf is - its set the leaf’s clause with its run-time key, built at the first row like the WHERE’s Params (lo_fix_qual()), its outcome the mask of the bits a row’s sets hold - at the price of a set a predicate rather than a set an outcome, and of no exclusion among them.
  • The competitor’s cold reads behind an anti join. §39 takes the competitor’s kind from the first join it meets (lion_competitor_kind()), so a Hash Anti Join over the nested loop into the dimension is priced as a hash join’s, whose rate stands and whose loop’s cold reads are not weighed: the node meets the loop beneath at its warm price. And lion_nestloop_cold() charges no cold read to a loop whose inner side is a Memoize. Both are §39’s, for every lion path priced against such a plan.
  • Labels shared with the WHERE’s leaves where the two read the same index: each set is built for itself.
  • A per-key cache of the outcome (a Memoize’s): the fact rows' keys repeat, and each repetition is a descent and a set test, a few microseconds; a cache keyed by the fact key would save the descent but not the visibility test of a page the map does not call all-visible.
  • The spill of a set is not exercised by the regression test, whose sets stay far below 64 kB; it is lion_ordered.c’s, which ordered.sql spills.
  • The cold discount’s reads count a leaf a probe in the primary key at the key’s share of the cache, as the loop’s do (§39’s “Not done”, “One residency for both sides' common reads”): against a loop that probes the same key they weigh alike and keep the discount from the cap, where the heap pages the node avoids would reach it.
  • The loop’s weighed cost keeps the anti join’s inner side and the aggregates over the join, which the node pays at core’s price too; only the fact’s scan is taken off.

Tests. test/sql/probejoin.sql: a 6,000-row dimension with lion indexes on (region, tier), a bool, an enum and an int array, every 23rd region, 31st bool and 29th array NULL; a 30,000-row fact table with no lion index, every 50th key NULL and every 47th past the dimension’s; a 120-row inner side of keys, keys of no row and NULLs. Every query is run through the node (the join methods off) and through core’s plan (the node and lion’s pushdowns off, sequential scans only) and the answers compared as multisets: a CASE over two columns beside a fact column; booleans with NULL truths; a WHERE alone, no GROUP BY at all; arithmetic over CASEs with the fact’s own WHERE and a fact predicate; an enum, a multi-key column and a HAVING; a sum and a max of fact columns; a WHERE lion can only bound (tags <@ ...), its set inexact and every member rechecked on the heap tuple (Rows Removed by Lion Recheck); the anti join on the fact’s key and on the dimension’s, with a filter of its own; a generic plan’s Param in the WHERE; a correlated subquery rebuilding the sets for each value (5 builds); pages not all-visible - HOT and non-HOT updates, deletes and inserts on half of the table, unvacuumed, then vacuumed (Heap TIDs Rechecked: 0); the parallel form (Finalize HashAggregate, Gather, Partial HashAggregate, the node over a Parallel Seq Scan and the inner side whole); the plan chosen on price with nothing disabled, an 80,000-row dimension of wide rows under 3,000 fact rows; and the node not offered - a dimension column grouped by, under a THEN, in an aggregate; a predicate of a column with no lion index, or one no posting set decides (a range whose bound is no constant), or of both tables; a WHERE no lion index answers; a LIKE lion answers only by its prefix’s range, alone or beside an exact clause, against the range written out, which the node takes; a LEFT JOIN; a join on a column with no unique index; no set to build; the switch off. Ranges (section 9): a 6,000-row dimension with lion indexes on a numeric with NULLs and NaN, a date, an int declared NOT NULL, a text of “C” collation and an enum, each with NULLs where it may; one bound beside a fact column, its truths compared too; bounds of every strategy, BETWEEN and a constant on the left; a range beside =, IN, <>, NOT IN and IS DISTINCT FROM of the same column, with empty gaps; a date’s, a text’s and an enum’s order; the tier of the shape above - ranges, an overlap and an IN list under a gate, rounded - with a range in the WHERE and the NOT EXISTS; EXPLAIN ANALYZE of the labels; the parallel form; tier >= 4, which a count by cells declines; and a generic plan’s Param as a bound, not offered. EXPLAIN is printed with its counters where a run leaves them the same and as > 0 where an earlier run’s killed index entries change them. test/sql/competitor_cold.sql prices the node against core’s loop on a dimension ten times effective_cache_size and one the cache holds: below a hash anti join and a nested loop anti join, through a Memoize (and in a subquery the planner cannot pull up), under a scan of a large fact, and in parallel (§39, “The join that reads the dimension”).

Measured (2026-10-23; PostgreSQL 16, the dev cluster - a VM of four cores shared with other work, shared_buffers 128 MB, the tables in the OS cache: warm; EXPLAIN (ANALYZE, TIMING OFF), three runs each). A 2,000,000-row dimension (a 422 MB heap, a 43 MB primary key) with lion indexes on a five-value text column, a five-value int, a bool and a four-value enum; a 1,000,000-row fact table with no lion index, its keys spread over the dimension; 2,000 keys in the NOT EXISTS. The query is the shape above, WHERE d.active AND d.mood IN (three of four) keeping half the dimension, with f.id <= n choosing the fact rows:

fact rows planned its time core’s plan, the switch off core’s nested loop
1,000,000 core’s hash join 1.06 - 1.48 s (the same) 4.5 - 5.1 s
1,000,000 the node, the joins off 2.61 - 2.92 s
300,000 the node 1.04 - 1.18 s hash join, 0.67 - 0.68 s
100,000 the node 0.34 - 0.39 s hash join, 0.61 - 0.66 s
10 the node, the joins off 48 - 62 ms

The ten-row line is the fixed part, the sets' builds - five sets, 2.2 million members in 3,303 containers - and the fact’s scan; a probe then costs 2.6 to 3.5 us, the descent most of it, as §40.10 measured its own. Warm, the node beats core’s nested loop by 1.7 times and core’s hash join where the fact rows are few against the dimension’s WHERE; the descent’s price, core’s own for an index scan’s, makes it some 1.7 times too cheap against the hash join’s in a warm cache, so that at 300,000 rows - 15% of the dimension - it is chosen and runs 1.6 times the hash join’s time. Cold is not measured; the pages each plan reads say it goes the other way: the hash join reads the WHERE’s heap pages, half of the 422 MB here and most of a heap far larger than memory, and the nested loop a random heap page a fact row, where the node reads the primary key’s leaves and the lion indexes and no heap page of an all-visible dimension.

Measured, ranges (2026-10-24; PostgreSQL 18, the dev cluster - a VM of four cores shared with other work; EXPLAIN (ANALYZE, TIMING OFF); warm, shared_buffers 2 GB, the median of five to nine runs; cold, shared_buffers 128 MB, the server restarted and the dimension’s heap dropped from the OS cache before each run, the median of three). A 1,000,000-row dimension of wide rows (a 503 MB heap, a 21 MB primary key) with lion indexes on twelve columns - texts of 20, 8, 8 and 4 values with NULLs, an int of 5, a text array of one to four of 30 tags, a numeric score from 0 to 1, a date since, two booleans, the WHERE’s status and country - and a 3,000,000-row fact table with a B-tree on (grp, ver, fk) and no lion index; 60,000 rows in the NOT EXISTS’s table, 10,000 of them its keys. The query is the shape above with a tier over a gated, rounded score of nine CASEs - score > s (true of 53% of the rows), since >= DATE '2015-01-01', tier IN (1, 2), an overlap, IN lists and equalities - beside f.level, under d.status = 'active' AND d.country IN ('us', 'ca') (187,000 rows) and the NOT EXISTS, for the fact rows of one grp (42,858), of ten (300,006) and of the largest (750,000). Before ranges the node was not offered and core’s plan was the plan; a count by cells of the same tier over the dimension alone is still declined.

fact rows form planned (cost) warm; cold ms core’s plan (cost): warm; cold core’s nested loop: warm; cold
42,858 serial the node (60,033) 391; 481 hash join (105,398): 393; 690 367; 2,839
42,858 3 workers the node (71,556) 257 - 307; 383 nested loop (80,137): 142 - 168; 1,152 (the same)
300,006 serial hash join (147,427) 614; 1,034 (the same) 1,821; 7,226
300,006 3 workers the node (103,782) 514; 826 hash join (131,788): 316; 903 494; 2,521
750,000 serial hash join (241,445) 981; 1,612 (the same) 1,184; 2,202
750,000 3 workers hash join (195,118) 420; 986 (the same) 557; 948

The node forced where it is not planned - the joins off - took 1,472 and 3,126 ms warm serially at 300,006 and 750,000 rows (1,769 and 3,992 cold): the price is right to leave it. Its fixed part is the sets' builds - 21 sets, 3,748,275 members in 20,140 containers, 227 ms of a 517 ms run with timing on - and a range’s set builds slower a member than an equality list’s, its entries many (§28): the score’s labels, score <= s and its NULLs, 469,190 members in some 45 ms (10,000 values); since >= DATE '2015-01-01', 320,184 in 30; tier IN (1, 2), 399,692 in 6. Each participant builds every set for itself, so the parallel form’s warm time is mostly one build, and against the parallel nested loop warm it is 1.8 times slower: within the cold discount’s warm floor (pg_lion.cold_max_discount, §39), and 3.0 times faster cold, where the loop reads a random heap page a probe. With effective_cache_size at 186 MB and random_page_cost 1.1 the node is planned serially at 70,912 against the hash join’s 103,670, and in parallel at 53,935 against 57,893 - the nested loop, under a Hash Anti Join that hides its cold reads (“Not done”); at random_page_cost 4 the serial plan is the hash join, and the parallel one the node, at 82,323 against 98,691.

47. An ordered walk across a key join (LionOrderedJoin, lion_ordjoin.c, 2026-10-23)

The shape is

SELECT f.id, f.score, d.pk, d.lvl
  FROM fact f JOIN dim d ON f.fk = d.pk
 WHERE f.grp = 2 AND f.tags @> '{a1}' AND d.lvl = 3 AND d.flag
   [AND f.fk IN (SELECT pk FROM flt WHERE x < 3)]
 ORDER BY f.score DESC NULLS LAST LIMIT 20

  • a page of a fact table joined to a dimension by the dimension’s key, in the order of a B-tree of the fact, both tables filtered by clauses their lion indexes answer, perhaps with tables that only filter the key. §30’s walk takes the fact alone: it walks fact_grp_score over the fact’s set and stops at the twentieth member. Across the join core has two plans, and neither filters the dimension before reading it. A nested loop under the LIMIT walks the B-tree with the fact’s filter as a Filter, each row fetched to be thrown away or kept, and looks each kept row’s key up in dim_pkey - an Index Scan that reads the dimension’s heap row to test lvl = 3 AND flag, rejecting most of them; Memoize saves a repeated key’s descent, not the heap rows of the rest. Or a hash join of the two bitmap heap scans of the sets, the rows sorted and the first twenty kept, which reads every fact row of the set however few the page needs. §40.10’s probe answers the dimension from its sets where a B-tree covers what the query reads of it, and here the query reads lvl, which the primary key does not hold.

    Limit -> Custom Scan (LionOrderedJoin) Ordered By: fact_grp_score Index Cond: (f.grp = 2) Lion Cond: (f.tags @> ‘{a1}’::text[]) Lion Indexes: fact_tags Dimension: dim d via dim_pkey Dimension Filter: d.flag Dimension Lion Cond: (d.lvl = 3) Dimension Lion Indexes: dim_lvl

LionOrderedJoin is the join rel’s CustomScan. It walks the fact’s B-tree in order over the fact’s lion set - §30’s walk, with its met-before bits, its lazy set, its early stop and its fetch-and-sort switch - tests each member’s key against the key sets, descends the dimension’s unique B-tree for the key, tests the TID it finds against the dimension’s own lion set BEFORE it reads the dimension’s heap, fetches and rechecks that row, and returns the two rows together in the walk’s order, until whatever is above it stops pulling. Core puts the rest above it: the Limit, an Incremental Sort for an ORDER BY that goes on past the walk’s columns, left joins and NOT EXISTS to other tables. Each side is a LionOrdered plan built for the join (lion_ordered.c, lion_ordered_side_path()), kept as a child the join runs and prints itself.

47.1 What qualifies

  • The join (oj_key_join()). An inner join of two tables by exactly one clause, every clause between them counted: a strict equality of two plain Vars (binary-compatible casts allowed), f.fk = d.pk either way round, whose dimension column is the one column of a unique B-tree (lion_column_unique_index(), lion_fkjoin.c, which §46 descends too) and whose operator, with the key on the left, is that B-tree’s family’s equality for the key’s type and the fact column’s. A join with a second clause - AND f.ver < d.kind - is declined: the node tests nothing between the rows. Either table may be the fact; the hook is called for each orientation and each is tried.
  • Each side (oj_side_ok(), lion_ordered_rel_ok()). A table LionOrdered may scan (§30.1, lo_rel_ok()): a heap table or a materialized view, or a partitioned table plan-time pruning leaves one leaf of, which the node plans on (lion_single_leaf(), §16, “A parent pruned to one leaf”); no TABLESAMPLE, no security quals or RLS, not the query’s result relation, no LATERAL reference, not proven empty, under no old_snapshot_threshold; of the heap AM. None of its restriction clauses is pseudoconstant - core would gate the side’s scan with a Result - or holds a SubPlan, whose plan EXPLAIN could not show under the join; its target is its own plain columns. Neither side needs a lion clause: a side without one is walked or probed with no set, every row it meets fetched and filtered. One of them has a set, or a key set filters; with neither, core’s nested loop already is the plan.
  • The order (lion_ordered_orders()). The fact’s ordered B-tree paths as §30.2 takes them - core’s unparameterized IndexPaths with pathkeys, and those LionOrdered’s own paths walk, which add_path() may have dropped for them - whose pathkeys, as the join’s (build_join_pathkeys()), give the query at least its first sort key; an Incremental Sort above sorts the rest.
  • The query. A SELECT with no row marks (FOR SHARE OF d is declined); a target of the two tables' plain columns, no PlaceHolderVar at the join; a join rel with no LATERAL reference. A LATERAL subquery that holds the whole join is planned on its own, its outer reference a Param of the node’s quals (§47.4, “Rescans”).
  • Declined, and why. An outer, semi or anti join as the join itself: the node returns a row only where both have one. A parallel plan: the node has no partial path and is not parallel-safe (§47.7). A parameterized path: the node is never the inner side of core’s nested loop by a Param of its key, which is §40.10’s probe. UPDATE, DELETE, MERGE and row marks: v1 leaves them to core, as §30.8 does.

47.2 Planner (oj_set_join_pathlist(), oj_add_paths(), oj_add_path())

A set_join_pathlist_hook, chained, offered when pg_lion.enable_ordered_join is on and the query has pathkeys. It answers two calls.

  • Two tables: outerrel and innerrel both base rels, an inner join (jointype and the SpecialJoinInfo’s both JOIN_INNER), the join rel their union. The fact is outerrel, the dimension innerrel; the other orientation is the hook’s other call.
  • The two tables and key sets (§47.6): outerrel a join of base rels alone - at most OJ_MAX_RELS - 1 of them, no outer join’s relid among them - and innerrel a base table, inner-joined or the inner side of a semi join (JOIN_SEMI, the SpecialJoinInfo’s JOIN_SEMI too: the unique-ified forms are core’s calls for the same join rel). Every ordered pair of outerrel’s tables is tried as the fact and the dimension, their clauses from lion_fkjoin_pair_clauses() (exported for this from lion_fkjoin.c); every other table of the join rel must make a key set, and innerrel must have the highest relid of them, so that the join rel is offered its paths in one call of the many core makes for it. oj_special_ok() then asks that every special join whose inner side is in the join rel be a key set’s semi join: the fact and the dimension are inner-joined.

For each of the fact’s orders, each of the fact’s lion accesses (lion_ordered_accesses(), §30.2, step 2; none, NULL, where it has none) and each of the dimension’s, one path is priced (§47.3) and handed to add_path(), which keeps what it keeps. The path’s pathkeys are the walk’s as the join’s; its target the join rel’s; it is unparameterized and not parallel-safe. Its custom_paths are the fact’s side, the dimension’s side and each key set’s table’s cheapest total path. The sides (lion_ordered_side_path()) are LionOrdered paths never added to their relations: the fact’s an ordered walk of the IndexPath with its set, index-only where lion_ordered_walk_price() found the B-tree to hold every column of the fact the join needs (§40.1, lo_needed_attrs() over the fact’s reltarget, which holds the key); the dimension’s a probe of an IndexPath of its unique B-tree with no clause, in heap mode, whose one index qual is pk = NULL - the key the join hands it before each descent (lion_ordered_probe_key()). Each carries a weight in a fourth custom_private member that lo_plan_path() reads for the side’s lazy set (§47.4).

The plan (oj_plan_path()) is a CustomScan with scanrelid 0 whose custom_scan_tlist is the fact’s side’s target, then the dimension’s, then a resjunk entry a key set, its key, which EXPLAIN reads to name the table and each row fills with the fact’s key, for a target that reads it (§47.6). The join’s target list is evaluated over that slot (CUSTOMPATH_SUPPORT_PROJECTION). custom_private is positional behind a marker: the counts and the key’s position among the fact’s columns, the dimension’s B-tree, each key set’s hashable equality, collation and key column in its plan’s target.

A side is a child the join keeps out of custom_ps, not a parameterized plan of its own: finalize_plan() refuses a CustomScan child whose quals reference a PARAM_EXEC its parent sets, and the dimension’s descent key, as a Param, would be one. The pk = NULL qual is a Const, and the key is written into the scan key’s argument at run time.

47.3 Cost (oj_cost())

In the planner’s units, on the walk’s own price, with no §39 rate; against core’s nested loop, put into the loop’s units with its Memoize at the hits the LIMIT meets, and cold weighed by §39’s cold reads (“The competitor” below). With factrows the fact’s rows after its quals as lion estimates them (lion_probe_rel_rows()):

  • The fact’s walk: lion_ordered_walk_price(), which is lo_cost() (§30.3, §40.3) over the whole walk - the set’s build, the entries walked, the members' fetches (in index-only mode the pages not all-visible, and every candidate where an inexact set’s columns are not in the B-tree), the filter, the fact’s target - with no set where the fact has no lion access (lion NULL, as §46 prices it). The set’s build is priced as a probe’s (lo_cost()’s probebuild): its lion access’s bitmap index scan price less lo_build_excess(), the TIDs past LO_BUILD_PIECE_TIDS a container piece, as the dimension’s set, §40.10’s probe and §40.11’s correlated subquery’s calls (which lo_cost() finds itself, the flag or more than one call) are priced - 3,772 down to about 960 for §47.9’s chan = 3, 8,401 to about 2,250 for tags @> '{a1}'. A plain LionOrdered walk keeps the bitmap scan’s price (§30.3): its walk is most of its price, where the LIMIT scales this one’s walk and not its build. A cpu_operator_cost a fact row for its key’s test.
  • The keys: keys = factrows * P(fk IS NOT NULL) (nulltestsel()). Each key set adds its plan’s total cost and a cpu_operator_cost a row to the startup, a cpu_operator_cost to each key still tested, and lets through pass of them, the descents left (oj_keyset_pass()): core’s own selectivity of the fact’s semi join to the key set’s table by the key (eqjoinsel_semi() through clause_selectivity(), on a clause and a JOIN_SEMI SpecialJoinInfo made for it, so that the query’s RestrictInfo keeps its own cached selectivity), over P(fk IS NOT NULL), at most 1. It matches the fact’s most common keys against the table’s where both have them, and otherwise takes the table’s distinct keys after its clauses against the fact’s, so a key set of a few keys each of which many fact rows hold, or of a few keys none of which the fact holds, is priced at what it lets through and not at its share of the dimension’s rows. A key set’s keys are taken to be dimension keys, so each key it lets through names a row, and the orphan share is taken at the first key set only. Where the table’s clauses pick out the keys the fact holds most of, which no statistic of either table records, pass is low; where they pick out keys the fact lacks, high (§47.7).
  • The distinct keys: a key set holds each key once, setkeys of them, estimate_num_groups() over its plan’s rows (what core prices a GROUP BY of them at), at most those rows. They size the set against work_mem (§47.6) and the fold’s leaves (below). An IN over a table whose keys repeat holds far fewer keys than it has rows.
  • The key sets folded (§47.6, “Folded into the walk”; oj_fold_share()): the walk is priced over the AND of its set and the rows of the key sets' keys, keysel = P(fk IS NOT NULL) * Π pass of the table, taken as independent of the fact’s quals (lo_cost()’s keysel): its members, pieces, bytes and fetches are the AND’s, its rows factrows * keysel, every key a dimension key that each key set passes (a test each all the same). Each key set’s leaf is priced in the start-up (lion_ordered_keys_price()): its setkeys keys' entries located in the key’s lion index, index_pages_fetched() over the index’s pages at random_page_cost as genericcostestimate() prices an IN list’s descents, a descent’s comparisons a key, and a piece a container key its rows reach; and an operator a key for the array. Its pages join the cold reads. Folded and not are both offered (oj_add_paths()), and add_path() keeps what it keeps: a page the walk meets in a few hundred entries does not pay a thousand keys' lookups back.
  • The fetches the key sets reject, not folded: the walk fetches each member to read its key, and the key sets keep kept = descents / keys of them. Core’s Limit takes s of a run priced whole, which charges those fetches s of the pages Mackert and Lohman give the whole walk’s - each page once, a page for the seventy fetches the whole walk makes of it - where the LIMIT’s share makes s * fetches of them, which read the pages that many fetches read: a walk that fetches a million rows for a hundred was charged a fifth of the table. So the pages of the share’s fetches at their own count (lion_ordered_heap_price(), lo_cost()’s heap price with its correlation) past s of the whole walk’s are charged, for the share 1 - kept of them the key sets reject, at 1 / s of it in the run, which the Limit takes s of. Not alone under the LIMIT (s 1) or with nothing rejected it is 0; folded there is nothing to reject.
  • The key sets tested on the index tuple (§47.6, “Tested on the index tuple”; oj_itup_col()), not folded, where the walked B-tree holds the key: the walk fetches only the members the key sets keep, so its heap price is that of kept of its fetches - lo_cost()’s heap price over the whole walk (lion_ordered_heap_price()) for them in place of all of them, and a cpu_tuple_cost less for each fetch not made - and nothing is fetched for nothing: the term above is 0. The key’s test is a hash probe, as after the fetch.
  • A descent, as btcostestimate() prices one: ceil(log2(index tuples)) comparisons, (tree_height + 1) * 50 operator costs for the pages descended, an index tuple, an operator; and the B-tree’s pages the descents read, index_pages_fetched(descents, index pages) at random_page_cost. A cpu_operator_cost a descent for the dimension’s set’s test, where it has one.
  • The dimension’s set: lion_ordered_set_info() - its build as lion prices a set (lion_ordered_set_price() and lion_ordered_spill_price(), §46’s), in the startup; the share of the dimension’s rows it holds (candsel); a recheck’s price when it is inexact (lion_ordered_set_exact()); the clauses it leaves to the side’s filter (lo_residual()).
  • The members: descents * hit * candsel, each a heap row read at random (Mackert and Lohman over the dimension’s pages, index_pages_fetched()), a cpu_tuple_cost, the recheck and the filter.
  • The rows: descents * hit * min(1, dimrows / dim.tuples), dimrows the dimension’s rows after its quals; each a cpu_tuple_cost and the join’s target. The dimension’s key is unique, but nothing makes the fact’s a foreign key: a key that names no dimension row costs its descent and returns nothing. Of the fact key’s nd distinct values (get_variable_numdistinct()) at most dim.tuples are the dimension’s, so found = min(1, dim.tuples / nd) of the keys name a row (oj_found_share(); 1 where nd is a default), and hit, the share of the descents that do, is found, or 1 once a key set has let through only its own keys. Core’s eqjoinsel_inner(), with no MCV list to match, takes the join at keys * dimrows / max(nd, dim.tuples), and the node takes the same: found is 1 where the fact’s distinct keys are no more than the dimension’s rows, and the orphan share where they are more. The members, a set’s test of a TID and core’s loop’s dimension heap reads (below) are counted over the keys that name a row.

The startup and total are divided by pg_lion.pushdown_margin, as LionOrdered’s paths' are; core’s Limit takes its share of the run. Where the node is not alone under the LIMIT (§47.10) the startup is the total: the whole walk, which no Limit above discounts. The path’s rows are the node’s estimate, not the join rel’s. Not priced, as lo_cost() prices neither: the fetch-and-sort switch, which the walk takes at run time when its entries per member say it pays (§30.4), and the early stop once the walk has met every member. enable_indexscan = off disables the path (lion_path_core_switches(), LION_CORE_INDEXSCAN): it is a walk of one index and descents of another, and reads the rows; enable_indexonlyscan = off takes the fact’s index-only mode away. The join methods' settings do not touch it.

The share the LIMIT takes, min(1, limit_tuples / rows) (oj_limit_share()), is not a stop and not in the price: it decides each side’s set, built at once or lazily (§47.4), and it is what the competitor’s Memoize and the cold reads are weighed over (below). The fact’s side weighs its walk by it; the dimension’s weighs the descents, probes * share - the descents the LIMIT’s share of the walk makes, not the whole walk’s (2026-10-23 review). It is taken only where the node is alone under the LIMIT (§47.10): the left joins and NOT EXISTS above it keep its rows about one to one with the LIMIT’s. Under a join or a clause above that filters, the share is 1, the whole walk, and the price all start-up. §30’s lo_limit() is stricter, LionOrdered’s limit being a hard stop: the node the query level’s only table. Where the share is wrong all the same the lazy error is the cheaper one: a lazy set whose walk goes on builds itself once its probes have cost what the build would (§30.4), one build more, where a set built at once for a walk that stops after a few hundred entries costs the whole build.

The competitor (oj_competitor()) is the plan the LIMIT would take of the join rel’s paths as core leaves them when the hook runs, found before the node’s first path is added (add_path() frees the paths a new one dominates): unparameterized and not disabled, with no lion path inside (lion_path_has_lion()) but for a nested loop’s outer LionOrdered walk of the fact (oj_loop_over_walk()), the plan core takes where LionOrdered walks the page; one whose first sort key is the walk’s at its startup and the LIMIT’s share of its run - all of it where the node is not alone under the LIMIT, as for the node - any other at its total, which the Sort above it reads whole. It is not lion_competitor_path()’s, the cheapest total, which here is a hash join at §39’s rate for one, 0.5: under a LIMIT core takes neither that plan nor its whole price. Only a nested loop in the walk’s order is weighed, for its Memoize and its cold reads; a hash join and its Sort read the two sides through bitmap heap scans, which §46 does not weigh either.

Its Memoize (§39, “A Memoize under a LIMIT”): core, the loop’s price of the LIMIT’s share, counts the hits of a Memoize over its inner scan at core’s ratio over the whole walk; it lacks memo, what the share’s calls do not hit (lion_memoize_limit_extra()). The node’s startup and total are multiplied by memoscale = core / (core + memo), unbounded, as lion_units_price() puts a path into such a loop’s units.

Cold reads (the end of oj_cost()), as §46 weighs them (§39, “A nested loop’s cold reads”, lion_units_cold()), over the LIMIT’s share s of the walk (*share), with walked the entries of the fact’s B-tree its Index Cond selects:

  • the loop’s reads (corereads): a leaf of the dimension’s key for each key it keeps and a dimension heap page for each that names a row, index_pages_fetched(s * keys) over the B-tree’s pages and index_pages_fetched(s * keys * found) over the dimension’s; where its outer side is the fact’s index scan, a fact heap page for each entry it walks, index_pages_fetched(s * walked) over the fact’s pages - in an index-only scan those not all-visible, 1 - allvisfrac of them; and where it is a LionOrdered walk, that walk’s reads (lion_ordered_walk_reads()): lo_cost() taken again for its B-tree and lion access, the heap pages of the LIMIT’s share of its fetches and its set’s lion index pages - the reads lo_cold_price() weighs, less the B-tree’s, which the node’s walk reads too and which are left out of both. An index-only walk whose price already carries lo_cold_price()’s discount, its reads weighed cold against a bitmap heap scan’s, adds none: they are not weighed twice;
  • the node’s (ownreads): the fact heap pages of its fetches (lo_cost()’s fetches of the whole walk, fetches_p), a leaf a descent and a dimension heap page a member, each index_pages_fetched() of s of them; the lion index pages the two sets' builds read (lo_lion_pages(), LionOrderedSet.lionpages); and the pages each key set’s plan reads (OjKeySet.reads, “The feed” in §47.6).

Each page is charged lion_units_cold_page(), perpage: pg_lion.cold_page_cost less the random_page_cost core charged it, in the share of the query’s tables and the two B-trees effective_cache_size cannot hold, 0 where it holds them all. The startup and total, after memoscale, are then multiplied by

(own + ownreads * perpage) * (core + memo) / ((core + memo + corereads * perpage) * own)

own the node’s price of the LIMIT’s share (startup + s * run): at most 1, a node that reads more than the loop keeping its price, and at least 1 / pg_lion.cold_max_discount. Warm it is 1.

47.4 Execution (oj_begin(), oj_exec(), oj_rescan())

  • Begin. The custom_private shape is checked against the plan’s children and target, and EXECUTE is asked of the current user, plain EXPLAIN included, on the functions of the join’s clause and of each key set’s (OJ_PRIV_EXECUTE, oj_execute_functions(); §9, “EXECUTE on what a count replaces”): the node descends the dimension’s B-tree with a scan key and tests a key set through a hash table, so nothing initialises those clauses as the joins it replaces would. Then the sides are initialised as children of the join (ExecInitNode(), neither backward nor marked) and kept out of custom_ps; each key set’s plan goes into custom_ps, and EXPLAIN shows it under the join. LionOrdered’s lo_begin() checks its side’s flags: a serial B-tree walk, and the dimension a probe in heap mode with exactly one scan key, pk = NULL, an equality on the first column.
  • The sets, lazily or at once (lo_plan_path(), from the side’s weight). The fact’s set is built at once when the share is at least LO_LAZY_MAX_SHARE (0.2) and its share of the walk at least LO_SWITCH_MIN_WALK (10,000) entries, lazily otherwise; the dimension’s at once when the descents are at least LO_SWITCH_MIN_WALK, lazily otherwise, as a nested loop’s probe’s is (§40.10). A lazy set builds itself on §30.4’s rules - the probe budget, the walk’s length, a walk out of heap order - and Lion Set says which (after N keys probed (probe budget)).
  • A row (oj_exec()). The key sets are built first, if they are not (§47.6); an empty one ends the scan before the walk. Then, for each row the fact’s walk returns - a member, visible, passing the fact’s filter and recheck, in the walk’s order: its key is read from the fact’s slot; a NULL key skips the row, as the strict equality matches nothing; a key some key set does not hold skips it (Keys Rejected by Key Sets); otherwise lion_ordered_probe_key() writes it into the dimension’s scan key and rescans the probe, and the probe’s next row is the dimension’s row of that key or none: the descent meets the key’s entries, each TID tested against the set (lazy or built) before the heap, the heap read for a member (index_fetch_heap(), the HOT chain followed to the version the snapshot sees), rechecked where the set is inexact and filtered. The key is unique, so one descent finds the row or none. The scan slot’s values are copied from the two sides' slots, fact first, and each key set’s column is the fact’s key (§47.6); neither side’s slot changes until the join asks it for its next row, and the key stays valid in the fact’s slot through the descent. The join’s target is projected from it.
  • Memory. The key sets' hash and equality run in the node’s per-tuple memory, reset each row the walk returns and each row a key set’s plan returns, as a hash join’s do (ExecHashGetHashValue()): a type’s own functions may leave what they allocate behind. Each set and its copied keys are in a context of its own, reset at each build.
  • Correctness. Each side’s is LionOrdered’s (§30.5, §40.5): a B-tree’s entries and a set’s members are both HOT chains' root TIDs, and a member’s row is the version the snapshot sees. The dimension’s key is unique, so of the entries a descent meets at most one leads to a version the query’s snapshot sees; that the descent stops at the first member that passes is right. A fact row whose dimension row the dimension’s quals reject - or that has none - is not joined, which is the inner join’s semantics.
  • Rescans (oj_rescan()). The walk restarts; the dimension is rescanned at its next descent. A changed Param reaches each side, which rebuilds its set only when its lion quals use it (§30.4), and each key set’s plan, whose set is dropped and rebuilt at the next row only when the plan depends on it (Key Set Builds). The join hands its chgParam to the sides and to the key sets' plans itself: core hands it only to a plan’s outer and inner children.

47.5 EXPLAIN (oj_explain(), lion_ordered_explain_side())

The fact’s side’s lines as LionOrdered prints them - Ordered By: <index>[ (index only)], Index Cond, Filter, Lion Cond, Lion Indexes - then Dimension: <table> [<alias>] via <index> and the dimension’s side’s lines behind Dimension (Dimension Filter, Dimension Lion Cond, Dimension Lion Indexes; its index and its descent’s key, the join’s clause, are not repeated), then Key Sets: <table>, ... and, where they are folded into the fact’s set, Key Set Index: <index>, or, where the fact’s walked B-tree holds the key and they are tested on its index tuples, Key Sets Tested On: <index> (§47.6). Under ANALYZE: the fact’s counters - Index Entries Walked, Lion Set Hits, Heap Fetches, Rows Removed by Lion Recheck, Rows Removed by Filter, Lion Set: N containers, exact|rechecked[, after N keys probed (probe budget)][, N builds], Switched to Fetch and Sort and the sort’s method and space, Scans Stopped Early; then Key Set Keys (the last builds' keys, summed over the sets), Key Set Builds and Keys Rejected by Key Sets; then the dimension’s behind Dimension: Dimension Descents, Dimension Lion Set Hits, Dimension Heap Fetches, Dimension Rows Removed by Lion Recheck, Dimension Rows Removed by Filter, Dimension Lion Set. The key sets' plans are the join’s children.

The sides' expressions are deparsed with the join at the head of a copy of the ancestors. The first version put it at the head of the caller’s list (lcons() extends a list in place), and a nested loop’s inner side printed after the join found the join where its own parent should be: its Param’s outer Var unresolved, Index Cond: (pk = $0) for (pk = d.pk), and on an assert-enabled build ruleutils' Assert(param->paramkind == PARAM_EXTERN) (CI on master, 2026-10-23).

47.6 Key sets (oj_keyset_of(), oj_keyset_build())

A table that only filters the join’s key - f.fk IN (SELECT pk FROM flt WHERE x < 3), EXISTS (SELECT 1 FROM flt x WHERE x.pk = d.pk AND x.x = 2), or JOIN flt x ON x.pk = d.pk with nothing of x wanted - is folded into the node: its plan is run once, before the walk, and its keys go into a hash set each fact row’s key is tested against before the descent. Without it core joins the node’s rows to the table above it, and every fact row the table rejects has been walked, fetched, descended and its dimension row read. A table becomes a key set when:

  • it is a base relation of the query, a table, with no LATERAL reference and no TABLESAMPLE, with an unparameterized path, and its target returns the key: the plan of it the node picks (“The feed”, below) is the key set’s child;
  • it is the whole inner side of a semi join - minimal and syntactic, an IN or an EXISTS of it alone - whose outer side is in the join rel, and in no other special join’s sides (oj_semi_of(): a second IN has the first’s table in its syntactic outer side, which is let pass); or it is in no special join, inner-joined, and its key unique among the rows its own clauses let through (oj_keyset_unique()), so that no row is repeated - core turns a semi join on a unique key into that inner join itself (reduce_unique_semijoins()). The key alone unique under the clause proves it (lion_column_is_unique()), and so does a unique B-tree of more columns whose others the table’s restriction clauses pin to a constant each (oj_column_pinned(): col = <pseudo-constant> either way round, by an equality of the column’s family in the index, under the index’s collation for it) - kt (fk, kind) under kind = 2, the key set holding those rows' keys alone; immediate, B-tree, and partial only where the clauses imply its predicate, as relation_has_unique_index_for() takes them;
  • nothing of it is in the join rel’s target but its key, and the key only where core put it there for its equivalence class alone: core makes each member of a class available wherever the class’s tables are joined (generate_base_implied_equalities()), so a table joined to the key above the join rel - a LATERAL subquery on the key that a strict WHERE clause made an inner join
    • puts the key set’s key in the target though the query reads nothing of the table. The key’s attr_needed is within the class’s relations, its type has image equality (lion_type_equalimage()) and the class’s equality is its type’s default: the node fills the column with the fact’s key, which equals it, and under such an equality is the very value the key set’s row holds;
  • exactly one clause joins it to the rest of the join rel, and it is of the key’s equivalence class: whichever of f.fk, d.pk it was written against, it filters the fact’s key;
  • its key is of the fact key’s type, with an equality in the class’s families that hashes both sides with one function: the set is a hash table of that type’s values, under the class’s collation;
  • its distinct keys (§47.3, oj_keyset_new()), at the bytes a simplehash entry and a copied key take, fit work_mem.

A key set is a simplehash of Datums (ojkeys), each non-NULL key copied once into the set’s context; a NULL key matches nothing. Key Set Keys counts them. Planning takes the key sets of a join rel whole: a join rel some of whose tables are not key sets gets no path, and core’s join of the node’s rows to them above is the plan.

A join above (2026-10-28; oj_keysets_above(), oj_keyset_above()). A query of more tables than join_collapse_limit (or from_collapse_limit) is planned in parts: the fact and the dimension in one, with the tables written next to them, and an EXISTS or an IN elsewhere in the query joined above that part’s plan. The key set’s table is then not in any join rel the node is offered for, and without it the node’s walk passes every fact row the EXISTS would drop; raising the limits plans the query whole, and also lets core reorder every other join in it, which can cost another query its plan. So a table outside the join search under way - not in the first level of join_rel_level, or in GEQO’s initial rels; another extension’s join search gets none - and joined above it by the key is a key set too: its plan is the key set’s child, its keys are tested as any key set’s are, and the join stays above and tests the node’s rows again. A row the key set drops is one that join would drop where:

  • the table is the whole inner side, minimal and syntactic, of a semi join whose syntactic outer side holds the join rel - an IN or an EXISTS of it alone - or on no special join’s inner side, an inner join (core makes one of an EXISTS on a unique key, reduce_unique_semijoins(); a key that repeats is fine here, the join above being the one that returns the rows);
  • the join rel is on no special join’s inner side but those wholly within it, and on no side of a FULL join reaching past it: its rows reach that join, never null-extended, holding the key;
  • the table is a base relation with no LATERAL reference, TABLESAMPLE, volatile function in its clauses or PlaceHolderVar computed at it; its key the key class’s member of it, a plain column of the fact key’s type with a hashable equality; and nothing of it wanted above but that key - no join clause of its own, no other column in its target. A table a column of which the query reads
    • a dimension of another join - or an EXISTS that tests more than the key is not one.

The node’s rows are then those its key sets keep, and the joins above return aboverep rows for each: one for a semi join, and for an inner join one for each of its table’s rows with that key, setrows * P(key IS NOT NULL) / setkeys (estimate_num_groups() counts no group for NULL), at least one. A row of the join rel gives abovepass rows above, the product of the key sets' pass times aboverep. The LIMIT’s share of a walk is taken over the rows above: the node’s own output times aboverep (oj_cost(), and os_cost()’s limit divided by it), the competitor’s rows times abovepass (oj_competitor()). The tables are not counted against the node being alone under the LIMIT (oj_alone(), §47.10). A table inside the join search is joined there and is a key set of the join rel that holds it, as before, so a query planned whole keeps the plan it had. EXPLAIN shows the key set under the node and the join above with its own scan of the table: the table is read twice.

Measured (2026-10-28; PostgreSQL 18.6 with assertions, the shared VM, random_page_cost 1.1, work_mem 256 MB, competitor_pricing off; warm, three runs). A synthetic query of ten tables, a 2,000,000-row fact walked by a B-tree under LIMIT 100, a 3,000,000-row dimension, and an EXISTS on a 10,000,000-row table filtered by a lion index, whose matches crowd on a few thousand keys; four filters of that table. Before, at the default limits the EXISTS was a nested loop semi join above the walk; at 16 the query was planned whole and the table was a key set.

filter before, default limits before, at 16 after, default limits after, at 16
1 2.9 to 6.6 s 0.88 to 0.97 s 0.89 to 1.11 s 0.89 to 0.94 s
2 1.9 to 2.2 s 0.40 to 0.52 s 0.36 to 0.39 s 0.35 to 0.37 s
3 1.9 to 2.0 s 0.18 to 0.30 s 0.17 to 0.44 s 0.17 to 0.19 s
4 1.9 to 2.0 s 0.35 to 0.51 s 0.31 to 0.38 s 0.31 to 0.36 s

Under the first filter the key set holds 3,177 keys and rejects 923,434 of the walk’s keys; setkeys took it at 179,000, the matches spread over the table’s distinct keys at the statistics' rate where they crowd on a few (§47.7). The plan is the same either way.

The feed (2026-10-28; oj_keyset_feed(), oj_feed_reads()). A key set’s plan runs once, before the walk, and reads its table’s matching rows whatever the walk’s LIMIT. Core’s cheapest path of the table prices its pages as if cached: cost_bitmap_heap_scan() never looks at effective_cache_size. So a bitmap heap scan that reads most of a large table at random beats a covering walk of the key’s B-tree (§40) that reads a tenth of the pages in order, and runs several times slower when the table is not in the cache - unless the walk’s price carries the bitmap heap scan’s cold reads, which it does (lo_cold_price(), §39), and wins.

The node picks the plan itself, among the table’s unparameterized paths and a Gather over each of its partial ones (core gathers only the cheapest, and add_path() may have freed the Gather over a parallel walk), at their total_cost, as core’s cheapest_total_path ranks them. The chosen plan keeps its own total_cost in §47.3’s price, and the pages it reads are §47.3’s ownreads for the key set, in place of the table’s pages (oj_feed_reads()): a lion walk’s, LionOrdered or LionBtreeScan, serial or under a Gather or Gather Merge (lion_ordered_scan_reads(): lo_cost() taken again); a bitmap heap scan’s heap and index pages (lion_units_bitmap_pages()); an index scan’s index pages and its rows' heap pages (index_pages_fetched(), those not all-visible where it is index-only); and the table’s for anything else.

Until 2026-10-29 the plans were also ranked by their own cold reads where pg_lion.competitor_pricing was off, the walks' prices then carrying none; that went with the switch (§49).

Measured (2026-10-28; PostgreSQL 18.6 with assertions, 16 vCPU, shared_buffers 512 MB, effective_cache_size 1 GB, random_page_cost 1.1, competitor_pricing off; core’s joins off so that the node is the plan). A 16,000,000-row key table of 2.9 GB whose filter a lion index answers, with a B-tree on its key; f.fk IN (SELECT fk FROM kt WHERE <filter>) with 240,000 matching rows (1.5%), the fact of 5,000,000 rows walked by a B-tree under LIMIT 50. Cold is a restart with every file of the cluster dropped from the kernel’s cache; three runs each.

key set’s plan cold warm
before Bitmap Heap Scan of 223,000 heap pages, priced 181,818 3.4 to 6.5 s 1.34 to 1.61 s
after Gather over a parallel LionOrdered of the key’s B-tree, index-only, 19,700 pages, priced 278,244 and 70,126 cold 1.2 to 1.5 s 0.49 to 0.51 s

With effective_cache_size large enough to hold the query’s tables and the plan’s indexes nothing is charged and the bitmap heap scan is taken, at core’s price.

Folded into the walk (2026-10-25; oj_key_index(), lion_ordered_keyset_keys()). Tested after the fetch, a key set leaves the walk to fetch every member of its set to read the key, and a selective key set has it fetch many rows for each it keeps. Where the fact’s key has a lion index the key sets are ANDed into the walk’s set instead, and an entry whose key a key set lacks is not a member: never fetched. The index (oj_key_index()): a lion index of the fact, not partial unless the query implies its predicate, with a key column that is the key itself, a plain column, scalar - an entry a row, not multikey - of the key’s type, whose equality strategy is each key set’s hashable equality, under the key sets' collation; so the rows of a key there are the rows that key set’s test passes, and an index that answers a key’s rows differently is not taken. The keys are known only at run time, so the fact’s plan carries a leaf for each key set (the fact side’s third member, their count LO_PRIV_INTS‘ fifth): fk = ANY (NULL::type[]) on that index, ANDed at the root of its tree before the lion access’s - or the tree alone where the fact has none - in the key sets’ order. When the join builds a key set (oj_exec()) it hands its keys to the walk as an array of the key’s type in the set’s memory (oj_keyset_array()), the leaf’s scan key’s argument (lion_ordered_keyset_keys(), as lion_ordered_probe_key() hands the dimension its key), which drops the walk’s set and restarts its walk; the set is then made from them as §30.4 makes any set, lazily or at once - a list longer than a batch built - an AND that a key set with no row empties before the lion access is read. The walk refuses to make its set while a leaf still holds the planned NULL. The join’s test stays and passes every row: the posting union is exact for a scalar equality, and the test costs a hash probe. What follows is LionOrdered’s: members are root TIDs and a member’s row the version the snapshot sees (§30.5) - an update of the key is not HOT and puts the new version under its new key; NULL keys are in no key’s postings; repeated keys are one array element each, the set’s; an empty key set ends the scan before the walk, its empty array handed all the same so that no leaf keeps a freed array; a rescan keeps the set unless a key set is rebuilt (its plan’s Param) or a lion qual’s Param changed. And the walk’s early stop and its fetch-and-sort switch (§30.4) now see the key sets: a set of a few thousand members switches to fetching and sorting them after 32 entries walked a member it has not met, which is the filter first, at run time. EXPLAIN prints Key Set Index: <index> after Key Sets, and the fact’s Lion Indexes leaves that index’s leaves out; under ANALYZE Lion Set Hits, Heap Fetches and Lion Set are the AND’s, Keys Rejected by Key Sets 0. pg_lion.enable_key_set_fold (on) offers it; the price is §47.3’s.

Measured (2026-10-25; PostgreSQL 18.6 release, the shared VM of four cores, shared_buffers 1 GB, work_mem 4 MB; warm, the median of seven runs, and buffers-cold - every buffer evicted with pg_buffercache_evict_all() before each run, the kernel’s cache warm - of five). A stand-in for the page: a 2,000,000-row fact (165 MB) walked by a B-tree of a random rank that does not hold the key, a 1,000,000-row dimension with a three-value lion tier, and a key table whose 746 keys hold 1,487 fact rows, 0.075% of the entries; ... WHERE f.fk IN (SELECT fk FROM kt WHERE prod = 7) ORDER BY d.tier, f.rank DESC LIMIT 100, and 19,817 keys (2%) for product 3. “Before” is the parent commit; core’s plan is with the node off.

the key’s index key set before after core’s plan
none 746 keys split, 419 ms (371,712 walked, all fetched, 371,412 rejected); cold 717 core’s hash join, 220 ms; cold 250 224 ms
lion 746 keys core’s, 37 ms (the split, forced: 473) core’s, 36 ms (the split folded, forced: 52 - 46,560 walked, 1,487 fetched, 0 rejected; cold 85 against 635) 37 ms
lion 19,817 keys split, 60 ms (13,342 walked and fetched) split folded, 77 ms (275 fetched) 328 ms
B-tree 746 keys core’s, 51 ms core’s, 32 ms 33 ms
B-tree 19,817 keys split, 72 ms; cold 123 core’s, 326 ms; cold 547 328 ms
none 19,817 keys split, 61 ms split, 66 ms 524 ms

Without an index of the key the page goes to core’s plan, 1.9 times faster warm and 2.9 cold. With the key’s lion index core’s plan is the key table first and the key’s lion index looked up for each key, 37 ms, and stays; the split folded runs 52 ms, nine times the split not folded. Folding the 2% key set reads 19,817 keys' postings to save 13,067 fetches, and is chosen at 77 ms where the split not folded ran 60. The 2% key set over a B-tree of the key is §47.7’s warm case. Ordered by the rank alone, with d.region < 10, the node gave way to core’s hash join without an index of the key (266 against 255 ms); at a work_mem of 64 MB core’s nested loop over the rank’s B-tree, a Memoize under its semi join, came out under both and ran 946 ms (§47.7).

Tested on the index tuple (2026-10-26; oj_itup_col(), lion_ordered_entry_test()). The walked B-tree may hold the key it joins on - (a, b, rank DESC) INCLUDE (fk) - while a lion index has the key as one of many columns and a plain B-tree has it alone. Folding reads the postings of every key; the B-tree’s index tuple has the key already. Where a column of the fact’s walked B-tree - a key column or an INCLUDE one - is the key column itself, stored as the key’s type and returnable (oj_itup_col(), and lion_ordered_entry_test() checks it again at run time), the join, not folded, has the walk hand it each member’s key from the index tuple before the fetch (oj_entry_test(): each key set’s hash probe, in a memory context of its own reset an entry), and an entry a key set rejects is turned away unfetched. The walk asks its B-tree for index tuples (xs_want_itup) to read the key, as an index-only walk asks for them to read its columns; that does not make it index-only, and it fetches the rows the key sets keep. Where the walk is index-only the test comes before the visibility map’s, and a rejected entry on a page not all-visible is not visited for its visibility either. It is right for the same reason the fold is: every version a TID of the B-tree reaches holds the key its index tuple holds, an update of an indexed column - an INCLUDE one too - being never HOT, so the version the snapshot sees, if any, would have been rejected after the fetch; a NULL key is rejected as the strict clause rejects it, and not counted. A rejected row takes no tuple predicate lock, which core’s plan, fetching it and filtering, would: under SERIALIZABLE the B-tree’s own lock on the leaf page it read stands for it, as for core’s index-only scan, and an update that moves the row’s key into a key set writes a new entry there (test/isolation/keyset_index_tuple_serializable.spec ends in the serialization failure core’s plan ends in). The join’s test after the fetch stays, and passes every row. A rejected entry whose row is dead is not marked dead in the B-tree either, as a fetch that finds its chain dead marks it (kill_prior_tuple), so a later walk meets it again until VACUUM, as core’s index-only scan leaves the entries it does not fetch: in ordjoin.sql a query walked twice walks 60 entries more the second time than it did when the first fetched them. EXPLAIN prints Key Sets Tested On: <index>, and under ANALYZE Heap Fetches are the kept members and Keys Rejected by Key Sets the entries a key set turned away; an entry whose key is NULL is in neither, turned away uncounted as the join’s test after the fetch leaves it uncounted. Folded paths are still offered beside them; add_path() keeps the cheaper.

Measured (2026-10-26; the stand-in above, its indexes of that shape: the key the third column of a multicolumn lion index (prod, x, fk), a plain B-tree on the key, and the walked B-tree (rank DESC) INCLUDE (fk); release build, the median of five, warm, and buffers-cold as above).

key set before the fold folded (§47.6 above) tested on the index tuple core’s plan
746 keys core’s, 35 ms; cold 61 core’s, 32 ms; cold 56 core’s, 25 ms; cold 49 34 ms
19,817 keys split, 57 ms (13,342 walked and fetched, 13,067 rejected); cold 493 split folded, 75 ms (275 fetched); cold 92 split, 27 ms (13,342 walked, 275 fetched, 13,067 rejected unfetched); cold 35 307 ms

Without the price of §47.3 for the rejected fetches the fold is not chosen (the split not folded, 52 ms warm), which is the plan that read the disk for the workload’s rejected rows; tested on the index tuple, the rejected rows are not fetched and need no price.

47.7 Not done

  • A key set past work_mem. Declined at plan time from the estimate; at run time a set larger than planned is held all the same, never spilled. A hash join spills; a key set’s rejections would have to be held back until its batches are read, which the walk’s order forbids.
  • The fact heap under a LIMIT. Core’s index scan’s heap pages are Mackert and Lohman’s over the whole walk’s fetches, each page counted once though the walk fetches it about seventy times, and the LIMIT takes its share of that: 343 pages for the medium page’s 26,257 fetches, which meet about 22,000. The node’s 3,030 member fetches are priced the same way, 343 pages for about 3,000. The model at the share’s own fetches would charge core’s loop some 86,000 more and the node 10,500. The Memoize’s hits are corrected (§39, “A Memoize under a LIMIT”); these pages are not, and with them the node’s startup, its sets' builds, keeps the medium pages core’s warm (§47.9).
  • The switch’s price. The fetch-and-sort switch acts at run time on what the walk meets; the path is priced as a walk to the end.
  • The dimension index-only. The probe always reads the member’s heap row. Where a B-tree on the key holds every column the query reads of the dimension, §40.10’s nested loop probe (LionBtreeScan) answers it from the index; a covering form of the join’s probe is not built.
  • A key set’s estimates under correlated clauses. pass and setkeys (§47.3) take the table’s clauses as independent of its key: a filter whose matches crowd on a few keys, or on the keys the fact holds most rows of, is estimated at many times the keys it holds, as core’s own estimate of the semi join is. That leans toward declining a key set for work_mem and pricing it as passing more than it does; nothing corrects it at run time but the walk’s own switch.
  • Key sets of another type: a key column of a type other than the fact key’s, or an equality whose two sides hash with different functions, is declined rather than cast.
  • A semi join’s inner side of several tables, a key set by IN over a join: only one table.
  • Parallel, parameterized paths, a partitioned table several leaves are left of (one leaf: §16, “A parent pruned to one leaf”), an outer join as the join itself, PHVs and LATERAL references at the join: declined (§47.1).
  • More than one dimension. A second key join to another dimension is core’s, above the node.
  • A walk that fills slowly, at run time. Not alone under the LIMIT the node is priced at its whole walk (§47.10), and alone a key set’s keys are taken to be spread over it; nothing at run time turns from the walk when its rows come slowly - the node having returned many times the LIMIT’s rows and still pulled (a filter above), or a key set’s walk past many times the entries priced. Under a key set the turn could be to the filter first: its keys looked up in a B-tree of the fact’s key and their rows sorted in the walk’s order past its position (the B-tree’s order is the score’s, then the TID’s), arm by arm under the split - about core’s plan, after the walk before it. Folded into the walk’s set (§47.6) that is the fetch-and-sort switch’s, where the fact’s key has a lion index; without one it is not done. Under a filter above, which the node does not see, it could only be the rest of the join at once, the two sets hash-joined and sorted: core’s plan of the join without the page’s filters, a bound, but the dense page’s 6.5 seconds on §47.10’s tables. Either needs a price for the switch, as §30.4’s has, and a sort past work_mem. Where the filter above is an EXISTS or IN whose table could be a key set, the turn is to the key set (§47.11): the set deferred, and built once the walk has run past the rows it was priced for.
  • A key set’s keys tested before the fact’s heap, where neither index holds the key. Folded where the key has a lion index, tested on the index tuple where the walked B-tree holds it (§47.6). With neither, the walk fetches the member’s row and the join reads its key.
  • The rejected fetches on a warm machine. Not folded, and where the walked B-tree does not hold the key (§47.6, “Tested on the index tuple”), the fetches a key set rejects are priced at their own pages (§47.3), each page a random_page_cost, which is what they cost where the walk reads the disk, and many times what they cost where the table is in memory. There the node can lose a page it runs faster: §47.6’s measurements, the 2% key set over a B-tree of the key, 72 ms through the split and 326 through core’s plan. And core’s own nested loop over the fact’s B-tree, its key table’s semi join Memoized above, keeps its Limit’s share of the whole walk’s pages: where the key has no index and the ORDER BY is the B-tree’s, that loop can come out under the node and core’s filter first and be the slowest of the three (946 ms at a work_mem of 64 MB, against the node’s 354 and the hash join’s 292; the split’s order has no such loop). At random_page_cost = 1.1 the 2% key set keeps the split, and the 0.075% one keeps it too (405 ms against core’s 224).
  • A key set’s other columns. A table a column of which the query reads is not a key set, even a column its clauses pin to a constant, which the node could fill as it fills the key: §47.10 says why that would cost the page its plan. A key that repeats under the table’s clauses is not one either: the rows would have to come out once for each of its rows.
  • Arms a key set passes over. Under the split, an arm none of whose dimension rows a key set’s keys name walks to its end (§47.10, “What a key set leaves”) - folded (§47.6), to the last member of its set, or to the switch, which fetches and sorts them. The key set’s keys could be descended once into the dimension and tested against each arm’s set, and an arm with none passed over; it is not.

47.8 Tests

test/sql/ordjoin.sql: a 100,000-row dimension with a primary key, lion indexes on a ten-value int, a bool, a text array and a hundred-value int, and a wide payload, so that its heap is many times its B-tree; a 500,000-row fact table, every 101st key NULL and every 89th score, a B-tree (grp, score DESC NULLS LAST) INCLUDE (fk) and a lion index on a text array. Each query is run through the node, lion’s paths priced at a thousandth so that the join is the node wherever it may be, and through core’s plan with the node off, and the answers compared row by row, in order, every ORDER BY ending in a unique key; oj() says whether the first plan was the node and how many key sets it had. The plan chosen with nothing forced: the node for a page the walk fills early, core’s hash join and sort for one whose rows are rare; core’s nested loop for a third warm, the node at effective_cache_size = 1MB (§47.3’s cold reads: the node’s price 4.2 times core’s warm and 0.53 of it cold, on PostgreSQL 17 and 19 alike, a margin that a plan’s prices moving with the sample ANALYZE takes does not cross); core’s nested loop over a Memoize for a fourth with pg_lion.enable_memoize_limit off, the node with it on (the node’s price 1.11 and 0.90 of core’s: no page of this table that the term flips clears core’s by more, warm, at a LIMIT of 20 to 200). The walk forward and backward, NULLs where the order puts them; an OFFSET; a side with no lion clause, each way; an inexact dimension set, rechecked; a fact filter; an Incremental Sort above for a trailing dimension key, a LEFT JOIN and a LEFT JOIN … IS NULL to a third table, and under a LEFT JOIN with LIMIT 5 the dimension’s set lazy and built on its probe budget (after 20 keys probed (probe budget), §47.3’s share); a generic plan’s Param in a lion qual; a LATERAL subquery rescanned for four outer rows, the dimension’s set, which depends on their value, rebuilt for each (4 builds) and the fact’s built once. Key sets: two at once (an IN on the fact’s key, an IN on the dimension’s of a table whose keys repeat), an IN, an EXISTS, an inner join to a unique column, an empty set ending the scan, a set rebuilt for each value of a LATERAL rescan (Key Set Builds: 4); and what is not a key set - an inner join to a column that is not unique, a table a column of which the query wants, a NOT EXISTS, keys past work_mem (64 kB). Two different tables make the two key sets: PostgreSQL 19 merges two semi joins of one table into one. The fact’s walk index-only, with a set, with none, and with an inexact set whose columns the B-tree lacks (Heap Fetches its candidates); the switch, made due at once by pg_lion.ordered_switch_ratio = 1. The rows out where half the fact’s keys name no dimension row (orph, 200,000 keys against the dimension’s 100,000): the node’s estimate is about the join’s count (0.7 to 1.4 times it, the check coarse enough that ANALYZE’s sample does not move it), with a key set and without; taking every non-NULL key to name a row would make it about twice (1.6 to 2.4). Declined: a second join clause, a key that is not unique, FOR SHARE, no lion clause on either side, the setting off (core’s nested loop over a Memoize). Section 9b, a key set joined above (§47.6, “A join above”): at a join_collapse_limit of 1 an EXISTS, an IN, two of them (one of a table whose keys repeat) and an empty one, each a key set of a node its semi join is planned above; not one: a table whose clauses call a volatile function, a NOT EXISTS, the fact and the dimension on a left join’s nullable side. A nine-table query - four left joins, a LATERAL subquery, an anti join and an EXISTS - at the default limits, a key set from above (the EXISTS on a unique key an inner join, and on a key that repeats a semi join, its table read twice in the plan: the key set’s child and the semi join’s), and not with an EXISTS that tests more than the key; at 16 the same query planned whole, its key set inside the join; under GEQO.

btreescan_param.sql’s cursor section turns the node off: under that section’s pushdown_margin = 1000 the node took the join from the LionBtreeScan probe the section tests.

test/sql/ordjoin_alone.sql, §47.10, in a schema of its own: a 100,000-row wide dimension with lion indexes on a bool flag and a hundred-value lvl; a 350,000-row fact table of four slices (grp, ver), the pages' (1, 2) holding the keys up to 50,000 and the others every key, with a B-tree (grp, ver, score DESC NULLS LAST) INCLUDE (fk), a unique (fk, grp, ver) and a lion index on a three-value rec; a key table unique by (fk, kind), kind 2 some 8,300 keys, all past the slice’s but forty, kind 1 some 3,900 over every key; a side table by key and day whose 'high' rows in the slice are those of its fifth of lowest scores and its NULL ones; the suppressed keys, one in fifty. Each query runs through the node, forced, and through core’s plan, the answers compared in order; oja_plan() names the plan with nothing forced, LionOrderedSplit, LionOrderedJoin or core’s, and its key sets. Core’s plan, with the node priced at 3.4 to 25 times core’s: the page (the key table, the side table’s clause read, the anti join) in the split’s order and the join’s, and its counters forced - both arms walked, 100,000 entries, 99,920 rejected by the key set, 80 descents; the side table’s clause alone; a clause on the side table that is not strict, so the left join stays one with a filter above; the key table with a column read, and with kind IN (2, 3), which pins nothing. The node, priced 1.4 to 47 times under core’s: the split with only a left join and an anti join above, and with a NOT EXISTS; the join, ordered by the score, under a left join and a NOT EXISTS; the split with kind 1 a key set (Arms Run: 1, 332 descents); and with kind 2 a key set, alone, its walk both arms long (§47.10, “What a key set leaves”). Not alone and still taken: the split under a LATERAL subquery’s clause, its LIMIT 1 keeping it from being pulled up, at 2.2 times under core’s price. A key set’s key in the target: with nested loops and merge joins off, the side table’s hash join above reads the key set’s key, Hash Cond: (s.fk = kt.fk), which the node fills.

test/sql/ordjoin_keyfold.sql, §47.6’s fold, in a schema of its own: a 20,000-row dimension with lion indexes on a three-value tier and a ten-value region; a 200,000-row fact table, ten rows a key, every 97th key NULL, with a B-tree by a random rank that does not hold the key and a lion index on a twenty-value prod; a key table of sixty keys for product 7 (600 fact rows) and 2,000 for product 3, and one of 40 keys repeated and NULLs. Each query through the node, forced, and through core’s plan, answers compared in order. Without a lion index of the key, the split led by the tier under product 7’s key set walks 18,275 entries, every one fetched, 18,043 keys rejected; with nothing forced the plan is core’s, the walk priced at its share’s fetches (§47.3), and for product 3’s two thousand keys the split’s. With fct_fk, a lion index of the key: the same split forced walks 17,792 entries over a set of the key set’s 610 rows, 54 met, then switches and fetches the 556 left, 610 fetches in all and no key rejected; the join with a fact and a dimension lion clause besides, its set the AND of both; with nothing forced core’s plan for product 7 - the key table, then the key’s lion index - and the split folded for product 3. A key set of repeated keys and NULLs; two key sets, an IN on the fact’s key and an EXISTS on the dimension’s, two leaves; an empty one; the whole join without a LIMIT; a LATERAL rescan whose key set is built and handed again for each of three rows (Key Set Builds: 3); a key updated and a row inserted in the transaction before the query. Not folded: a partial lion index of the key whose predicate the query does not imply, and folded into where it does; a key table of bigint keys, no key set; pg_lion.enable_key_set_fold off. Section 6, the key tested on the index tuple: with no lion index of the key and the B-tree (rank DESC) INCLUDE (fk), the split under product 7’s key set walks the same 18,275 entries and fetches 55, 18,043 keys rejected unfetched - the other 177 entries NULL keys, turned away uncounted - and with nothing forced the split is the plan, for product 3 too; repeated keys and NULLs, two key sets, a LATERAL rescan; a key updated, a row inserted and rows of the set’s keys updated HOT in the transaction; and the key a key column, (rank DESC, fk). test/isolation/keyset_index_tuple_serializable.spec: the write skew of a row whose key an update moves into the key set, after the join turned its entry away unfetched, ends in a serialization failure in either order of the commits. ordjoin.sql’s B-tree fact_grp_score holds the key (INCLUDE (fk)), so its key sets are tested on the index tuple too: 191 heap fetches where there were 19,944, 2,750 where there were 82,302.

47.9 Measured (2026-10-23; PostgreSQL 17, the dev cluster - a VM of four cores, shared_buffers 2 GB; warm and cold; EXPLAIN (ANALYZE, TIMING OFF))

A 5,000,000-row fact table (554 MB; its key uniform over the dimension, every 101st NULL; score uncorrelated with the heap) with a B-tree (score DESC) INCLUDE (fk, id) (193 MB) and lion indexes on an eight-value int chan and a text array tags, one of four a’s and one of nine b’s; a 200,000-row dimension of 576-byte payloads (120 MB, a 4.4 MB primary key) with lion indexes on a ten-value kind, a hundred-value lvl, a bool flag and a five-value region; a 20,000-row key table. The query is SELECT f.id, f.score, f.amount, d.pk, d.region FROM bfact f JOIN bdim d ON f.fk = d.pk WHERE ... ORDER BY f.score DESC LIMIT 20, amount in no index, so that the fact’s walk is in heap mode; without it the walk is index-only. “Walked” is the node’s Index Entries Walked. Warm: every table and index in shared buffers, the median of seven runs after one. Cold: effective_cache_size = 400MB, which the tables and B-trees, about 870 MB, are twice, as a 50-million-row fact table is twice the default 4 GB; the cluster restarted and the kernel’s page cache dropped before each run, the median of three. “Planned” is the plan with nothing forced, at the default 4 GB and at 400 MB; the node’s time where it was not planned is with pg_lion.pushdown_margin = 1000. Core’s cold time is the plan with the node off, which for the medium pages, the key set’s and the deep one is core’s nested loop over lion’s LionOrdered walk of the fact and a Memoize.

page WHERE walked core’s plan warm: core the node planned, 4 GB planned, 400 MB cold: core the node
dense f.chan = 3 AND d.region = 'north' 720 nested loop, Memoize 0.88 ms 1.67 ms core node 45.0 ms 17.6 ms
dense f.tags @> '{a1}' AND d.region = 'north' 491 nested loop 1.03 ms 1.82 ms core core 36.1 ms 15.8 ms
dense f.chan = 3 AND d.lvl < 50 248 nested loop 0.59 ms 7.61 ms core core 18.1 ms 19.4 ms
dense, a key set the first, AND f.fk IN (SELECT k FROM bwatch WHERE active) 2,829 nested loops 2.42 ms 3.43 ms core node 29.1 ms 28.6 ms
medium f.chan = 3 AND d.lvl = 3 AND d.flag 26,257 nested loop 34.0 ms 11.6 ms core node 343 ms 195 ms
medium f.tags @> '{a1}' AND d.lvl = 3 AND d.flag 9,595 nested loop 11.6 ms 6.9 ms core node 244 ms 159 ms
medium, index-only f.chan = 3 AND d.lvl = 3 AND d.flag 26,257 nested loop 31.7 ms 9.0 ms node node 190 ms 39.0 ms
the dimension’s only, index-only d.lvl = 3 AND d.flag 3,902 nested loop, Index Only Scan 6.9 ms 5.1 ms core node 189 ms 27.8 ms
dense, OFFSET 2000 f.chan = 3 AND d.region = 'north' 81,388 nested loop 101 ms 49 ms core node 893 ms 599 ms
sparse f.tags @> '{a1}' AND f.tags @> '{b2}' AND d.lvl = 3 AND d.kind = 3 730,643 hash join, Sort 67.8 ms 173 ms core core 534 ms 1,184 ms

Where the page is neither dense nor sparse the node runs 1.4 to 3.5 times faster than core’s nested loop, which fetches every fact row the walk meets to test the fact’s filter and reads the dimension’s heap row of every key it keeps: the node fetches a member’s row, and the dimension’s of a member’s key, one dimension row in about a hundred descents here. The index-only walk takes the fact’s heap out (0 heap fetches, 9.0 against 11.6 ms). A deep page gains as much, 2.1 times. Where the page is dense core reads a few hundred fact rows and about a hundred dimension rows, and the node pays its sets: the scalar ones lazily, each built after 64 keys by §30.4’s heap-order rule (1,109 and 241 containers, about a millisecond), the multi-key fact set at once; a range on the dimension (lvl < 50, fifty posting lists) is always built at once, 7 ms; a key set’s table is read whole first, a Seq Scan of 20,000 rows. Where it is sparse the walk meets 730,000 entries for twenty rows, and core’s hash join of the two sets and its sort is 2.6 times faster.

Warm, at the default 4 GB, the planner takes the node for the medium index-only page (9.5 ms, timed again, against 31.7) and keeps core’s plan elsewhere: rightly where the page is dense or sparse, wrongly for the other medium pages, the dimension’s only and the deep one. The fact set’s build (§47.3: chan = 3’s 3,772 now about 960) and core’s Memoize (§39, “A Memoize under a LIMIT”) are corrected. Per row kept, the two prices of the fact’s heap agree: the node’s fetches are Mackert and Lohman’s over the walk’s 625,000 members and the table’s 70,900 pages, about 0.45 a fetch, and core’s Index Scan fetches all five million rows of its walk over the same pages, about 0.06 a fetch, eight to a member. The medium scalar page: the node 1,150..818,180 for 4,129 rows before the Memoize’s scale of 0.715, its Limit 5,110; core’s loop 0.9..587,688 for the same rows, its Limit 2,847, and 3,982 with the hits its 3,030 calls meet where core counted 70% of them. The node’s run under the LIMIT, 3,960, is now core’s corrected 3,982; what is left is its startup, the sets' builds, 1,150, against core’s fact heap, which Mackert and Lohman price at the whole walk’s share (§47.7, “The fact heap under a LIMIT”). The multi-key page’s set is built at once (2,600 of startup); the deep page comes to 11,040 against 9,594; and the dimension’s only page ties, 235.2 against 235.3, where add_path() keeps the loop’s lower startup. A descent of the node’s is btcostestimate()’s, (tree_height + 1) * 50 operator costs and its comparisons, 0.39; core’s probe, the same index scan under the Memoize, was priced 0.19 on its hit ratio over the whole walk. The dense page f.chan = 3 AND d.region = 'north': the node 1,280..861,590 for 123,747 rows, its Limit 1,420, its sets' builds most of it; core’s loop over a Memoize, its Limit 95.8.

Cold, the planner takes the node for every medium page, the dimension’s only, the deep page, the dense scalar page and the key set’s, and each flip is right: the medium pages 1.8, 1.5 and 4.9 times faster than the plan without the node, the dimension’s only 6.8 times (27.8 against 189 ms), the deep page 1.5 times (599 against 893 ms), the dense scalar page 2.6 times, the key set’s a tie. The deep page’s competitor is core’s loop over lion’s LionOrdered walk, weighed with that walk’s reads (§47.3): 42,317, which its Memoize’s hits under the LIMIT bring to 68,059, against the node’s 52,452 before the cold reads' 0.96, 31,306 in the loop’s units. Core’s plan stays, rightly, for the sparse page (534 against 1,184 ms) and the dense range (18.1 against 19.4 ms); and wrongly for the dense multi-key page (36 against 16 ms): core’s loop at 944 under the LIMIT, 1,264 with its Memoize’s hits, against the node’s 3,219, most of it the multi-key set built at once, of which the cold reads take 0.54. Not measured: other majors.

47.10 Alone under the LIMIT (oj_alone(), oj_limit_share(), 2026-10-24)

What the LIMIT gives the node - the share of the walk it takes (§47.3: each side’s set lazy or at once, the competitor’s Memoize under the LIMIT, the cold reads over that share) and the split’s shape (§48.3) - it gives only where the node is alone under the LIMIT: where the rows the LIMIT counts are the node’s, one to one, or about.

The rule (oj_alone(), asked once for each join rel in oj_add_paths(), the answer in OjJoin.alone). After reduce_outer_joins(), which makes a left join whose inner side a strict WHERE clause reads an inner join, and a left join under IS NULL of its key an anti join:

  • every other table of the query level is on the inner side of a left join or an anti join that the join rel is not inside. A left join returns each of the node’s rows; an anti join drops those it matches, and is taken to match few, the exclusions a NOT EXISTS lists. A table joined by an inner, semi or full join, or by none, is not;
  • the join rel is on no special join’s inner side but its own key sets' semi joins (oj_special_ok());
  • every clause that reaches past the join rel (its joininfo) is an outer join’s own ON clause (not is_pushed_down), or is evaluated within the inner side of such a left or anti join, where it filters only that side; a WHERE clause on an inner side’s columns carries the outer join’s relid and is neither. No equivalence class joins the join rel to another table (a class with a constant makes no join clause);
  • the query level asks nothing more of the rows: no HAVING, no set-returning function in its target, which may return no row for one, and no WITH TIES.

Grouping, aggregates, DISTINCT and windows leave root->limit_tuples at -1, so there is no share to take; row marks are declined (§47.1). A Sort or Incremental Sort above for the rest of the ORDER BY is core’s, priced from the node’s own start-up, total and rows. The key sets are inside the join rel (§47.6), so a table folded into one filters nothing above it. lo_limit()’s other conditions (§30.4) are its hard stop’s and not a price’s: a LIMIT of a Param is priced at core’s guess, as core’s own plans are, and the order past the node’s is the sort’s above.

Not alone, the share is 1 (oj_limit_share()): the sets are chosen for the whole walk (at once where it is long, §47.4), the competitor’s Memoize gets no correction for the LIMIT (lion_memoize_limit_extra() adds nothing at a share of 1), the cold reads are weighed over the whole walk, and the path’s start-up is its total (oj_add_path()), so that core’s Limit, which takes startup + (total - startup) * n / rows of its input, pays all of it. The split is priced as without a LIMIT, every arm, its start-up its total too (os_cost(), §48.3). Both are still offered. At the price of the walk to its end the node is taken only where core’s plans cost more: where they sort a large join for the page, or run a subquery for each row of a large first group of the ORDER BY. The walk then costs at most what it was priced at, wherever the page’s rows are.

Why core’s estimate is no price under a LIMIT. Core prices a join or a clause above the node by the share of rows it expects through, sel, spread evenly over the rows beneath, and its Limit takes n / (rows * sel) of everything below: the walk is priced at the entries it takes to meet n survivors at that density. Under an ordered walk the survivors are where what the filter keeps falls in the walk’s order, and nothing in the statistics says where that is: a small table’s keys among the fact rows the walk meets last, a side table’s flagged rows those of the lowest scores, a key table of the dimension rows of one value of the split’s column. Where they come late the walk runs on, possibly to its end - every entry fetched, descended and joined above - priced at a few hundred. The page of the tests (§47.8) is this: core takes the filters above the walk to pass some 3,000 of its rows, ten do, and the walk’s two arms run to their ends for them. Core’s own plans for it are a filter first and a sort, which read what they read whatever the order; core’s ordered nested loops under a LIMIT carry the same estimate, and it is theirs.

What is pessimistic. A filter above that is spread over the walk as core takes it would let the walk stop early, and the node is priced as if it would not: a selective clause on a side table’s column that is not correlated with the score is priced as the whole walk, and core’s plan is taken (the measurements below: 968 ms against 37 ms for the walk). So is an inner join that drops no row, a foreign key’s to the referenced table’s key with the fact’s key NOT NULL.

What it still takes on trust. An anti join that drops many rows is taken to drop few. And the joins above the node are core’s, priced at core’s density: not alone under a clause on a LATERAL subquery that core cannot pull up (its LIMIT 1), the split is taken at the price of its whole walk, under core’s plans, which sort the whole of the flag’s first value and run the subquery for each of its rows; the nested loop of the subquery above the split takes the LIMIT’s share of its own run, and the tests' page of that shape runs 195 ms against core’s 105 to 150.

What the key sets take back (§47.6). A table folded into a key set is inside the node: its rejections are the node’s, an entry of the fact’s walk and a hash probe each, before any descent, and it filters nothing above. The page’s small key table was kept out of §47.6 for two reasons, both lifted:

  • its key unique only with its primary key’s other columns pinned - kt (fk, kind) under kind = 2 - which lion_column_is_unique(), a unique B-tree of the key alone, does not prove: oj_keyset_unique() takes a unique B-tree whose other columns the table’s own restriction clauses pin to a constant each, the key set holding only those rows' keys;
  • its key in the join rel’s target, which the query does not read: a LATERAL subquery on the key, which a strict WHERE clause made an inner join, joins the key’s class above the join rel, and core makes every member of the class available there, the key set’s among them. Where nothing outside the class needs the key, its type has image equality and the class’s equality is the type’s default, the node fills the column with the fact’s key (§47.6).

Not lifted:

  • a key that repeats - kind IN (2, 3), which pins nothing: each fact row would have to come out once for each of the key’s rows, a count a hash set does not hold;
  • a column of the table read, even one its clauses pin to a constant. The node could fill a pinned column with its constant as it fills the key; that would fold the page’s side table - s.level read and pinned to 'high' by the WHERE clause, its key unique with its other columns pinned - into a key set, and make the page’s own shape alone, under the key set’s estimate, which is the one that fails there (below);
  • a second join clause, a key of another type, a key in the target whose type lacks image equality, a set past work_mem (§47.7), and a join rel core never forms as the rest joined to the key set of the highest relid (§47.2), as GEQO, or a join order the collapse limits keep, may not.

What a key set leaves. Its share of the walk is still core’s estimate: its keys are taken to be setrows / dim.tuples of the dimension’s rows, spread evenly over the walk and over the split’s arms (§47.3, §48.3 “Independence”). Where they are not - the slice holding forty of kind 2’s keys, or a key table of flag = false rows under a split led by flag DESC - the walk runs to its end as it would under the join above, but each entry it rejects is an index entry and a hash probe, not a fetch, a descent, a dimension heap read and a join: 2.3 seconds where the join above took 20 and core’s plan, the key table first, under one (below). Bounded by the walk, not by the page.

Measured (2026-10-24; PostgreSQL 17, §47.9’s cluster and tables, warm: every table and index in shared buffers, the median of five runs after one; EXPLAIN (ANALYZE, TIMING OFF)). §47.9’s tables with an index on the fact’s key, bfact (fk), as the page’s own fact has one, and three tables of the page’s shape: bib (fk, kind, since), primary key (fk, kind), kind 3 the keys of 12,000 flag = false rows, kind 1 4,000 keys spread over all; bsig (fk, day, level), primary key (fk, day), 'high' for a tenth of the keys, spread; bsup (fk, reason), every 50th key. The page is SELECT d.pk, d.flag, d.region, f.id, f.score[, s.level] FROM bdim d JOIN bfact f ON f.fk = d.pk LEFT JOIN LATERAL (SELECT s.* FROM bsig s WHERE s.fk = f.fk AND s.day = 3) s ON true [JOIN bib ON bib.fk = d.pk AND bib.kind = K] LEFT JOIN bsup ON bsup.fk = d.pk AND bsup.reason = 1 WHERE bsup.fk IS NULL [AND s.level = 'high'] ORDER BY d.flag DESC, f.score DESC, f.id LIMIT 100, the walk index-only. “Taken under any LIMIT” is the same build without oj_alone() and the two key set rules above. Core’s plan, with the node off, is the key table or the side table first, the fact by its key and a Sort, but for the dense page’s, a Sort over the hash joins of the whole fact.

page above the node taken under any LIMIT §47.10 core’s plan
the page bib kind 3, s.level = 'high', s.level read split, 19,884 ms (5,019,060 walked, 4,969,363 descents) core’s, 151 ms 149 ms
the side table’s clause alone s.level = 'high', read split, 37.4 ms (1,807 walked) core’s, 968 ms 983 ms
the same, s not read s.level = 'high' split, 36.1 ms split, key set bsig, 47.7 ms 954 ms
the key table alone bib kind 3 split, 20,116 ms (4,952,137 descents) split, key set bib, 2,344 ms (5,001,659 walked, 297,188 descents) 767 to 867 ms
the page, s not read bib kind 3, s.level = 'high' split, 18,189 ms split, key sets bib and bsig, 1,597 ms (5,019,060 walked) 151 ms
dense, a key set bib kind 1 split, 78.4 ms (7,681 descents) split, key set bib, 10.4 ms (165 descents) 359 to 385 ms
dense the left join and the anti join split, 8.8 ms (170 walked) split, 9.2 ms 6,452 ms

The page goes to core’s plan, 130 times faster; the dense pages keep the split, 35 to 700 times faster than core’s; the dense page with the key table gains the key set, 7.5 times. The side table’s clause alone is the rule’s pessimism: spread as core takes it, it costs the walk 26 times over. Read, the side table stays a join above; not read, it is a key set (its key unique with day pinned) and the split is taken, 47.7 ms. The key table alone, and the page with the side table not read, are a key set’s estimate: 8.6 and 11 times faster than the join above, about 3 and 11 times slower than core’s plan. Ordered by the score alone, the page is core’s either way (50 and 54 ms). Not measured: cold, other majors.

47.11 Deferred key sets (oj_add_paths(), oj_defer_at(), 2026-10-29)

A key set is built before the walk: its plan runs to its end, and the node’s start-up is that plan’s whole price (§47.3). For an EXISTS whose filter matches many rows - hundreds of thousands over a large share of the keys - the plan reads every one of them, while a page under a LIMIT is met in a thousand or so rows of the walk, a tenth of which the EXISTS passes. Built first, the set costs the node more than core’s plan of the page, which joins the whole fact to the dimension and sorts it, and core’s plan is taken: about 1.4 seconds against 0.07 on the measurements below. Where the EXISTS is joined above the node (§47.6, “A join above”), or inside the join search but above the join rel, the semi join above tests the node’s rows anyway; the set only spares the walk the descents of the rows that join would drop.

The deferred paths. Under a LIMIT each join is offered again with the key sets above deferred (oj_add_paths()): the tables of §47.6’s “A join above”, and, for these paths alone, the table of an EXISTS or IN inside the join search, the whole inner side of a semi join above the join rel, by the same rules (oj_keysets_above() with insearch; an inner-joined table inside the search is not taken, its join being the one that returns its rows). A deferred set is not built before the walk, its keys are not tested, and it is never folded (§47.6), whose fold needs the keys before the walk; the key sets of the join rel itself are built as before. The joins above filter the node’s rows, which they drop abovepass of (§47.6), and the node is alone under the LIMIT where the deferred sets' tables are its only other filters (§47.10).

Price (oj_cost(), os_cost()). A deferred set adds nothing to the start-up and no test a key, and lets every key through: the descents, members and rows are the walk’s without it. The LIMIT’s share is taken over the rows the joins above return, out * abovepass (oj_above_rows()), in place of out * aboverep for the sets built first; the split’s arms are priced to meet limit / abovepass rows. The semi join above is core’s, at core’s price of its probes. Both ways are offered and add_path() keeps what it keeps: a narrow EXISTS, whose rows the walk meets in many thousands of entries, keeps its set built first.

The turn (oj_defer_at(), oj_exec()). The node counts the rows it returns, over every rescan - the split’s arms are rescans - and at deferat of them builds the deferred sets; from the next row on, each key is tested as a key set’s is, on the walked B-tree’s index tuple where it holds the key (§47.6, “Tested on the index tuple”), and an empty set ends the scan, as one built first does: the join above would drop every row the node returned. deferat is twice the rows the LIMIT is expected to take of the node, limit / abovepass, or, where more, the rows that cost what the sets' plans do, as a lazy set builds itself once its probes have cost what the build would (§30.4): a row costs its share of the node’s run and, for each deferred set, the join above’s test of it, the cheapest parameterized path of the set’s table (a probe of its key; a cpu_operator_cost where it has none, a hash join’s probe). A walk whose rows the joins above keep fewer of than they were priced at - the EXISTS’s keys among the rows the walk meets last - is then given up on at about twice its price, plus the build, for the walk the built sets filter. Nothing the set drops is a row the join above would keep: before it is built the node drops nothing, after it only rows whose key the EXISTS’s table lacks.

EXPLAIN prints (deferred) after each deferred set’s table in Key Sets, and Key Sets Deferred For: <N> rows; Key Set Builds is 0 where the walk never reached them. pg_lion.enable_key_set_defer (on) offers the deferred paths.

Tests (ordjoin_defer): a broad tag deferred and never built, with the EXISTS above the explicit join (join_collapse_limit 1) and inside the search, and as an IN; a tag on the keys of the fact’s lower half of scores, priced as met in two hundred rows, deferred, built once at the turn, the same rows; a narrow tag not deferred; the switch off and no LIMIT, nothing deferred; with a lion index on the fact’s key, a deferred set never folded, the split’s paths as the join’s. Every answer against core’s plan with the node off, row by row.

Measured (2026-10-29; PostgreSQL 18.6 built without assertions, 4 cores, shared_buffers 4 GB, warm, the median of three runs after one; random_page_cost 1.1, work_mem 256 MB). A dimension of 3M rows, 80% flag; a fact leaf of 2M of its keys with a score B-tree including the key; a child table of 20M rows over the dimension’s keys, 10M of them on 5,000 keys, a tsvector of three tags each; the page of §47.10’s shape with eight tables, EXISTS (SELECT 1 FROM child c WHERE c.doc @@ q AND c.fk = d.pk), ordered by d.flag DESC, f.score DESC, d.rel DESC, LIMIT 100:

q collapse limits 8 (EXISTS above) limits 16 (inside the search)
(s1 | s2) & (f1 | f2) (554,462 rows, 9% of the keys) 1,409 ms (core’s sort of the join) to 64 ms, deferred 1,472 ms to 69 ms
s1 & f1 & w1 (144 keys) 980 ms, set built first, either way 29 to 30 ms, either way

The same rows in every case, and with the node off. The narrow tag at limit 8 walks the whole leaf (3.9M entries) for its keys: the leaf’s lion index has no column of the key, so the key set cannot be folded (§47.6), and every entry is tested on the B-tree’s tuple; with a lion index on the key it takes 300 to 340 ms. Not measured: cold, other majors.

48. An ORDER BY led by the dimension: the join an arm at a time (LionOrderedSplit, lion_ordjoin.c, 2026-10-24)

The shape is §47’s with the ORDER BY led by a column of the dimension:

SELECT f.id, f.score, d.pk, d.flag
  FROM fact f JOIN dim d ON f.fk = d.pk
 WHERE f.grp = 2 AND f.tags @> '{a1}' AND d.lvl < 30
 ORDER BY d.flag DESC, f.score DESC NULLS LAST, f.id LIMIT 20

asks for the fact’s best rows among the flagged dimension rows first, then among the rest. No B-tree of the fact gives that order, so §47’s walk is not offered (§47.1, “The order”), and core has one plan: the hash join of the two sets' bitmap heap scans, every joined row sorted for the first twenty: 7,525 rows here, and on §48.8’s tables 625,000 fact rows read for twenty. But the leading column has few values, and lion holds each value’s rows as an entry of its directory, in the column’s order (§30.11). Among the rows of one value the ORDER BY is the fact’s walk: §47’s join with the dimension narrowed to that value’s rows returns them in the order wanted, and run for each value in turn, in the ORDER BY’s order, the joins' rows one after another are the query’s.

Limit
  ->  Incremental Sort
        Sort Key: d.flag DESC, f.score DESC NULLS LAST, f.id
        Presorted Key: d.flag, f.score
        ->  Custom Scan (LionOrderedSplit)
              Split By: dim_flag (flag, backward, nulls first)
              Arms Max: 8
              ->  Custom Scan (LionOrderedJoin)
                    Ordered By: fact_grp_score
                    Index Cond: (f.grp = 2)
                    Lion Cond: (f.tags @> '{a1}'::text[])
                    Lion Indexes: fact_tags
                    Dimension: dim d via dim_pkey
                    Dimension Lion Cond: (d.lvl < 30)
                    Dimension Lion Indexes: dim_lvl

LionOrderedSplit is the join rel’s CustomScan over one LionOrderedJoin, its child. It walks the column’s lion entries in the ORDER BY’s direction with §30.11’s walk, the NULL entry where NULLS FIRST or LAST puts it; for each entry - an arm - it narrows the child’s dimension to the entry’s rows, the entry’s TIDs ANDed with the dimension’s own set, and runs the child from the start of its walk; past the arm’s last row, the next entry is the next arm. The dimension’s set is made once and kept, as is the fact’s; only the fact’s walk restarts. Core puts the rest above it: the Limit, and an Incremental Sort for what the ORDER BY has past the walk - here f.id, which fact_grp_score does not hold.

48.1 What qualifies (os_split_of())

  • The join. Everything §47.1 asks of the join, its sides and the query - one key join to a unique B-tree, each side a table LionOrdered may scan, the target, no row marks - with its key sets (§47.6); what §47.1 declines the split declines (FOR SHARE OF d, a LEFT JOIN). Neither side needs a lion clause and no key set is needed: the arm’s narrowed set is a set of the dimension’s.
  • The column (lion_ordered_column_order(), lion_ordered.c; lo_column_gives(), the rule of §30.11’s walk of a table’s own column). The ORDER BY’s first sort key is of an equivalence class that holds a column of the dimension; the column is a plain column, not an expression of it (d.kind % 2); a key column of a lion index, ordered and scalar - not multikey - of the column’s type, under the column’s collation; its comparison its type’s default B-tree comparison; and its pathkey in that order of the sort key’s class and operator family: another family (USING ~<~) or another collation (COLLATE "POSIX") is declined. A partial index qualifies only where the query implies its predicate. The walk gives either direction and either place for NULLs.
  • The fact’s order (lion_ordered_orders_for(), lion_ordered.c). The rest of the ORDER BY leads with the order of a B-tree of the fact, at least its first key; an Incremental Sort above sorts what follows. Core builds no ordered path of the fact for an order the query’s does not lead with (truncate_useless_pathkeys() finds the B-tree’s pathkeys useless), so the paths are built on a scratch copy of the fact that sees only its B-trees, with root->query_pathkeys and root->sort_pathkeys both the rest of the ORDER BY while create_index_paths() runs: it judges an order by the one or the other, depending on the major. An order that holds the column’s own class (a fact column the WHERE equates with it) is passed over.
  • The arms (os_arms(), §48.3). At most pg_lion.ordered_split_max (8; 0 offers no split) are expected: the column’s distinct values, and its NULL entry where the walk takes it. A column whose distinct count is a default is declined. The cap is the planner’s alone: at run time every entry of the column is an arm, however many.

48.2 Planner (os_add_paths(), os_add_path(), os_plan_path())

oj_add_paths() asks os_split_of() for each join §47.2 recognizes - two tables, or two and their key sets - before it takes the join’s own orders. For each order of the fact (§48.1), each of the fact’s lion accesses and each of the dimension’s, paired as §47.2 pairs them, one path is priced (§48.3) and handed to add_path(); the competitor of each order (oj_competitor(), §47.3) is found at the split’s pathkeys before the first path is added. Where a column of the fact is in the column’s class the join’s own paths are offered too, and add_path() keeps what it keeps.

  • The path’s pathkeys are the column’s sort key and then the order’s pathkeys, as far as they give the rest of the ORDER BY; its target is the join rel’s; it is unparameterized and not parallel-safe, and enable_indexscan = off disables it (lion_path_core_switches(), LION_CORE_INDEXSCAN) as it does the join’s.
  • Its one child (custom_paths) is the arms' LionOrderedJoin, built as §47.2 builds the join’s path (oj_join_path()): its pathkeys the order’s alone, its rows and price an arm’s - the arms' rows divided by their count, the first arm’s start-up and the mean arm’s run - and its fact’s side weighted for its lazy set by the arms the LIMIT takes (§48.3).
  • The NULL entry’s place (os_nulls()) is where the sort key puts NULLs, or nowhere when a restriction clause of the dimension is strict in the column (find_nonnullable_vars(): kind IS NOT NULL, kind IN (2, 7)), which no NULL passes.

The plan (os_plan_path()) is a CustomScan with scanrelid 0 over the child, whose custom_scan_tlist is the child’s target list; the split’s target is projected over it (CUSTOMPATH_SUPPORT_PROJECTION). custom_private is positional behind a marker: the column as its lion index numbers it, whether the walk is backward, the NULL entry’s place (LION_ORDER_NULLS_*), the cap the plan was made under, and the index.

48.3 Cost (os_arms(), os_cost())

The arms. With nd the column’s distinct values (get_variable_numdistinct(): lion’s count of the column’s entries where it has one, §33, ANALYZE’s otherwise), ANALYZE’s MCV list and its null fraction:

  • an arm for each of max(ceil(nd), MCVs) values, and one for the NULL entry where the walk takes it and the column has NULLs;
  • a most common value’s arm has its frequency, freq. The list is sorted by the type’s B-tree comparison under the column’s collation and taken in the walk’s direction; the values past it share what is left of the rows, 1 - nullfrac - Σ freq, evenly, and are taken after it; the NULL arm has nullfrac, first or last. The list is used only where statistic_proc_security_check() lets the comparison run on the column’s statistics, as core’s estimators ask (the user may read the column, or the comparison is leakproof); elsewhere every value shares 1 - nullfrac evenly, the split priced less well rather than refused;
  • arm i’s share of the rows the dimension’s quals pass is share_i = freq_i / Σ freq over the arms walked.

An arm is priced as §47.3 prices the join (oj_cost(), its competitor’s Memoize and the cold reads with it), the dimension’s set taken to hold share_i of what it holds: the members, their heap reads and the rows out are scaled by it, and each descent’s TID is tested against a set even where the dimension has no lion access, since the narrowed set is one. Each arm’s walk starts at the start of the fact’s order, and the LIMIT’s share of it is the share its own rows give.

The arms in order. The first arm’s start-up is paid once: the sets' builds, the key sets', and the column walk’s descent, two random_page_cost (§30.11). Each later arm restarts the fact’s walk with a descent of its B-tree, (tree_height + 1) * 50 operator costs and a random page. Each arm reads its entry first (os_entry_cost()): LION_RANGE_UNION_ENTRY_COST, two cpu_operator_cost a TID - handed out, gathered into the arm’s set - and the entry’s TIDs' pages, two bytes a row, at seq_page_cost; and it holds its narrowed set beside the dimension’s, within hash_mem and past it in the arena (§48.4, “The arm’s memory”), so it pays what the two sets take past hash_mem together less what the dimension’s alone takes, which the set’s build has paid, at lo_spill_cost()’s price (§30.3, §30.4 “Spilling”). The arm’s set holds share_i of the dimension’s set’s members, or without a set of the dimension the entry’s rows, freq_i of the table, its bytes lion_ordered_set_bytes()’s (lo_set_bytes() over at most a container for each LION_BLOCKS_PER_CONTAINER pages); the arms' sets are given back one after another, so each arm is priced against the dimension’s set alone. With out_i the rows of arm i, run_i its price past its start-up, and S the first arm’s start-up with the walk’s descent, the price of the LIMIT’s n rows (limit_tuples, the OFFSET’s with the count’s) is

C(n) = S + Σ (entry_i + restart_i + run_i)  over the arms before m
         + entry_m + restart_m + run_m * (n - Σ out_i over the arms before m) / out_m

with m the first arm whose rows, added to those before it, reach n (the first arm has no restart).

The shape. Core’s Limit takes startup + (total - startup) * n / rows of its input. The path’s start-up is S and the first arm’s entry, its rows R the arms' sum, and its total startup + (C(n) - startup) * R / n, at which the Limit’s price is C(n). Without a LIMIT, or with one the arms are not expected to meet, the total is every arm’s. Where the split is not alone under the LIMIT (§47.10: a join or a clause above it filters) nothing is shaped: n is taken as no LIMIT, the fact’s set weighed by every arm, and the start-up is the total, every arm’s, which core’s Limit pays whole; the child’s start-up is its own total too. Both are divided by pg_lion.pushdown_margin, as the join’s are.

The sides' sets (§47.4). The fact’s is weighed for its lazy set by the arms up to m, their shares of its walk summed - two whole arms weigh 2. The dimension’s is built at once under a split, whatever its weight (§48.4, “Built at once”), and oj_cost() prices its build whole, as it does for the join. Key sets folded into the fact’s set (§47.6) are priced in it as for the join (§47.3), once: the set is kept from arm to arm, the key sets built once; and each arm’s run, not folded, carries the pages of the fetches its key sets reject at the arm’s own share (§47.3).

Where the price and the run part:

  • Independence. The dimension’s quals are taken to pass the same share of each value’s rows - share_i of the rows they pass are arm i’s - which a correlation of the quals and the column breaks: the LIMIT may be met in another arm than the priced one.
  • The values past the MCV list are priced after it, evenly; the walk meets them in their place. ANALYZE keeps every value it samples of a column of a few values in the list, so only a value its sample missed, a rare one, is misplaced.
  • The shape is for core’s Limit. It is exact where the Limit is right above the split. An Incremental Sort between them is priced by core from the split’s start-up, total and rows, and hands the Limit a price near C(n). A plan that reads every row of the split below the Limit - the split as a hash join’s inner side - takes a total that is the LIMIT’s price stretched over R rows, not every arm’s, and may be under it or over.
  • An arm that returns fewer rows than priced walks further: an entry whose dimension rows no fact row of the set joins walks the whole fact, or until the fact’s walk has met every member of its set (§30.4’s early stop).
  • An entry whose rows are all dead, or invisible to the snapshot, still has members of the dimension’s set: its arm runs and returns nothing. Only an entry with no member is passed over.
  • A filter above the split, a join or a clause that drops rows of the arms: §47.10 prices the split at every arm. A key set filters inside it, and its keys are taken to be spread over the arms as the dimension’s quals are; a key set of one arm’s rows walks the others to their ends (§47.10, “What a key set leaves”) - folded into the fact’s set (§47.6), each to the last member of the set, or to the switch.

48.4 Execution (os_begin(), os_exec(), os_next_arm(), os_rescan(); lion_ordered_narrow_begin(), _add(), _end(), lion_ordered.c)

  • Begin. The custom_private shape is checked; the child is initialised without EXEC_FLAG_BACKWARD or EXEC_FLAG_MARK - the split supports neither, and core puts a Material above it for a scroll cursor - and must be a LionOrderedJoin, whose dimension’s side, the probe, is what the arms narrow. The column’s lion index is opened.
  • An arm (os_next_arm()). The column’s walk begins with the first arm (lion_order_walk_begin(): the column, no bounds, the direction, the NULL entry’s place). For each entry: the child is rescanned first, when it has run since its last rescan or a Param of the split’s has changed; then lion_ordered_narrow_begin() on the probe, lion_ordered_narrow_add() for each of the entry’s TIDs, and lion_ordered_narrow_end(), which returns the narrowed set’s members. An entry with none is passed over (Entries Skipped), the child’s rescan, which reads nothing, not repeated for the next; otherwise the child runs, and its rows are the split’s until it returns none. The walk has read the next entry’s first TID, and holds no pin or lock while the arm runs (§30.11).
  • The narrowing. narrow_begin gives back the last arm’s narrowed set, then makes the dimension’s set unless it has it - built at once (below) - a changed Param of its lion quals making it again first. The narrowed set is built from the entry’s TIDs a container key at a time, as they come in heap order (§30.11), each key’s together, and each container is ANDed (lion_container_and()) with the dimension’s set’s container of the key; without a set the container is kept whole. While an arm runs, the probe tests a descent’s TID against the narrowed set alone, a subset of the dimension’s. A container key below the one before it would be an entry out of heap order, which only damage makes: ERRCODE_INDEX_CORRUPTED.
  • The arm’s memory. The narrowed set’s containers and its directory are the set’s memory (lo_set_push(), lo_set_alloc()): charged against hash_mem with the dimension’s set, and in its arena past it (§30.4, “Spilling”) - neither held without bound nor a reason to decline the plan. It matters most without a set of the dimension, where an arm’s set is its entry whole: on a dimension of hundreds of millions of rows, a value of a column of a few holds tens of millions of TIDs, tens to hundreds of MB of containers, held for as long as the arm runs. Hence the order in narrow_begin. The last arm’s set is given back first (lo_set_drop()): its bytes in memory to the charge, its blocks in the arena to the arena’s free lists, where the next arm’s take them, and the spill is put back as the set’s build left it, so that an arm past hash_mem does not send the next, which may fit, to the file. Then the set: its build after a changed Param lets go of all of the set’s memory and the arena’s before it starts (lo_set_reset()), and of the arm’s set with it. Then the new arm’s set, charged on top of the set’s. The node’s end lets go of what is left with the set. EXPLAIN ANALYZE counts the arms whose set had bytes in the arena, and the most of one there (§48.5); the arena’s bytes the node holds are counted for that (arenabytes), added at each block lo_set_alloc() takes there and taken off at each it gives back, since the arena’s own measure (lion_arena_used()) is the file’s high-water mark, which blocks taken again from the free lists do not raise.
  • Built at once. Under a split the dimension’s set is built at once, whatever the weight its plan carries (§48.3). Each arm ANDs its entry with the set at every container key the entry has, which for a column of a few values is most of them, so a lazy set would evaluate nearly every key before the first arm’s first descent; its memo would sit outside hash_mem while the arm runs; and its conversion to a built set in the middle of an arm (lo_lazy_convert()) would reset the set’s memory, the arm’s set with it. oj_cost() prices the dimension’s build whole either way.
  • The fact’s side is the child’s (§47.4): its rescan restarts its walk and keeps its set, which no Param reaches (§30.4).
  • A row. The child’s row is copied into the scan slot and the split’s target projected; the child does not touch its slot until it is asked for its next row.
  • Correctness. The column is scalar, not multikey: each row version is under exactly one of its entries (§30.11), the NULL entry holding the NULLs, so the arms divide the join’s rows by the column’s value, and each arm’s rows in the fact walk’s order are a run of the query’s order. The narrowed set is a subset of the dimension’s set, so an arm returns only rows the join returns, and every row the join returns is in the arm of its dimension row’s value: the walk reads the index, not the statistics, so a value ANALYZE never saw is an entry in its place, and a row inserted earlier in the same transaction is in the index. An update of the column is not HOT and puts the new version’s TID under the new value: the arm of the old value meets the old version’s TID, which a snapshot that sees the new version does not see, and the arm of the new value the new one. A TID met under two entries is a slot reused after the walk passed it, its row invisible to the snapshot (§30.11).
  • Rescans (os_rescan()). The column’s walk ends, and begins again at the next arm. A changed Param is handed to the child, whose rescan at the next arm hands it to its sides (§47.4) before the entry is narrowed: narrow_begin consumes the probe’s with a rescan of it, which drops the dimension’s set when its lion quals use the Param, and makes the set for the new value, once; the arm’s descents find no Param left to rescan for. Rescanned after the narrowing instead, the child would hand the Param to the probe again, and its first descent would drop the set and build it a second time. Each side keeps its set unless its lion quals use the Param; a key set is kept unless its plan does.

48.5 EXPLAIN (os_explain())

Split By: <index> (<column>[, backward][, nulls first|nulls last]), the column’s walk as planned, with no NULLs placement where a clause is strict in the column, and Arms Max: N, the cap the plan was made under; under ANALYZE, Arms Run and Entries Skipped, the entries with no member of the dimension’s set; and, where any arm’s narrowed set had bytes in the arena (§48.4, “The arm’s memory”), Arm Sets Spilled: <arms>, <N> kB, those arms and the most bytes of one there - what the dimension’s set itself spilled is its own line’s. The child is printed beneath it once, as a LionOrderedJoin with its own lines (§47.5), its counters summed over the arms and its loops theirs; its Dimension Lion Set is the dimension’s set, not an arm’s.

48.6 Not done

  • An ORDER BY of the column alone (ORDER BY d.flag DESC LIMIT 20): an arm needs an order of the fact to walk, and here any would do. Core’s plan stays.
  • Two columns of the dimension leading (ORDER BY d.flag, d.kind, f.score): arms of the first column only, and the fact’s order then does not lead the rest, so no split. An arm for each pair of entries, the two ANDed, is not built.
  • Arms at once. The arms run one after another; none is run ahead, merged or in parallel, and the split is not parallel (§47.7).
  • A cap at run time. None: a column with more entries than planned - values inserted since ANALYZE - runs every one.
  • A multikey column, whose rows are under several entries: declined.
  • Arms a key set passes over, and a switch from a slow arm to the filter first: §47.7 - the switch is the fetch-and-sort’s where the key sets are folded into the fact’s set (§47.6).

48.7 Tests

test/sql/ordjoin.sql, section 13, on §47.8’s tables and a nullable copy of the dimension (dimn: every 11th flag, 13th kind and 17th tier NULL, kind four values and tier three, each with a lion index). Each query is run through the split and through core’s plan with the node off and the answers compared row by row, in order; oj() says LionOrderedSplit. d.flag leading, DESC and ASC: the split with nothing forced for a page in the first arm, an OFFSET into the second (Arms Run: 2), and the whole join without a LIMIT (7,525 rows); a trailing d.lvl sorted by the Incremental Sort; d.kind leading, ten values, declined at the default cap and split at ordered_split_max = 10, and kind IN (2, 7) passing over the two entries above 7 (Entries Skipped: 2). On the copy: NULLS FIRST and LAST, DESC and ASC; a text column, tier; kind IS NOT NULL, whose walk takes no NULL entry (Split By: dimn_kind (kind)); a value inserted after ANALYZE (kind 9, tier ‘zinc’) and one inserted in the same transaction before the query (kind -1, tier ‘aluminium’), each returned in its place. A key set built once for two arms (Key Set Builds: 1); a LATERAL subquery rescanned for each outer row, the dimension’s set made again for each value once however many arms the row runs (seven rows of two arms each: 7 builds), and one that passes over entries between rescans (d.kind IN (x.x, x.x + 4)); a generic plan’s Param. Arms past a 64 kB hash_mem, on a dimension of 640,000 narrow rows (dimbig, 55 containers): without a set of the dimension, each arm’s set the entry whole (Arm Sets Spilled: 2, 21 kB); with d.lvl < 4, the set in memory and the arm’s past it (1, 23 kB); with d.lvl < 5, the set spilled too and the arms' sets in the arena whole; and a LATERAL subquery whose set, rebuilt for each of seven outer rows, lets go of the arms' with its own (7 builds, 8, 83 kB). Declined: another operator family, another collation, an expression of the column, FOR SHARE OF d, a LEFT JOIN, ordered_split_max 1 and 0, and enable_ordered_join off (core’s Sort over a Hash Join). test/sql/ordjoin_alone.sql (§47.8) tests the split alone and not alone under the LIMIT (§47.10).

48.8 Measured (2026-10-24; PostgreSQL 17, §47.9’s cluster and tables; warm and cold; EXPLAIN (ANALYZE, TIMING OFF))

SELECT f.id, f.score, f.amount, d.pk, d.region FROM bfact f JOIN bdim d ON f.fk = d.pk WHERE ... ORDER BY <the column>, f.score DESC LIMIT 20 on §47.9’s 5,000,000-row fact table and 200,000-row dimension. Warm: the median of seven runs after one. Cold: effective_cache_size = 400MB, the cluster restarted and the kernel’s page cache dropped before each run, the median of three; the reads are EXPLAIN’s shared buffers read. “Planned” is the plan with nothing forced, at the default 4 GB and at 400 MB. Core’s plan, with pg_lion.enable_ordered_join off, is in each case a Sort over the hash join of the two sets' bitmap heap scans. “Walked” is the child’s Index Entries Walked; every page below took one arm.

page WHERE ORDER BY walked warm: core the split cold: core the split planned, 4 GB / 400 MB
dense f.chan = 3 AND d.region = 'north' d.flag DESC 1,282 243 ms 3.3 ms 1,488 ms (86,490 reads) 23.9 ms (532) split / split
dense the same d.flag 1,770 243 ms 3.1 ms 1,350 ms 30.6 ms split / split
dense, five values f.chan = 3 AND d.lvl < 50 d.region 1,079 323 ms 8.2 ms 1,473 ms 29.0 ms split / split
medium f.chan = 3 AND d.lvl = 3 d.flag DESC 26,257 167 ms 12.7 ms 805 ms (72,973 reads) 208 ms (4,085) split / split
medium the same d.flag 32,336 192 ms 13.2 ms 757 ms 218 ms split / split
sparse f.tags @> '{a1}' AND f.tags @> '{b2}' AND d.lvl = 3 AND d.kind = 3 d.flag DESC 1,187,000 61.5 ms 265 ms core / core

Core reads every row of both sets, joins them and sorts the join for twenty: chan = 3 is 625,000 fact rows. The split’s first arm fills each dense and medium page, its walk §47’s for that value: the medium d.flag DESC page’s arm is §47.9’s medium page (d.lvl = 3 AND d.flag), the same 26,257 entries, in about the same time (12.7 against 11.6 ms). Warm the split is 13 to 78 times faster than core’s plan, cold 3.5 to 62 times, and the planner takes it for each of those pages at both sizes: its Limit 2,197 against core’s 104,372 for the dense d.flag DESC page, 5,785 against 90,083 for the medium. It keeps core’s plan for the sparse page, rightly: the arm walks 1.19 million entries for its twenty rows, 4.3 times slower than core’s plan, and is priced 128,890 warm and 190,970 cold against 77,161. Not measured: other majors.

49. A lion path’s own price, and the heap probe (2026-10-25)

§39 prices a lion path against the cheapest core plan it meets: its own price times that plan’s rate, divided by that plan’s cold reads, scaled by the rows that plan’s probes of a skewed key meet and by its Memoize’s hits under a LIMIT. Each correction reads the competitor’s plan, so each new shape of core plan needs a new reading (§39’s “The join that reads the dimension” is the latest), and a path’s price moves with a plan it is not part of. This section starts replacing that with a rule: a lion path is priced by the work it does, in core’s units, and core’s plans by core.

  • Its own work, itemized. No factor comes from another path.
  • What it shares with core, at core’s prices. Heap pages through index_pages_fetched() and the tablespace’s page costs, a B-tree descent through the index’s amcostestimate, a row at cpu_tuple_cost, quals at cost_qual_eval(). A table read cold is core’s to price, through effective_cache_size and the tablespace’s random_page_cost, and then it is priced so for every plan alike.
  • What is lion’s alone, at its constants (§31), fitted in core’s units.
  • One set of row estimates. What lion knows better than core (n_distinct, §33; the rows of an intersection, pg_lion.enable_rows_correction) goes to every path or to none.
  • Scans that core composes. Where lion can offer a path of a relation that core’s join search combines - the inner side of a nested loop, under a Memoize, a semi, anti or left join - its price meets core’s alternative for the same relation and parameterization in add_path(), both made by the same formulas, and nothing needs to read the plan around it.

The switch (pg_lion.competitor_pricing)

On by default, §39 as it was; off, a lion path took nothing from its competitor but its kind, for the margin: no rate, no cold reads, no Memoize scale, no skewed key’s rows. It was the ablation of §39 in one setting, and is gone with the rates and the skew estimate (§39, “Without the rates”); the cold reads and the Memoize under a LIMIT stay, unconditionally. The measurements below were made with it.

On the fixtures' shapes at scale (2026-10-26; PostgreSQL 18.6 release, the shared VM of four cores, shared_buffers 1 GB, work_mem 64 MB, effective_cache_size 4 GB; bench/calib/joins.sql and joins.py through matrix.py). A 4,000,000-row dimension of 1,563 MB with probejoin.sql’s, ordjoin.sql’s and heapprobe.sql’s lion columns, all-visible; an 8,000,000-row fact of 833 MB, two rows a key, walked by §47.9’s (score DESC) INCLUDE (fk, id) and by a B-tree of a rank without the key; key sets of 1,000 and 80,000 keys; competitor_cold.sql’s dimension at 8,000,000 rows, not vacuumed. 48 queries of §40.10’s, §46’s, §47’s, §47.6’s, §48’s and §49’s shapes and competitor_cold.sql’s, each planned at the four settings of this switch and pg_lion.enable_heap_probe and timed with every plan core’s and lion’s forced arms make. Warm, the tables in the kernel’s cache but not in shared_buffers: medians of seven runs, interleaved, and a second run of five that made the same choices. Cold - before each run matrix.py --cold ran a script that stopped the server (pg_ctl stop), ran sync, dropped each file of the database’s directory from the kernel’s cache (posix_fadvise(POSIX_FADV_DONTNEED)) and started the server - medians of three, 27 of the queries, planned at effective_cache_size 4 GB and 256 MB. Cold this disk reads 1.5 GB/s in sequence and a random page in about 70 us.

With the switch off, two warm choices change, both to slower plans: LionProbeJoin under a NOT EXISTS of 150,000 keys, priced at its own 596,693 against the hash anti join’s 323,929 where the hash join’s rate, 0.5, had it at 319,902 (2,021 ms against 879 warm); and LionOrderedJoin’s medium page, which without the Memoize under a LIMIT loses to core’s loop, 5,917 against 5,573 (44.0 against 14.0 ms warm, 1,304 against 330 cold). At effective_cache_size 256 MB two more: a bitmap heap scan’s cold reads had taken LionBtreeScan’s covering walk under the hash join, 795 ms cold against the bitmap heap scan’s 1,751; and a nested loop’s cold reads had taken the dense page’s LionOrderedJoin, 34.5 against 75.9 ms. Off took one plan to a faster one, and only cold with the planner told the tables were cached: the same LionProbeJoin, 4,113 ms cold against the hash join’s 2,343, which reads its 1.2 GB in sequence. Off costs the warm matrix 1.2 s and the cold one at 256 MB 1.0 s, and moves nothing faster that the planner was told the truth about. So four of §39’s corrections still earn their place: the hash join’s rate, the Memoize under a LIMIT, and a bitmap heap scan’s and a nested loop’s cold reads; the switch stays on until each is priced as work. The skew factors and the rates other than the hash join’s moved nothing here (no key of these tables is skewed, and no count competes with a hash aggregate); whether they earn theirs is for a workload that has them to say.

The heap probe (pg_lion.enable_heap_probe)

§40.10’s probe is index-only: a nested loop’s inner LionBtreeScan descends a covering B-tree to the outer row’s key and tests the entry against the relation’s lion set. It declined the heap mode as “a plan that loses to core’s wherever the set does not” reject most rows. That is a judgement for the price, and the price can make it: in heap mode the probe is core’s parameterized Index Scan of the same B-tree with the lion test in place of the Filter, the heap row fetched for a member alone. Its fetches are a subset of the Index Scan’s, priced by the same formula (lo_cost() with loops), so its price is lower by the rejected rows' pages and quals and higher by the set’s build, spread over the scans (§40.10), and an entry’s test. Where the B-tree does not cover what the scan must return, lo_set_rel_pathlist() takes its parameterized paths alone (lo_btree_paths(); its unparameterized walk would read every member’s row) and offers each paired lion access in heap mode: iocols NIL, no LO_FLAG_INDEXONLY; the executor’s heap fetch, recheck and EvalPlanQual are §30’s. It needs enable_indexscan. This is the shape LionProbeJoin (§46) and LionOrderedJoin’s dimension (§47) each build inside a join of their own; as a path of the relation, core composes it with any join.

The probe’s members over the loop. lo_cost() counted a probe’s members and their pages a scan at a time, each clamped to one: a set that passes one key in a hundred was charged a heap fetch and a page an outer row - core’s Index Scan’s - plus its own test, and lost to it everywhere. A key finds its entry in every scan and the set passes a share of them, so a probe’s members are now counted over all of the loop’s scans and shared out (fetched, candidates), and a scan that fetches less than a row reads that share of a page (pages_corr). The index-only probes are priced by the same counts. In competitor_cold.sql three plans moved to LionProbeJoin, whose probe is priced the same way (lion_ordered_probe_price()): the anti join’s nested loop at ten times the cache with the cold charge at 0, and the Memoize loop in the cache and at ten times it uncharged. Warm, on PostgreSQL 18.6 (release), medians of nine over three rounds: the anti join ties (node 1.54 to 1.89 ms, loop 1.46 to 1.92), and the Memoize query runs faster through the node (5.22 to 5.67 against 5.79 to 7.00).

Measured (heapprobe.sql’s tables: a 100,000-row dimension on 2,619 pages, 200,000 fact rows; PostgreSQL 18.6 release, warm, hash and merge joins off, pg_lion.competitor_pricing off; medians of five to seven runs on a shared VM whose runs vary by a third, so a difference below that is a tie):

outer rows the set passes heap probe core’s Index Scan the planner takes
3,585 0.7% 6.9 ms 11.1 ms the probe
3,585 10% 7.3 to 10.5 10.6 the probe
40,000 10% 65.9 80.2 core’s, at a price 1.8% below the probe’s (“The probe’s own work”, below)
200,000 10% 267 406 core’s, at a price 1% below the probe’s
4,000 100% 12.3 13.0 core’s (the set rejects nothing)
200,000 100% 388 376 core’s
3 7% core’s (the build is not paid back)

With 200,000 outer rows over a 2,619-page table both plans read every page about once, which is all index_pages_fetched() charges them, and each heap fetch past that is a cpu_tuple_cost: the 180,000 fetches the probe saves, about 0.7 us each here, are priced at a hundredth of a unit, and the two come out within 1%. That is core’s price of a cached heap fetch, which core’s own plans take too; by this section’s rule the probe takes it as well, and the near tie goes to core’s plan. Where the pages are not all read - fewer outer rows, a larger table, a cold one at its random_page_cost - the fetches the probe saves are pages, and its price shows them.

At scale (the fixtures of “The switch”, above). On, the loops of the table above move to the probe and run faster on the 200,000-page dimension: 39,600 outer rows 1.1 to 1.8 times warm and 1.6 to 2.9 times cold, 3,960 tied warm and 1.5 times faster cold; at 198,000 the probe is priced 2% over the hash join and runs 1.4 times faster warm, 2.0 cold. Core’s hash join over the dimension’s bitmap heap scan in competitor_cold.sql’s subquery, 784 to 860 ms warm, gives way to the probe under a Memoize, 118 to 139 (cold the hash join is 12% faster). Under a LIMIT the probe loses: on LionOrderedJoin’s medium page core’s ordered loop over a Memoize of the probe is priced 3,635 against the node’s 5,475 and runs 3.1 times slower warm, 3.7 times cold, because the probe’s set - 14,990 to build - is spread over the 883,021 scans the whole loop would make, of which the LIMIT charges 2,403, while it is built whole at the first (loops * expected >= LO_SWITCH_MIN_WALK, from the whole loop’s rows). The dense page and the medium page index-only lose the same way, 2.8 to 3.3 times; the sparse page, 5.6 times, to core’s price of the outer Index Scan’s cached fetches, which the probe’s lower inner price exposes. On, the warm matrix comes out level with the default, the cold one 3.8 s worse at 4 GB and 6.7 s better at 256 MB: the probe stays off until its set under a LIMIT is priced as it is built.

Calibration (bench/calib/ops.py)

A lion constant is a multiple of a core cost setting; what multiple is set by the core operation the lion operation stands in for. Core’s plans do not run at one rate. On PostgreSQL 18.6 (release), on the shared VM above, a cached sequential scan’s quals take 4.9 to 5.5 ns a cpu_operator_cost - the slope of its time over the number of a <> -k quals a row - and a nested loop of cached Index Scans 12 to 15 ns: core prices a descent and a heap fetch at less of their time than a qual. A lion operation that competes with one of them is priced at that one’s rate - its time over core’s price of it - so that the planner weighs the two as their times weigh. Priced at the sequential scan’s rate, a probe’s few hundred nanoseconds cost two and a half times what core charges for the same time in the Index Scan the probe replaces. A ratio measured on one machine carries to another far better than a rate of cost units a millisecond, which is the machine’s. This is not §39’s rates: a constant is fitted once, offline, against the core operation its own shape replaces, and no price reads the plan it meets.

bench/calib/ops.py checks the measurements into the tree. It prints the sequential scan’s rate for comparison; its first lion operation is the probe: the heap probe over a set that passes every row, against core’s Index Scan of the same B-tree with no filter - the same descents and fetches - as the slopes of their times and of core’s price over 50,000, 100,000 and 200,000 outer rows, so that the set’s build stays out; of a unique key (calib_ops_dim) and of a key of eight entries on one page (calib_ops_many), which apportion the probe’s own work between the scan and the entry.

The probe’s own work. Under callgrind (a single-user backend, 10,000 and 30,000 outer rows, with the cache and branch simulators, the difference over 20,000) a scan of the probe runs no more instructions than core’s Index Scan, but 134 instruction cache misses to core’s 69: the node’s code - lo_next(), lo_rescan(), the set’s lookup - runs beside core’s between one outer row and the next, and the two no longer fit the first-level cache. An entry costs about 370 instructions more: the set’s lookup, two binary searches - the container key’s (lo_set_find(), 96 instructions and 1.6 mispredicted branches an entry) and the member’s in its container (lion_container_contains(), 110 and 1.4) - and the node’s projection of its target into a slot of its own (about 110 with its bookkeeping), where core’s Index Scan returns the heap tuple through a physical target list. The key’s search now starts within what sorted, distinct keys allow - an index no further from either end than the key is from that end’s key - so a set with a container at every key of its range, as a set of a share of most blocks' rows has, finds it with no search: one run before measured an entry at 93 ns over core’s, seven after at 31 to 107 (median 49).

Over those seven runs, at the Index Scan’s rate, an outer row of a unique key costs the probe 14 to 29 cpu_operator_costs more than core’s (median 20), one of eight entries 40 to 70 (median 51): an entry 4.4, a scan 15.6. They are two settings: pg_lion.probe_entry_cost (4.5), charged by lo_cost() for each entry an index parameterized by the outer row walks, in place of the walk’s LO_PROBE_TUPLES (a probe marks nothing), and pg_lion.probe_rescan_cost (16), charged a scan of LionBtreeScan’s probe. A probe of a unique key was charged 9 an entry and nothing a scan, and is now charged 20.5. LionProbeJoin’s probes (lion_ordered_probe_price()) take the entry’s price; a scan of them is the join’s own loop, not a node’s rescan, and is not measured.

The descent is startup. lo_cost() puts the B-tree’s descent in the walk’s run, so LionBtreeScan’s probe offered add_path() a first row at 0.00 against core’s Index Scan’s 0.29 - for the same descent - and, its total within add_path()’s 1% of core’s, won on it: at 40,000 outer rows the plan above was the probe’s at a price 1.8% above core’s. The probe’s path now takes the descent as its startup, as cost_index() has an Index Scan’s (ipath’s startup, in lo_set_rel_pathlist()). The walks keep it in their run - one descent beside a set’s build - and lion_ordered_probe_price(), which takes a probe’s price as its total less its startup, the build, keeps it in the probe’s price.

What it costs. Priced so, the probe keeps the 3,585-row loop of the table above and loses the 40,000-row one to core’s Index Scan, at a price 1.8% below the probe’s - re-timed, the probe 70.5 ms against core’s 77.9 - and the 196,000-row one (359 against 419). The probe’s saving is heap fetches of cached pages, which core prices at an Index Scan’s entry and its fetch, cpu_index_tuple_cost + cpu_operator_cost + cpu_tuple_cost: 7 cpu_operator_costs, about 95 ns at the Index Scan’s rate, against about 300 ns measured for each row past the first of a key’s eight on one page (the calib_ops_many loop), and more for a row on a page of its own. A probe that saves most of them wins by more than its price says, and the near ties go to core’s plan. That is core’s price of a cached fetch, which every core plan takes too; by this section’s rule the probe takes it as well, and how lion’s plans fare against core’s on a warm machine rests on core’s page costs as the deployment sets them.

The probe’s projection (2026-10-26). Core’s Index Scan under a plain nested loop returns the heap tuple it fetched, through a physical target list, and the loop’s own projection reads what it needs from it; the probe projected its target into a slot of its own first. Core gives a CustomPath no physical target list (use_physical_tlist(): “the custom path creator can set up the path’s pathtarget that way”), and a pathtarget of the relation’s columns cannot be the probe path’s own: PlanCustomPath is not told whether its parent needs an exact target list (an Append’s child) or a small one (a Memoize’s, which caches the rows) - the flags create_scan_plan() reads. The parent is known one level up: create_nestloop_plan() asks its inner side for no particular list. So the join hook (lion_ordered_join_probes()) gives each nested loop core made whose inner side is a probe a copy of the probe whose pathtarget is the relation’s columns (lo_physical_target(), under use_physical_tlist()’s conditions: a plain table of the query, no system column, whole-row Var or placeholder wanted of it; and none of its columns dropped or missing from older tuples, which tlist_matches_tupdesc() would project anyway). Its plan’s target list then matches the scan tuple, and ExecAssignScanProjectionInfo() leaves the node no projection: heap mode returns the buffer’s heap tuple, index-only mode the virtual tuple it fills from the index tuple - the columns the index does not hold NULL, which nothing above reads, since the loop reads the relation’s target alone. A Memoize’s or an Append’s probe keeps the relation’s target, and its projection. The path’s width is the relation’s row, which no nested loop’s price reads. EXPLAIN VERBOSE shows the probe’s Output as every column of the relation, as it shows a core scan’s physical target list.

Under callgrind (§49’s harness, the difference of 30,000 and 10,000 outer rows), an outer row of a key of eight entries took the probe 23,157 instructions, 145 instruction cache misses and 37 mispredicted indirect branches - the expression interpreter’s dispatch - against core’s Index Scan’s 20,150, 75 and 4; returning its scan tuple, 22,274, 135 and 8. An entry is 110 instructions and 3.6 mispredicted branches cheaper; a unique key’s outer row 106 instructions and 8 cache misses. Timed (ops.py, seven runs on the VM shared with another benchmark, load 2), the probe over core’s Index Scan came to 21.6 cpu_operator_costs an outer row of a unique key (-3 to 58) and 58 of eight entries (39 to 81), against 20 and 51 before: the runs spread wider than the saving, so probe_entry_cost and probe_rescan_cost stay at 4.5 and 16 until a quiet machine measures it.

Not done.

  • Column values from the sets. A query that reads a low-cardinality lion column of the dimension still fetches the row for it; the probe could return its value from the column’s per-value sets, as LionProbeJoin’s labels give a predicate’s truth, and be covering for it.
  • The comparison on the private workload with the switch off, for the skew factors and the rates other than the hash join’s, which the fixtures do not reach (“The switch”, above).
  • LionProbeJoin’s scan: its overhead over core’s nested loop, which probe_rescan_cost does not price.
  • The harness’s next operations - the set’s build a container (lo_build_excess()’s LO_BUILD_PIECE_TIDS is a correction of §30.3’s price, not a measurement of its own), a container’s AND, a count - each against the core operation it stands in for, at that operation’s rate.
  • A probe’s set under a LIMIT (“At scale”, above): priced the LIMIT’s share of its build, built whole at the first scan. LionOrderedJoin’s dimension side, the same probe, prices its build in the startup and builds it lazily while the LIMIT’s share of its descents is under 10,000 (§47.3, §47.4). Deciding lazy or at once at run time, as §30.4’s lazy set does, and pricing a scan at the lesser of a lazy lookup and the build’s share, is right for any share of the loop a LIMIT runs.
  • LionProbeJoin’s competitor under the heap probe: where the probe’s loop takes the join, every path of the upper rel has a lion path inside, lion_competitor_path() finds none, and the node’s rate rises from 0.5 to 1 - its price doubled (65,055 to 129,373) for nothing of its own: one more reason the rates are to be priced as work.
  • LionOrderedJoin’s rejected fetches (§47.3, ks.noinc.*): priced at the LIMIT’s own pages where core’s loop doing the same fetches is priced its share of the whole walk’s - 18,897 against 1,158 for 5,000 fetches, the node running 1.7 to 2.2 times faster than the loop chosen. By this section’s rule the two are to be priced alike.

Tests. test/sql/heapprobe.sql: every query through the heap probe and through core’s plan, answers compared - an inner join, a left join, a Memoize chosen at the prices, an ORDER BY the loop keeps, a residual filter, FOR SHARE of the dimension, rows updated after the VACUUM - and the choice at the prices between the probe and core’s Index Scan. costrates.sql: with the switch off no rate moves a count, a GROUP BY, an FK-side join or a semi or anti join, each at its reference price. competitor_cold.sql: with the switch off LionProbeJoin’s price at ten times the cache is its price with no cold charge.

50. Bit-plane arithmetic (lion_planes.[ch], 2026-10-26)

n values of nbits bits each are nbits bit-planes: plane j holds bit j of every value, bit i of its word w being bit j of value 64 w + i. A set of rows F, as a bit vector over the same rows, then weighs the values it selects as

sum over i in F of x_i  =  sum over j of 2^j popcount(F AND plane_j)

Two parts of lion compute that sum. lionvec’s plane-1 estimate (§44.3) is it with F a row’s sign bits, one a dimension, and the planes the query’s four bits a dimension; a bit-sliced index’s sum(x) under a filter, the section after this one, is it with F the filter’s container and the planes the column’s slices, a roaring container each. The identity is one; the operands are transposed. lionvec’s many-bit operand is the query: made once a scan, in memory, short (dimsp / 64 words, 24 at 1536 dimensions), met by thousands of rows' sign bits, and kept interleaved, a word’s four planes side by side, planes[w * 4 + j], so that one vector holds them against the row’s word broadcast. A bit-sliced index’s many-bit operand is the stored column: a slice is a posting set of its own, a container per container key in whatever representation suits it - the high slices of small values sparse ARRAYs or nothing, the low ones BITSETs or NARROWs (§38) - inserted into and vacuumed one slice at a time as any posting set is, so its planes are strided: each its own 512-word payload (a BITSET’s; 64 k words a NARROW’s), wherever its page holds it. Interleaving slices on disk would make one item of 4 KB a slice that no container operation reads, that every insert rewrites whole, and whose empty high slices still cost their words; de-interleaving lionvec’s query would cost its kernel the one-vector-a-word load. So the layouts stay as their owners need them, and this module takes both: one kernel for the identity, below the container library and lionvec, frontend-safe as lion_container.c is (c.h, pg_bitutils and libpgcommon).

The interface (lion_planes.h)

/* per-plane counts: counts[j] = sum over w < nwords of popcount(bits[w] AND plane j's word w) */
void   lion_planes_count_interleaved(const uint64 *bits, const uint64 *planes,  /* [w * k + j] */
                                     uint32 nwords, uint32 k, uint32 *counts);
void   lion_planes_count_strided(const uint64 *bits, const uint64 *const *planes, /* [j][w] */
                                 uint32 nwords, uint32 k, uint32 *counts);
/* sum over j < k of 2^j counts[j], modulo 2^64; k <= 64 */
uint64 lion_planes_dot_interleaved(const uint64 *bits, const uint64 *planes, uint32 nwords, uint32 k);
uint64 lion_planes_dot_strided(const uint64 *bits, const uint64 *const *planes, uint32 nwords, uint32 k);
/* 64 x 64 bit matrix, in place: bit c of word r to bit r of word c */
void   lion_planes_transpose64(uint64 *m);
/* n values of nbits (1 .. 64) to nbits planes of ceil(n / 64) words, and back */
void   lion_planes_scatter_interleaved(const uint64 *values, uint32 n, uint32 nbits, uint64 *planes);
void   lion_planes_scatter_strided(const uint64 *values, uint32 n, uint32 nbits, uint64 *const *planes);
void   lion_planes_gather_interleaved(const uint64 *planes, uint32 n, uint32 nbits, uint64 *values);
void   lion_planes_gather_strided(const uint64 *const *planes, uint32 n, uint32 nbits, uint64 *values);

STRIDED takes an array of k pointers, so that k payloads that lie on k pages, or anywhere, need no copy; a caller whose planes are one array at a stride builds the k pointers. nwords and k are any, 0 included: a BITSET’s 512, a NARROW’s 64 k, lionvec’s dimsp / 64. A scatter pads the last word with zeros past value n - 1 and leaves out a value’s bits from nbits up; a gather writes exactly n values and does not read the padding as one. Nothing allocates; every array is the caller’s.

Counts, not sums. A count is at most 64 nwords, a uint32 below 226 words. A weighted sum is not bounded so: 64 slices over a container’s 32,768 rows weigh to (264 - 1) 32,768, past any uint64, and a column’s sum over many containers further. So the count functions return per-plane counts and the caller weighs them as its sum needs - a bit-sliced index into int128, a signed column’s top slice negatively - and the dot functions are the convenience for a caller that knows (2^k - 1) 64 nwords is below 264, lionvec’s being at most 121,920 (k = 4, 127 words); past that they wrap, deterministically, and every implementation wraps alike.

Kernels

Each count function runs one of four implementations (below), every one returning the same counts to the bit. Every kernel takes the planes four at a time, a group, whose four sums stay in registers through one pass over the words; the last k % 4 planes are a shorter group. bits is read once a pass, from L1 after the first.

  • portable and POPCNT: a word at a time, four sums, one a plane, so that neither their adds nor POPCNT’s false dependency on its destination chain the words; the word’s popcount lion_popcount64() (the builtin where the target has the instruction, pg_popcount64() where not, as §3’s portable kernel counts) or the POPCNT instruction.
  • AVX2: each byte counted a nibble at a time with VPSHUFB (Muła, as §3 and lionvec count), the byte counts added over 31 vectors, 248 a byte at most, before VPSADBW sums them into 64-bit lanes. INTERLEAVED, lionvec’s: a word’s four planes are one vector, ANDed with the bits' word broadcast, lane j counting plane j; up to four groups a pass, a word’s 16 planes, so that a pass over 64 interleaved planes reads each cache line once - a group a pass read each line twice and took a fifth longer. STRIDED: four words of the bits against four of each of the group’s planes, one vector a plane, as §3 counts an AND.
  • AVX-512: AVX2’s, each lane’s word counted whole by VPOPCNTDQ into its 64-bit sum, on 256-bit vectors (avx512f,avx512vl,avx512vpopcntdq, as lionvec’s, §44.3): no 512-bit instruction runs.

Where a vector would reach past the caller’s array - the last nwords % 4 words of a STRIDED pass, the last k % 4 planes of an INTERLEAVED one at its last words - AVX2 reads a zero-padded copy and AVX-512 a masked load (_mm256_maskz_loadu_epi64), which reads nothing past the lanes it keeps; elsewhere the last planes' vector reads the next word’s planes into its spare lanes, whose sums are dropped. The unit test holds every array in an exact-size heap allocation, so a sanitized build sees a read one byte past.

Two things the first version did and this one does not, both found by the timings. Each group’s sums were arrays of vectors indexed in a loop over the group’s planes; GCC kept them in memory, and four STRIDED planes took longer than four calls of one (AVX2 671 ns against 450, AVX-512 397 against 250, 512 words); they are named registers now (446 and 195). And at lionvec’s shape - four planes, 24 words - the weighted sum went through the counts of any k: a call more, 64 counts on the stack with its canary, a loop of variable shifts, 22 - 28 ns against lionvec_plane_dot()’s 11 - 15 with AVX-512. Every INTERLEAVED kernel now has a pass of constant stride for k = 4, and the weighted sum a function for each implementation that weighs its lanes in registers (planes_dot4_*()): 16 ns against 15. (2026-10-26, when lionvec took its plane 1 from here, §44.3: a third. The portable planes_dot4_portable(), the one of the four without a target of its own, was inlined into lion_planes_dot_interleaved(), and its loop’s registers made the dispatch save six and grow a frame on every call, whichever kernel it then jumped to - AVX2’s estimate 39.2 ns against lionvec’s 36.5. It is pg_noinline now, and the dispatch a test of k, a load of the choice and a jump.)

One choice of implementation (lion_simd_*)

The extension chose its SIMD kernels twice, by the same tests written twice: lion_container.c’s bits_kernel_best() (AVX2, else POPCNT, else portable, §3) and lionvec’s lionvec_kernels_init() (AVX-512 with VPOPCNTDQ, AVX2, POPCNT, §44.3). The choice is now one, here:

  • LionSimdImpl - LION_SIMD_PORTABLE, _POPCNT, _AVX2, _AVX512 (AVX2 and AVX-512F, VL and VPOPCNTDQ), and LION_SIMD_AUTO, the best - moves from lion_container.h, where it was the unit tests' alone; lion_simd_has() is whether the build has one and the CPU runs it (__builtin_cpu_supports(), which also checks that the OS saves the registers); lion_simd_best() the best of those; lion_simd_current() what every kernel runs, resolved at its first call to the best - as PostgreSQL’s pg_popcount64 resolves at its first - and read through an inline function, a load and a compare; lion_simd_force(), in a -DFRONTEND build, makes every kernel run another, or the best again, and refuses (false, nothing changed) one the build or the CPU lacks; lion_simd_f16c() is whether the build knows F16C (GCC 11, clang 18) and the CPU has it, for lionvec’s halves, not an implementation of its own nor forced.
  • The compile-time side moves with it: LION_HAVE_X86_SIMD (GCC 5 or clang 6 on x86-64, and not LION_NO_SIMD), LION_HAVE_X86_AVX512 (GCC 8, clang 8), LION_HAVE_X86_F16C, LION_TARGET(), and the portable kernels' lion_popcount64().
  • lion_container.c keeps its seven whole-bitset passes - COUNT, RUNS, AND_COUNT, AND, OR, ANDNOT, OR_NEW - as they were, and picks among their three kernels by lion_simd_current() (bits_kernel_of()) where a function pointer started at a resolver did; AVX-512 runs its AVX2 kernel, which is what bits_kernel_best() picked on such a CPU. Each kernel stays a function the dispatch calls (pg_noinline), as it was behind the pointer. lion_container_simd_force() and lion_container_simd_current() keep their signatures: force() is lion_simd_force(), and so forces every kernel that follows the choice, and current() reports the kernel the passes run, AVX2 under AVX-512. The passes take what they took: through the library, one pair of BITSETs in L1, the fastest of six alternated runs before and after, AVX2 and_cardinality() 94.1 / 94.2 ns, and_raw() 109.2 / 109.0, bitset_recount() 88.9 / 89.3, or_inplace() 113.8 / 113.4 (this section’s machine, below; on the Cascade Lake Xeon at 2.8 GHz without VPOPCNTDQ where this work began, 125.0 / 126.3, 128.2 / 130.2, 104.3 / 105.8, 134.4 / 134.7), and container_test passes under every forced implementation with the same 3,139,381 checks as before (2,887,036 without the x86-64 kernels), under AddressSanitizer and UBSan too.
  • lion_vec_math.c is unchanged: lionvec still chooses its own set, under its own LIONVEC_HAVE_* tests, until it takes its kernels from here (“Not done”). (Since 2026-10-26 it does, §44.3, “Kernels”: every kernel of lionvec’s dispatches on lion_simd_current(), under LION_HAVE_X86_*, and its plane 1 is lion_planes_dot_interleaved(bits, planes, nwords, 4).)

The transpose

The 64 x 64 bit matrix, word r its row r and bit c its column c, transposed in six rounds as Hacker’s Delight’s 7-3 transposes 32 x 32: round j = 32, 16, .., 1 swaps the off-diagonal j x j blocks of every 2j x 2j block, word k’s columns from j up for word k + j’s below j (bit j of k clear) - a mask, two shifts and three XORs a pair, 192 pairs. Round j exchanges bit j of a bit’s row index with bit j of its column index, so the rounds commute, and all six exchange every bit of the two: the transpose. The planes of 64 values are the first nbits rows of the transpose of the values (scatter), and 64 values the transpose of their planes padded with zero rows (gather); for nbits up to p, a power of two below 64, each needs only part of the work. The scatter needs only rows below p: round j < p mixes rows below p among themselves, and round j >= p, which runs first, need make only its pairs' first rows, j half-swaps. The gather runs the rounds the other way, j = 1 .. 32, on rows of which those from p up are zero: rounds j < p mix rows below p, and round j >= p splits each row k below j into itself and row k + j. A word of 4-bit values - lionvec’s query, a column of 16 values - takes 60 half-swaps and four swaps rather than 192.

Measured

Intel Xeon at 2.10 GHz (family 6, model 207: Emerald Rapids; AVX-512 with VPOPCNTDQ, and F16C), a VM of four cores shared with other builds; gcc 13.3 -O2 (make planes-bench, the unit test’s program with --bench), PostgreSQL 16’s libpgcommon. Each figure is ns a call, the median of three runs of the median of nine rounds of about 2 ms; operands warm, random at half density, each plane 4 KB, 64 of them 256 KB, in L2. “A plane a call” is the STRIDED count called once a plane, k calls, which is what the container library’s AND_COUNT pass a slice would cost:

512 words x k planes k = 4 k = 32 k = 64
portable, STRIDED 2,706 21,662 40,487
portable, INTERLEAVED 2,539 22,984 42,095
portable, a plane a call 3,317 25,844 53,562
POPCNT, STRIDED 688 5,807 11,392
POPCNT, INTERLEAVED 668 6,269 15,594
POPCNT, a plane a call 1,028 7,439 16,144
AVX2, STRIDED 446 3,642 7,636
AVX2, INTERLEAVED 512 3,897 7,866
AVX2, a plane a call 540 4,512 9,077
AVX-512, STRIDED 195 2,181 4,274
AVX-512, INTERLEAVED 279 2,657 5,245
AVX-512, a plane a call 325 3,165 6,276

So the sum of a 64-slice column over one container key of a filter takes 4.3 us with AVX-512 and 7.6 with AVX2, 0.13 and 0.23 ns a row: a third less than a slice at a time with AVX-512, a sixth with AVX2. The portable kernel is x86 built for its baseline, a pg_popcount64() - an indirect call on PostgreSQL 16 - a word; on aarch64 and under -mpopcnt it is POPCNT’s loop. INTERLEAVED is the slower layout at 32 and 64 planes, a word’s planes being a stride of 256 or 512 bytes; STRIDED is what the large k have.

At lionvec’s shape, 24 words (1536 dimensions) by four INTERLEAVED planes, the weighted sum against lionvec_plane_dot(), the same program: portable 144 / 135, POPCNT 38.9 / 34.3, AVX2 27.2 / 29.0, AVX-512 16.3 / 15.1.

The transpose takes 320 ns. A container’s 32,768 values to STRIDED planes and back: 4 bits 40.0 and 48.4 us, 16 bits 58.3 and 63.4, 32 bits 91.5 and 90.6, 64 bits 211 and 206 - from 78 ns a word of 64 values at 4 bits to 410 at 64, the loads and stores of a word’s 64 values and nbits planes beside the transpose.

Tests

test/unit/planes_test.c (make unit, with planes_test_nosimd built as container_test_nosimd is, -DLION_NO_SIMD and -mpopcnt on x86-64), every count against one a bit at a time - each row the bits select, each plane’s bit there - and every weighted sum against the values the planes hold summed over those rows, modulo 264:

  • the choice: the best is what runs before any force; forcing each of the four returns what lion_simd_has() says and takes effect, or changes nothing; LION_SIMD_AUTO restores the best; a build without the x86-64 kernels has the portable one alone and no F16C;
  • under every implementation the build and CPU have, both layouts: k of 1, 3, 4, 5, 32 and 64 planes over 1, 3, 4, 24, 100 and 512 words, 32 cases each - the bits and each plane its own of 16 patterns: none, all, the first or the last bit of every word, a single bit at either edge of a word, random words, densities from 0.05% to 99.95% - in exact-size heap buffers at every 8-byte alignment modulo 32, each STRIDED plane its own allocation; the inputs left as they were; a single set bit on each side of every word boundary of 1, 3, 24, 100 and 512 words against 1, 5 and 64 random planes; every bit set at every word count from 1 to 130 and 1 to 8 planes, the most a byte count adds up, across every run and tail length; and no words or no planes;
  • the transpose against one a bit at a time on 200 matrices, every single bit, and twice the identity; scatter and gather at every width of 1 to 64 bits for ten counts of 1 to 1,000 values, and at 4,113 and 32,768 values for 13 widths, both layouts, against planes made a bit at a time, the padding zero, a value’s bits past its width ignored, and the gather blind to padding set to ones;
  • the bridge to lionvec, lion_vec_math.c linked as vec_test links it (-ffp-contract=off): the INTERLEAVED weighted sum of four planes is lionvec_plane_dot() under every pair of the two’s implementations, at every word count an index takes (1 to 127) on random and full inputs; and for 45 queries of 64 to 7,232 dimensions and every metric, the scatter of the query’s 4-bit codes - recomputed from qc, qlo and qdelta as lionvec_query_init() rounds them - is the planes it built, to the bit, and the gather of its planes the codes. (Since 2026-10-26 lionvec_plane_dot() is that weighted sum, §44.3, so the bridge pairs this module’s implementations against one another through lionvec’s call; vec_test checks lionvec’s against its own count a bit at a time, as before.)

113,714 checks on this section’s machine, 37,169 without the x86-64 kernels, under AddressSanitizer and UBSan as well (make unit SANITIZE=1). Deliberate faults - a byte run of 33 vectors, a wrong round mask, a tail copied a word short, a half-swap range one short in either direction of the transpose, the plane order of a group swapped, a weighed count shifted mod 32 - each failed the test.

Not done

  • lionvec onto this module (the next change), which would take from it: lion_simd_current() for lionvec_kernels_init()’s choice, so that the extension chooses once and its tests force one (lionvec_kernel_force() becomes lion_simd_force(), its sets this enum’s); LION_HAVE_X86_* for its LIONVEC_HAVE_X86/_AVX512/_F16C; lion_simd_f16c() for its halves; and lion_planes_dot_interleaved(bits, planes, nwords, 4) for lionvec_plane_dot - every pair agrees to the bit (the bridge test), at the same time within a nanosecond or two with AVX2 and AVX-512. Its 8-bit sums, l2sq, rotation and halves stay its own kernels, under the one choice. lionvec_query_init() may keep building its planes in registers as it does, which the scatter reproduces to the bit; the scatter would need the codes in an array first, and the time is the floats' arithmetic, not the planes'. (Done 2026-10-26, §44.3, “Kernels”, as this says, with no macro of lionvec’s own left and lionvec_kernel_force() keeping its numbers; one fix here (“Kernels” above: the portable pass out of line). Plane 1 at 1536 dimensions with AVX-512 took 16.6 ns before and 12.9 after, AVX2 25.4 and 26.7, POPCNT 30.4 and 30.1; vec_test’s checks, a rebuilt index’s pages and the regression suite’s plans are as they were, and a query and a build on 100,000 rows of 1536 dimensions took what they took, within the noise.)
  • The bit-sliced index (§51), which would call lion_planes_count_strided() per container key with the filter’s BITSET words and its slices' payloads - slices of other representations through the container library’s AND cardinality, a NARROW against the filter at its width - and weigh the counts into int128; lion_planes_scatter_strided() to build a container key’s slices from its 32,768 values, each slice then lion_container_optimize()d; and the gather, or a bit test a slice, to decode a row’s value. Its own: a signed column’s offset or two’s complement, min and max by the slices from the top, and comparisons as slice algebra.
  • A faster scatter for few bits. At 4 bits a word’s planes are four movemasks of its 64 values as bytes (VPMOVMSKB after a shift), which would beat the 78 ns a word of the partial transpose several times over; nothing needs it yet.
  • A scalar INTERLEAVED pass of eight planes, a word’s cache line, for portable and POPCNT at 32 and more planes (POPCNT’s a third slower than STRIDED at 64), which no caller uses.

51. Bit-sliced columns: SLICE entries (format version 15, 2026-10-26)

A lion column answers count(*) under any filter from its posting sets, and since §37 sum, min and max of a column with no WHERE, each key’s value weighted by its entry’s rows. Under a filter neither helps: the aggregate needs each row’s value, which a posting set of a key holds only as membership in that key’s set, so SELECT sum(amount) FROM events WHERE country = 'JP' reads the rows - a bitmap heap scan over the filter’s TIDs, or the table. A BIT-SLICED INDEX keeps the values themselves as posting sets: for a column of nbits-wide integers, slice i is the set of rows whose value has bit i set, and

sum(x) over F  =  Σ_i 2^i · |S_i ∧ F|

for any set of rows F - nbits counted ANDs, whatever the column’s cardinality, against a read of |F| rows; min and max under F follow by a greedy descent from the top bit, one counted AND a bit, and ranges, percentiles and top-k by the algorithms of O'Neil and Quass (1997, “Improved query performance with variant indexes”) and Rinfret, O'Neil and O'Neil (2001, “Bit-sliced index arithmetic”). This section is the storage half: the slices, their format, build, insert, VACUUM, verify, and a direct SQL function that proves them exact end to end. The planner’s half - LionCount answering sum, avg, min, max and ranges from slices - is the next piece of work (§51.14). The decision taken before it began: a slice is an ordinary directory entry of a new kind, so that placement, splits, posting trees, VACUUM, the WAL records, their redo and page recycling carry it with no new record type, exactly as §32 did for summaries.

51.1 What a slice holds, and the encoding (lion_slice_enc.h)

A sliced column of a type nbits wide keeps nbits + 1 SLICE entries after its VALUE entries and its summaries:

  • bit slice i (0 <= i < nbits): the rows whose value’s ENCODING has bit i set;
  • the existence slice E: every row whose value is not NULL.

The encoding is the value as a signed integer of the type’s width with the sign bit flipped: (v mod 2^nbits) XOR 2^(nbits-1), lion_slice_encode(), and back lion_slice_decode(). It maps the type’s range onto [0, 2nbits) preserving order, so the smallest value is 0 and the largest all ones; the greedy descent then finds the type’s minimum and maximum as the smallest and largest encodings, and a sum of encodings is the sum of values plus count · 2nbits-1 (lion_slice_value_sum()). NULL rows are in no slice: the NULL entry of §14 holds them already.

Why E. A row whose encoding is 0 - the type’s minimum, -2147483648 for an int4 - is in no bit slice, so the bit slices alone cannot count a column’s rows; and the descent has to start from the filter’s rows that are not NULL, which without E is F minus the NULL entry, a count of F alone that holds no pin and so trusts no map (§9: “a count with no pinned positive source”). E is O'Neil and Quass’s existence bitmap (their B_nn). It costs one more posting-set insert a row and a set as large as the column’s union of keys, a bitset per container key of a dense column (§51.11 measures it), and makes count(x) = |E ∧ F| one counted AND whose positive source carries the pin.

The allowlist: int2 (16 bits), int4 and date (32), int8, time, timestamp and timestamptz (64) - pass-by-value types whose btree order is the signed-integer order of the Datum’s bits, infinities included (-infinity and infinity are the int64 extremes of the timestamp types and the int32 extremes of date). lion_slice_type_nbits() is the list; nbits is the type’s width, so an int4 column has 33 slices and an int8 or timestamp column 65. Not oid, whose order is unsigned (an encoding without the flip would serve, a later change); not float4 or float8, whose bits are not in order for negative values and which have NaN (the usual fix - flip every bit of a negative, the sign bit of a positive - is not done); not numeric or interval, which are not one integer. A key column of another type under slices = on is simply not sliced, without an error, and lion_index_stats().sliced says which are. A domain over an allowed type is sliced as its base type is, since the key type of its opclass is the base’s; so is an expression of an allowed type.

51.2 The entry and its order (lion_entryfmt.h, lion.h, lion_dir.c)

A SLICE entry carries the flag LION_ENTRY_SLICE (0x4000, the last but one free bit of the flags), its kind is LION_KIND_SLICE (9), after SUMLAST, and its key is ONE BYTE: the bit position, or LION_SLICE_EXISTS (64) for the existence slice, which so sorts after every bit slice. The directory order becomes

MINF < NULL < EMPTY < EMPTYDIM < EMPTYINNER < NULLELEMS < VALUE < SUMMARY < SUMLAST < SLICE

so a column’s slices are the last run of its entries, after its summaries.

Where the position goes. Two places were weighed. The HASH WORD, as a summary carries LION_SUMMARY_HASH: but no entry stores its hash since §21’s format 10 - the comparison recomputes it from the key - so the position would have to be stored as a key anyway and then read back as a hash by a special case in the hash term of lion_cmp_prefix(), in lion_entry_hash() and in every search key a slice lookup makes. A SMALL FIXED KEY needs one special case, and it is the one taken: lion_cmp_prefix() compares a slice’s one-byte key bytewise as part of the PREFIX, after the kind, where every other kind without a column key returns 0. Two slices of a column therefore never tie on the prefix, which makes a slice exactly what a reserved entry is to every path that finds entries: one entry per (column, kind, position), found by lion_dir_find()’s prefix run with no equality call - lion_dir_scan_run_ext()’s “any kind but VALUE is found on a prefix tie” already says so - and placed by the find-or-create every new entry takes. A pivot made from a slice keeps the flag (LION_ENTRY_KINDFLAGS gained it, so lion_pivot_kindflags() and lion_entry_chain_shape() copy it) and the key, so it routes by the position as the entry sorts by it; nothing about splits, high keys or downlinks changed. The key is not a value of the column, and nothing reads it as one: LION_KIND_HAS_KEY() stays VALUE and SUMMARY, so no search key built from a slice (lion_search_key_exact()) calls the opclass, and lion_entry_key() on a slice - a reader that missed the audit below - fails lion_check_key() (one byte is no int4) rather than reading garbage. A search key with no stored form compares below every slice of its column, which is where a walk of them starts.

lion_owner_stamp() gives a slice’s posting-tree pages 0, as every kind but VALUE; the root block identifies the set (§18). lion_make_slice_entry() builds one, lion_find_slice_entry() and lion_posting_set_lookup_slice_col() find one (lion_search_key_slice() is the search key). An INLINE slice costs 13 bytes before its payload, a CHAIN one 32 in all.

51.3 The meta page and the format

LionMetaPageData had no reserved word left (§32 took the last two, and §33 put its counts after the struct rather than move it), but it had a dead one: unused_nbuckets, the hash directory’s bucket count, written 0 by every build since format 4 (§21) and checked to be 0 by verify. It is now slice_cols, bit i - 1 for a sliced key column i, at byte 12 of the struct (36 of the page) - on every existing meta page exactly “no column is sliced” - and the struct keeps its 56 bytes.

An index with at least one sliced column is format 15; one with none is format 14, byte for byte what it was (test/sql/slices.sql reads the version and the word of an index built with and without the option, and REINDEXed each way). This build reads both. lion_read_meta() refuses a 15 that names no column and a 14 that names one, lion_meta_check_version() and lion_index_other_format() take 14 and 15 and nothing else (so the planner keeps both in view, §21 “An index of another format”), and lion_fill_index_state() refuses a named column whose key type lion_slice_type_nbits() does not take, with the REINDEX hint. Format 15 moves the number for the reason §32’s 7 did: a build before this one would take a SLICE entry for a VALUE entry - the kind an unknown flag reads as - with a one-byte key, and fail on it, or count it. Downgrade: REINDEX with slices = off first; upgrade: REINDEX with it on.

51.4 The option

slices (boolean, default off), per index, read at BUILD time only like summaries (lion_settle_slices()): on slices every key column of the allowlist, an empty index’s too (its first insert makes the entries), and the meta page records which; ALTER INDEX ... SET (slices = ...) changes what the next REINDEX builds; an insert gives slices only to the columns the meta page names. lionbuildempty() records them for an unlogged table’s init fork.

Chosen conservatively, flagged for the owner (§51.13): off by default, and per index. Per column would need a column-level option the AM interface does not have, or an opclass parameter.

51.5 Build: the uniform route (lion_spool.c, lion_build.c)

A row’s slices are its extra keys. The spool of §24 (“Build: appends, not a sort”) already keeps the reserved entries outside its hash table, found by kind; the slices are kept the same way, LionAccCol.slices[] indexed by position, and lion_acc_add_slices() appends the row’s code to the slice of every bit its value’s encoding has set and to E: a varbyte delta each, one or two bytes a code for a dense slice. They are in the column’s memory, counted by the budget and spilled with the column’s entries as one run, after its values in position order (lion_acc_sorted()); a run record of kind SLICE has its one byte for a key (lion_cursor_next_record() checks it), and the merge orders slices of one position across runs by code as it orders a key’s (lion_merge_prefix_cmp()). HOT chains' root offsets and the parallel participants' interleaved chunks are handled by the code that handles them for every key. The write pass hands each slice to an ordinary builder (lion_builder_create() with the kind, lion_make_slice_entry() in lion_builder_flush()); the first SLICE group closes the column’s summaries first (lion_build_emit()), since they sort before it. A parallel worker writing a sliced column writes its slices into its file like any entry, and the leader replays them; lion_build_add_entry() counts neither a slice nor a summary as a key (§17’s guard) or among the rows (§33).

Measured, and why it stays (§51.11): the uniform route builds a 5M-row int4 column with slices in 1.2 to 1.3 times the time without, an int8 one in 1.3 to 1.5 times (1.9 at worst, in one of three runs), inside the 2x set beforehand as the point past which a dedicated route would be written. So it was not: the dedicated route - nbits in-progress bitsets per sliced column for the current container key, each appended to its slice as an item when the key changes, which is where the plane kernel’s lion_planes_scatter_strided() (§50) will go - is the seam marked in lion_acc_add_slices().

51.6 Insert (lion_insert.c)

lioninsert() puts a sliced column’s row under its key (lion_insert_one()), then into its bucket’s summary when the column has summaries, and then into the slice of every bit its value’s encoding has set, in ascending position, and into E last (lion_slice_insert()): each a separate insert through lion_slice_insert_one(), which finds or creates the slice entry as every insert finds its key’s (lion_dir_find() for write, lion_writer_check() for SERIALIZABLE, lion_dir_add_entry() for a new one) and puts the member in with the same lion_insert_inline() / lion_insert_chain() paths - so the in-place fast paths of §4 and their records and redo are the slices' too: a dense slice is BITSETs, and most of its inserts set one bit in place, a record of a few bytes. Nothing is held between the steps, as between a multi-key row’s keys (§17, “Build and insert”) and between a key and its summary (§32): the inserting transaction has not committed, so no snapshot that can see the row runs before the last step. A failure between steps leaves a dead row in some of the sets and not the others, which no snapshot sees and the next VACUUM takes out of all of them. KEY FIRST, as §32 has it, is what verify’s argument below rests on: a row in a slice is under its key.

Cost, measured in §51.11: popcount(encoding) + 1 descents and posting-set inserts a row, some 17 for an int4 of values spread over the type and 33 for an int8. An INSERT of such values into a lion index on an int4 and one on an int8 costs 9 times what it costs without slices with the custom WAL records (some 1.5 µs and 91 bytes of WAL a slice insert), 12 times with generic WAL, where the page deltas are half of it; single-row INSERTs, 6 and 10 times. That is the price of a BSI. One descent per slice is the plain version: the slices of a column are a few leaves at most, and a row’s could be filled with the leaf held, or a multi-row insert transposed into planes (lion_planes_scatter_strided()) and added a word a plane - §51.14. (2026-10-27: filled with the leaf held - one descent a column, the in-place adds several slices to a record - and the existence slice’s RUNs kept on the fast path: 4.6 and 2.8 times with the resource manager, 10.9 and 10.7 with generic WAL, from 10.0, 5.9, 15.7 and 13.5 measured again on that day’s machine; “Insert cost, revisited”, below.)

51.7 VACUUM

Nothing new: slices are entries. The bulk-delete pass takes dead TIDs out of them as out of a key’s set, a CHAIN slice’s empty leaves leave its tree (§22, “Empty leaves”), and an emptied slice is deleted as an emptied entry is (§18) - unlike the open summary bucket there is nothing a slice must keep, and the next insert of a row with that bit makes it again. A CHAIN slice VACUUM is working on is found again after a split by its exact key, which the probe of lion_vac_ref_relocate() now builds with the SLICE flag. Two counters had to learn the kind: the rows a VACUUM reports (lion_vac_counts_rows()) and the rows of the cleanup’s key count (§33) leave slices out as they left summaries out. Page recycling sees pages, not kinds.

51.8 The reader audit

Every place that tests LION_KIND_SUMMARY, LION_KIND_SUMLAST, LionEntryIsSummary(), LION_ENTRY_SUMKINDS, LION_KIND_HAS_KEY(), lion_entry_kind() or walks a column’s entries, and what it does with a slice. LionEntryIsDerived() (a summary or a slice; LION_ENTRY_DERIVED) and LION_KIND_IS_DERIVED() are new for the places that meant “after the values”.

  • The entry scans (lion_range.c, lion_entry_scan_selects()): a walk of values stops at kind >= LION_KIND_SUMMARY, now LION_KIND_IS_DERIVED() - the same test, as SLICE is above both - so every walk built on them meets no slice: the GROUP BY walk and its batches (§10), the two-column pairs (§20) and the decoded walk (§34), count(DISTINCT) (§26), the top k (§36), the aggregates over keys (§37), the sum over every row (§35), lion_index_count_group_stats(), lion_index_facets() (§7), the FK-side join’s group walks (§27), the multi-key prefix walks. Changed: the summed walk’s SUMS phase skipped entries below a summary and judged every entry at or above one as a bucket; past the open bucket it now stops at a slice instead of comparing its one byte as a key (kind > LION_KIND_SUMLAST). A column with summaries and slices is in test/sql/slices.sql’s ranges.
  • The summary landings (lion_range_landing() for E_j, lion_summary_shape(), lion_summary_insert()’s descent, verify’s lion_verify_summary_after()): each takes the first item at or above (SUMMARY, key) and stops at anything not a summary of the column, which a slice is not; a column whose summaries are all gone lands on its first slice and is “no summary”, as it was when it landed on the next column.
  • The bitmap scan’s leaf walk (lion_scan.c, LionLeafWalk: IS NOT NULL, ranges, a whole column): stopped at LionEntryIsSummary(). Changed to LionEntryIsDerived() - without it an IS NOT NULL bitmap scan of a sliced column with no summaries handed every slice’s TIDs to the executor (harmless to the answer, a recheck, but a scan of every slice). The ordered walk of a lion column (§30, kind > LION_KIND_VALUE) needed nothing.
  • The planner (lion_selfuncs.c): the endpoint probes classify by kind != VALUE, and lion_dir_value_end() descends to (attno, SUMMARY), above which a slice sorts; n_distinct and the MCV frequencies come from the meta page and from lookups of VALUE keys (§33). Nothing changed. The cost model reads the index’s pages, which now include the slices' (§51.13).
  • §33’s key counting: the build (lion_build_add_entry()), ambulkdelete (lion_vac_count()) and the cleanup’s walk count lion_entry_kind() == LION_KIND_VALUE entries only; the ROWS they record left out summaries by LionEntryIsSummary() / LION_ENTRY_SUMKINDS. Changed to LionEntryIsDerived() / LION_ENTRY_DERIVED in all three, and the build’s §17 guard likewise - a sliced index’s row count would otherwise have been a multiple of its rows.
  • Locating a set (lion_set.c): a located entry’s stored key is copied for its relocation; a slice has none, as a reserved entry has none (changed: LionEntryIsSlice() beside LionEntryIsReserved()), and nothing relocates a NOPIN slice: every slice is located singly, with its pin.
  • The SERIALIZABLE locks (lion_reader.c): lion_reader_lock_leaf() locks the roots of every CHAIN entry of the column on a leaf a walk reads, slices included - a false conflict at most, never a missed one; a slice’s insert checks its own leaf or root (lion_writer_check()).
  • lion_index_stats(): counted a summary apart; changed to count a slice apart too (slice_entries, slice_tids, slice_bytes, slice_pages, and sliced), so entries, ntids, the containers and the pages still describe the column’s keys.
  • VACUUM (§51.7), the build (§51.5), insert’s summary step (lion_summary_insert(), whose landing is above) and verify (§51.9).
  • The WAL records and redo (§25), page deletion (§21), splits and pivots: by item, not by kind; LION_ENTRY_ALLFLAGS gained the flag for the checks that refuse unknown bits.

51.9 Verify (lion_verify.c, lion_verify_slice.c)

Structure. A slice item - entry or pivot - has a one-byte key, a position below its column’s width or 64, a column the meta page names, and no reserved, summary, positions or alignment flag, nor one this build does not know (lion_verify_dir_item()); a slice entry is checked like any entry for its counts and payload, and the duplicate check sees two slices of one position as two entries of one prefix.

What they hold (lion_verify_slices(), after the summaries): for each sliced column, the codes of every VALUE entry, each with its key’s encoding, go through a tuplesort by code (a quarter of maintenance_work_mem, spilling past it); the slices are then read as streams in container-key order beside them, and at each container key the slices' containers, made bitset images, are compared word by word with the images the keys' codes make (lion_verify_slice_ckey()): bit slice i must be exactly the codes whose encoding has bit i set, E exactly the codes under any key. The check holds the sort and 2 · (nbits + 1) images of 4 kB, whatever the column’s size.

Beside writers the keys and the slices cannot be read at one instant, so every difference is a CANDIDATE, as §32’s are: once the walk is done - or the candidates fill a quarter of maintenance_work_mem - the writers that may explain them are waited for (lion_verify_wait_for_writers()) and each candidate row looked at again: the slices it is in NOW spell an encoding, which must be the key’s it was found under, or, for a row the walk found under no key, decode to a key it is now under; and it must be in E. A DEAD row is never a difference (lion_verify_row_state(), which §32’s settle exported): an insert that failed between its steps, or a VACUUM interrupted between a key’s leaf and the slices', leaves one. KEY FIRST makes the one direction a reader of slices relies on - a row in a slice is under its key - hold at every instant on a primary, since VACUUM waits for verify there. On a standby, where replay may hold an insert between its steps, a candidate whose inserting transaction is still running is settled as such, as §32’s are. The report names the TID and, for a superuser only, the bit (a bit of a value is the value’s, §7).

heapallindexed: every row the heap scan sees is looked up in every slice of its column and in E (lion_verify_row_slices()): in bit slice i exactly when its value’s encoding has bit i set. A row the scan’s fresh snapshot sees committed before it, so its insert went through every step: no candidate. It is nbits + 1 lookups a row, which makes heapallindexed of a sliced int8 column some 65 times its keys' lookups; it is a check, not a path.

lion_debug_slice_insert(idx, attno, pos, tid), which the tests create from the library as they create lion_debug_key_skew(), puts a row into a slice whatever its value - the damage verify is shown to find (on PostgreSQL 16 too, where test/sql/corrupt.sql’s eviction is not available).

51.10 The SQL function (lion_slice.c)

lion_index_slice_agg(idx regclass, filter text DEFAULT NULL, attno int2 DEFAULT 1,
                     OUT count bigint, OUT sum numeric, OUT min text, OUT max text)

stands for SELECT count(x), sum(x), min(x)::text, max(x)::text FROM t WHERE <filter> (with a partial index’s predicate ANDed in), and answers it from the slices alone:

  1. The filter is lion_index_facets()’s - parsed and analysed against the table, taken apart into posting sets of its lion indexes by lion_facets_filter_tree(), collected once into a pinless copy F by lion_sources_collect() - now the shared LionTextFilter (lion_text_filter_begin(), lion_facets.c), which the facets use as before, with its errors naming a “slice filter” here.
  2. The counts: every slice of the column, E included, located with its pin (lion_posting_set_lookup_slice_col()) and counted ANDed with F a batch at a time in one walk of F’s container keys (lion_count_groups_copy(), §10 “The groups of a walk, counted together”), as the facets count their keys; without a filter, each alone (lion_count_sources_cached()). count = |E ∧ F|; sum = Σ_i 2^i |S_i ∧ F| - count · 2^(nbits-1), in 128 bits (lion_slice_value_sum()) and made a numeric exactly - numeric arithmetic where the compiler has no int128 - NULL for the date and time types, which have no sum.
  3. min and max, when count > 0, by the greedy descent (lion_slice_extreme()): the candidates C start as a collected copy of E ∧ F, and from the top bit down each step counts |C ∧ S_i| with S_i located with its pin. For max, bit i is set when the count is not 0; for min, it is clear when the count is below |C|. C is collected again - C ∧ S_i, or C ∧ ¬S_i for the min - only when the step splits the rows C shows; a bit every candidate has, or none has, leaves them as they were. Collections hold no pin and are only ever counted ANDed with a pinned slice. On a hot standby, where nothing is collected (§10), each step intersects E, the filter’s sets and the slices of the steps that split, all located again.

Why it is exact. Every number is a count of the count engine (§9): a located set with its pin - a slice - ANDed with a pinless copy, counted against the visibility map under the pin and in the heap under the caller’s snapshot for every TID on a page the map does not vouch for, the argument of §10’s collected WHERE (a dead TID the copy still lists is gone from the slice’s container, or the slice’s page was read before VACUUM’s ambulkdelete passed it, so its heap page is not all-visible and the recheck decides). A row the snapshot sees is in E and in exactly the bit slices of its value: its insert went through every step before its transaction committed, and nothing but VACUUM removes a TID from a set, which it does only for a row no snapshot sees. So |S_i ∧ F| counted is exactly the visible rows of F with bit i set, and the sum, the count and the descent’s steps are exact. A step’s verdict and the collection that follows it both read S_i, at different moments; a row the snapshot sees is in S_i at both or at neither, and only rows it does not see can come or go between, so the rows C shows after the step are exactly those of C before it with or without bit i. This transaction’s own rows are rows its snapshot sees (or, deleted, does not), decided in the heap as every other. test/sql/slices.sql compares the function with the query over every type of the allowlist, with NULLs, negatives, each type’s extremes and infinities, under filters of every shape, on a dirty heap and after VACUUM, inside a transaction that inserted, deleted and updated rows and after its rollback; test/isolation/slices_snapshot.spec compares them under one REPEATABLE READ snapshot while another session writes, commits, rolls back and VACUUMs.

The contract (§9, “Direct readers”): lion_count_open_indexes_ext() with ownfuncs - the facets' checks (SELECT on what the query reads, a partial index’s predicate columns included, EXECUTE on the index’s expressions, row-level security, an unpopulated materialized view, snapshot eligibility, predicate locks) without count(*)’s and the equality’s EXECUTE, which the query does not call - and EXECUTE on count("any"), sum, min and max of the key type, which it does (lion_slice_check_execute()). reader_contracts.sql, reader_checkxmin.spec and reader_serializable.spec have a line for it. A key column that is not sliced, a multi-key one, a column the index does not have are refused. DEBUG1 reports the slices, the counts, the candidate sets collected, and the heap blocks counted from the map and the TIDs rechecked.

51.11 Measured

bench/slices.sql (psql -X -v rows=5000000 -f bench/slices.sql, into a scratch database): a table of 5,000,000 rows (51,547 heap pages, 403 MB) with int4 and int8 columns of three kinds - u, uniform over the type (an int8’s two halves drawn apart, since random() has 52 bits); s, the row number (times 1,000,003 for the int8), so in heap order; k, skewed (power(random(), 4) times 106, or 1015 for the int8, most rows near 0) - and g100, g10 and g2 for filters of 1%, 10% and 50%, under a lion index of their own. PostgreSQL 16.15 built at -O2 without assertions, four cores, 15 GB, shared_buffers 2 GB, maintenance_work_mem 1 GB, no parallel workers; every number the best of three, warm. The machine was shared with another run’s test suite (a load of two to three on its four cores), so a timing moves by 10-20% from run to run; a ratio less. 2026-10-26.

Build and size, a one-column index of each column, slices off and on (the slices' columns from lion_index_stats(), for the three indexes the run keeps):

column without slices with slices build size slice entries rows in them slice pages
u4 6,921 ms, 193.0 MB 9,081 ms, 222.5 MB 1.31x +15% 33 85.0 M 3,776
s4 5,747 ms, 193.1 MB 6,913 ms, 201.1 MB 1.20x +4%
k4 4,272 ms, 47.5 MB 5,225 ms, 66.3 MB 1.22x +40% 22 48.7 M 2,404
u8 6,958 ms, 193.4 MB 10,318 ms, 252.1 MB 1.48x +30% 65 165.0 M 7,520
s8 4,967 ms, 193.4 MB 7,379 ms, 219.5 MB 1.49x +13%
k8 7,244 ms, 193.2 MB 9,630 ms, 239.4 MB 1.33x +24%

Two other runs of the builds gave 1.20x to 1.88x; the 1.88x was the first run’s u8, whose values then came through float8’s 15 digits (bit 0 never set, so 64 slice entries), and 1.56x the second time. The uniform route is within the 2x, so it ships (§51.5). A row costs its column popcount + 1 varbyte appends (17 on average for a u4, 33 for a u8) where its key costs one hash probe and one append, and the slices' codes then go through the builders as every key’s do.

The slices of a column are about the same size whatever its keys: a bit slice of half the rows is a bitset at every container key, some 0.83 MB at 5M rows (u4: 26.8 MB of items in 33 slices); a slice of every row - E, or a bit every value has - is a RUN container a container key, 0.21 MB (measured on an index of (id * 0): two slices, 806 RUN containers each); a bit no value has is no entry (k4’s eleven bits 20 to 30). Sequential values set their higher bits in long runs of the heap, so s4’s slices are 8 MB. So the slices weigh most against an index of few keys - k4’s 875,000 entries, 40% - and least against one of distinct values - u4’s 5 million, 15%.

Insert: 100,000 rows of uniform int4 and int8 values into a table of 1,000,000 rows with a lion index on each column (two indexes, so some 50 slice inserts a row: 17 + 33), by one INSERT ... SELECT and by 10,000 single-row INSERTs, in both WAL modes (§25):

insert no index lion, no slices lion, slices slices / no slices
INSERT ... SELECT 100k, generic 54 ms 2,168 ms 25,744 ms 11.9x
… its WAL 7.2 MB 204.8 MB 588.0 MB 2.9x
INSERT ... SELECT 100k, rmgr 59 ms 901 ms 8,174 ms 9.1x
… its WAL 7.2 MB 47.6 MB 501.7 MB 10.5x
10,000 INSERTs, generic 56 ms 276 ms 2,630 ms 9.5x
10,000 INSERTs, rmgr 58 ms 163 ms 907 ms 5.6x

In rmgr mode a slice insert is some 1.5 µs and 91 bytes of WAL ((8,174 - 901) ms and (501.7 - 47.6) MB over 5 million of them): a descent, a bit set in place, a record. In generic mode a sampled profile of the sliced INSERT ... SELECT (40 stacks) had 22 in GenericXLogFinish(), computing the page deltas, and 9 in the descents (lion_dir_find()); the rmgr records are the cheaper way to keep slices, as they are for every insert.

The function against the query it stands for (SELECT count(x), sum(x), min(x), max(x) FROM slc_b WHERE <filter>, the plan the planner chose; both answers compared equal):

column filter rows lion_index_slice_agg() the query the query’s plan
u4 g100 = 7 (1%) 50,256 28.1 ms 30.5 ms Index Scan of the filter’s lion index
u4 g10 = 3 (10%) 500,164 35.3 ms 161.6 ms Bitmap Heap Scan, 51,542 heap blocks
u4 g2 = 1 (50%) 2,500,515 39.7 ms 303.3 ms Bitmap Heap Scan, 51,547 heap blocks
u8 1% 50.2 ms 30.0 ms
u8 10% 42.4 ms 148.0 ms
u8 50% 51.6 ms 327.0 ms
k4 1% 25.8 ms 29.8 ms
k4 10% 44.0 ms 137.5 ms
k4 50% 46.3 ms 318.1 ms

The query’s time follows the rows it reads; the function’s hardly moves. Its work is its counts - 97 for an int4 (33 for the count and the sum, 32 steps of each descent), 193 for an int8 - each a walk of the filter’s container keys whose heap blocks are counted from the visibility map (DEBUG1: 0.76 million block counts for u4 at 1%, 2.3 million at 50%, 1.4 and 4.0 million for u8), and some 30 to 50 candidate sets collected. So it is 3x to 8x faster at 10% and 50%, level at 1% for an int4, and slower at 1% for an int8, whose 128 descent steps cost more than the query’s 50,000 rows. That is the cost the planner task has to weigh: (3 · nbits + 1) counts over the filter’s container keys against the filter’s rows read, and only sum and count - 33 or 65 counts, no descent - where the query asks for no min or max.

Both WAL modes pass the whole suite on this build - 178 regression tests and 31 isolation specs (the injection-point specs skipped), the rmgr run on the dev cluster restarted with shared_preload_libraries = pg_lion after proving pg_lion among its resource managers and an index built there in mode rmgr, as make installcheck-rmgr does.

51.12 Tests

  • test/sql/slices.sql: every type of the allowlist with NULLs, negatives and each type’s extremes and infinities, the function against the query (slc_cmp(), the query with every lion path off) under nine filters each - none, an equality, an IN list, an OR with a multi-key operator, an AND with one, IS NULL, no row, an equality and an IN list on the sliced column itself; the version and the sliced-columns word; lion_index_stats()’s slice columns; verify with and without heapallindexed; the refusals (a column not sliced, a type that is not, a multi-key column, a missing column, filters with a range, a NOT, two statements; a table) and the allowlist’s others (oid, float8, numeric, interval: not sliced); an index built empty and filled by dense, sparse and single-row inserts into the containers they made, and an unlogged table’s; updates and deletes with a dirty heap and after VACUUM; this transaction’s rows, rolled back; REPEATABLE READ; a partial index; a multicolumn index of a sliced and an unsliced column; a slice damaged by lion_debug_slice_insert() - a bit its value has not, a NULL row in E - found by verify; the walks of keys from an index with slices and summaries against the same rows' index without slices, each with the plan that ran it: GROUP BY one and two columns, the top k, count(DISTINCT), the aggregates over keys, summed ranges, <>, bitmap scans of IS NOT NULL and of a range, the facets, lion_index_stats() and the planner’s n_distinct, and again after VACUUM empties a key; ALTER INDEX ... SET (slices) and REINDEX each way; the format’s refusals, written on the meta page under the server (a 14 naming sliced columns, a 15 naming none, a 16).
  • test/isolation/slices_snapshot.spec: the function beside the query under one snapshot while another session writes rows at every column’s extremes, commits, rolls back and VACUUMs.
  • reader_contracts.sql, reader_checkxmin.spec, reader_serializable.spec: the new reader, an aggregate’s EXECUTE revoked; verify.sql’s meta page case reads the repurposed word.
  • Both WAL modes pass the whole suite (§51.11’s last paragraph says on what).

51.13 Open for the owner

  • slices = off by default, and per index: an insert into a sliced column costs a posting-set insert per set bit, and per column needs an option the AM interface lacks.
  • The existence slice (§51.1): one more set and one more insert a row, for a count from a pinned set and a descent with no NULL rows in it. The alternative - F minus the NULL entry - has no pin for the map.
  • The width is the type’s, so a date column of a few decades keeps 32 slices of which a dozen are all rows or none, and a time column 64 for 37 bits of values. Narrower encodings (an offset and a width recorded per column at build, re-encoded when an insert falls outside) would save the constant slices but make the encoding a column’s state that inserts can outgrow; constant slices are a RUN container a container key, cheap to keep (0.21 MB at 5M rows, §51.11), and nothing in the format stops a later version.
  • One descent per slice on insert (§51.6). (2026-10-27: one a column, and several slices to a record; “Insert cost, revisited”.)
  • The cost model counts the slices' pages (lion_amcost.c): an index scan’s pages are priced as tuples * index->pages / index->tuples, and index->pages now includes the slices' - 13% of the u4 index’s pages in §51.11, 23% of the u8’s, 28% of the k4’s - so a sliced index’s scans are priced dearer than the same index without slices though they read the same pages. Pricing them from the pages before the first SLICE entry needs a page count the meta page does not keep; nothing here changes the cost model, and the planner task that reads the slices is the place to decide it. (2026-10-26, §52.6: the planner is handed the index’s pages less an estimate of the slices', from their entries' headers, within 2% of lion_index_stats()’s count here.)

51.14 Not done

  • The planner integration: LionCount answering sum, avg, min, max, ranges (x BETWEEN a AND b as the O'Neil range over slices) and percentiles from slices - the next task. What it needs: lion_posting_set_lookup_slice_col(index, attno, pos, &ps) locates a slice with its pin (pos a bit, or LION_SLICE_EXISTS); LionState.sliced and .slicebits say a column’s slices; lion_slice_enc.h has the encoding, frontend-safe for unit tests; lion_slice_counts() and lion_slice_extreme() are the counts and the descent - static in lion_slice.c, over a LionSliceRun whose filter is LionTextFilter’s collected copy, so a LionCount plan, whose filter is its own sources, lifts them into an interface that takes a collected copy (or, on a standby, constraint lists) - and §51.11’s numbers are what its cost model has to weigh: (3 · nbits + 1) counts against the filter’s rows. (2026-10-26, §52: done for count, sum, avg, min and max under the node’s WHERE and per group of a GROUP BY of one or two columns, through LionSliceAgg - the interface both the function and the node now use. Ranges on the sliced column and percentiles are still to do, §52.10.)
  • The plane kernel (§50): the counts of k slices' BITSETs ANDed with the filter’s image at a container key in one strided pass (lion_planes_count_strided(), at the seam in lion_slice_counts(); slices that are not BITSETs keep the container AND), and the build’s and a bulk insert’s transposition of a container key’s values into slice words (lion_planes_scatter_strided(), at lion_acc_add_slices() and lion_slice_insert()). (2026-10-26, §52.4: the counts take the kernel - BITSETs and NARROWs of one width at a key the map vouches for, pg_lion.enable_slice_kernel; the scatter at build and insert is still to do.)
  • Other types: oid (unsigned: no flip), float4/float8 (the sign-magnitude flip, NaN’s place), numeric and interval (not one integer).
  • Per-column control (§51.13), and an ALTER INDEX that adds slices without a REINDEX.
  • The standby path of the descent is exercised by no test here: the recovery harness’s standby readers do not call the function yet. (2026-10-27: they do - the harness’s checks compare a sliced index’s count, sum, min and max with a scan on the primary, on a hot standby and on the promoted node; “Insert cost, revisited”, below.)

Insert cost, revisited (2026-10-27)

§51.11 measured an INSERT ... SELECT of 100,000 rows into a table with a sliced int4 index and a sliced int8 one - some 50 slice inserts a row - at 9.1 times the time without slices with the resource manager and 11.9 with generic WAL, 48 MB of WAL become 502 (rmgr) and 205 become 588 (generic), single-row inserts at 5.6 and 9.5 times; the generic profile spent 22 of 40 stacks computing page deltas and 9 descending. A row did one directory descent and wrote one record per slice: 17 of each for an int4 spread over the type, 33 for an int8. This section cuts that without changing what is stored - the same entries, the same members in the same containers, the same counts; the one thing that differs on disk is how much growth slack some items are given (3 below). Three changes, in the order they were tried, each measured, then two the profile of the second pointed at.

1. One descent a column (lion_slice_insert(), lion_dir_find_next()). A column’s slices sort together, by position, after its summaries (§51.2), so they are one run on one leaf or a few adjacent ones, and a row’s positions, ascending, are visited in that order. The first is found by the descent every insert makes (lion_dir_find() for write); each next one from the leaf the walk holds, by what a descent does on the leaf it lands on: move right while the high key does not exceed the key, finishing every unfinished split the walk steps onto, then search the leaf. That is the leaf every writer of that slice descends to, because the held leaf’s range begins at or below the key - it held an earlier key of the walk, and a split of it while it was held (an INLINE slice that outgrew the leaf, a new slice entry) kept its lower bound and moved what sorts above the cut to the right - and a slice key’s prefix run is one entry (§51.2), so no run spans pages and find-or-create needs no guard: holding that leaf serialises the slice’s writers, as lion_dir_find() does when a run stays on one page. Moving right releases the leaf before it locks the sibling, as the descent’s move-right does. The leaf is held from the first slice to the last; each slice has its own SERIALIZABLE check (lion_writer_check()), and its posting pages are locked after the leaf, as for one insert, so the lock order and the split protocol are those of one insert repeated. Holding the leaf through the walk keeps the slices' other writers out for a row’s walk rather than a step, but they queued on the same leaf for every step anyway. The search on the held leaf starts at the previous slice’s offset (hint) once the item before it is seen to sort below the key - so whatever happened on the leaf since, the first item at or above the key is at or past it - and is a binary search otherwise; a row’s next slice is a step or two on.

2. Several slices to a record (lion_slice_batch_add(), lion_slice_batch_inline(), lion_slice_batch_write()). A dense slice is a posting tree, and a row appended to the table goes into the last container of each of its slices, on the tree’s tail, in place: before, a record of that container page and the entry’s leaf (lion_insert_container_inplace()), one per slice, and the leaf is the same for every one of them. Now the in-place adds of a row into the slices on one leaf are gathered - each slice’s tail locked, its item found by the decision a single insert makes (lion_chain_target(), shared, so a member goes where it went), checked for room by the first half of the in-place insert (lion_container_inplace_check(), lion_segment_inplace_check()), everything fallible done - and made in ONE ITEM_SET record: the posting leaves first, each with its CONTAINER_ADD or SPARSE_INS (and MINMAX), then the leaf. Three posting leaves a record in generic mode, GenericXLog’s four buffers in all; seven with the resource manager, which opens a wider record when asked (lion_wal_begin_ext(), LION_WAL_MAX_BLOCKS_WIDE = 8, the bits of the header’s masks; §25). On the leaf a CHAIN entry changes in nothing but its ntids, so the counter is raised where it is and logged as a SETBYTES of the byte or two of it that changed (lion_slice_count_one()) - nine bytes, where the DELTA of a rewritten entry is some fourteen and costs a copy and a comparison besides. An INLINE slice whose grown payload fits the bytes its entry has - its growth slack, §4 - joins the record as a rewrite of the entry at its length, which moves nothing and cannot fail; one that has to grow, or spill, goes the ordinary way. So does everything else that is not an in-place add: an item without room, a new segment, a member past a NARROW’s width, a tail that is not its set’s rightmost leaf or does not own the container key (the append hint of lion_insert_lock_chain_page()), a new slice - each of which may split the leaf and move the batch’s entries, so the batch is written first, and also before the walk leaves its leaf. When every slice of a batch was INLINE the record is an ENTRY of the leaf alone.

The WAL format is unchanged: no new record type, no new operation, no new redo. ITEM_SET and ENTRY have always replayed any stream on any block (lion_redo_apply()), and SETBYTES had a redo and no writer; the catalogue (§25) now says ITEM_SET takes up to eight blocks. A record of more than five buffers needs XLogEnsureRecordSpace(), which may allocate, so the shim calls it before the critical section opens; nothing else in redo, rm_mask() or the shim changed. Every operation names its bytes as before, so masking is what it was, and wal_consistency_checking checks the new shapes where it can only check anything, in replay: a crash of a cluster loaded with slice inserts of both kinds (dense and INLINE, appended, after a VACUUM, single rows) replayed with it, in rmgr mode, and the recovery harness, whose fixture now has a sliced index filled by aminsert (below).

Locks. The one thing new is a backend holding posting leaves of several sets at once, so a tail after a batch’s first is only TRIED (ConditionalLockBuffer()); a busy one has the batch written and let go, and the slice starts a new batch, waiting for its tail with nothing but the leaf held. The batch therefore waits for exactly what a single insert waits for, in the same order, and adds no edge to any wait-for graph on the primary - VACUUM’s pass 2 holds a posting page only while it tries the leaf, as before (§11). On a standby redo takes the posting leaves and then the leaf (§25, “Replay takes a posting set’s pages BEFORE the entry’s leaf”) and holds them all to the end of the record, which closes no cycle either: no reader holds a content lock on a posting page while it waits for another lock - a cursor copies a page and lets the lock go (lion_cursor_take_page()), a descent lets the parent go before it locks the child, a walk right lets the page go before the next. Each entry the batch rewrites has to be alone in its bytes; a single insert checks that of its one entry by a pass over the leaf’s line pointers (lion_page_check_alone()), and the walk, which changes the leaf by nothing but same-length rewrites while it batches, checks every item of the leaf once instead (lion_page_check_items()) until it moves or changes the leaf otherwise - some 13% of the batched inserts' time in a profile, at 17 or 33 passes over a leaf of a few hundred entries a row.

3. The fast paths, counted. A counting build (LION_SLICE_STATS, a patch not kept) sorted every slice insert of §51.11’s benchmark by the path it took. Rows appended to the table land in the last container of each slice, so an in-place add is the rule, and the question is how often it fails and why:

where the slice inserts went INSERT ... SELECT, before after 10,000 INSERTs, before after
slice inserts 5,000,448 5,000,448 500,201 500,201
in place: a container 4,932,286 4,971,710 492,110 496,675
in place: a sparse segment 1,568 1,568 196 196
general: a new container key’s first pair 784 784 98 98
general: its fourth, which promotes it 784 784 98 98
general: an ARRAY out of slack 16,550 15,923 1,856 1,780
general: a RUN out of slack 48,476 9,679 5,843 1,354
off the fast path 1.33% 0.54% 1.58% 0.67%

(Nothing else: no tail that was not its set’s rightmost leaf, no busy tail, no INLINE or new slice on a table of a million rows.) The paths a single insert takes are the batch’s - the decisions are the same functions - so “before” is the code before this section. The RUNs were the existence slices': a quarter of their inserts took the general path. A row extends the last run of its container key, which changes no size, but the in-place add asks for room for one more run whatever it does (lion_container_inplace_need(), which its redo asks too, so the writer cannot ask less without a WAL change); a new heap page’s run used up the slack, the general path then rewrote the item at the SAME length, since it still fit (§4: an item written back over itself keeps its length), and so with no room, and every later row of that heap page took the general path again. An insert’s rewrite now keeps the length only while it leaves LION_ITEM_SLACK_MIN bytes to grow into, and takes fresh slack otherwise when the page has room (lion_tree_put_items_locked()); what is left of the RUNs is tails too full for fresh slack, and the ARRAYs are the 64 bytes of slack an item may have (§4), a rewrite every 32 members, and the new container keys every 64 heap pages are inherent. It is one record in 75 to 185 either way, so it shows little in the time below; it is the general path’s descent of the posting tree and its rewrite each time, and not slices' only: any key whose rows come in heap order is a RUN that met the same thing.

Measured (2026-10-27): §51.11’s benchmark (bench/slices_insert.sql, one run of the insert half of bench/slices.sql, seeded, so every run inserts the same rows and its WAL is the same to a few hundred bytes), 100,000 rows by one INSERT ... SELECT and then 10,000 single-row INSERTs into a table of 1,000,000 rows with a lion index on an int4 column and one on an int8, both sliced; each change a build of its own, the builds alternated within each of four rounds (the cluster restarted for each, as the resource manager’s library is read at start), medians. PostgreSQL 16.15 at -O2, a 2.8 GHz Xeon of four cores with AVX-512, 15 GB, shared_buffers 2 GB, fsync and synchronous_commit off; the machine had other clusters on it, a load of 1.2 to 2.0 through the runs. It is not the machine of §51.11, which is why “before” is measured again: it is slower, by a quarter with the resource manager and by more than half with generic WAL, whose byte-wise comparison of every page it logs is most of its time. Milliseconds, and MB of WAL:

insert no index lion, no slices before 1. one descent 2. + several to a record 3. + the slack rule after slices / no slices, before → after
INSERT ... SELECT 100k, rmgr 70 1,017 10,189 6,915 6,015 6,032 4,711 10.0x → 4.6x
… its WAL 7.2 47.7 453.1 453.1 298.5 296.1 223.2 9.5x → 4.7x
10,000 INSERTs, rmgr 72 241 1,429 887 794 827 674 5.9x → 2.8x
… their WAL 0.7 14.1 54.8 54.8 39.4 39.1 31.8 3.9x → 2.3x
INSERT ... SELECT 100k, generic 77 2,622 41,229 38,822 29,699 28,971 28,650 15.7x → 10.9x
… its WAL 7.2 136.4 515.2 515.2 360.7 358.7 358.7 3.8x → 2.6x
10,000 INSERTs, generic 81 399 5,373 5,429 4,626 4,369 4,289 13.5x → 10.7x
… their WAL 0.7 23.5 61.8 61.8 46.4 46.2 46.2 2.6x → 2.0x

“Before” is the code of §51; the build of §52, which changes no insert, measured 9,504, 1,169, 41,677 and 5,784 ms in the four rows of time, the same within the noise. “2.” is three posting leaves a record in both modes and the entries' DELTAs; “after” adds the leaf’s check once a walk and the hint (rmgr 5,182 and 720, generic 28,154 and 4,342 ms), then the wide records and the counters as SETBYTES (rmgr 4,929 and 714 with the first, 4,711 and 674 with both; generic, which takes neither, is as it was within the noise). With the resource manager a slice insert went from 1.8 µs and 81 bytes of WAL to 0.74 µs and 35 bytes ((10,189 - 1,017) ms and (453.1 - 47.7) MB over 5,000,448 of them, against (4,711 - 1,017) and (223.2 - 47.7)); with generic WAL from 7.7 µs and 76 bytes to 5.2 µs and 44. A row’s 50 slices write some 8 records with the resource manager and 17 with generic WAL, not 50. The descents went first, 3.3 s of the 10.2 with the resource manager, as the profile said; then the records mattered more in generic mode, where a record is a page image of each block it names and three slices a record took 9.1 s, than with the resource manager, where they took 0.9 and the leaf’s check and the hint as much again.

Where it stands. With the resource manager the INSERT ... SELECT costs 4.6 times what it costs without slices and a single-row INSERT 2.8 times; with generic WAL, 10.9 and 10.7 times. All but the resource manager’s single rows are above the 3x this work aimed at, for two different reasons. Generic WAL pays a page image a block of a record, copied in and compared byte by byte at the end (GenericXLogFinish()), and a row’s slices are 50 posting leaves, each a different page: batching took the leaf’s images from one a slice to one a batch of three, 100 images a row to some 67, and that is the floor while every slice insert writes its own posting leaf. With the resource manager what is left is the work of 50 in-place adds a row - a buffer pinned and locked, the container found and checked, a bit set, a few bytes logged - at some 0.74 µs each: no longer a descent or a record a slice, but a posting page a slice. Neither moves further without a row’s slices sharing a page.

The deferred tail - designed, not started (2026-10-27). §23 kept one form of batching - “per-key deferred appends kept inside the entry tuple (a small sorted TID tail merged into the containers on the next insert that finds it full, or by VACUUM), which stays inside the pinned-page protocol and keeps counts exact” - and slices are where it would pay: a CHAIN slice entry would carry, after its header and key, a tail of up to T codes (a new entry flag, so format 16), and a row’s slice inserts would append its code to each slice’s tail on the leaf - one record of one page a column, in either mode, the 50 posting leaves a row replaced by two leaves and a T-th of the posting writes. A tail that is full is merged into the tree by the insert that finds it so: its codes go into their containers by the paths above, a record a posting leaf, and the tail is emptied in the record of the last. What it costs is every reader of a slice: a slice becomes its tree AND its tail, so lion_posting_set_lookup_slice_col() and the counts of §52 (the batch counts, the kernel, the decode route) would copy the tail from the leaf under its lock and keep the leaf’s pin as the §9 interlock for its codes, as for an INLINE set; VACUUM’s pass 1 would filter tails on the leaf under its cleanup lock, and the merge would move codes from the leaf to a posting page, which is the spill’s hazard (§18, §25: a reader holding a stale copy of the tail while VACUUM cleans the page its codes moved to), answered as the spill’s is - VACUUM cleanup-locks the leaf before it walks its entries' trees, and the standby barrier carries it; and verify reads tails as members. It is not started here: it is a format change whose whole cost is in the readers of slices, which are the count engine’s code §52 has just written (lion_count*.c, lion_exec_slice.c, lion_slice_decode.c), and the measurements say what it would buy - generic WAL from some 67 page images a row to 2 or 3, the resource manager from 50 posting-page visits a row to 2 leaf visits and 50 appends - but not yet that inserts into sliced columns are what someone is waiting on. A bulk insert transposed into planes (lion_planes_scatter_strided(), §51.14) is the other route, and it needs an end-of-statement hook (aminsertcleanup, PostgreSQL 17) and a reader of the statement’s own rows to wait for it, which 16 does not have.

Tests. test/sql/slices_insert.sql: an int8 column of sixteen values whose bits look random, under an index built empty, whose 64 INLINE slices of about 2 kB fill one directory leaf after another, so that a row’s walk crosses leaves and the slices that grow out of their slack split the leaf the walk holds, until they spill; dense int4 and int8 slices taking an INSERT ... SELECT and single-row inserts in place, with ARRAYs out of slack and new container keys in the middle of walks; INLINE slices beside CHAIN ones in one column with summaries; a rollback, deletes, NULLs, the types' extremes and a VACUUM - each followed by lion_index_verify() with heapallindexed and lion_index_slice_agg() against the query. walrecords.sql pins what the records are in each mode: a row’s 32 CHAIN slices in records of four blocks (generic) or eight (rmgr), at most 14 records a row (some 12 or 6 in fact, the value’s own insert with them), and its INLINE slices in records of the leaf alone. The recovery harness’s fixture has a sliced index of three columns filled by aminsert - an int8, a nullable int4, and an int4 expression negative one row in 97, so that INLINE slices sit beside dense ones - which every crash round of phase 1 replays, the hot standby of phase 2 replays (in rmgr mode under wal_consistency_checking), and whose count, sum, min and max from the slices the checks compare with a scan on the primary, the standby and the promoted node. On this build make recovery-check-rmgr passed the phases a server without injection points runs (1, 1f, 1h, 2, 4 and 5), 930,119 index records replayed across its crashes and the standby’s replay compared page by page under wal_consistency_checking = 'pg_lion', and so did make recovery-check WAL_CONSISTENCY=1 in generic mode (690,840 records); the whole suite passed in both modes (192 regression files and 32 isolation specs, the injection points' skipped).

52. Aggregates and ranges from slices in the count pushdown (2026-10-26)

§51 stored a bit-sliced column’s values as posting sets and proved them exact through a SQL function; this is the planner’s half. The LionCount node (§10) answers count(x), sum(x), avg(x), min(x) and max(x) of a column a lion index slices, under any WHERE the count pushdown answers from posting sets and per group of a GROUP BY of one or two lion-indexed columns, from the column’s slices - beside the count(*) and the filtered counts it answered before:

SELECT sum(amount), avg(amount), max(amount), count(*) FROM events
 WHERE country = 'JP' AND tags && '{mobile}';
SELECT country, sum(amount), min(amount) FROM events
 WHERE device = 'ios' GROUP BY country;

The one row is the column’s width in counted ANDs - every bit’s slice counted against the rows a sum is over, one AND a bit for each extreme - over the WHERE’s container keys, against a read of its rows (§52.2). A GROUP BY is that a group at a time, a few dozen counted ANDs a group (§52.3), or, past a few groups, a walk a batch of groups that reads the slices once and every row’s value off them (§52.4, “the decode route”); the price decides (§52.6). The index’s pages the planner prices a sliced index’s scans by now leave the slices out (§52.7). Three of the four levels the task set are done; the fourth - a range on the sliced column itself, as a count’s source and as a filter, by O'Neil’s descent with container algebra - is not, nor are percentiles (§52.12).

52.1 What is taken (lion_sagg_classify(), lion_count_path_sagg(), lion_find_sliced_index())

  • The aggregates: count(x); sum(x) and avg(x) of int2, int4 and int8 - sums of integers only: the date and time types have none, and the allowlist’s others are not sliced; min(x) and max(x) of those and of date, time, timestamp and timestamptz - of the bare column (a RelabelType aside), with no DISTINCT, FILTER or ORDER BY. They are told by aggfnoid (F_SUM_INT4, F_MAX_TIMESTAMPTZ, …), so a user’s aggregate of the same name is not taken. Without int128 in the compiler, sum and avg are left to the other plans.
  • The column: a key column of a whole (not partial) lion index that slices it at its type’s width (LionState.sliced, slicebits == lion_slice_type_nbits(typid)), not multi-key. Several columns, of one index or several, in one query; a column may also be the grouping column.
  • The rest of the query is what the count pushdown takes on one table (§10): equalities, IN lists, IS [NOT] NULL, <> and the other subtractions (§35), OR trees, the multi-key operators, a range on another column - summed from its entries (§28) or collected as a source (§32) - parameters of a generic plan and stable expressions; count(*) and count(*) FILTER (WHERE ...) beside the aggregates and a HAVING over them; a GROUP BY of one or two lion-indexed columns, an IN list on the grouping column driving its groups (§15), or none.
  • Declined (lion_count_path_slices_ok()): a join (the FK-side join of §27, an anti join), a partitioned table’s GROUP BY (each partition would answer its own and core’s partial aggregates add them up, which needs the partial states of avg and sum(int8) serialized - not written; the one row of a partitioned table is taken, §52.13), a residual clause (§10: the slices would have to be counted over the rows the heap keeps), the aggregates over keys in the same target list (§37), count(DISTINCT), the top k (§36), the decoded walk of §34, three grouping columns, a filtered count beside a GROUP BY, a coalesce group or cells (§10), a parallel plan, and a hot standby, where the pushdown collects nothing (§10) - a plan made on the primary and run there all the same counts every slice beside the WHERE’s located sets (lion_slice_column()’s standby branch), exact and slow. With no WHERE and no GROUP BY this and the aggregates over keys (§37) are both offered, and the price decides.
  • The switches: pg_lion.enable_count_slices (on) turns the whole of it off, and the planner builds what it built before; pg_lion.enable_slice_decode (on) the decode route alone (§52.4); pg_lion.enable_slice_kernel (on) the plane kernel alone (§52.5).

The planner builds such a query’s LionCount path in a mode of its own first (lion_try_count_path_scoped(), slicemode), with the sliced columns' aggregates as the node’s outputs (LION_TL_SAGG(j) among the target-list kinds; custom_private member 25, SLICES: the columns, their indexes and key columns, and each aggregate’s kind and column). A query it declines gets the paths it got before - the aggregates over keys, the counts with an Agg above them, or none - and add_path() keeps the cheapest of everything offered.

52.2 The one row (lion_exec_slice.c, lion_slice.c)

lion_index_slice_agg()’s statics are an interface now (LionSliceAgg, lion_slice_counts(), lion_slice_extreme(), lion_count.h) over a filter that is a collected copy or, on a standby, a list of located sources (LionSliceFilter); the SQL function and the node both run on it. For each sliced column of the plan, once the node has located its WHERE (lion_locate_where()):

  1. C0 = E ∧ WHERE, collected (lion_sources_collect()) from the column’s existence slice E, located with its pin, and the WHERE’s sources as the node located them for its own count - every clause, an OR tree, a multi-key query, a list, the negated clauses that subtract, a range taken as a source (collected, §32), and a range summed over its entries (§28), collected for this from its summaries and entries (lion_range_collect()). Budgeted like every copy the node makes, and spilled past it (§27). E ∧ WHERE rather than the WHERE alone: the copy holds no row whose value is NULL, so count(x) = |C0| and a descent starts from it.
  2. The counts: every slice located with its pin and counted against C0 a batch at a time in one walk of C0’s container keys (lion_count_groups_copy_ext(), §10 “The groups of a walk, counted together”) - E’s count beside them, and every bit’s for a sum or an average. A bit no row has is a slice with no entry, counted 0.
  3. min, max: §51.10’s descent from C0 (lion_slice_extreme()), a step a bit, |C ∧ S_i| counted with S_i located, C collected again only where the bit splits what C shows.
  4. The results as core computes them (§37, “The results”): count an int8; sum(int2|int4) the int128 sum modulo 264, which is the int8 core’s wrapping adds come to in any order; sum(int8) the int128 made a numeric (lion_slice_int128_numeric()); avg core’s - the sum over the count, numeric_div() of two numerics, as int8_avg() and numeric_poly_avg() compute it; min and max the encoding decoded to the type’s Datum (lion_slice_value_datum()), infinities included. NULL over no row but count, which is 0.

The WHERE’s own count (count(*), the filtered counts) runs as before where the target list has one (needcount), and is skipped where it has none. A WHERE that selects nothing (wheremissing) answers 0 and NULLs without locating a slice (lion_slice_none()). A LATERAL rescan locates and collects again.

Why it is exact. §51.10’s argument with the node’s WHERE for the text filter, and §10’s for a collected WHERE: C0 is a copy made after the snapshot was taken, pinless, and only ever counted ANDed with a slice located with its pin, and every TID on a page the map does not vouch for goes to the heap under the node’s snapshot (§9). A summed range is collected from the entries the node would have walked, under the same snapshot. On a standby nothing is collected, and every count intersects E, the WHERE’s located sources and the slice - and the descent’s steps' slices.

52.3 GROUP BY, a group at a time (lion_slice_group())

Each group with a visible row is answered as the one row is, its rows the WHERE’s: its copy C = G ∧ C0 (G1 ∧ G2 ∧ C0 for a pair of two columns, §20), collected from the group’s sets as the walk located them, with their pins, and C0, made once a relation in the node’s relation memory (relcxt, released with the WHERE by lion_release_where()). Then the counts and the descents against C: 33 counted ANDs for the sum of an int4, 33 more for each extreme, each over C’s keys. Every place a group’s row is made calls it while the group’s sets are still located: the walk a group at a time (lion_next_group()), §10’s batches, which keep each group’s answers in the batch’s memory until its row goes up (lion_slice_keep(), lion_slice_put()), an IN list’s groups (lion_next_group_inlist()) and the pairs' nested loop (lion_next_group2()). A HAVING over the aggregates is the node’s filter on its rows. Exact as the one row is: C is a copy made after the snapshot, counted only ANDed with a pinned slice, and its visible rows are the group’s rows the WHERE keeps (§10’s argument for a group’s count against the collected WHERE).

A group at a time reads every slice at every key the group’s rows lie on. A group of a column of 200 values under a 10% WHERE lies at all 806 keys of §52.10’s table, so each group read the 32 slices of an int4 whole - 1.73 s for 200 groups where the ordinary plan took 181 ms (§52.10). That is what the decode route is for.

52.4 The decode route (lion_slice_batch(), lion_count_groups_slices(), lion_slice_decode.c)

The task’s alternative to the pairs was §34’s decoded walk with the slices gathered at each key by lion_planes_gather_strided(). The measurements of §52.3 said many groups need it, so it is here - not inside §34’s walk, whose rows are partial counts below a Finalize Agg, which the slices' aggregates would need serialized partial states to go through, but as a walk of its own over §10’s batches of a column’s groups, whose rows go up final:

  • The batch: §10’s batch of the walk’s entries (lion_group_batch_fill(), up to lion_count_groups_batch() of them - the pin and memory budget of one count’s cursors, 256 at most), located with their pins, taken a batch at a time whether or not the WHERE has been collected yet (lion_slice_batch_ok()): the WHERE is collected before the first batch.
  • One walk a sliced column (lion_count_groups_slices()): lion_count_groups_decode()’s walk with one column, the batch’s groups, and the collected WHERE (or every row, with none). At each key the WHERE leaves rows at, each group’s container is decoded into the group of each of its rows, and each of the column’s slices - every bit’s and E - is read there by a cursor that holds no pin: a BITSET or a NARROW where it lies, its word of 64 positions read at block b, word s of it when s is below its width (lion_slice_plane_word()); an ARRAY or a RUN made an image. Then each 64-row word: E’s word says which rows have a value, and their values are gathered 64 at a time by a transpose of the word’s 64 × nbits bits (lion_planes_gather_strided(), §50) where it holds 4 such rows or more, or read a bit at a time where it holds fewer (LION_SLICE_GATHER_MIN: a transpose costs about what four rows read bit by bit do), and added to its group’s count, sum of encodings, least and greatest (lion_slice_decode_add()). The first column’s walk also counts every row of each group, value or not: the groups' own counts, so the batch is walked once for them and the first column together.
  • Visibility: the map asked at a key after every group’s container there was copied under its pin, and a row on a block it does not vouch for sent to the heap with its group, its value and whether it has one (lion_recheck_visible()), after the walk or when the list is full - the decoded walk’s machinery, and its argument: every row counted is in exactly one group’s pinned set, and the slices, the WHERE’s copy and E only label rows of pinned sets. A row the snapshot sees with a value is in E and in exactly the slices of its value’s bits, which its insert wrote before its transaction committed and only VACUUM removes, for rows no snapshot sees; read after the snapshot, a slice lacks no visible row’s bit and holds none it should not. A TID a slice holds for a row the snapshot does not see labels a row the heap or the map rejects.
  • Where not: two grouping columns (their pairs would be the decoded walk’s combinations, whose partial rows the Finalize Agg adds - the partial states again), groups an IN list or equality on the grouping column drives (one located at a time, §15: the planner offers the route only where no WHERE clause names the grouping column), a WHERE with no copy (one that selects nothing by its sets, left to the row filter: answered a group at a time), and a standby, whose counts cannot read the slices without their pins (a group at a time, which on a standby counts every slice beside the sets).

A batch of 200 groups of §52.10’s int4 under a 10% WHERE is one walk of 806 keys: 33 slices read once, 200 groups' containers decoded, 500,000 rows' values gathered - 41 ms against 1,734 a group at a time and 181 for the ordinary plan.

52.5 The kernel (lion_count_groups_copy_ext(), lion_groups_count_clean())

The slices' counts of §52.2 and §52.3 are §10’s batch walk with an option the slices ask for: at a container key of C whose blocks the map calls all-visible - nothing to recheck, and not SERIALIZABLE, whose blocks take their predicate locks one at a time below - every slice standing there is counted without its intersection being built. The BITSETs and NARROWs of one width together, by one pass of lion_planes_count_strided() (§50) over their payloads against C’s container as an image of that width - C’s own words where it is a NARROW of that width, or the first width words of each block of its full image (lion_groups_width_image()), which is exact since a NARROW of width k has no member past offset 64·k of a block; every other container, and every one with pg_lion.enable_slice_kernel off, by lion_container_and_cardinality() (an ARRAY against the image by a bit test a member, lion_container_and_image_count()). The blocks the map vouches for count in blocks_skipped_via_vm for each group with a row there, as before. lion_planes.[ch] are unchanged; the kernel and its unit tests are §50’s. What it buys end to end is §52.10’s kernel off column: 0 to 20% (the median 7%; once 1.6 times, once 0.86, both within this machine’s noise), because the walk’s time goes to the cursors' page copies (lion_cursor_next_item(), memcpy(): most of a profile of u4 all 10%) and the counting the kernel replaces was some 10% of it. It is kept for the shapes where the counting is the work - many slices of a dense column against a dense WHERE, a cache that holds the slices' pages - and its answers are the container AND’s, to the bit (slices_plan.sql runs both).

52.6 Cost (lion_cost_slice_path(), nine settings)

The path is the count’s path (lion_count_path_add(): the WHERE’s count, or the grouped count, priced as before) with the slices' work added - in the count’s place when the target list has no count and no GROUP BY - in the pushdown’s units (§39, 500 a millisecond), from what the slices hold: their entries' rows and items, read at plan time from the directory (lion_slice_shape(): a descent and the entries' headers, four leaves at most; a column it cannot read whole is taken as every slice holding half of every key’s rows). For each column:

  • The collection of C0: the WHERE’s merge again - half its price beside a count, whose pages it reads cached, all of it without one - E sought at each of C0’s keys (LION_PROBE_COST) and a container of the copy written there (fkjoin_copy_container_cost); with no WHERE, E’s containers read and written. Once for every column: they share the WHERE.
  • The walk: each of C0’s keys a batch of up to 64 slices (pg_lion.slice_key_cost, 120: the copy’s image, the map, the cursors' heap) and each slice read at those keys (lion_slice_read_cost()): the containers met (slice_container_cost, 15), the posting pages they lie on - a page read (lion_index_page_cost(), §39’s resident pages) and a copy of it walked (slice_page_cost, 200) - and, where the keys are fewer than the pages, a descent of the posting tree to each page sought. A container takes a bitmap of the heap’s width at most, and an ARRAY’s two bytes a row below that.
  • A descent: nbits steps (slice_step_cost, 1000: the slice located and the count set up), each - but a bit no row has - a count of the candidates, slice_step_key_cost (50) a key and slice_step_row_cost (0.9) a row of them, the rows capped at 1,024 a key, which a step that splits them narrows to the rows kept: those with the bit for the maximum, without it for the minimum, taken to have it as often as E’s rows have it (the slice’s rows over E’s) - so a uniform column’s halve at every bit of its range, and a skewed one’s minimum keeps most of them for long. The candidates' collections are in the two step prices, as fitted.
  • A group at a time (§52.3): one group’s share of C0’s rows, at the keys it lies at, its copy collected there (a probe, a container and a copy a key) and counted and descended as the one row is, times the planner’s estimate of the groups.
  • The decode route (§52.4): a walk a batch (lion_count_groups_batch() of the grouping index, read at plan time): every slice read at the WHERE’s keys and a key’s overhead; each group’s containers at its keys (slice_container_cost); each 64-row word that holds rows, the lesser of its rows read a bit at a time (slice_decode_bit_cost, 0.3 a bit) and a transpose (slice_gather_cost, 12 for 32 bits); and each row added to its group’s answers (slice_decode_row_cost, 1). Both routes are offered for a GROUP BY of one column, and add_path() keeps the cheaper.

The constants were fitted on §52.10’s 5M-row table: the shapes there run at 300 to 900, the median about 600 (a sum over a range collected for C0 at 210, §52.12) units a millisecond of the price the planner gave them under the lion paths, against the 500 the pushdown’s units stand for; the decisions are in §52.10’s plan column.

52.7 The index’s pages (lion_index_slice_pages(), lion_leave_out_slice_pages())

§51.13 left this open: the planner prices a lion index’s scans by index->pages - prorated by the selectivity in genericcostestimate(), shared out among the columns by the AND of sets and the count pushdown - and a sliced index’s pages include its slices, 13% of the u4 index’s in §51.11, 23% of the u8’s and 28% of the k4’s, which no scan of its keys reads. So the hook that follows get_relation_info() (lion_get_relation_info(), build_simple_rel_hook from 19) hands the planner each sliced lion index’s pages less its slices', as estimated from their entries' headers (lion_slice_shape()): each slice’s containers, sixteen bytes of header each and the smallest of the container kinds for its members a key - an ARRAY’s two bytes a member, a bitmap of the heap’s width, or a RUN of the rows it lacks there, which is what a slice every row of a key has (a constant bit, or E with no NULL) comes to - at 87% of a page filled. An index without slices is not touched (its meta page’s slice_cols is 0) and costs nothing to ask; DEBUG1 reports each estimate. The meta page keeps no page count of the slices, and lion_index_stats() walks every posting tree for one; the estimate against it:

index pages slice pages (stats) estimate error
u4, 5M rows (§51.11) 28,482 3,776 3,767 -0.2%
k4 8,484 2,404 2,358 -1.9%
u8 32,275 7,520 7,531 +0.1%
slices_plan.sql’s 7 columns, 30k rows 1,088 233 257 +10%
slices.sql’s slc_w1_k, 8k rows 66 13 (inline) 22 +70%

It is high where the values run in heap order (slices.sql’s hourly timestamps), whose slices are RUNs a bitmap’s estimate overstates, and on a small index whose slices are inline in its directory entries. Of slices_plan.sql’s sliced index, the bitmap index scans of a range and of an AND of two columns, priced 1.4 and 1.3 times the same index’s without slices, are priced the same now (to a tenth). slices.sql’s summed range over slc_w1_k, which the LionCount node took at about the bitmap scan’s price, went to the bitmap scan with that index 9 pages cheaper than its unsliced twin; the test turns the bitmap scan off there. The suite’s plans over unsliced indexes did not move (§52.11).

52.8 EXPLAIN

Custom Scan (LionCount)
  Lion Indexes: slp_g (g), slp_h (h = 1)
  Group Key: g
  Aggregates From Slices: slp_v (b: every bit counted, min descent, max descent)
  Slice Groups: decoded by batch

a column at a time: its index, and what is counted - every bit counted for a sum or an average, existence counted for a count, a min or a max alone - and the descents; for a GROUP BY, one at a time or decoded by batch. EXPLAIN ANALYZE adds Slices Located and Slices Absent (bits no row has), Slice Counts (slices counted against a copy) and Slice Containers Counted, Slice Containers by Kernel, Slice Kernel Keys (the keys the kernel ran at) and Slice Kernel (lion_simd_name(): avx512, avx2, popcnt, portable), Slice Descent Steps, Slice Candidate Sets (the descents' collections), Slice WHERE Copies (C0 and the groups' copies), Slice Batches Decoded, Slice Keys Decoded and Slice Rows Decoded (and Slice Rows Rechecked), and Slice Heap TIDs Rechecked and Slice Heap Blocks Rechecked where the map did not vouch for a page.

52.9 Not on a standby, a join or a partitioned table’s groups

Restated from §52.1 because the task asked for each: on a hot standby the planner offers none of this (the pushdown collects nothing there, §10), and a plan made on the primary runs exact and slow - every slice counted beside the located sets, the descent’s steps intersected again, and the groups a group at a time; a join (the FK-side join of §27, a semi or anti join) and a partitioned table’s GROUP BY are declined, the latter for want of serialized partial states for core’s Finalize Agg (§52.12). A partitioned table’s one row is answered (§52.13).

52.10 Measured

bench/slices_pushdown.sql (psql -X -v rows=5000000 -f bench/slices_pushdown.sql, then -v setup=off; -v only=1 or only=2 for one level): 5,000,000 rows on 51,547 pages - §51.11’s u4, k4 and u8 (uniform int4, skewed int4, uniform int8) each under a one-column sliced index, g100, g10 and g2 for filters of 1%, 10% and 50% and h2, h20 and h200 for groups of 2, 20 and 200 (two multicolumn lion indexes), and r with summaries for a range. PostgreSQL 16 at -O2 without assertions, four cores shared with another session’s test suite (a load of two), no parallel workers; each number the best of three, warm. plan is what the planner chose at its defaults; lion the LionCount node with every other scan off (the planner’s choice among its paths), kernel off the same with pg_lion.enable_slice_kernel off, one at a time with pg_lion.enable_slice_decode off; ordinary the plan with the pushdown off. Every answer of each was compared with the ordinary plan’s.

The one row (only=1); sum,count is sum(x), count(x), all adds avg, min and max:

shape plan ms lion kernel off ordinary plan ordinary ms
u4 sum,count, 1% LionCount (slices) 12.3 7.7 7.6 Aggregate > Index Scan 36.0
u4 sum,count, 10% LionCount (slices) 9.8 7.0 9.0 … > Bitmap Heap Scan 166.0
u4 sum,count, 50% LionCount (slices) 6.7 5.8 7.0 … > Bitmap Heap Scan 340.3
u4 all, 1% LionCount (slices) 13.8 13.1 14.9 Aggregate > Index Scan 23.5
u4 all, 10% LionCount (slices) 27.2 28.2 27.6 … > Bitmap Heap Scan 174.9
u4 all, 50% LionCount (slices) 33.1 33.6 34.3 … > Bitmap Heap Scan 369.1
u8 all, 1% LionCount (slices) 23.1 22.2 24.6 Aggregate > Index Scan 28.4
u8 all, 10% LionCount (slices) 47.8 33.5 53.8 … > Bitmap Heap Scan 189.8
u8 all, 50% LionCount (slices) 41.6 38.2 37.6 … > Bitmap Heap Scan 332.3
k4 all, 1% LionCount (slices) 16.4 16.5 17.8 Aggregate > Index Scan 26.4
k4 all, 10% LionCount (slices) 41.2 54.0 46.4 … > Bitmap Heap Scan 143.4
k4 all, 50% LionCount (slices) 34.7 37.5 38.9 … > Bitmap Heap Scan 379.1
u4 max, 10% Result (min/max) 0.1 0.1 0.1 Result 0.1
u4 sum, IN, 2% LionCount (slices) 8.7 8.0 9.4 Aggregate > Index Scan 64.1
u4 sum, range 10% (§32) LionCount (slices) 41.4 39.6 41.0 … > Bitmap Heap Scan 167.1
u4 sum, max, no WHERE LionCount (slices) 25.7 18.8 22.4 Aggregate > Seq Scan 325.1
k4 sum, max, no WHERE LionCount (slices) 12.0 11.6 12.3 Aggregate > Seq Scan 331.9

GROUP BY (only=2), of h2, h20 or h200; decoded marks the decode route, one at a time the same with pg_lion.enable_slice_decode off:

shape plan ms lion one at a time ordinary plan ordinary ms
u4 sum, 10%, 2 groups LionCount (slices) 14.0 15.2 15.2 … > Bitmap Heap Scan 185.7
u4 sum, 10%, 20 groups LionCount (slices) decoded 39.5 39.3 129.1 … > Bitmap Heap Scan 187.4
u4 sum, 10%, 200 groups LionCount (slices) decoded 52.1 51.8 903.0 … > Bitmap Heap Scan 189.1
u4 all, 10%, 2 groups LionCount (slices) decoded 40.1 39.5 47.7 … > Bitmap Heap Scan 181.9
u4 all, 10%, 20 groups LionCount (slices) decoded 42.5 41.0 219.5 … > Bitmap Heap Scan 186.0
u4 all, 10%, 200 groups LionCount (slices) decoded 45.6 41.1 1,734.2 … > Bitmap Heap Scan 181.4
k4 all, 10%, 20 groups LionCount (slices) decoded 37.5 36.8 294.0 … > Bitmap Heap Scan 194.7
u4 max, 1%, 20 groups LionCount (slices) decoded 20.1 21.0 62.0 Aggregate > Index Scan 29.5
u4 sum, no WHERE, 2 LionCount (slices) 12.0 11.1 11.0 Aggregate > Seq Scan 428.9
u4 sum, no WHERE, 20 LionCount (slices) decoded 47.8 50.6 112.6 Aggregate > Seq Scan 479.4
u4 sum, no WHERE, 200 LionCount (slices) decoded 71.0 70.4 1,135.5 Aggregate > Seq Scan 693.3
u4 sum, 10%, pairs 2 × 20 … > Bitmap Heap Scan 181.4 237.8 271.1 … > Bitmap Heap Scan 200.2
u4 sum, 50%, pairs 10 × 20 … > Bitmap Heap Scan 439.9 1,327 1,360 … > Bitmap Heap Scan 437.2

(ordinary aggregates are HashAggregates for the GROUP BYs. Timings move by 10-20% from run to run on this shared machine - u8 all, 10% took 33.5 and 47.8 ms in two columns that ran the same plan - which is also the size of what the kernel changes.)

What the tables say:

  • The one row wins from 1% up, by 1.2 to 51 times: its work hardly grows with the WHERE’s rows - the slices are read at the WHERE’s keys, which a 1% WHERE of rows spread over the heap already spans - where the ordinary plan’s does. A sum alone is a walk of 33 counts (6 to 8 ms); the extremes add two descents of 32 steps (13 to 54 ms). The int8’s 1% is the closest (23 against 28 ms). With no WHERE the slices beat the sequential scan 13 to 28 times and were chosen over §37’s walk of keys, which a near-unique u4 makes a walk of five million entries.
  • max alone goes to core’s min/max optimization, a Result over an ordered scan of the lion index (§30): 0.1 ms, which nothing here competes with.
  • GROUP BY: a group at a time wins at 2 groups and loses from about 20 on - 1.73 s at 200 against 181 for the ordinary plan; the decode route stays near 40-70 ms from 2 groups to 200, 1.5 to 10 times faster than the ordinary plan. The planner took the decode route wherever it was the faster, a group at a time for the sum of 2 groups, and the ordinary plan for the two-column pairs, which are a group at a time and lose.
  • The kernel: 0 to 20% (§52.5).

52.11 Tests

  • test/sql/slices_plan.sql: every answer against the ordinary plan (the pushdown off, a sequential scan) as multisets, with the plan that ran it named (slp_q(): from slices, decoded, over keys, counts only, or no pushdown): each type of the allowlist with NULLs, negatives and each type’s extremes and infinities; several columns of one index at once; a WHERE of every kind (equality, IN, <>, NOT IN, IS [NOT] NULL of the column and of others, OR, multi-key @> and &&, a range on another column summed and as a source, an equality on the sliced column, a stable expression, no row); no WHERE, against §37’s aggregates over keys; beside count(*), filtered counts, a HAVING and expressions over the aggregates; the kernel off; generic plans' parameters (an array, a range with its bounds crossed, NULL) and a LATERAL rescan; deletes and updates on a dirty heap, this transaction’s rows at the extremes and its deletes, rolled back, and VACUUM; a GROUP BY of fourteen shapes each way (slp_groups(): a group at a time, then decoded) - one column with few groups, a thousand in four batches, an IN list’s, a group of NULL, groups the WHERE empties of values, the sliced column itself as the group, HAVING, two columns - again on a dirty heap with this transaction’s rows; EXPLAIN of each shape and EXPLAIN ANALYZE’s counters (the kernel’s name masked); the index’s pages (§52.7), the bitmap scans' prices against the same index without slices; EXECUTE on max(int4) revoked, refused by the node as by core’s Agg; the declines - a column not sliced, a float column, an expression, DISTINCT, FILTER, ORDER BY, a residual clause, a join, a partitioned table’s GROUP BY (its one row is answered, §52.13), a filtered count beside a GROUP BY, three grouping columns, a grouping expression.
  • test/isolation/slices_pushdown.spec: the node’s answers beside the same aggregates of x + 0, which only the ordinary plan computes, in one statement: a REPEATABLE READ reader across a writer’s rows at every column’s extremes, updates and deletes, committed, rolled back, and VACUUMed under the reader’s snapshot - for the one row and for each group of a GROUP BY, decoded and a group at a time.
  • test/sql/slices.sql keeps §37’s plan with the path off and on - on an all-visible heap the walk of the entries' headers is the cheaper answer for every row of its 8k-row table - and shows the slices' plan under its summed range, which turns the bitmap scan off (§52.7). The test commits synchronously, as the suite’s tests that VACUUM do: the dev cluster commits asynchronously, and a VACUUM that runs before the WAL writer’s next flush cannot set all-visible, which left the §37 walk priced as a count of every entry and the plan to a race (2026-10-27, seen on 19 in rmgr mode).
  • costgucs.sql lists the nine settings; priv_decode.out (make hookcheck) the plan’s new member.
  • The whole suite: 191 regression and 32 isolation tests in generic WAL mode and again with the resource manager registered (shared_preload_libraries = pg_lion, an index built there reporting rmgr), the injection-point specs skipped on this build as always; hookcheck’s tableam, priv_decode and exec_hook with nothing preloaded.

52.12 Not done

  • Ranges on the sliced column itself (the task’s fourth level): x > c, x BETWEEN a AND b and x <> c as a count’s SOURCE and as a FILTER of these aggregates, by O'Neil’s range algorithm over the slices. What it takes: at each container key of the rows that matter (the WHERE’s copy, or E), the slices' containers read as §52.4 reads them and folded from the top bit into two images, GT and EQ - for each bit, GT |= EQ ∧ S_i where c’s bit is 0 and EQ &= S_i or EQ &= ¬S_i by c’s bit - two word operations a bit and a word, the range’s container then GT, GT ∪ EQ, E ∖ (GT ∪ EQ) and so on, collected into a LionMatSet as §32’s range union is (lion_range_union_*) and spilled past the memory as it is; a source like §32’s collected range, pinless and counted only beside a pinned set (E itself serves), on §52.4’s argument; the planner offering it beside §28’s walk of entries and §32’s summaries for a range on a sliced column, priced at nbits × the keys' words against the entries walked - which wins for a wide range over a near-unique column (u4’s 5 million entries) and loses to §32’s summaries for a narrow one; and the measurement with and without summaries the task asked for. (2026-10-27, §53: done - over E’s container keys rather than the WHERE’s copy, so that the range is a source like any and nothing downstream changed; the comparisons LT of the lower bound, GT of the upper and EQ of each <>, each stopping where its EQ empties; the copy built as lion_sources_collect() builds one, not as a union, since it comes a container a key in key order; and ANDed with E, which pins it. Measured with and without summaries in §53.7.)
  • Percentiles (percentile_disc(f) WITHIN GROUP (ORDER BY x), percentile_cont): the k-th value by Rinfret, O'Neil and O'Neil’s descent - §51.10’s min descent with the rank in place of the count: at bit i, |C ∧ ¬S_i| against the rank still to go, the bit clear and C narrowed to C ∧ ¬S_i when the rank lies among them, set and the rank less that count otherwise; one counted AND and at most one collection a bit, as a minimum. percentile_cont interpolates between the k-th and the (k+1)-th, two descents (or one that keeps both while they agree). The planner has to take an ordered-set aggregate whose argument is the column cast to float8 (percentile_cont has no integer form) and a constant fraction, k from the count; per group, as the extremes are, and on the decode route, which would need each group’s values sorted or a descent per group.
  • The decode route for two grouping columns and for an IN list’s groups: the pairs and the listed values stay a group at a time, so GROUP BY g, h of many combinations goes to the ordinary plan (u4 sum, 50%, pairs 10 × 20: 1.3 s against 437 ms). §34’s decoded walk with the slices beside its columns would answer it in one pass, if its tally did not spill (a dense tally of the pass’s combinations, whose rows go up final), or with serialized partial states below a Finalize Agg.
  • Partitioned tables' groups, joins (the one row of a partitioned table: done 2026-10-28, §52.13): each partition answering its own needs partial states core’s Finalize Agg can combine - sum(int8) and avg keep internal states, which the node would have to emit serialized (int8_avg_serialize(), numeric_poly_serialize()); min, max and the int8 sums of sum(int2/int4) combine as they are. The FK-side join’s counts per dimension row (§27) would each need the slices - a group at a time, or decoded by fact key.
  • The decode route on a standby, which cannot collect the WHERE: a walk with the row filter and the located sources would do, as §34’s does without a copy.
  • A summed range’s collection for C0 is underpriced: u4 sum, range 10% runs at 210 units a millisecond, priced as half the summed walk where collecting the range from its summaries and entries costs more; the decision there is right by a wide margin (40 ms against 167). (2026-10-27, §53.8: priced now as §32 prices a range taken as a source, once beside the WHERE; underpriced, it kept the summaries' route against ranges from slices twice as fast.)
  • The competitor’s warm pages: at a 0.1% WHERE, min and max were priced 6,746 against core’s index scan at 15,040 and ran 10.5 ms against 3.7 warm - core’s random page cost for pages that are in shared buffers is §39’s to correct, not this section’s.
  • The walk’s time is the cursors' page copies (lion_cursor_next_item(), memcpy()), not the counting: a cursor that reads a posting page in place under its pin would serve the slices' counts, a group at a time and decoded, more than any kernel. A column asked for count(x) alone on the decode route reads every bit slice where E would do.
  • The slices' page estimate (§52.7) overstates a column in heap order; an exact count would need the meta page to keep the slices' pages, a format change.
  • The plane scatter of §51.14 (build and bulk insert) is still to do.

52.13 Partitioned tables (2026-10-28)

The one row of §52.2 over a partitioned table, as one LionCount node that counts every leaf the planner keeps (§16):

SELECT count(b), sum(b), avg(b), min(ts), max(ts) FROM parted WHERE g = 3;

  Custom Scan (LionCount)
    Partitions: parted_1, parted_2, parted_3
    Lion Indexes: (g = 3)
    Aggregates From Slices: (b: every bit counted), (ts: existence counted, min descent, max descent)
  • What is taken: the aggregates, columns, WHEREs and switches of §52.1, with no GROUP BY, wherever the pushdown takes the partitioned table (§16: equalities, IN, ranges, ORs and the rest on lion-indexed columns of every leaf, a clause on the partition key a leaf’s bounds imply left out of that leaf’s runs, plan-time pruning). Every live leaf must have a whole lion index that slices the column at its type’s width (lion_find_sliced_index() on the leaf, the column’s number translated through lion_leaf_expr(), lion_count_path_slice_parts()), each leaf its own, at whatever key position and under whatever column numbering - a leaf attached with dropped columns, or with an index of its own listed before the one the parent’s index made. A leaf without one - no index slicing the column, a partial one, an expression’s - and the whole query is declined (to the counts with an Agg above them, or core’s plan): an aggregate over some leaves' slices and others' rows would be a second executor.
  • The plan: member 25 (SLICES) keeps one index and key-column pair per sliced column on each leaf, leaf-major in PARTS order (nrel × ncol, nrel 1 for a table), its one-index-per-column lists otherwise unchanged; LION_PRIV_MAGIC changed with it.
  • The executor: each leaf’s slice indexes open with its relation and stay open for the node’s life, as §16 keeps every relation the node reads (lion_open_parts()), so a REINDEX of one under an open cursor is refused as it is for the leaf’s other indexes - where a table’s opens its slice index for each run. A leaf’s turn (lion_run_partition(), or run 0 of lion_count_runs()) locates its WHERE, counts it if the target list asks a count, and then lion_slice_part() answers the leaf’s aggregates as lion_slice_column() answers a table’s - C0 = E ∧ WHERE collected against the leaf’s existence slice, every bit counted for a sum, the descents for the extremes - before the leaf’s sets are released, so that the §9 interlock is a leaf’s at a time as before. The answers are added up as they come (LionSCol.tcount, tsum in int128, the least minenc and greatest maxenc of the leaves with a row, §51.1’s encodings being order-preserving), and after the last leaf lion_slice_parts_end() hands the totals to lion_slice_results(), which builds the row as for a table: sum(int8)’s numeric and sum(int4)’s int8, avg’s scale, NULL where no leaf had a row. A leaf with no rows, or none that pass, adds nothing; a parent pruned to nothing is core’s dummy rel and its one row of NULLs, never this node; a leaf whose index slices the column at another type or width is an error, since the planner checked every leaf.
  • The price: the sum of the leaves' own (§49): each leaf’s runs priced as a table’s (lion_count_path_price_runs() over that leaf’s target alone) and its slices by lion_cost_slice_path() with the leaf’s rows, its index’s pages and its sliced ranges (lion_cost_slice_rel()), then §39’s units once over the total. Nothing else’s price changed; a parent pruned to one leaf is priced as that leaf named alone.
  • EXPLAIN names no index in Aggregates From Slices, as it names none in Lion Indexes: each leaf reads its own.
  • Not taken: a GROUP BY (§52.12: core’s Finalize HashAggregate combines the leaves' partial groups, and avg and sum(int8) have internal partial states the node would have to serialize; an in-node merge of every leaf’s groups is what §16 removed for its memory); a clause on the partition key with enable_partition_pruning off, or one a kept leaf’s bounds do not imply, which stays residual (§16, “Residual clauses over a partitioned table”); a hot standby, as for a table. A range on a sliced column, as the WHERE of these aggregates or of a count, is §53.11’s.

Tests: test/sql/slices_partitioned.sql compares every answer with the pushdown forced (sequential, bitmap and plain scans off) against the pushdown off, as multisets of text, and names the plan that ran it - over RANGE, LIST (a NULL partition, a default) and HASH partitioning and two levels of it, an empty leaf, a leaf attached with dropped and reordered columns and its own index, every type of the allowlist with NULLs, extremes and infinities, int8 sums past bigint; WHEREs on other indexed columns, pruning by the key, the key implied, pruned to nothing, pruning off, a generic plan and a LATERAL rescan; a dirty heap - updates that move rows between leaves, deletes, this transaction’s rows rolled back - and VACUUM, with EXPLAIN ANALYZE’s counters; and the declines (a leaf with an unsliced, a partial or an expression’s index, a GROUP BY, a residual clause, the switch off; a REINDEX under a cursor refused). test/sql/slices_partitioned_leaf.sql has the cases pruned to one leaf (of a sub-partitioned parent, an empty one, a LIST’s NULL partition, one of a HASH’s), apart because PostgreSQL 18.0 to 18.6 drop a disabled path’s disabled_nodes in an Append of one child (fixed upstream for 18.7): the plan forced off is not off there and is the cheaper on data this small, so the file stops at once on those releases. test/isolation/slices_partitioned.spec runs §52.11’s snapshot checks against a parent of three leaves, the writer’s updates moving rows between them; slices_plan.sql’s partitioned table now gets its one row from the slices and its GROUP BY from core.

53. Ranges from slices (2026-10-27)

§52.12 left the fourth level of the slices' planner task undone: a range on the sliced column itself. Until this section a range on any lion column was answered from its VALUE entries - the walk of every key it covers that bounds the count (§28), or the union of those entries collected into a copy where it is a source (§32) - or, on a column with summaries, from the summaries of the buckets it covers whole and the keys at its two ends. Both grow with the range: a walk with the keys it covers, a collection with the containers its sets hand out. A sliced column keeps its values as bits, and O'Neil and Quass’s range algorithm (1997, “Improved query performance with variant indexes”, Algorithm 4.3) answers any range from them with a fixed amount of work at each container key, whatever the range covers:

SELECT count(*) FROM events WHERE amount BETWEEN 100 AND 5000;
SELECT sum(amount), max(amount) FROM events WHERE amount > 100;
SELECT country, count(*) FROM events WHERE ts >= $1 AND amount <> 0 GROUP BY country;

x > c, x >= c, x < c, x <= c, BETWEEN and x <> c on a column a lion index slices (§51) are now collected from its slices where the planner prices that cheaper than walking or summing the entries: folded into a private copy key by key, the copy a source like §32’s collected range, ANDed with the column’s existence slice E located with its pin. Nothing downstream changed: the WHERE collected once and the group batches of §10, §52’s slice aggregates, the parallel count and the spill of §32 take the copy as they take any collected range.

53.1 The fold (lion_slice_range_collect(), lion_planes_bsi_step())

A range on a sliced column of nbits bits is, once its bounds are encodings (§53.2), the rows of E whose encoding lies in [lo, hi] and equals none of the holes h1 … hk (<>). For one constant c O'Neil’s descent keeps two sets - EQ, the rows whose bits so far are c’s, and an accumulator - and folds one slice S_i a step, from the top bit down:

LT (rows below c):  c_i = 1: LT |= EQ ∧ ¬S_i;  EQ &= S_i      c_i = 0: EQ &= ¬S_i
GT (rows above c):  c_i = 0: GT |= EQ ∧ S_i;   EQ &= ¬S_i     c_i = 1: EQ &= S_i
EQ (rows equal):    EQ &= S_i or ¬S_i by c_i

and the range is

E ∧ ¬LT(lo) ∧ ¬GT(hi) ∧ ¬EQ(h1) ∧ ... ∧ ¬EQ(hk)

A one-sided range needs one comparison, LT of its lower bound or GT of its upper, a BETWEEN two, and <> one EQ a hole. The fold runs at each container key of E (lion_slice_range_collect(), lion_slice.c), on images of the key’s rows:

  • E’s container is the start. Every slice holds rows of E only, so the images are laid out at the WIDTH of E’s container there (§38: a heap block’s words, 2 for a table of 100 rows a page, 128 words a key where a full bitset is 512), and a slice’s container of another width or kind is laid out to it (lion_slice_range_plane(): a BITSET’s or a NARROW’s words where the widths agree, read in place; another width’s block by block; an ARRAY’s members or a RUN’s runs set word by word). A member past E’s width is no member of E and is left out, which the AND with E’s rows makes exact.
  • A step is one or two word operations a word (lion_planes_bsi_step(), lion_planes.c, plain loops the compiler vectorizes; a slice with no entry, or none at the key, is a slice of no row).
  • A comparison stops where EQ empties: once no row is equal to the constant on the bits so far, no later step can add to its accumulator. A key’s rows that agree on their top bits keep EQ alive through them, and past those the rows halve about every step, so a comparison takes the bits the key’s rows need to differ from the constant, some log2 of the rows a key and two: about 14 of an int4’s 32, or of an int8’s 64, at 7,700 rows a key of values spread over the type (§53.7). Where the column is in the heap’s order, the rows at a key share their values' top bits and part from the constant at the first bit they differ in: the bits every row of the column shares and two or three more (11.7 of a sequential int4’s 32, 23.8 of an int8’s 64 whose values take 43). Each slice is read by a cursor that holds no pin (lion_cursor_init_at() with droppins), sought to the key only when a comparison reaches its bit.
  • The key’s rows - E less what the comparisons rule out - are made a container (lion_bits_to_container()) and appended to the copy in key order (lion_collect_add()): within the memory the relation’s collected sources share (lion_collect_within(), §32), and past it into a temporary file, as lion_sources_collect() spills (the copy-building half of that function, factored out as lion_collect_begin(), _add() and _finish()).

The bounds that leave every row of E in (x >= -2147483648 on an int4, x < 5000000000 on one) need no fold: the source is E itself, located with its pin. Bounds that cross, a NULL bound, and one past the type’s range on the side that excludes everything select nothing, and the source is missing (§53.2).

53.2 The bounds as encodings (lion_slice_range_add(), _settle())

A bound’s value is the column’s own type’s, or for an integer column one of the integer family’s other types (§28’s cross-type comparisons, int4col < 5000000000::int8); every type of the allowlist orders as the signed integer its Datum is (§51.1), so x op v holds of a row exactly when it holds of the two integers. The bounds are clipped to the type’s range [-2nbits-1, 2nbits-1 - 1] as they are added - x > v with v at or past the greatest value selects nothing, x < v with v past it bounds nothing - ANDed into one closed interval [L, H] of values, and encoded once they are all in (lion_slice_encode()): lo = enc(L), hi = enc(H), LT needed where lo is not 0 and GT where hi is not all ones, a hole kept only inside [L, H]. The infinities of date, timestamp and timestamptz are their types' extremes and need nothing of their own: ts < 'infinity' is every row but the infinite ones, ts > 'infinity' none. A NULL bound (a parameter’s) makes the range empty, every operator involved being strict; BETWEEN 9 AND 3 crosses, and is empty. The executor builds the bounds at every locate of the WHERE (per run and rescan, the parameters being values then), from each bound clause’s value and type (lion_rangesrc_range()); a bound of another type, which no shipped family has, sends the range to §32’s collection instead.

53.3 A source that pins itself, and why it is exact

The copy holds no pin, as every collected range (§32, “What it needs beside it”), so a count with nothing else that carries the §9 interlock would recheck every row in the heap; the planner took §32’s ranges only beside a pinned source. A range from slices brings its own: its source is the AND of the copy and E, located with its pin (lion_locate_range_sliced()), and an AND is pinned when any of its sets is (lion_plan_node()). E holds every row the copy can hold, so the AND counts what the copy does. count(*) WHERE x > c needs nothing beside it, and an OR’s leaf from slices is pinned as any leaf.

Why it is exact. The slices and E are read without a pin, after the snapshot was taken. A row the snapshot sees is in E and in exactly the slices of its value’s bits: its insert wrote every one of them before its transaction committed (§51.6, key first), and only VACUUM takes a TID out of a set, for a row no snapshot sees. Whenever each slice is read, then, the fold classifies a visible row by its value, and the copy holds it exactly when its value is in the range. A row the snapshot does not see may be classified by slices read before and after a write moved it, or VACUUM took it from some of them: it is in the copy or not, and is counted beside E located with its pin, whose container the map is asked under - the collected copy’s argument (lion_sources_collect()): a dead TID the copy still lists is gone from E’s container or on a heap page whose all-visible bit VACUUM cannot yet have set, and the recheck under the snapshot rejects it. The transaction’s own rows are rows its snapshot sees, decided in the heap as every other; a TID whose row was replaced after the snapshot is a row the snapshot does not see. On a hot standby the planner offers none of this, as it collects nothing there (§10), and an executor that met such a plan there would collect the range from its entries instead.

53.4 The planner (lion_try_count_path_scoped(), lion_count_path_strategy())

A build of the count path of its own (LionCountPathBuild.slicerange), beside each of the builds §52 made - with the slices' aggregates and without - in which every range on a column whose index slices it is a source from the slices:

  • Which ranges. A range on a column whose clauses are matched to an index that slices it at its type’s width (lion_range_sliced_index(): the first whole lion index of the column, which is the one lion_match_index() takes), with bounds of types the encoding takes - outside the scope of a partial index. Not one that bounds the walk of a GROUP BY or count(DISTINCT) of its own column, whose entries the walk reads anyway; never the summed driver of §28. An OR’s range leaf likewise (a union holds the §9 pin, which it now does). x <> c on such a column, outside the ORs and the filtered counts' conditions, is made a range with a HOLE (§35) - a positive source, where it is otherwise a negated one that only subtracts - and joins any other range on the column in the one fold (lion_index_answers() takes strategy 10 for a RANGESRC). Every bound is marked LION_CLAUSE_SLICED in the plan’s clause kinds, beside §32’s LION_CLAUSE_WALK, which a range from slices never is.
  • Declined where the build has no range to answer so (no duplicate of the other build), and on an FK-side or anti join, a SELECT DISTINCT’s groups and a hot standby (lion_query_may_slice_ranges()), and with pg_lion.enable_slice_ranges off. A partitioned table takes it partition by partition (§53.11).
  • The price (§53.5) is the build’s own; add_path() keeps the cheapest of it, the build that walks and sums the entries, the summaries' collections, and core’s paths. That is the whole of “summaries at selective ranges, slices at wide ones”: each prices what it reads.

pg_lion.prefer_slice_ranges (off; for the tests) leaves the other builds out where the build from slices is made.

53.5 Cost (lion_cost_slice_range(), four settings)

A range from slices is a range taken as a source (§32, “Costs”): its rows, the planner’s selectivity of its bounds, are a source of the AND like any, priced by the count model as before. What differs is the collection, which lion_count_rel_range_source() prices by lion_cost_slice_range() for a range whose first bound the build marked (the planner hands the cost model the list of them, lion_cost_set_sliced(), around its pricing of the build), in place of §32’s lion_cost_range_source() - and which is never priced walked. From what the slices' entries record (lion_slice_shape(), as §52.6 reads it):

  • the keys: E’s container keys - its rows over the heap’s keys - each folded at pg_lion.slice_range_key_cost (700 cpu_operator_cost, 1.4 us at 500 units a millisecond: E’s container laid out, the comparisons' images begun, the key’s rows counted for a container);
  • the steps: one comparison for a range bounded on one side, two on both, each taking at a key the bits every row of the column agrees on from the top (topconst: the slices of no row and of every row, which the shape says), two more, and - where the column is not in the heap’s order, its correlation’s square interpolating - log2 of the rows a key, which a uniform column’s need to part from the constant: 15 of an int4’s 32 at 7,700 rows a key (14.3 measured), 11.7 measured for a sequential int4 (its nine constant top bits and 2.7), 23.8 for a sequential int8 (21 and 2.8). A step is pg_lion.slice_range_step_cost (8) and slice_range_word_cost (0.1) a word of the key’s width: 20.8, 42 ns, at a width of 2;
  • the slices read: E and the top steps slices at E’s keys, as §52’s counts read a slice (lion_slice_read_cost(): each container met, each posting page read and copied, the seeks);
  • the copy: a container of the range’s rows a key they lie at - a key of the heap’s each for a column in no order, their share of the keys for one in order, the correlation interpolating as for §32’s union - at pg_lion.slice_range_container_cost (900, 1.8 us: the image made a container and optimized, copied into the copy, and its AND with E’s container in the count), and E read again at those keys; within what the ranges before it left of a hash table’s memory, or past it into a file written and read once.

Fitted on §53.7’s 5M rows and five columns - uniform and sequential int4 and int8, and a timestamp’s microseconds in heap order - with a one-sided range on each side and a BETWEEN at 1%, 10%, 50% and 90% of each, sixty ranges in all, by least squares over what the rest of the price leaves of each one’s time at 500 units a millisecond: 680 a key, 20.4 a step at width 2, 874 a container. With the settings rounded to those, the price of the sixty runs at 344 to 770 units a millisecond, median 498, and the steps it counts were 0.94 to 1.06 times those EXPLAIN ANALYZE counted. DEBUG1 reports each estimate’s parts: the keys, the steps a comparison, the comparisons, the words, the copy’s containers, and the read, fold and copy prices.

53.6 EXPLAIN

Custom Scan (LionCount)
  Lion Indexes: t_x (x >= 100 AND x <= 5000), t_g (g = 3)
  Range From Slices: x >= 100 AND x <= 5000 (32 slices)

Range From Slices names each range the plan folds - a WHERE’s, and an OR arm’s - with its bounds and the slices its column has (its width). EXPLAIN ANALYZE adds Slice Range Descents (collections: a range once a relation, run, rescan and participant), Slice Range Keys (E’s container keys folded), Slice Range Steps (the comparisons' steps: a slice’s container laid out and folded at a key, the container operations the price counts) and Slice Range Copies Spilled; the range is among the Range Sources Collected (and Spilled) of §32 too.

53.7 Measured

bench/slices_range.sql (psql -X -v rows=5000000 -f bench/slices_range.sql, then -v setup=off): 5,000,000 rows on about 41,700 pages - 651 container keys of E, 7,700 rows a key - with five columns: u4 and u8 uniform over their types, s4 and s8 the row number (s8 times 1,000,003) in the heap’s order, and t8 a timestamp’s microseconds, a row a second with a second’s jitter, in the heap’s order. Each under a one-column lion index with slices, without summaries (phase 1), with them (phase 2) and with a btree beside (phase 3); each range a BETWEEN centred in the column’s values, of 0.1% to 90% of the rows. PostgreSQL 16 at -O2 without assertions, four cores of a 2.80 GHz Xeon shared with another session’s builds and tests (a load of about two), no parallel workers, work_mem 64 MB; each number the median of five runs, the shapes taken in turn, warm. slices is the range from slices (pg_lion.prefer_slice_ranges, the other scans off); walk the same count with pg_lion.enable_slice_ranges off, on the index without summaries (§28’s walk of the range’s entries); summaries the same on the index with them (§32); ordinary the plan with the pushdown off; btree the same with a btree on the column beside (an index-only scan); plan what the planner chose at its defaults, and its time, on the index without summaries and with them. Every answer of each was compared with the ordinary plan’s. ms:

count(*):

col % slices walk summaries ordinary btree plan, no summaries plan, summaries
u4 0.1 7.7 1.6 1.9 7.2 0.7 walk 1.6 summaries 1.9
u4 1 8.4 14.0 3.6 47.2 5.9 slices 8.3 summaries 4.0
u4 10 7.8 139.3 7.6 216.2 57.9 slices 7.7 summaries 7.6
u4 50 7.4 712.3 32.3 361.8 283.1 slices 7.4 slices 7.4
u4 90 7.0 1,304.0 55.7 395.7 359.8 slices 7.8 slices 7.1
u8 0.1 7.0 1.6 0.5 7.3 0.7 walk 1.6 summaries 0.5
u8 1 7.9 13.9 2.2 47.4 5.9 slices 8.2 summaries 2.2
u8 10 7.4 137.3 7.2 202.6 56.7 slices 7.5 summaries 7.4
u8 50 7.3 746.4 34.8 386.3 284.2 slices 7.4 slices 6.9
u8 90 7.1 1,251.8 61.6 374.8 422.0 slices 7.5 slices 8.0
s4 0.1 2.6 1.3 1.4 2.1 0.5 Index Scan 2.1 Index Scan 2.1
s4 1 2.7 12.0 1.4 20.4 4.1 slices 2.8 summaries 1.4
s4 10 3.4 128.7 1.3 204.2 39.4 slices 3.6 summaries 1.3
s4 50 6.0 629.8 2.0 327.5 195.1 slices 6.3 summaries 2.0
s4 90 8.3 1,054.3 2.7 356.1 397.1 slices 8.4 summaries 2.7
s8 0.1 3.2 1.3 1.4 2.1 0.6 Index Scan 2.1 Index Scan 2.2
s8 1 3.2 11.7 1.3 20.0 4.1 slices 3.4 summaries 1.4
s8 10 3.9 115.7 1.4 198.7 47.2 slices 4.0 summaries 1.3
s8 50 6.6 575.7 2.0 329.4 237.6 slices 6.6 summaries 1.9
s8 90 9.0 1,081.0 2.7 379.4 393.9 slices 9.0 summaries 2.7
t8 0.1 4.5 1.3 1.5 2.2 0.5 Index Scan 2.2 Index Scan 2.2
t8 1 4.7 11.8 1.4 20.5 4.3 slices 4.7 summaries 1.4
t8 10 5.4 117.2 1.4 204.1 42.7 slices 5.3 summaries 1.3
t8 50 8.1 610.0 2.0 360.2 199.7 slices 8.1 summaries 2.0
t8 90 10.4 1,077.0 2.6 390.5 406.5 slices 10.5 summaries 2.6

(Index Scan is the lion index’s own plain scan under an Aggregate, §29.) With the btree beside, the planner took its index-only scan at 0.1% (0.6 to 0.8 ms) and the LionCount node from 1% up - the summaries, and the slices at 50% and 90% of u4 and u8 (7.7 to 8.0 ms).

sum(x) under the range, end to end (§52’s sum of the slices over the copy, against its sum over C0 collected from the summaries and entries; the medians of three such runs, after §53.8’s price of that collection; btree and ordinary from the first):

col % slices summaries ordinary btree plan, summaries
u4 0.1 14.3 10.1 7.2 0.7 summaries 10.1
u4 1 15.6 14.2 45.4 6.7 summaries 14.2
u4 10 15.2 33.3 154.9 63.7 slices 15.6
u4 50 14.1 66.3 386.4 291.6 slices 14.3
u4 90 14.5 91.3 468.3 384.2 slices 14.8
u8 0.1 20.6 14.7 8.1 0.8 summaries 14.7
u8 1 21.5 20.6 45.8 6.2 summaries 20.8
u8 10 21.2 36.4 161.4 75.6 slices 21.0
u8 50 21.0 68.8 393.9 409.2 slices 21.5
u8 90 21.4 99.3 393.2 422.5 slices 21.3
s4 0.1 2.9 2.3 2.3 1.0 Index Scan 2.6
s4 1 3.1 2.6 6.7 4.4 Index Scan 7.1
s4 10 4.2 3.6 57.0 40.9 summaries 3.4
s4 50 9.4 10.4 280.5 198.7 slices 9.7
s4 90 14.9 17.8 412.4 368.8 slices 14.8
s8 0.1 3.2 2.2 2.4 0.5 Index Scan 2.4
s8 1 3.6 2.4 7.0 4.3 Index Scan 7.2
s8 10 5.0 3.8 54.7 41.5 summaries 4.0
s8 50 13.0 12.6 276.1 208.8 slices 13.4
s8 90 20.1 21.5 418.3 472.4 slices 19.7
t8 0.1 4.7 2.3 2.3 1.0 Index Scan 2.5
t8 1 4.8 2.5 7.1 4.4 Index Scan 7.2
t8 10 6.7 4.1 56.0 48.8 summaries 3.9
t8 50 15.3 13.4 268.5 215.3 summaries 13.5
t8 90 23.3 23.0 425.6 455.4 slices 22.8

What the tables say:

  • A range from slices costs about the same at any width on a column in no order: 7.0 to 8.4 ms for the count at 0.1% and at 90% of u4 and u8 - every key of E folded through about 14 steps a comparison (14.3 measured, for an int8 as for an int4: the steps are what 7,700 rows need to part from the constant, not the type’s bits), and a container of the copy at every key. On a column in the heap’s order it grows with the keys the range spans, 2.6 to 10.4 ms, the keys outside it stopping at their first bit that differs from the bounds' and giving no container.
  • Against the walk of its entries the range from slices is 1.7 to 186 times faster from 1% up; at 0.1%, 5,000 entries walked (1.3 to 1.6 ms) are faster, and the planner walks them.
  • Against the summaries it depends on the column’s order. A column in no order spreads each bucket’s rows over the whole heap, so what the summaries read grows with the buckets a range covers: 0.5 to 3.6 ms at up to 1%, a tie at 10% (7.2 to 7.6 ms), 32 to 62 ms from 50% up, where the slices are 4.4 to 8.7 times faster. A column in the heap’s order keeps a bucket’s rows in a few keys, whose summaries are a few containers: 1.3 to 2.7 ms at any width, 1.9 to 4 times the slices' speed. The planner chose the faster wherever the two were not tied - summaries for the selective ranges and for the ordered columns, slices for the wide ranges of the uniform ones, §53.4’s “each prices what it reads” - but at 0.1% of an ordered column (below).
  • Against the btree’s index-only scan: 7.4 to 12 times faster at 10%, 39 to 59 at 90%; at 0.1% the btree’s 0.5 to 0.8 ms wins, and the planner takes it when it is there.
  • sum(x): the slices' sum under a range from slices is 14 to 16 ms on u4 and 21 on u8 at any width - the range’s fold, then §52’s 33 (or 65) counted ANDs at the copy’s keys. Over C0 collected from the summaries and entries it grows with the range, 10 to 91 ms (u4); from 10% up the slices are 2.2 to 6.3 times faster and the planner takes them, below it keeps the summaries, 1.1 to 1.4 times faster. On the ordered columns the two are within 20% of each other from 50% up, and the planner takes either. (Before §53.8’s price of C0’s collection it took the summaries at 10% of u4 and u8, 31 and 38 ms against the slices' 15 and 22.)
  • Where the planner was wrong, it was in prices this section did not set: the lion index’s plain scan (§29) taken at 0.1% of an ordered column, 2.1 ms against the walk’s 1.3, and for the sum of 1% of one, 7.1 to 7.2 ms against the summaries' 2.4 to 2.6; and the summaries' sum at 0.1% of u4 and u8, 10.1 and 14.7 ms, against the ordinary plan’s 7.2 and 8.1 (§53.10).

53.8 Beside it

  • The parallel count (§10, “A count in parallel”) divides the WHERE’s keys among its participants; each folds the range over E for itself, as each collects §32’s ranges, and counts its share. A GROUP BY beside a range taken as a source stays serial, as beside any.
  • §52’s summed range is priced as collected. The slices' aggregates collect a range that bounds the sum over its entries for their WHERE (C0, lion_slice_where_sources()), which lion_cost_slice_path() took to be within half of the WHERE’s price; it is now priced as §32 prices a range taken as a source (lion_cost_range_source(), its memory and spill), once, beside the WHERE (§52.12’s “underpriced” bullet). Without it the build from slices lost to that one at 10% of a uniform column’s sum, whose collection it priced at a fifth of its time (§53.7).
  • The copy’s building - lion_collect_begin(), lion_collect_add() and lion_collect_finish() (lion_count.c) - is factored out of lion_sources_collect_keys(), which calls them as before; the fold hands them its containers a key at a time.
  • A range from slices is never served from a private copy of E on its second use, as a located set may be (nomaterialize): E carries the source’s pin.
  • New settings: pg_lion.enable_slice_ranges (on), pg_lion.prefer_slice_ranges (off, not in the sample file), and the four costs of §53.5.
  • lion_planes.c gains lion_planes_bsi_step() and its unit tests; lion_slice.c gains the counting side (lion_slice_range_*()), nothing on the insert side.

53.9 Tests

  • test/sql/slices_range.sql: every answer against the ordinary plan (the pushdown off, a sequential scan) as multisets, with how its ranges were answered (slr_q(): from slices, not, or no pushdown), under pg_lion.prefer_slice_ranges: each type of the allowlist with NULLs, negatives, each type’s extremes and the next ones in, and the infinities - every operator, BETWEEN and <>, bounds at, beside and past the type’s range (an int4 or int8 bound on an int2 column, an int8 one on an int4), reversed and crossed bounds, several bounds and holes on one column; beside equalities, lists, null tests, <> and ranges of other columns, two ranges from slices, a range and an equality or a list on one column; a GROUP BY of another column, of two, and of the range’s own (still a bound), the top k and count(DISTINCT); §52’s aggregates of the same and of other columns, the one row and a GROUP BY’s groups decoded and a group at a time; an OR’s leaves, BETWEEN SYMMETRIC; a generic plan’s parameters (NULL, crossed, past the type), stable expressions, a LATERAL rescan; deletes and updates on a dirty heap, this transaction’s rows at the extremes, deleted and updated, rolled back, and VACUUM; a copy past a 64 kB work_mem, spilled; the plans and EXPLAIN ANALYZE’s counters; a parallel count’s Gather, and a GROUP BY beside it kept serial; the planner’s choice, with nothing preferred, over 100,000 unique values with summaries and slices and with slices alone, at 0.1% and 60%, and with the switch off; the declines: a float column, a join, a column no index slices, the switch off; a partitioned table’s range from slices (§53.11, whose tests are slices_partitioned.sql’s).
  • test/isolation/slices_range.spec: ranges from slices, a <>, an OR and a sum under a range, beside the same counts of expressions of the columns, which only the ordinary plan computes, in one statement: a REPEATABLE READ reader across a writer’s rows at every column’s extremes, updates that move rows across the bounds and deletes, committed, rolled back, and VACUUMed under the reader’s snapshot.
  • test/unit/planes_test.c: lion_planes_bsi_step()’s LT, GT and EQ folded over random columns of 8 to 64 bits and 1 to 512 words, against the values compared one by one, stopped where EQ empties and run to the last bit.
  • test/sql/slices.sql’s wide range on its sliced column is now answered from the slices; it shows that plan and keeps its answers on the summaries' walk with the switch off.
  • costgucs.sql lists the four settings.
  • The whole suite: 192 regression and 33 isolation tests in generic WAL mode and again with the resource manager registered (shared_preload_libraries = pg_lion, an index built there reporting rmgr), the injection-point specs skipped on this build as always; the unit tests.

53.10 Not done

  • The AM’s own scans (amgetbitmap, amgettuple) of a range on a sliced column still walk its keys or its summaries (§28, “Bitmap scans”; §29.3). What it would take, on §28’s template: in liongetbitmap(), a range key on a sliced column priced (in lioncostestimate()) cheaper folded than walked would be folded at each container key of E - or, beside other columns' sets, at the keys of each window of their stream (lion_emit_intersect()), E and the slices sought there - and the key’s rows added to the caller’s bitmap, or ANDed with the window’s container first; the bitmap scan needs no §9 interlock, since the executor visits every TID it emits, so the fold holds no pin, as here. The plain scan’s stream (§29.3) would take the folded containers a window at a time the same way. Neither changes the format; the price is §53.5’s with the copy left out.
  • Partitioned tables, joins: each partition would fold its own index’s slices, and the planner’s build decide per partition - the slices' index is matched per relation already (§16), but the build is declined whole, as §52’s is. (2026-10-28, §53.11: partitioned tables done, every partition’s index slicing the column or the build declined.) The FK-side join’s fact filters (§27) and the anti join’s take ranges as §32’s sources only. A filtered count’s condition (§10, “Filtered counts”) takes its ranges from the entries.
  • The fold at the WHERE’s keys alone: beside a selective clause, E’s keys are mostly keys the count never reaches. Folding only at the keys of the other sources (sought as the probe of §32 seeks a summed range’s sets) would make the copy the WHERE’s and no longer a source of its own, which is what kept everything downstream unchanged; the planner’s choice of the summaries or the walk for a selective range covers the case where it matters most. Likewise each participant of a parallel count folds the whole of E, where its share of the keys would do.
  • A BETWEEN’s two comparisons share their EQ while lo and hi agree on their top bits, and could be folded as one there: a BETWEEN of u4 took 9.4 ms where one bound took 7.5 in §53.5’s runs, the steps the sharing would save.
  • The full domain at plan time: bounds every value meets are E alone at run time (§53.1), but the planner prices the fold all the same; and an x <> c alone, which is E less c’s rows, is priced and run as a fold with one EQ, where E and c’s entry, subtracted, would do with no fold.
  • The competitors' prices (§53.7): the lion index’s plain scan (§29) is taken for a range of 0.1% of a column in the heap’s order, 2.1 ms, over the walk’s 1.3, and for the sum of 1% of one, 7.1 ms, over the summaries' 2.6; the summaries' sum of 0.1% of a uniform column is taken at 10 to 15 ms over the ordinary plan’s 7 to 8. Neither is this section’s price, nor changed by it.
  • On a hot standby the planner offers none of this, as none of §32’s collections (§10).
  • Percentiles and the rest of §52.12 stand.

53.11 Partitioned tables (2026-10-28)

The ranges of §53.4 over a partitioned table, in the one LionCount node that counts every leaf (§16), each leaf folding its own index’s slices:

SELECT count(*) FROM parted WHERE k > 50 AND k < 350 AND v <> 17;

  Custom Scan (LionCount)
    Partitions: parted_1, parted_2a, parted_2b, parted_3
    Lion Indexes: (k > 50 AND k < 350), (v <> 17)
    Range From Slices: k > 50 AND k < 350 (32 slices), v <> 17 (32 slices)
    Implied by Partition Bounds: (k > 50) in 3 of 4 partitions, (k < 350) in 3 of 4 partitions

It fell out of what was there: the executor resolves a range against the index the leaf’s turn matched its bounds to (lion_rangesrc_range()), and decides from that index’s column whether the slices answer it; the cost model prices a range source per relation by its clause (lion_cost_sliced), and a partitioned count is priced as the sum of its leaves' counts (§49), so a range from slices is priced in each leaf by §53.5’s formula with the leaf’s index and rows. What changed:

  • The planner asks the first live leaf where it asked the table - whether any whole lion index slices a column (lion_query_may_slice_ranges()), and whether the one a range’s column is matched to slices it at width and takes the bounds' type (lion_count_slice_range_ok(), the column translated as §52.13’s are) - and, once the leaves are known, every leaf (lion_count_path_slice_range_parts()): the index its target matched each bound to has to be the one lion_range_sliced_index() finds on the leaf, at the same key column. A leaf whose index does not slice the column declines the build; the build that collects or walks the range from the entries is offered beside it as always, and the executor’s fallback - a range from the entries where the column is no longer sliced, as after a REINDEX - is never planned. A leaf that leaves a bound out, its partition bounds implying it (§16), reads nothing for it, and a range whose every bound is so left out in every leaf is not shown in EXPLAIN. Not inside a partial index’s scope (§45), which is a leaf’s at a time; not on a hot standby.
  • The pin (lion_plan_partition.c): a leaf that leaves clauses out must keep one that selects rows and carries the §9 interlock (lion_leaf_selects(), lion_pinned_not_implied()), which a range collected from its entries does not - but a range from slices does, ANDed with E located with its pin (§53.3), alone or as an OR’s leaf. Before this, count(*) ... WHERE v > c on a partitioned table had no clause to carry the pin and was declined.
  • EXPLAIN’s (N slices) is read from a leaf’s index (the plan holds none for a partitioned table), and a bound every leaf leaves out is not printed.

The ranges compose with §52.13’s aggregates (sum(w), max(ts) ... WHERE g = 3 AND v BETWEEN a AND b, from slices both), with the counts' partial groups of a GROUP BY (g, count(*) ... WHERE v > c GROUP BY g, the Finalize Agg above), and with pruning. Over 9,000 rows in 4 leaves, count(*) WHERE v > 1000 was priced 60.71, the sum of the four leaves named alone (15.61, 14.71, 14.79, 15.61), against 633.57 collected from the entries. Tests: section 6 of test/sql/slices_partitioned.sql (each type, beside equalities and lists, <>, an OR’s leaf, bounds of another type, crossed and past the type, ranges on the partition key implied by some leaves and by all, a GROUP BY, §52.13’s aggregates, a generic plan with a NULL bound, a dirty heap and VACUUM, the plans and EXPLAIN ANALYZE’s counters, and the declines: a leaf whose index does not slice the column, until pruning leaves it out, and the switch); slices_range.sql’s partitioned table now answered; and the isolation spec slices_partitioned.spec with a count and a sum under a range from slices beside the aggregates.