Limits

Generated by scripts/gen-limits.sh from the registry SIFT_LIMITS in src/core/limits.h; do not edit by hand. scripts/gen-limits.sh --check fails when this file is out of date or when a *_MAX_* / *_LIMIT* macro of src/ has no row (or a reason in scripts/limits.allow).

“Normal plan” means the statement runs on the plan PostgreSQL would choose without the scan, with the same result; sift.explain_why names the slug. A limit that is an error says so. Limits of PostgreSQL itself (1 GB per value, 1600 columns) are not listed.

In a testing build (make SIFT_TESTING=1) SET sift.test_limits = 'id=value,...' lowers the limits marked “yes” in the last column, so that a test reaches them on a few rows; sift._test_limits() lists every limit with its default and the value in force.

Aggregate pushdown (src/agg)

id Limit Default Unit At the limit Changed by Source Test
partitions leaf partitions per SiftAgg scan, after PostgreSQL’s pruning 32,768 partitions normal plan (slug partition_count, the detail has the count) agg/part.h SIFT_AGG_MAX_PARTS yes
hash_part_combos value combinations the remainders of a hash level are computed for (once per level) 1,048,576 combinations the partitions of the level are kept (not pruned) agg/part/prune.c HASH_MAX_COMBOS yes
part_key_tests keys times range or list partitions a list of keys is tested against exactly 20,000 tests only the range of the keys prunes agg/part/prune.c KEYS_MAX_TESTS yes
group_nospill_factor group table of a scan that cannot spill (a two level aggregate, grouping sets, ordered, generic and numeric or text min, max and sum over combos) or of the node that groups the rows of a plan, as a multiple of work_mem times hash_mem_multiplier (8 MB at least): estimated by the planner, held by the scan 32 times hash memory normal plan (slug group_memory, the detail names the groups and the limit); ERROR 54000 naming work_mem and hash_mem_multiplier when the table the scan holds passes it work_mem, hash_mem_multiplier agg/spec.c GROUP_NOSPILL_FACTOR yes
group_spill_parts partitions a group table that outgrows hash memory is written to (the planner’s groups over the groups the first table held, a power of two from 8) 128 partitions a partition that does not fit is partitioned again work_mem, hash_mem_multiplier agg/spill.c SPILL_MAX_PARTS yes
group_spill_depth levels of partitions a partition of a spilled group table is partitioned again (a hash of the same key values, another level each time) 8 levels ERROR 54000 naming work_mem and hash_mem_multiplier work_mem, hash_mem_multiplier agg/spill.c SPILL_MAX_DEPTH yes
group_nospill_base the hash memory group_nospill_factor is applied to when work_mem times hash_mem_multiplier is less (the memory the defaults give) 8,388,608 (8 MiB) bytes as group_nospill_factor agg/spec.c GROUP_NOSPILL_BASE yes
joins lookup-joined tables per scan 32 tables normal plan (slug shape_width) agg/spec/expr.h SIFT_AGG_MAX_JOINS yes
combos value combinations per scan 128 combinations normal plan (slug shape_width) agg/spec/expr.h SIFT_AGG_MAX_COMBOS yes
combo_entries distinct input tuples of one value-combination cache 16,777,216 entries the caches start again (re-evaluates the values they held) agg/combo/int.h COMBO_MAX_ENTRIES yes
join_keys equality columns of a lookup join 3 columns normal plan agg/spec/expr.h SIFT_AGG_MAX_JKEYS
join_cols columns of the joined table that expressions read 16 columns normal plan (slug shape_width) agg/spec/expr.h SIFT_AGG_MAX_JCOLS yes
join_index_eqs equalities of a joined-table column with a value, for the index scan that loads it 3 equalities normal plan agg/spec/expr.h SIFT_AGG_MAX_JIDX
lookup_kb estimated memory of the rows of a lookup table (a sub-select, VALUES list or function, a first-row child) or of a spine 262,144 (256 MiB) kB normal plan (slug join_child, spine_rows) sift.agg_lookup_max_kb agg/plan/finish.c
lookup_shared_bytes a lookup table shared with the parallel workers through DSM 1,073,741,824 (1 GiB) bytes each backend loads its own sift.agg_lookup_shared_kb agg/par/int.h SHARE_MAX
lookup_shared_slots lookup tables shared through DSM at one time 64 tables each backend loads its own sift.agg_lookup_shared_kb agg/lookshm.c LOOKSHM_SLOTS
topn_rows LIMIT + OFFSET a top-N scan keeps in a heap 100,000 rows the scan sorts every row that passes (in memory up to hash_mem, then a tuplesort at work_mem) and the plan is priced with it agg/topn.h SIFT_TOPN_MAX_ROWS yes
topn_keys ORDER BY columns a top-N scan keeps in a heap 8 columns the scan sorts every row that passes (in memory up to hash_mem, then a tuplesort at work_mem) and the plan is priced with it agg/topn.h SIFT_TOPN_MAX_KEYS yes
topn_code_entries entries of the dictionary of a segment a top-N scan makes an order code for (8 bytes an entry, up to 8 dictionaries at a time) 1,048,576 entries the rows of its units are compared with the bar as bytes, as before the order code agg/topn/int.h ORD_MAX_ENTRIES yes
skip_pending the entries an OFFSET over a global order leaves to the heap to confirm, and the entries of one group of an order read group after group that it checks at once (12 bytes each); a longer one is walked 1,048,576 entries the entries the skip leaves are read one by one, as without the skip (the rows are the same) agg/gord/int.h GORD_PEND_MAX yes
blob_bytes the partial result a worker of a parallel aggregate sends in one message: the groups of its table go in messages of about this many bytes (a pair of a count(DISTINCT) takes 8); a group that alone takes more than four times this (1 GB at most) ends the scan 268,435,456 (256 MiB) bytes more messages (the leader adds them one after the other); ERROR 54000 for a single group that takes more than four times it, naming max_parallel_workers_per_gather agg/blob/dump.c BLOB_CHUNK_BYTES yes
first_keys sort columns after the DISTINCT ON columns 4 columns normal plan agg/order/int.h FIRST_MAX_KEYS
first_order_cols ORDER BY columns of a first-row lookup child 8 columns the sub-select runs once per key, as a lookup subquery agg/spec/join.h SIFT_AGG_MAX_FORD yes
first_out_cols output columns of a first-row lookup child that map to the table 16 columns normal plan agg/spec/join.h SIFT_AGG_MAX_FOUT
group_keys group keys (grouping sets count with their keys) 32 keys normal plan (slug too_many_keys) agg/spec/expr.h SIFT_AGG_MAX_KEYS yes
num_in_list constants of a numeric IN list compared one by one; a longer list is tested as a hash set 64 constants a hash set of the scaled values agg/plan_pred.c AGG_MAX_NUM_IN yes
bits_set_values members of an IN list from which a range wider than a byte table (4 million values) is tested in a bit table; a shorter list is a hash table 1,024 values a hash table of the values agg/pred_in.c MIN_BITS_VALUES yes
dist_narrow_words words of a group in a narrow count(DISTINCT) set (64 values a word, one window of values for all the groups); 0 turns the form off 8 words the set is a bitmap, a directory and chunks of 4096 bits a group agg/dist_int.h DIST_NARROW_WORDS yes
arms UNION ALL arms of a fused or two-level scan 16 arms normal plan (slug shape_width) agg/spec/expr.h SIFT_AGG_MAX_ARMS yes
virtual_leaves virtual leaves (expressions) per scan 16 leaves normal plan (slug shape_width) agg/spec/expr.h SIFT_AGG_MAX_VLEAVES yes
layout_leaves leaves of the layout of one scan 128 leaves normal plan agg/colsrc/meta.h SIFT_COLSRC_MAX_LEAVES
scan_filters filters of a scan that skip units 64 filters the others do not skip units agg/colsrc/exec.h SIFT_COLSRC_MAX_FILTERS
wide_conjuncts conjuncts on wide leaves per scan 8 conjuncts normal plan agg/spec/expr.h SIFT_AGG_MAX_WIDE
generic_leaves leaves one argument of a generic aggregate reads 4 leaves normal plan agg/spec/expr.h SIFT_AGG_MAX_GLEAVES
generic_args arguments of a generic aggregate (PostgreSQL FUNC_MAX_ARGS) 100 arguments normal plan agg/spec/expr.h SIFT_AGG_MAX_GARGS
float_ops operations of one float8 aggregate argument 16 operations normal plan agg/spec/expr.h SIFT_AGG_MAX_VX
combo_leaves leaves one value combination reads 8 leaves normal plan agg/spec/expr.h SIFT_AGG_MAX_CLEAVES
rows_shared aggregates sharing one rows-set (count(DISTINCT) grouping) 32 aggregates normal plan (slug rows_shape) agg/rows.h SIFT_ROWS_MAX_SHARED yes
text_prune text conditions of a scan that prune units by prefix 16 conditions the others do not prune units agg/chunkfilter.h COL_MAX_STR_PRUNE
filter_set values of an IN set sent as a zone filter 8 values a larger set is not used as a zone filter agg/init_filter.c FILTER_MAX_SET

Column index (src/col)

id Limit Default Unit At the limit Changed by Source Test
gord_seek_pct the bytes a seekable chunk of an order array (a fence key every 32 entries) may take, against the plain chunk of the same entries: the writer keeps the seekable form of a chunk up to this; 0 writes none 115 percent the chunk is written plain and read by the old walk (the rows are the same) col/gorder.h SIFT_CGORD_SEEK_MAX_PCT yes
gord_pos_pct the bytes a positional chunk of a grouped order array (its entries as positions in the plain order of the leaf, read through it) may take, against the other form of the same entries: the writer keeps the positional form of a chunk up to this; 0 writes none 75 percent the chunk keeps the keys and addresses of its entries (the rows are the same) col/gorder.h SIFT_CGORD_POS_MAX_PCT yes
gord_pos_gap the mean distance between the positions of a group in a chunk of a grouped order array that the writer keeps positional (as many entries as two chunks of the plain order): farther apart, each entry a scan reads is in a chunk of the plain order of its own 1,024 entries the chunk keeps the keys and addresses of its entries (the rows are the same) col/gorder.h SIFT_CGORD_POS_MAX_GAP yes
gord_mem_rows rows a top-N, range or count by an order array sorts in memory for one source: the write buffers' and those of the segments the overlays left without the array (a text key under a locale: at least 16,384 or 8 for each row wanted) 2,097,152 rows the scan reads the units, and the planner prices no walk of the array (the rows are the same) col/gorder.h SIFT_CGORD_MEM_MAX yes
gord_free_rows rows of the segments that keep no order array of a leaf (the overlays written lean) that a top-N, range or count by the array of the leaf decodes and sorts in memory for each scan 65,536 rows the scan reads the units, and the planner prices no walk of the array (the rows are the same) col/gorder.h SIFT_CGORD_FREE_MAX yes
merge_disk_peak disk the index needs while a merge or a rewrite runs: the output is written before the inputs are freed (measured 2.02, 1.86 and 2.07 times the live index), and the next VACUUM gives the end of the file back 2 times the live index the step fails with PostgreSQL’s disk-full error before it replaces a segment col/maint/maint.c
col_leaves leaf slots of a sift_col index and columns of a sift_table (127 are usable) 128 leaves CREATE INDEX, CREATE TABLE or ALTER TABLE rejected col/api/format.h SIFT_COL_MAX_COLS yes
string_bytes one string value of a column leaf 1,073,741,823 (~1024 MiB) bytes ERROR 54000 at build or merge col/api/format.h SIFT_COL_MAX_VALUE yes
unit_bytes string bytes of one leaf in one column unit 1,073,741,823 (~1024 MiB) bytes the values after it are stored as overflow and read from the heap (NOTICE) col/api/format.h SIFT_COL_UNIT_MAX_BYTES yes
unit_rows rows of one column unit (256 heap pages) 74,496 rows a new unit starts col/api/format.h SIFT_COL_UNIT_MAX_ROWS
dict_entry longest string that is a dictionary entry 255 bytes the unit stores its strings inline col/api/format.h SIFT_COL_DICT_MAX_ENTRY
dict_bytes one segment dictionary 67,108,864 (64 MiB) bytes the unit stores its strings inline col/api/format.h SIFT_COL_DICT_MAX_BYTES
par_build_rows rows a participant of a parallel CREATE INDEX of a sift_col index takes at the least (a worker is launched for each further 150,000 rows of the table) 150,000 rows fewer workers are launched, down to the build of the leader alone: one segment max_parallel_maintenance_workers, the parallel_workers option of the table col/build_par.c PAR_BUILD_ROWS yes
pre_census_units units of each segment of an index the planner counts the pre-aggregate sections of (spread over the segment) 16 units the share of the units that hold a section is that of the units counted; a segment of fewer units is counted whole col/am/cache.c yes
dict_cache_kb parsed segment dictionaries a backend keeps for its next query, with the lookup table and the sorted order built over them 65,536 (64 MiB) kB the least recently used unpinned dictionary is dropped; one over a half of it is kept for its scan only and searched entry by entry; a dictionary another backend parsed is copied from the shared ring (col/dictshm.c) and counts here too col/dictcache.c CDICT_CACHE_MAX_KB yes
dict_lookup_min strings a segment dictionary needs before it is written in blocks (and with the codes in byte order for a lookup structure), so that a fetch or a lookup reads it in place (column format minor 14) 1,024 strings the dictionary is written in the compact form and a fetch or a lookup loads it whole dict_lookup option col/seg_write/seg_write_int.h CDICT_LOOKUP_MIN yes
dict_tsig_min bytes of the strings a segment dictionary in blocks needs before it is written with a trigram signature of each block, once the dictionary of its leaf, scaled to the index, no longer fits the parsed ones a backend keeps (dict_cache_kb), so that a substring test reads the blocks that may hold the part and not all of them (column format minor 29) 1,024 (1 MiB) kB no signature: a substring test reads every block of the dictionary col/seg_write/dict.c CDICT_TSIG_MIN_KB yes
lst_min_units units a segment needs before it keeps its leaf statistics after its sections, so that a plan reads a few pages of it, not the zone arrays of every leaf 64 units no statistics block: the plan reads the zone arrays of the segment (a few pages) col/seg_write/seg_write_int.h LST_MIN_UNITS yes
mirror_bytes bytes of the header page of a segment that keep a copy of the small structures a plan and a first read of the segment open (the schema and the unit directory, the point directories, the order headers, the headers of the pre-aggregates), so that they cost no pages of their own 6,144 (6 KiB) bytes no copy: each of them is read from its own page of the segment col/seg_write/seg_write_int.h SEG_MIRROR_BYTES yes
lst_share_div the share of its segment a leaf statistics block may take: a segment whose block would take more than 1/n of its bytes (3% at 33; units of a few rows and many leaves) keeps none 33 1/n of a segment no statistics block: the plan reads the zone arrays of the segment col/seg_write/seg_write.c LST_SHARE_DIV yes
int_digits precision of a numeric the integer leaf class holds exactly 18 digits the wide leaf class holds it col/api/leaf.h SIFT_COL_MAX_SCALE
wide_digits precision of a numeric in a wide leaf 38 digits ERROR 54000 at build or merge col/wide.h SIFT_COL_MAX_WIDE_SCALE
cube_dims dimensions of the cube of an index 16 dimensions option rejected at CREATE or ALTER INDEX cube option col/preagg.h SIFT_PAGG_MAX_DIMS
rollup_dims dimensions of one rollup 6 dimensions option rejected at CREATE or ALTER INDEX rollup option col/preagg.h SIFT_PAGG_MAX_RDIMS
rollups rollups of one index 8 rollups option rejected at CREATE or ALTER INDEX rollup option col/preagg.h SIFT_PAGG_MAX_ROLLUPS
cube_measures measures of a cube or rollup 8 measures option rejected at CREATE or ALTER INDEX cube option col/preagg.h SIFT_PAGG_MAX_MEAS
cube_groups groups of a default cube dimension in one unit 256 groups the unit has no histogram for the dimension (it is scanned) col/preagg.h SIFT_PAGG_DEF_GROUPS
calendar_groups groups of a calendar dimension (days, else months, else years) in one unit 1,024 groups the unit tries the next grain, then has no histogram for the dimension (it is scanned) col/preagg.h SIFT_PAGG_CAL_GROUPS
explicit_groups groups of a dimension of an explicit cube(…) in one unit 4,096 groups the unit has no histogram for the dimension (it is scanned) cube option col/preagg.h SIFT_PAGG_CUBE_GROUPS
rollup_tuples tuples of an explicit rollup in one unit 4,096 tuples the unit has no rollup (it is scanned) rollup option col/preagg.h SIFT_PAGG_ROLLUP_TUPLES
set_keys distinct keys of a column of key_columns a unit keeps exactly in a summary, a bloom filter beyond 32 keys a filter instead of the keys col/setsum.h SIFT_CSET_LIMIT
set_keys_max distinct keys of a stored exact summary 1,024 keys the summary is a bloom filter col/setsum.h SIFT_CSET_LIMIT_MAX
presence_pct share of a unit’s bytes the key lists and bitmaps of all its key_columns may take together 3 percent a column past the share has no key list or bitmap in the unit (it is scanned) key_columns option col/setsum.h SIFT_CSET_PRES_PCT
super_fan_max largest value of the superunit option: units in one group summary 256 units option rejected at CREATE or ALTER INDEX superunit option col/preagg.h SIFT_SUPER_MAX_FAN
shred_spec_bytes size of a jsonb shredding spec 65,536 (64 KiB) bytes ERROR at CREATE INDEX col/shred.h SIFT_SHRED_MAX_SPEC_BYTES
shred_leaves leaves of one jsonb shredding spec 256 leaves ERROR at CREATE INDEX col/shred.h SIFT_SHRED_MAX_COLS
shred_groups arrays and wildcards of one jsonb shredding spec 64 groups ERROR at CREATE INDEX col/shred.h SIFT_SHRED_MAX_GROUPS
shred_steps steps of one jsonb path 16 steps ERROR at CREATE INDEX col/shred.h SIFT_SHRED_MAX_STEPS
shred_text one text value of a jsonb leaf 1,048,576 (1 MiB) bytes the value is OVERFLOW (read from the heap) col/shred.h SIFT_SHRED_MAX_TEXT
shred_key_bytes a key of a jsonb wildcard object 255 bytes the value is OVERFLOW (read from the heap) col/shred.h SIFT_SHRED_MAX_KEY_BYTES
shred_keys distinct keys of a jsonb wildcard object per segment 65,536 keys the value is OVERFLOW (read from the heap); CREATE INDEX says how many units hold one (NOTICE) col/shred.h SIFT_SHRED_DEFAULT_KEYS yes

Search index, storage and queries (src/core, src/bm25)

id Limit Default Unit At the limit Changed by Source Test
segment_cache_kb docmap directories and term blobs of the segments that a backend keeps, in its own memory, for its next statement (one entry at most a quarter of it) 32,768 (32 MiB) kB the least recently used entry that no cursor holds is dropped; a blob that does not fit while every entry is held is read as before and kept by nobody bm25/segcache.c SEGCACHE_MAX_KB yes
array_bytes the largest array a build, merge or rewrite allocates in proportion to the rows of a segment or a run (a sort copy, the entries of a point directory or an order array) 1,125,899,906,842,624 (1.04858e+06 GiB) bytes none outside a test: a segment is cut at its memory budget, which counts these arrays, and one over 1 GB is allocated huge (a test lowers the limit: ERROR 54000 naming the size) maintenance_work_mem core/limits.h SIFT_ARRAY_UNLIMITED yes
tomb_log_kb the tombstone log of a sift_table that one backend keeps in memory (64 bytes an entry: the sorted entries, the scratch of a first look and the chain members), the entries of its own statement included 8,388,608 (8 GiB) kB ERROR 54000 naming sift.tomb_log_limit and VACUUM, which is not bound by it and drops the entries of the rows it cleared; a worker VACUUM is asked for when the log reaches a quarter of it sift.tomb_log_limit core/tomb.h SIFT_TOMB_LOG_KB
max_segments directory entries of a sift or sift_col index (a sift_table has sift.table_max_segments for a test) 128 segments writers merge the smallest closed segments from 16 entries below it; a forced seal leaves the write buffer open (NOTICE) core/sift.h SIFT_MAX_SEGMENTS yes
term_bytes a BM25 token 255 bytes the token is its first 236 bytes and a hash of all of it core/sift.h SIFT_MAX_TERM_BYTES
term_tf term frequency in one document 255 occurrences saturated core/sift.h SIFT_MAX_TF
doc_length length of one document 65,535 tokens saturated core/sift.h SIFT_MAX_DL
fields fields of a BM25 index 14 fields CREATE INDEX rejected fields option core/sift.h SIFT_MAX_FIELDS
prefix_index prefix_index option 15 characters option rejected at CREATE INDEX prefix_index option core/sift.h SIFT_MAX_PREFIX_INDEX
write_buffer_kb default write buffer; a sift_table row that does not fit it is accepted with sift.table_wide_rows (on) 4,096 (4 MiB) kB ERROR 54000 for a larger sift_table row only when sift.table_wide_rows is off (on, the default: up to 1 GB) sift.table_wide_rows core/sift.h SIFT_DEFAULT_WRITE_BUFFER_LIMIT
query_bytes length of a search query 262,144 (256 KiB) bytes ERROR 54000 bm25/query.h SIFT_QUERY_MAX_BYTES yes
query_leaves terms of a search query 256 terms ERROR 54000 core/sift.h SIFT_MAX_LEAVES
query_depth nesting of a search query 32 levels ERROR 54000 core/sift.h SIFT_MAX_DEPTH
phrase_terms terms of one phrase 128 terms ERROR 54000 core/sift.h SIFT_MAX_PHRASE yes
boost combined boost of a query term 10,000 factor ERROR 54000 bm25/query.h SIFT_MAX_BOOST
regex_bytes a regular expression of a query 1,000 bytes ERROR 54000 bm25/query.h SIFT_MAX_REGEX_BYTES
any_list_rows rows a query list (= ANY(array)) may match in one scan 100,000 rows normal plan bm25/planner_path.c ANY_LIST_MAX_ROWS
maint_backoff_ms longest wait before a failing background maintenance retries 3,600,000 ms retried once an hour bm25/maint.c BACKOFF_MAX_MS
semi_set rows of a table whose joined keys a ranked scan tests the INCLUDE column against (the planner offers a table of at most this many rows, the scan stops at this many keys) 65,536 rows the join keeps every row the scan returns (the rows of the table are checked after it, as without INCLUDE columns) bm25/planner.h SIFT_SEMI_MAX_SET

sift_table (src/table)

id Limit Default Unit At the limit Changed by Source Test
wide_dict_bytes the distinct values of a wide column (a numeric past 18 digits, an unconstrained numeric, money, an interval) in the rows of one unit of a sift_table, as the scan keeps them to compare and group 1,073,741,823 (~1024 MiB) bytes ERROR 54000 naming the column table/wide.c yes
table_vblocks virtual blocks of TIDs of a sift_table, never reused (sift.table_space: vblocks_left) 4,294,967,294 blocks ERROR 54000, VACUUM FULL renumbers them table/wbuf.c