Contents

• Fuzzing and sanitizers
  • Server under ASan
  • Targets
  • Coverage-guided targets (test/fuzz)

Fuzzing and sanitizers

scripts/fuzz.sh [--scale N] [--seed N] [TARGET...] builds every fuzzer and randomized harness with AddressSanitizer and UndefinedBehaviorSanitizer and runs them. It prints PASS/FAIL per target and exits 1 on any finding.

Server under ASan

SANITIZE=1 scripts/pgtest.sh install    # make clean + build with -fsanitize=address,undefined
scripts/pgtest.sh stop; scripts/pgtest.sh start
scripts/fuzz.sh                         # all targets
scripts/pgtest.sh sanitizer-report      # exit 1 if the server logged a finding

The PostgreSQL server binary is not instrumented, so pgtest.sh start preloads libasan/libubsan (LD_PRELOAD). It sets ASAN_OPTIONS=detect_leaks=1:exitcode=0:abort_on_error=1: LeakSanitizer reports the malloc leaks of every server process as it exits, minus PostgreSQL’s own startup allocations (scripts/lsan.supp); palloc memory belongs to memory contexts and is not a malloc leak. The standalone harnesses (tools/analyze, tools/query, tools/codec) run with leak detection too; their palloc stubs free the arena at exit (tools/pgshim.c). Reports are written to $SIFT_TMP/sanitizer/. The canary target loads a deliberately overflowing module (asan_canary.c) and checks that a report appears, which shows the pipeline works. Plain scripts/pgtest.sh install switches back to the normal build.

ASan only sees malloc, stack and global memory. Overruns inside a palloc chunk stay invisible unless they reach the chunk header, which memory-context checks then catch in assert-enabled builds. The standalone harnesses below use malloc and get full coverage, so prefer them for pointer-heavy code.

Targets

Target	Owner	What runs
canary	tests	ASan pipeline self-test (server must be the ASan flavor)
codec	codec	tools/codec randomized property tests against sorted-array references, at every ISA level
blob	hardening	tools/codec/fuzz: bit-flipped, truncated and spliced term blobs; sift_blob_check must never read outside the blob, and every accepted blob must decode to the same strictly increasing tids on all ISAs, through the AND/OR/ANDNOT cursors, seeks and counts. fuzz -s SEED -i ITER replays one iteration
kernels	kernels	tools/kernels randomized and exhaustive equivalence of SIMD and scalar kernels (make SANITIZE=1)
analyze	analyze	tools/analyze/harness fuzz N SEED: random text through the tokenizer, checking its invariants
query	query	tools/query/gen.py --mode fuzz corpus through the standalone parser harness, then through the server (tools/query/fuzz.sql: fixpoint deparse, same matches)
colchunk	safety	test/fuzz/fz_colchunk.c: the column codec blobs, string and bit lists, unit chunks, unit records, segment dictionaries and schemas (7 modes in one input), at every SIMD level, levels must agree. Coverage guided, 6 s per --scale step
bm25	safety	test/fuzz/fz_bm25.c, mode 1: sift_blob_check then every cursor on every ISA; an accepted blob must stay inside itself
bm25-raw	safety	the same target on the query path: bytes nobody verified (sift_cur_blob, sift_post_open, tf, positions). Clean since 2.0.2; opt-in
shred	safety	test/fuzz/fz_shred.c: a path spec and a JSON text through the shredder’s event interface, the produced document checked (ranges, states, parents at every level), spec deparse a fixed point
sql	tests	test/fuzz/fuzz.sql in the server: hostile SiftQL to sift.parse_query (failures other than 42601/54000/2201B; disagreements with the SQL reference parser are reported, not fatal) and hostile text to sift.tokenize (invariants)

--scale multiplies the iteration counts of every target (default 10, about 3.5 minutes; codec runs 300 iterations and kernels 0.2 million per unit). --budget SEC runs scale-1 passes (about 25 s each) until SEC seconds have passed, the seeded targets with a new seed each pass; scripts/ci.sh quick uses a 20 s budget. Run scripts/fuzz.sh with no arguments, at nice 19, for long sessions. Every input of the sql target is logged as sift_fuzz input before use, so after a backend crash the script prints the input that killed it.

Coverage-guided targets (test/fuzz)

colchunk, bm25 and shred link the decoders without a backend: tools/pgshim.c and fz_shim.c stand in for the allocator (every palloc is its own malloc, so ASan sees an overrun of a palloc’d object; MaxAllocSize is enforced), for ereport(ERROR) (a longjmp to fz_try; the answer a decoder may give to corrupt bytes is DATA_CORRUPTED, INDEX_CORRUPTED, FEATURE_NOT_SUPPORTED or a limit, any other SQLSTATE is reported once as FINDING-LOW) and for the stream reader. They build with clang as libFuzzer binaries, with gcc (or FZ_DRIVER=1) against fz_driver.c, a coverage-guided mutation driver with the same command line. Always ASan and UBSan with -fno-sanitize-recover=all and without -fwrapv: signed overflow is a finding here.

nice make -j4 -C test/fuzz OUT=/tmp/x all                   # colchunk, bm25, shred
FZ_GEN=/tmp/seeds /tmp/x/colchunk                           # the seed corpus from the writers, then exit
/tmp/x/colchunk -max_total_time=60 -timeout=10 -rss_limit_mb=4096 -artifact_prefix=/tmp/art- /tmp/corpus
/tmp/x/colchunk /tmp/art-crash-...                          # replay
nice make -j4 -C test/fuzz OUT=/tmp/x regress               # replay regress/: the crashes of the fixed findings
nice /tmp/x/bm25 -fork=3 -ignore_crashes=1 -max_total_time=60 /tmp/corpus   # many findings in one run (libFuzzer)

Environment: FZ_MODE=n forces the mode of an input (the first byte otherwise), FZ_SKIP=a,b skips checks whose finding is recorded (none now; the name is the argument of fz_skip), FZ_NOERR=1 makes any raised error fatal (a seed that does not parse is a bug of the seed), FZ_STRICT=1 makes a wrong SQLSTATE fatal, FZ_TRACE=1 prints the digest after each step (to see which operation two SIMD levels disagree on), UBSAN_OPTIONS=suppressions=test/fuzz/ubsan.supp skips the UB findings already recorded. scripts/fuzz.sh sets the skips and suppressions of the findings that are not fixed yet; delete an entry when its fix lands.

Adding a fuzzer: give it a standalone harness under tools/<module>/ whose Makefile accepts SANITIZE=1 (or always sanitizes), add a function named after the module to scripts/fuzz.sh, and add a row here. libFuzzer entry points (LLVMFuzzerTestOneInput) need clang. gcc is the only compiler on this host, so the harnesses of tools/ drive their own generators; the targets of test/fuzz use libFuzzer where clang has it and their own driver else.