Benchmarks

Harnesses, configs, dataset scripts, the fake Pinecone and the write-ups behind pg_pinecone’s performance numbers. Nothing here is built or run by CI: .github/workflows/ci.yml formats and lints the Rust crates and runs the Rust tests only.

dir measures
vector/ Filtered and unfiltered ANN search: pgvector hnsw / ivfflat / halfvec against pg_pinecone’s native local engine and the remote engine; residual WHERE filters and the iterative scan, on synthetic data, SIFT1M, wiki_dpr 768d and Cohere wikipedia-22-12 768d

Common setup

  • State, downloads and scratch clusters live in bench/.data/ (gitignored). Move them with PG_PINECONE_BENCH_DATA=/path.
  • The Postgres under test is PG_PINECONE_BENCH_PG (an install prefix with bin/pg_ctl). vector/ defaults to a pgrx-managed PG 17 (~/.pgrx/17.10/pgrx-install) that has pgvector. pgrx’s own PG is a --enable-cassert build: fine for plumbing and recall, not for timing.
  • Build pg_pinecone into a private pgrx home (PGRX_HOME), never into one other worktrees install into.
  • The smoke below needs no Pinecone key and runs with env -u PINECONE_API_KEY. The remote configs read a key from a file and spend real money (see vector/box/README.md).
  • The Python venv is vector/.venv (verified with Python 3.14). Paths must not contain spaces.

Smoke command (verified 2026-10-12; sizes and timings in vector/README.md)

From bench/vector (PG_PINECONE_BENCH_PG needs pgvector and pg_pinecone installed: see “Install pg_pinecone” in vector/README.md):

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
export PG_PINECONE_BENCH_DATA="$(cd .. && pwd)/.data" PG_PINECONE_BENCH_PG=/path/to/postgres-install-prefix
scripts/download-datasets.sh sift                # 168 MB; `wikidpr` is 5.3 GB
env -u PINECONE_API_KEY .venv/bin/python run.py --config configs/smoke-native.yaml \
    --data-dir "$PG_PINECONE_BENCH_DATA/vector/pgdata-smoke" -v      # 50k x 768d, 411 s

Results land in vector/results/ (one JSON and one markdown file per run), gitignored. The committed write-ups are the records of past runs:

  • vector/REPORT-residual-filter.md: iterative scan vs pgvector, 1M x 768d, 2026-10-07.
  • vector/REPORT-halfvec.md: pgvector halfvec vs FP32 hnsw and native, 1M x 768d, 2026-10-16.
  • vector/REPORT-halfvec-filtered.md: the same on 1% and 10% filters, 0.10.0, 2026-10-22.
  • vector/REPORT-overram.md: search under a memory cap, 2026-10-22.

Not reproducible from this tree

  • Anything against real Pinecone needs a paid Pinecone project (and, for the box recipe, AWS): the remote_* configs and their phases in vector/box/phase.sh.
  • Absolute numbers. Reports quote a laptop (M4 Max, 36 GB, other apps running) or an AWS box; compare rows inside a table.