pgContext vs. pgvector vs. Qdrant benchmark harness
This harness generates real 384-dimensional MiniLM embeddings from SciFact and compares pgContext 0.1.0 with pgvector 0.8.5 in separate databases on one PostgreSQL 17 server and Qdrant 1.18.2 over local gRPC.
See the methodology, results, caveats, and reproduction guide.
docker run --rm --name pgcontext-bench-qdrant \
-p 6333:6333 -p 6334:6334 qdrant/qdrant:v1.18.2
benchmarks/pgvector_comparison/run.sh test
benchmarks/pgvector_comparison/run.sh prepare
PGCONTEXT_HNSW_EF_SEARCH=48 \
benchmarks/pgvector_comparison/run.sh run --trials 3
Additional lanes:
# Latency-vs-recall Pareto curves per system (writes sweep.json).
benchmarks/pgvector_comparison/run.sh sweep --ef-values 16,24,32,48,64,96
# Filtered ANN at 1%, 10%, and 50% selectivity (writes filtered-sweep.json).
benchmarks/pgvector_comparison/run.sh filtered-sweep
# Multi-client throughput and per-backend resident memory
# (writes concurrency.json).
benchmarks/pgvector_comparison/run.sh concurrency --workers 1,8,32
# Seeded synthetic corpus for scale lanes beyond SciFact.
benchmarks/pgvector_comparison/run.sh prepare --synthetic 100000 \
--output-dir target/pgvector-comparison-100k
# Update-churn stability: rewrite N% of rows per round, VACUUM, re-measure
# (writes churn.json; PostgreSQL systems only).
benchmarks/pgvector_comparison/run.sh churn --rounds 5 --churn-percent 5
# Cold-cache: restart PostgreSQL and record first-query latencies
# (writes cold-cache.json; PostgreSQL systems only).
PGCONTEXT_BENCH_RESTART_CMD="pg_ctl -D /opt/homebrew/var/postgresql@17 restart -w" \
benchmarks/pgvector_comparison/run.sh cold-cache
The harness uses fixed database and collection names, so never run two invocations against the same server concurrently — a second run drops and recreates the databases the first one is still using.
Generated datasets and full results are written under
target/pgvector-comparison/. Compact historical results live in results/.