Cohere vector CI
Cohere restores its existing pgBackRest heap snapshots into a source-built
ParadeDB Docker image. paradedb/pdbench selects recall operating points;
paradedb/benchmarker measures search latency and throughput.
The workflow runs the 1M and 10M snapshots independently. Each uses the existing
100 held-out vectors, exact ground-truth fixtures, and unfiltered, approximately
10% (year), and approximately 1% (battle) full-text filters. The index retains
eight segments, a 0.32 training sample ratio, and English stemming/stopwords on
text. Cohere, MS MARCO, HN, Wikipedia, and Stack Exchange share
setup-docker-benchmark-cluster. Cohere, MS MARCO, and HN start the paradedb
service from pdbench’s datasets/<dataset>/docker-compose.yml, which is the
overridden in CI for these datasets: 8 CPUs on cores 0–7, 32 GiB memory, 16 GiB shared memory, 8 GiB
shared buffers, 8 GiB maintenance memory, a 24 GiB effective cache, and eight
parallel workers in a 32-process pool. Index builds can use seven parallel maintenance workers;
queries can use four. VectorChord retains eight internal build threads. k6 runs on CPUs 64–67. Wikipedia and
Stack Exchange keep the Compose file on S3. Both tools use the same database
and index throughout a run.
The RunsOn host remains unchanged; the CPU and memory limits apply to the
database container. Result dataset and artifact names include -8cpu-32gb,
keeping both recall/router baselines and published metrics separate from the
previous 64-CPU runs. Run diagnostics include runtime-compose.yml with the
resolved container configuration.
The benchmark index enables vector_router_quantization = true in its
CREATE INDEX ... WITH (...) clause. This boolean index option defaults to false
and is stored in the index definition, so parallel builds, REINDEX, and
VACUUM FULL use the same setting without session configuration. Document-vector
quantization is independent. To change the router for an existing index, use
ALTER INDEX ... SET (vector_router_quantization = true) followed by REINDEX;
changing the option alone does not rewrite existing router data.
Router code and sidecar columns are file-backed: open reads only headers and list topology, and scoring fetches the row ranges for the lists actually probed. This avoids decoding the entire router during query preparation. Payload validation occurs when each batch is read; the persisted format is unchanged.
With paradedb.vector_stats enabled, per-segment
EXPLAIN (ANALYZE, VERBOSE, FORMAT JSON) reports:
routing_kindandrouting_lists_scannedidentify the router and its work.routing_approximate_scoredcounts one-bit scores across all router levels.routing_approximate_members_scoredcounts bottom-router member scores only.routing_approximate_prunedcounts scored members excluded from the padded shortlist, before any raw vector is read for reranking.routing_exact_scored,routing_exact_pruned, androuting_candidate_countcount reranked members, members rejected by reranking, and returned candidates.
For finite queries, approximate members scored equals approximate members pruned plus exact scores; exact scores equals exact pruned plus candidates. Unopened lists and members skipped by APS are excluded from the pruning count. Non-finite queries fall back to exact scoring and report zero approximate work. Flat and located scans do not invoke a router and omit these counters.
Router search policies can be adjusted at query time without rebuilding:
| Setting | Default | Effect |
|---|---|---|
paradedb.vector_router_rerank_factor |
2.0 | Fractional candidate multiplier before exact reranking |
paradedb.vector_router_min_beam_fraction |
0.5 | Minimum list fraction at each quantized router level |
paradedb.vector_router_aps |
on | Stop hierarchical routing after quiet lists |
paradedb.vector_router_audit |
off | Diagnostic: stored or flat routing with every segment routed |
The rerank factor accepts fractional values from 1.0 through 16.0. Retention
rounds up to whole candidates: for example, 1.25 retains up to four candidates
for router_k = 3. A factor of 1.0 removes padding but still computes exact
scores, which document pruning and quantized distance estimation require.
The beam setting leaves full-precision-only routing unchanged. Quantized routing
defaults to a 50% beam floor with 2 * router_k retained members. Global routing
shares this beam and the padded shortlist across segments, including mixed raw
and quantized routers and coarse partition centroids. It reranks retained
quantized members before the global router_k selection.
Wider selection compensates for errors in quantized centroid ordering;
retaining more members cannot recover members in unopened lists.
- Restore the existing Cohere heap snapshot, refresh the installed pg_search
extension, and run
pdbench-load finalizewith pdbench’sconfigs/load/cohere-<size>-paradedb.yaml. The index definition, its options, and the Postgres settings it is built under come from that config and pdbench’sdatasets/cohere/. Build time is the whole finalize run: dropping any old index, the build, andVACUUM ANALYZE. Snapshot generation continues to usedatasets/cohere/config.tomlandcreate_tables.sql. - Import the published ground-truth Parquets into pdbench’s fixture table.
pdbench searchloads the query vectors and sweepsparadedb.vector_cluster_max_probein absolute cluster-work units separately for each filter to r90/r95/r99. The sweep starts at 100 units and searches between 1 and 100,000 units, using only whole-number budgets. Its required positive tolerance is 0.000001, below one neighbor’s contribution to recall over 100 top-10 queries (0.001). After the sweeps, CI downloads diagnostics from the latest successful main push preceding this workflow run with an unexpired artifact for that corpus size.pdbench compare --htmlcompares each query at matched recall and saves offline HTML reports, text deltas, labeled JSON inputs, and commit provenance undercomparison/in the diagnostics artifact. The job summary identifies the baseline run and report filenames. The PR comment’s Reports section links the sweep comparisons under Sweep and the existing k6 graphs under Search. Missing baselines or queries and differing config fingerprints are reported as skipped comparisons; CI does not force comparisons across benchmark configurations. Main runs compare with the previous successful main run. This reuses recorded mainline sweeps rather than rebuilding and measuring main on the current runner. prepare_k6.pyexports pdbench’s selected points, SQL, and session settings. It rejects selections below their target and does not select probes or calculate latency. Unreached targets remain visible and do not run in k6.- k6 cycles over the same 100 vectors with one VU: 10 seconds of discarded warmup and 50 seconds of measurement per filter/target. The vector is bound as a parameter, with custom plans and no prepared-statement cache. The selected max_probe and all fixed search settings apply to every connection. CI verifies the parameterized plans and filter pushdown before measuring.
- Results and recall context enter the shared publication queue. The Wikipedia/StackExchange publisher posts p50/p95/p99, QPS, index build time, and index size with baseline comparisons and dashboard links. The comment shows the rows above 1.15x and collapses the operating points (recall, max_probe, and mean whole-query shared buffer hits/reads), the full results table, and the report links. Buffer diagnostics come from pdbench’s separate EXPLAIN pass over all 100 vectors.
The workflow runs daily on main and via benchmarks, benchmark-cohere-pg_search,
benchmark-benchmarker-cohere, or benchmark-cohere-router-quality; only the
daily run and benchmark-cohere-router-quality add the router-quality section.
Manual runs can select a size, pg_search commit, and tool refs, and add the
router-quality section with router_quality. Manual runs also accept
engine=pg_vectorsearch, vectorchord, or vectorchord-prefilter with size=10m
and router_quality=false. runner_type=on-demand selects an on-demand runner;
benchmark-nonspot controls PR-triggered pg_search runs.
The pdbench checkout uses the ParadeDB GitHub App’s read access to the private
repository. Both tool revisions are pinned and recorded in diagnostics. The
fixture import only populates pdbench’s ground-truth table; the corpus is restored
from its snapshot.
Results begin new benchmarker cohere-1m and benchmarker cohere-10m histories.
The old server-time samples are not comparable to k6’s client-observed latency
and throughput. Index build time (minutes) and size (MiB) have a separate
benchmarker cohere-<size> (index build) history. Its first run has no previous
baseline; subsequent PR runs compare with main and do not update its baseline.
The cohere-<size>-load-report artifact uploads immediately after the index build,
before recall sweeps and k6. Download it from the running workflow’s Artifacts
section (or the Upload Load Report step’s artifact URL) and open report.html.
The remaining diagnostics upload after plan verification, before the longer k6
pass, so recall sweeps can be reviewed while latency measurements continue.
Artifacts retain index-build logs, the pdbench-load run and its report.html, pdbench search reports (including
all sweep points, recall and buffer diagnostics), selected operating points,
query vectors, checked plans, and k6 JSON/HTML dashboards. The full workload is opt-in
on PRs; the small handoff tests can be run with:
python3 -m unittest discover -s benchmarks/cohere -p 'test_*.py'
Cohere 10M comparison engines
The comparison jobs install the pinned extension into the same paradedb
service and restore the same Cohere 10M heap as pg_search. Docker inspection
asserts the CPU limit, CPU affinity, RAM, and shared-memory size. Both the build
and recall profiles assert the common PostgreSQL settings before doing work.
The comparison pdbench pin includes VectorChord support and the same Compose
settings as the pg_search pin.
10m-vectorchord.toml and 10m-vectorchord-prefilter.toml use one global cosine
index with 16,000 lists, residual quantization, spherical centroids, eight build
threads, sampling_factor=64, hierarchical k-means, and clustering dimension 100. The list count targets roughly 625 vectors per partition at 10M rows;
the sample cap is 1,024,000 vectors. These are initial settings for the 32 GiB
budget, not a claim of optimal performance. Hierarchical clustering, reduced
sampling, and reduced clustering dimension follow the
VectorChord partitioning guidance.
Build and search use the same profile, and the two profiles differ only in
vchordrq.prefilter. Probes sweep from 1 through all 16,000 lists with
epsilon=1.9 and no scan-tuple cap.
pg_vectorsearch uses automatic nlist, default replication, and integer
prism.nprobe from 1 through 4096. Both engines reuse pdbench’s existing Cohere
comparison SQL: cosine ordering and PostgreSQL English full-text matching on
text for year and battle, with no scalar filter index. They are evaluated
against the same published ground truth as pg_search, without changing the
strict r90/r95/r99 gates. Pilot sweeps locate candidates; only operating points
validated against all 100 vectors may enter k6. Missed targets and exhausted
budgets remain visible. k6 retains one VU, 10 seconds of warmup, and 50 seconds
of measurement. Executed plans must use the selected ANN index and preserve
both text predicates; VectorChord’s prefilter setting is checked in the same
session as each plan.
Results use separate cohere-10m-pg_vectorsearch, cohere-10m-vectorchord, and
cohere-10m-vectorchord-prefilter identities. Diagnostics include build/search
profiles, tool and extension commits, container resources, executed plans,
recall outcomes, PostgreSQL logs, and k6 dashboards. The existing pg_search
nightly and router-quality paths retain their own profiles and tool pin.
Router quality
On the daily main run, with the benchmark-cohere-router-quality label, and on
manual runs with router_quality, CI compares the index’s stored router with a
flat ranking of every centroid on the same index after k6. Other labels skip it. paradedb.vector_router_audit selects the arm:
stored routes every segment through its stored router, and flat scores every
centroid of every segment exactly from the raw centroid rows. Both disable the
located path, so every segment routes, and both keep the caller’s router_k,
budget, and charging. The scan bounds both rankings the same way: a parallel
plan keeps the best router_k candidates, and a serial plan takes the whole
ranking. The setting is a diagnostic and is not documented for users.
router_quality.py configs writes three arm profiles from the size’s profile,
each adding only the audit setting and paradedb.vector_adaptive_probing:
flat and stored with adaptive probing off, and stored-adaptive with it on.
router_quality.py sweep finds each arm’s budget at r90/r95/r99 with
pdbench search (no HTML, no prewarm), each run in its own directory under
benchmark-artifacts/router/<arm>/<query>/ so no run reuses another’s cached
sweep values. Per query, the stored arm runs first:
- On the unfiltered query it is seeded from the budgets the existing sweep selected. That sweep is the stored router with adaptive probing on, and with it off the router probes a superset of the clusters at the same budget, so each target’s range ends at the seed and starts one below it.
- On the filtered queries the existing sweep may take the located path, so the stored arm sweeps the profile’s full range.
Flat and stored-adaptive are then seeded from the budgets the stored arm
selected, one pdbench run per target. stored-adaptive needs at least the
stored budget, so its range starts at the seed with the minimum one below it;
flat’s range spans half to twice the seed. The span is a policy. A target a
seeded range does not bracket (no exact hit and no probe below the selection,
unless it sits at the profile’s minimum) is swept again over the full range.
On the unfiltered query the existing sweep’s report stands in for
stored-adaptive when the query has no filter, every segment routed at every
point, and its configuration matches the stored arm’s beyond the audit and
adaptive settings; otherwise the arm is swept like the filtered queries.
pdbench saves each run’s report before building its console projection, which
can fail on a parallel plan with “representative layer0 has no scan stage”. The
sweep keeps such a run when it saved its report, notes it in the job summary and
the full report, and fails on any other error. sweeps.json records every run
(kind, seed, range, probes, seconds) and the point behind each selected budget.
pdbench stops at the first probe whose recall equals the target, so a seeded sweep can select another budget on a recall plateau than a full-range sweep, with the same recall.
router_quality.py measure reruns all 100 vectors at each selected budget under
EXPLAIN (ANALYZE, VERBOSE, FORMAT JSON) with paradedb.vector_stats on, after
one discarded warm pass, and sums each query’s per-segment statistics. It checks
that each statistic means what the report says: every segment routes, only the
flat arm reports routing_mode = flat, filtered queries satisfy
vectors_visited = layer0_eligible + pruned_filter + pruned_dead, fixed-probe
arms never stop adaptively, and each query runs the same plan shape (parallel or
serial) in every arm. Segment count and K come from every segment id the stored
arm reports.
router_quality.py render writes:
router_quality.md, the PR-comment section under the operating-point table: a verdict line, then one table, one footnote paragraph and a link to the full report, collapsed under “Router quality table”. The table has a row per filter level and target (Unfiltered, Filter 10%, Filter 1%, each at r90, r95, r99) and these columns: Flat and Router clusters probed and their Ratio (Router / Flat); Flat and Router vectors and their Ratio, where vectors are vectors probed on unfiltered rows and filter-valid vectors on filtered rows; Adaptive on vs off,stored-adaptiveover the stored arm on the row’s vectors; and Routing ms, the stored arm’s routing time with its share of query time in brackets. Flat is the flat arm and Router the stored arm, both with adaptive probing off. A target an arm did not reach printsnot reachedin that arm’s cells and leaves the ratios that depend on it empty.router_quality.html, the full report: every arm’s clusters, vectors probed, valid vectors, empty clusters, layer-0 scored, exact rows, routing time, centroids scored and adaptive stops per query and target, this commit’s values beside main’s, the checks, every sweep run with its seed, range and probes (including full-range reruns), tolerated runs, plan shapes and config. CI copies it into the sweep reports directory, so the publisher puts it on gh-pages underbenchmarker-runs/<commit>/cohere-<size>/sweep/router_quality.html.router-metrics.jsonandrouter-detail-metrics.json, unfiltered values per target for the benchmark history. The publisher stores them as two suites:benchmarker cohere-<size> (router)holds the clusters Ratio and Adaptive on vs off on vectors probed, and alerts like the other suites (115%, commenting only on runs without a PR);benchmarker cohere-<size> (router detail)holds Flat vectors probed, adaptive-on vectors probed and Router routing time, without alerts. Neither adds rows to the PR comment’s results table.router_quality.json, every number above with the commit and config fingerprint; later runs read the latest successful daily main run’s file as the baseline.
Only Unfiltered rows flag. Either Ratio above 1.15 turns bold with ⚠️ at r90, r95 and r99. With a main baseline it must also exceed main’s by more than 0.15, so a router already above 1.15 on main does not flag every run. Against main, at r90 and r95, a cell turns bold with ⚠️ and shows main’s value when Flat clusters or Flat vectors rose by more than 10%, when Adaptive on vs off rose by more than 0.05, or when Routing ms rose by more than 25% and 0.1 ms. Main’s values appear in the comment only in flagged cells. These are policy thresholds for review, not derived bounds.
The verdict names the against-main cell with the largest relative rise, followed by where to look (router for a Ratio, clustering for Flat, adaptive stopping for Adaptive on vs off, router speed for Routing ms); otherwise the worst flagged Ratio as the factor Router needs over flat (clusters when the clusters Ratio flags, otherwise vectors). With nothing flagged it gives the worst Ratio, or, when a Ratio sits above 1.15 within 0.15 of main’s, that Ratio beside main’s. Filtered rows are informational: every segment is forced to route, their plans are parallel, and routing time includes row gating. Nothing here fails the job.