mentat scale benchmark: final

Environment (abridged; full detail in env.txt)

instance_type:   c7i.8xlarge
kernel:          6.12.103-129.197.amzn2023.x86_64
cpu_model:       Intel(R) Xeon(R) Platinum 8488C
cpus:            32
sockets/numa:    1 1 
mem_total_gib:   61.8
hugepages:       HugePages_Total:=0 HugePages_Free:=0 Hugepagesize:=2048 
data_fs:         /dev/nvme0n1p1 xfs   150G   47G  104G  31% /
mentat_git:      final
duckdb:          1.5.5
postgres:        postgres (PostgreSQL) 16.15

Dataset and PG size vs shared_buffers

s: 984803 datoms (1600 users, 130000 issues, 6492 history updates)
m: 9850258 datoms (16000 users, 1300000 issues, 64986 history updates)

bulk_load

| backend | scale | datoms | wall s | datoms/s | ANALYZE s | VACUUM s | store size | note | |—|—|—:|—:|—:|—:|—:|—:|—|

point_lookup

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
pg s 984,803 1 read 33687 0.191 0.649 0.743 3.265 3,369 0 3
pg m 9,850,258 1 read 33550 0.191 0.650 0.743 3.236 3,355 0 3

ref_traversal

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
pg s 984,803 1 read 10289 0.915 1.254 1.365 5.449 1,029 0 3
pg m 9,850,258 1 read 8515 1.131 1.459 1.554 6.157 851.5 0 3

aggregate

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
pg s 984,803 1 read 1088 9.172 9.708 10.090 12.763 108.8 0 3
pg m 9,850,258 1 read 173 57.659 59.074 59.918 64.397 17.3 0 3

predicate_scan

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
pg s 984,803 1 read 205 48.908 50.596 51.199 57.870 20.5 0 3
pg m 9,850,258 1 read 99 612.583 627.577 682.471 682.471 1.6 0 3

pull

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
pg s 984,803 1 read 11592 0.845 0.963 1.027 4.137 1,159 0 3
pg m 9,850,258 1 read 11177 0.876 0.990 1.052 4.237 1,118 0 3

as_of

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
pg s 984,803 1 read 5977 1.643 1.991 2.142 10.047 597.7 0 3
pg m 9,850,258 1 read 5021 1.962 2.339 2.541 10.781 502.1 0 3

since

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
pg s 984,803 1 read 5243 1.890 2.004 2.097 4.442 524.3 0 3
pg m 9,850,258 1 read 3271 3.047 3.149 3.278 5.949 327.1 0 3

input_bindings

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
pg s 984,803 1 read 11411 0.814 1.160 1.255 4.063 1,141 0 3
pg m 9,850,258 1 read 10115 0.922 1.254 1.346 4.770 1,012 0 3

concurrency_sweep

backend scale datoms clients op n p50 ms p95 ms p99 ms max ms ops/s errors reps
pg s 984,803 1 read 12776 0.793 1.242 1.368 3.401 1,278 0 3
pg s 984,803 8 read 102347 0.823 1.239 1.381 6.487 10,235 0 3
pg s 984,803 32 read 325690 1.113 1.495 1.582 14.812 32,569 0 3
pg m 9,850,258 1 read 12625 0.867 1.215 1.317 3.276 1,262 0 3
pg m 9,850,258 8 read 90670 0.906 1.453 1.592 7.307 9,067 0 3
pg m 9,850,258 32 read 307182 1.113 1.700 1.810 13.666 30,718 0 3

Correctness checks

36 PASS, 0 FAIL, 0 SKIP; 2 backend check runs OK, 0 FAILED (all FAIL/FAILED lines, with annotations, in checks.txt)