# Vector engine benchmark: eshoponweb

Run 2026-10-07T22:14:29Z (run-all). Command line:

```
~/gvb-work/v8-run/bin/GenericVectorBuilder.Bench.dll run-all --pipeline eshoponweb --queries golden --seed 802 --out ~/ForClaude/GenericVectorBuilder/bench-results --repo ~/ForClaude/GenericVectorBuilder
```

- Machine: bench-host, Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz (8 logical CPUs), 62.7 GiB RAM, GPU NVIDIA GeForce RTX 3060, 12288 MiB, Ubuntu 24.04.5 LTS, kernel 6.8.0-142-generic, .NET 10.0.12, load average at start 2.66 3.30 3.46
- Data: 524 vectors x 1024 dims, collection `gvbbench_eshoponweb`. Read READ-ONLY from GenericVectorBuilder.dbo.gvb_eshoponweb in ChunkId order, vectors as native SqlVector<float>, 0.2 s.
- Queries: golden: 20 labelled questions from ~/ForClaude/evalkit/questions_golden.json, embedded with qwen3-emb-0.6b (cached in ~/gvb-data/bench-cache/golden-eshoponweb-68df42ca69efa079.json, no embedding calls). Top 10. Throughput at concurrency 1, 8 for 20 s each.
- Request speed on a small collection (524 vectors): Measured end to end through each engine's .NET client; at this size it reflects per-request cost including the client library, not index scaling.
- Client CPU per search is the CPU time the test's .NET client itself used for each search, measured in the same pass as the figure beside it, summed over every thread, so it can exceed the time per search. For an embedded engine (DuckDB, sqlite-vec) the engine runs inside the client process, so its figure includes the engine's own CPU time.
- Ground truth: brute force over every vector in memory, 0.0 s for all 20 queries.
- nDCG@10 of the exact answer itself (the ceiling for this embedder): 0.518

| engine | index | load rows/s | p50 ms | p95 ms | p99 ms | QPS@1 | QPS@8 | client CPU ms/search@1 | client CPU ms/search@8 | recall@10 | nDCG@10 | RAM | disk |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sql-diskann | DiskANN (preview) via VECTOR_SEARCH, cosine, build {"StartId":"306", "L":"48", "M":"8", "R":"48"}; the exact mode scans the same table | 532 | 3.56 | 4.00 | 5.00 | 277.0 | 668.4 | 1.01 | 1.08 | 0.965 | 0.526 | 530.3 MiB | 4.52 MiB |
| oracle | HNSW in-memory neighbor graph NEIGHBORS=16 EFCONSTRUCTION=128, EFSEARCH=100 per query, cosine; exact mode = FETCH EXACT FIRST (full scan); Oracle Free caps itself at 2 CPUs (cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit) | 1,701 | 0.94 | 1.18 | 1.82 | 1028.7 | 2820.4 | 0.89 | 0.65 | 1.000 | 0.518 | 2.22 GiB | 0 B |
| qdrant | exact scan: the builder's sink sends exact=true on every search, so no HNSW graph is used whether or not Qdrant has built one (see the index state) | 4,229 | 0.89 | 1.00 | 1.41 | 1113.5 | 3293.3 | 0.74 | 0.54 | 1.000 | 0.518 | 37.95 MiB | 196.08 MiB |
| elasticsearch | HNSW float32, no quantization, m=16, ef_construction=128, cosine; search k=top, num_candidates=100; 1 shard, 0 replicas; force-merged to one segment after the load (at 1,024 dimensions a segment under 1,043 vectors gets no graph) | 463 | 1.31 | 1.53 | 1.76 | 747.5 | 2731.9 | 0.46 | 0.50 | 1.000 | 0.518 | 2.54 GiB | 2.39 MiB |
| weaviate | HNSW maxConnections(M)=16 efConstruction=128, ef=-1 (dynamic: limit x 8 clamped 100..500), cosine, no quantization; approximate only (no exact mode) | 979 | 5.31 | 10.7 | 11.6 | 167.4 | 306.3 | 0.56 | 0.61 | 1.000 | 0.518 | 83.91 MiB | 2.96 MiB |
| pgvector | HNSW vector_cosine_ops m=16 ef_construction=128, hnsw.ef_search=100 per query, float32 vector(n), cosine; exact mode = same query with index scans off (sequential scan) | 594 | 0.87 | 1.03 | 1.31 | 1126.7 | 4006.3 | 0.69 | 0.54 | 1.000 | 0.518 | 100.4 MiB | 7.57 MiB |
| opensearch | faiss HNSW float32, no compression, m=16, ef_construction=128, cosinesimil; search k=top, ef_search=100; 1 shard, 0 replicas; graph built at any segment size (approximate_threshold=0); force-merged to one segment after the load | 384 | 2.06 | 2.28 | 2.83 | 479.9 | 1462.2 | 0.47 | 0.51 | 1.000 | 0.518 | 2.56 GiB | 9.57 MiB |
| sql | exact VECTOR_DISTANCE cosine, no vector index (full scan) | 605 | 3.93 | 4.43 | 4.94 | 249.7 | 606.5 | 1.21 | 1.19 | 1.000 | 0.518 | 531.1 MiB | 5.15 MiB |
| clickhouse | vector_similarity HNSW cosineDistance, quantization bf16, M=16 ef_construction=128, hnsw_candidate_list_size_for_search=256, rescoring off; exact mode = full scan with skip indexes off | 2,735 | 4.87 | 7.46 | 8.10 | 193.9 | 380.3 | 0.86 | 1.04 | 0.995 | 0.528 | 797.7 MiB | 4.18 GiB |
| typesense | HNSW float32 (hnswlib), m=16, ef_construction=128, cosine; search k=top, ef=100; exact mode = filter ordinal:>=0 with flat_search_cutoff; index held in memory | 1,006 | 4.16 | 4.50 | 5.01 | 237.4 | 576.3 | 0.52 | 0.58 | 1.000 | 0.518 | 118.1 MiB | 8.1 MiB |
| chroma | HNSW M=16 ef_construction=128, ef_search=100 (Chroma default), cosine; approximate only (no exact mode) | 878 | 2.12 | 2.38 | 2.62 | 467.6 | 778.5 | 0.48 | 0.47 | 1.000 | 0.518 | 49 MiB | 414.16 KiB |
| milvus | HNSW M=16 efConstruction=128, ef=100, metric COSINE, Strong consistency searches; approximate only (no exact mode) | 1,182 | 2.23 | 2.75 | 3.67 | 427.1 | 1274.0 | 0.55 | 0.63 | 1.000 | 0.518 | 190.9 MiB | 10.86 MiB |
| mariadb | VECTOR INDEX (HNSW variant) DISTANCE=cosine, M=16 (no ef_construction setting exists), mhnsw_ef_search=100 per statement (the ef 100 most engines here use, so the search effort matches; MariaDB's own default is 20; recall@10 at ef 100 falls as the set grows (random 1024-dimension vectors, measured 2026-10-04: 0.99 at 524, 0.89 to 0.92 at 2,000; an earlier run gave about 0.09 at 100,000)), mhnsw_max_cache_size 4G; exact mode = IGNORE INDEX full scan | 1,006 | 0.63 | 0.71 | 0.89 | 1563.2 | 5926.2 | 0.48 | 0.37 | 1.000 | 0.518 | 164.9 MiB | 22.01 MiB |
| duckdb | HNSW (vss extension) FLOAT[n] metric=cosine m=16 ef_construction=128, ef_search=100 per connection, persistent (hnsw_enable_experimental_persistence=true, checkpoint_threshold=256MB); exact mode = array_cosine_similarity sequential scan; score = 1 - cosine distance; searches run concurrently, one connection per searcher (opened as searchers arrive, at most 32), writes run one at a time and never overlap a search | 1,563 | 3.62 | 3.99 | 5.17 | 273.7 | 577.9 | 4.98 | 6.89 | 1.000 | 0.518 | - | 2.9 MiB |
| redis | HNSW TYPE FLOAT32 M=16 EF_CONSTRUCTION=128, EF_RUNTIME=100 per query, cosine; exact mode = FLAT index built on first exact query | 11,254 | 0.39 | 0.48 | 0.66 | 2587.0 | 5253.4 | 0.55 | 0.36 | 1.000 | 0.518 | 253.1 MiB | 2.43 MiB |
| qdrant-hnsw | HNSW m=16 ef_construct=100, hnsw_ef=server default, cosine; indexing_threshold_kb 1 and full_scan_threshold_kb 10 (server defaults are 10,000 each) so a small collection builds and walks its graph | 7,366 | 0.92 | 1.02 | 1.45 | 1074.9 | 3472.3 | 0.72 | 0.52 | 1.000 | 0.518 | 36.01 MiB | 164.1 MiB |
| sqlitevec | vec0 brute-force scan, no ANN index (exact), float32, cosine distance, default chunk_size=1024; score = 1 - cosine distance; searches run concurrently, one WAL reader connection per searcher (opened as searchers arrive, at most 32), writes run one at a time and may overlap searches | 4,360 | 1.89 | 2.10 | 2.29 | 520.6 | 1136.4 | 2.02 | 3.51 | 1.000 | 0.518 | - | 9.72 MiB |
| mongodb | vectorSearch index, HNSW maxEdges=16 numEdgeCandidates=128, float32 binData, cosine, numCandidates=20x hits (min 100); exact mode = $vectorSearch exact:true | 3,076 | 1.37 | 1.56 | 1.94 | 717.5 | 2150.0 | 0.30 | 0.29 | 1.000 | 0.518 | 639.7 MiB | 2.21 MiB |
| vespa | HNSW float32 tensor, prenormalized-angular (cosine), max-links-per-node=16, neighbors-to-explore-at-insert=128; search targetHits=top, ef=100 via exploreAdditionalHits; exact mode = approximate:false; vectors held in memory | 296 | 1.92 | 2.41 | 2.78 | 503.1 | 1555.7 | 0.91 | 0.87 | 1.000 | 0.518 | 2.69 GiB | 3.59 MiB |

## Details per target

### sql-diskann

- Engine: Microsoft SQL Server 2025 (RTM-CU9) (KB5122048) - 17.0.5005.3 (X64), Enterprise Developer Edition (64-bit), in container gvb-mssql (image mcr.microsoft.com/mssql/server:2025-CU9-ubuntu-24.04@sha256:2b5b581621126574f3d1f75e78d3eebe8d05aedb59ad0cfdf9aa42cb0634d726; cpuset 2-3,6-7 from its creation; SQL Server counts 4 CPU(s) and runs 4 visible scheduler(s), affinity AUTO) (compose)
- Index: DiskANN (preview) via VECTOR_SEARCH, cosine, build {"StartId":"306", "L":"48", "M":"8", "R":"48"}; the exact mode scans the same table
- Search settings: L=48, M=8, R=48, StartId=306
- Engine settings: HostConfig.Memory = 8589934592 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit))
- Image: mcr.microsoft.com/mssql/server:2025-CU9-ubuntu-24.04@sha256:2b5b581621126574f3d1f75e78d3eebe8d05aedb59ad0cfdf9aa42cb0634d726, id sha256:2b5b581621126574f3d1f75e78d3eebe8d05aedb59ad0cfdf9aa42cb0634d726 (the pinned id), tagged on this machine at 2026-10-04T22:02:11.293843649Z
- Data folder ~/gvb-data/engines/mssql: 105.06 MiB at the start, 505.89 MiB at the end
- Load: 524 rows in batches of 1000, 1.0 s of upserts (532 rows/s); count matched 0.0 s after the last upsert; index step 3.9 s (copied 524 rows to dbo.gvb_gvbbench_eshoponweb_ann (INT key) and built DiskANN, parameters {"StartId":"306", "L":"48", "M":"8", "R":"48"}, graph covers 524 rows)
- Search: 5540 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 1.01 ms, 8 searchers 1.08 ms
- Exact mode: 16,451 searches, p50 3.57 ms, p95 4.06 ms, recall 1.000
- RAM: 530.3 MiB (docker stats); disk: 4.52 MiB (table and its indexes, reserved pages)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started mssql.compose.yaml with every container created on CPUs 2-3,6-7: gvb-mssql cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [mssql.compose.yaml b7359663cc15]. This run created the engine from these files.
- Index after the load: ready, 524 of 524 indexed (DiskANN index built and used. sys.vector_indexes: vix_gvb_gvbbench_eshoponweb on dbo.gvb_gvbbench_eshoponweb_ann in GvbBenchDiskAnn, DiskANN, COSINE, enabled, build {"StartId":"306", "L":"48", "M":"8", "R":"48"}; graph table rows 524 of 524 table rows. Real VECTOR_SEARCH plan: Vector Index Seek on index vix_gvb_gvbbench_eshoponweb of GvbBenchDiskAnn.gvb_gvbbench_eshoponweb_ann (IndexKind DiskANN), 10 rows returned, 10 hits. The exact mode searches the SAME table (dbo.gvb_gvbbench_eshoponweb_ann, database GvbBenchDiskAnn) and its real plan is: Clustered Index Scan of GvbBenchDiskAnn.gvb_gvbbench_eshoponweb_ann through PK_gvb_gvbbench_eshoponweb_ann (IndexKind Clustered), 524 rows read, no vector index operator, 10 hits). Durability: A commit returns after its transaction-log records are written to disk (SQL Server write-ahead logging; delayed durability is DISABLED); a database created here copies the model database: recovery model FULL, page_verify CHECKSUM; no global trace flags are enabled; mssql.conf of container gvb-mssql (/var/opt/mssql/mssql.conf, on the host at ~/gvb-data/engines/mssql/mssql.conf) does not exist, so it sets: nothing; it has no [control] or [traceflag] entry, so SQL Server's own Linux defaults for flushing writes apply (Microsoft's Linux performance guide names trace flag 3982 as that default; read from the guide, not tested here). Not tested by cutting power; whether the disk's own write cache reaches the media was not checked. Container settings from its environment (names only): MSSQL_AGENT_ENABLED, MSSQL_MEMORY_LIMIT_MB, MSSQL_PID, MSSQL_RPC_PORT, MSSQL_SA_PASSWORD..
- Outside load when searching began: 0.16 CPUs on average over the last 6 s and 0.16 over the last few seconds (limit 0.3), a quiet box. The load average 2.29 3.19 3.42 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: exact 8,000 searches in 30.0 s, default@8 20,361 searches in 30.0 s, default@1 8,306 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. exact: warm-up 15.0 s and 4,059 searches (0 failed) at 1 searcher; p50 of the last windows 3.632, 3.612, 3.612 ms (windows of at least 2 s and 100 searches); trial of 819 searches in 3.0 s at 1 searcher: p50 3.610 ms against the settled p50 3.612 ms, 0% apart (limit 10%); the timed pass p50 3.573 ms, 1% from the settled p50 3.612 ms (limit 10% or 0.1 ms; 0.040 ms); default@8: warm-up 15.0 s and 10,380 searches (0 failed) at 8 searchers; QPS of the last windows 678, 700, 678 (windows of at least 2 s and 100 searches); trial of 2,107 searches in 3.0 s at 8 searchers: 700 QPS against the settled 678 QPS, 3% apart (limit 10%); the timed pass 668 QPS, 2% from the settled 678 QPS (limit 10%); default@1: warm-up 15.0 s and 4,164 searches (0 failed) at 1 searcher; p50 of the last windows 3.556, 3.563, 3.563 ms (windows of at least 2 s and 100 searches); trial of 823 searches in 3.0 s at 1 searcher: p50 3.570 ms against the settled p50 3.563 ms, 0% apart (limit 10%); the timed pass p50 3.564 ms, 0% from the settled p50 3.563 ms (limit 10% or 0.1 ms; 0.001 ms).
- Pass exact after rehearsal: 2026-10-07T22:16:35.310Z to 2026-10-07T22:17:35.312Z (60.0 s), 16,451 searches, 0 failed, p50 3.573 ms, mean 3.644 ms, p99 4.661 ms, 274.2 QPS (1000/QPS 3.647 ms).
- Pass default@8 after exact: 2026-10-07T22:17:53.444Z to 2026-10-07T22:18:13.452Z (20.0 s), 13,373 searches, 0 failed, p50 11.393 ms, mean 11.964 ms, p99 20.865 ms, 668.4 QPS (1000/QPS 1.496 ms).
- Pass default@1 after default@8: 2026-10-07T22:18:31.461Z to 2026-10-07T22:18:51.461Z (20.0 s), 5,540 searches, 0 failed, p50 3.564 ms, mean 3.608 ms, p99 5.003 ms, 277.0 QPS (1000/QPS 3.610 ms).
- Passes in the order run: exact, default@8, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 59,019 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (DiskANN index built and used. sys.vector_indexes: vix_gvb_gvbbench_eshoponweb on dbo.gvb_gvbbench_eshoponweb_ann in GvbBenchDiskAnn, DiskANN, COSINE, enabled, build {"StartId":"306", "L":"48", "M":"8", "R":"48"}; graph table rows 524 of 524 table rows. Real VECTOR_SEARCH plan: Vector Index Seek on index vix_gvb_gvbbench_eshoponweb of GvbBenchDiskAnn.gvb_gvbbench_eshoponweb_ann (IndexKind DiskANN), 10 rows returned, 10 hits. The exact mode searches the SAME table (dbo.gvb_gvbbench_eshoponweb_ann, database GvbBenchDiskAnn) and its real plan is: Clustered Index Scan of GvbBenchDiskAnn.gvb_gvbbench_eshoponweb_ann through PK_gvb_gvbbench_eshoponweb_ann (IndexKind Clustered), 524 rows read, no vector index operator, 10 hits).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-mssql (f86683dfa516) on network gvb-mssql_default at container-ip:1433, not through docker-proxy localhost:14330. Open after the passes: container-ip:1433 x9 (this target), localhost:1433 x2 (not this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-mssql: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-mssql: 86 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): exact 3492/3492 MHz, average 3493.3/3492.4, performance, outside load 0.14 before the warm-up, 0.15 during, client CPU 1.138 ms per search, engine CPU 3.515 ms per search; default@8 3492/3492 MHz, average 3492.0/3492.2, performance, outside load 0.15 before the warm-up, 0.13 during, client CPU 1.083 ms per search, engine CPU 5.330 ms per search; default@1 3492/3492 MHz, average 3493.3/3492.4, performance, outside load 0.12 before the warm-up, 0.14 during, client CPU 1.010 ms per search, engine CPU 3.445 ms per search.

### oracle

- Engine: Oracle AI Database Free 23.26 (23ai line, VECTOR FLOAT32) (compose)
- Index: HNSW in-memory neighbor graph NEIGHBORS=16 EFCONSTRUCTION=128, EFSEARCH=100 per query, cosine; exact mode = FETCH EXACT FIRST (full scan); Oracle Free caps itself at 2 CPUs (cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit)
- Search settings: EFCONSTRUCTION=128, EFSEARCH=100, NEIGHBORS=16
- Engine settings: HostConfig.Memory = 6442450944 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit)); cpu_count = 2 (read: docker exec gvb-oracle sqlplus / as sysdba, V$PARAMETER and V$INSTANCE at the instance); sga_target = 1610612736 (read: docker exec gvb-oracle sqlplus / as sysdba, V$PARAMETER and V$INSTANCE at the instance); oracle_version = 23.26.3.0.0 (read: docker exec gvb-oracle sqlplus / as sysdba, V$PARAMETER and V$INSTANCE at the instance)
- Image: gvenzl/oracle-free:23-slim-faststart, id sha256:f5ff19033860d662c821cb04eb10483fa94f14f78eae252d054291ea07028093 (the pinned id), tagged on this machine at 2026-10-03T14:17:05.132133694Z
- Data folder ~/gvb-data/engines/oracle: 6.9 GiB at the start, 6.9 GiB at the end
- Load: 524 rows in batches of 1000, 0.3 s of upserts (1,701 rows/s); count matched 0.1 s after the last upsert; index step 3.7 s (HNSW graph holds 524 of 524 rows, change log waiting: 0 inserts and 0 deletes, USER_INDEXES status VALID, plan of the default search: VECTOR INDEX HNSW SCAN > TABLE ACCESS BY INDEX ROWID, index used by 0 queries so far; Oracle Free caps itself at 2 CPUs (cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit); graph rebuilt in 2.4 s)
- Search: 20574 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 0.89 ms, 8 searchers 0.65 ms
- Exact mode: 23,511 searches, p50 2.49 ms, p95 2.97 ms, recall 1.000
- RAM: 2.22 GiB (docker stats); disk: 0 B added by this load (6.9 GiB (whole engine data folder), 6.9 GiB before)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started oracle.compose.yaml with every container created on CPUs 2-3,6-7: gvb-oracle cpuset 2-3,6-7, gvb-oracle-seed cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [oracle-init/01-vector-memory.sh 6452e4e1953e; oracle.compose.yaml 2a7fd367012f]. This run created the engine from these files.
- Index after the load: ready, 524 of 524 indexed (HNSW graph holds 524 of 524 rows, change log waiting: 0 inserts and 0 deletes, USER_INDEXES status VALID, plan of the default search: VECTOR INDEX HNSW SCAN > TABLE ACCESS BY INDEX ROWID, index used by 0 queries so far; Oracle Free caps itself at 2 CPUs (cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit)). Durability: By these settings a committed row should survive a crash or power loss. The sink uses a plain COMMIT and commit_logging, commit_wait and commit_write are unset in the database, so every commit waits for its redo to be written. Measured 2026-10-04: 50 separate client commits raised V$SYSSTAT 'redo synch writes' by 55 (the 50 commits plus the CREATE and DROP of the probe table), and the log writer and the datafile writer hold their files open with O_DSYNC (open flags 02110002, filesystemio_options none). The database runs NOARCHIVELOG (V$DATABASE.LOG_MODE), so redo serves crash recovery only and there is no point-in-time restore. The HNSW graph lives in the 768 MB vector memory pool (oracle-init/01-vector-memory.sh) and is not the durable copy; the table is. Not tested by cutting power; whether the disk's own write cache reaches the media was not checked. CPU: Oracle Free caps itself at 2 CPUs (measured 2026-10-04: cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit; this sink reads the live value when the index state is read)..
- Outside load when searching began: 0.14 CPUs on average over the last 6 s and 0.14 over the last few seconds (limit 0.3), a quiet box. The load average 2.50 2.82 3.20 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@8 54,796 searches in 30.0 s, exact 9,462 searches in 30.0 s, default@1 25,689 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. default@8: warm-up 15.0 s and 36,426 searches (0 failed) at 8 searchers; QPS of the last windows 1,595, 3,030, 2,727 (windows of at least 2 s and 100 searches) (its last 3 windows differed by more than 5%); trial of 6,759 searches in 3.0 s at 8 searchers: 2,251 QPS against the settled 2,727 QPS, 21% apart (limit 10%), so the warm-up was EXTENDED once (50.2 s and 133,731 searches (0 failed) at 8 searchers; QPS of its older 10 windows 2,680 (windows 2,374, 2,186, 2,760, 2,014, 3,403, 2,388, 3,000, 2,861, 2,540, 3,276) and of its newer 10 2,810 (windows 2,363, 3,358, 2,482, 2,932, 2,892, 2,504, 3,274, 2,392, 3,380, 2,524), 4.9% apart (limit 5%) (windows of at least 2 s and 100 searches)); second trial of 9,189 searches in 3.0 s at 8 searchers: 3,062 QPS against the settled 2,810 QPS, 8% apart (limit 10%), inside its newer windows' range 2,363 to 3,380 QPS; settled after the extension; the timed pass 2,820 QPS, 0% from the settled 2,810 QPS (limit 10%); exact: warm-up 15.0 s and 5,854 searches (0 failed) at 1 searcher; p50 of the last windows 2.484, 2.505, 2.508 ms (windows of at least 2 s and 100 searches); trial of 1,196 searches in 3.0 s at 1 searcher: p50 2.474 ms against the settled p50 2.505 ms, 1% apart (limit 10%); the timed pass p50 2.486 ms, 1% from the settled p50 2.505 ms (limit 10% or 0.1 ms; 0.020 ms); default@1: warm-up 15.0 s and 15,795 searches (0 failed) at 1 searcher; p50 of the last windows 0.915, 0.914, 0.920 ms (windows of at least 2 s and 100 searches); trial of 3,177 searches in 3.0 s at 1 searcher: p50 0.921 ms against the settled p50 0.915 ms, 1% apart (limit 10%); the timed pass p50 0.944 ms, 3% from the settled p50 0.915 ms (limit 10% or 0.1 ms; 0.029 ms).
- Pass default@8 after rehearsal: 2026-10-07T22:22:08.153Z to 2026-10-07T22:22:28.254Z (20.1 s), 56,690 searches, 0 failed, p50 1.616 ms, mean 2.834 ms, p99 3.501 ms, 2820.4 QPS (1000/QPS 0.355 ms).
- Pass exact after default@8: 2026-10-07T22:22:46.280Z to 2026-10-07T22:23:46.281Z (60.0 s), 23,511 searches, 0 failed, p50 2.486 ms, mean 2.550 ms, p99 3.989 ms, 391.8 QPS (1000/QPS 2.552 ms).
- Pass default@1 after exact: 2026-10-07T22:24:04.312Z to 2026-10-07T22:24:24.313Z (20.0 s), 20,574 searches, 0 failed, p50 0.944 ms, mean 0.970 ms, p99 1.818 ms, 1028.7 QPS (1000/QPS 0.972 ms).
- Passes in the order run: default@8, exact, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 343,640 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (HNSW graph holds 524 of 524 rows, change log waiting: 0 inserts and 0 deletes, USER_INDEXES status VALID, plan of the default search: VECTOR INDEX HNSW SCAN > TABLE ACCESS BY INDEX ROWID, index used by 404402 queries so far; Oracle Free caps itself at 2 CPUs (cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit)).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-oracle (16943ca32f1f) on network gvb-oracle_default at container-ip:1521, not through docker-proxy localhost:1521. Open after the passes: container-ip:1521 x14 (this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-oracle: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-oracle: 85 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): default@8 3492/3492 MHz, average 3186.5/3255.4, performance, outside load 0.17 before the warm-up, 0.16 during, client CPU 0.649 ms per search, engine CPU 0.720 ms per search; exact 3492/3492 MHz, average 3492.3/3492.4, performance, outside load 0.16 before the warm-up, 0.14 during, client CPU 0.936 ms per search, engine CPU 2.159 ms per search; default@1 3492/3492 MHz, average 3492.3/3492.1, performance, outside load 0.14 before the warm-up, 0.16 during, client CPU 0.892 ms per search, engine CPU 0.704 ms per search.

### qdrant

- Engine: Qdrant 1.17.0 in container gvb-qdrant (image qdrant/qdrant:v1.17.0@sha256:f1c7272cdac52b38c1a0e89313922d940ba50afd90d593a1605dbbc214e66ffb; cpuset 2-3,6-7 from its creation; 3 search thread(s)) (compose)
- Index: exact scan: the builder's sink sends exact=true on every search, so no HNSW graph is used whether or not Qdrant has built one (see the index state)
- Search settings: exact=true
- Engine settings: HostConfig.Memory = 8589934592 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit)); search_threads = 3 (read: threads named search-* of the qdrant process in container gvb-qdrant (/proc))
- Image: qdrant/qdrant:v1.17.0@sha256:f1c7272cdac52b38c1a0e89313922d940ba50afd90d593a1605dbbc214e66ffb, id sha256:f1c7272cdac52b38c1a0e89313922d940ba50afd90d593a1605dbbc214e66ffb (the pinned id), tagged on this machine at 2026-10-04T22:03:16.973657523Z
- Data folder ~/gvb-data/engines/qdrant: 357 B at the start, 196.08 MiB at the end
- Load: 524 rows in batches of 1000, 0.1 s of upserts (4,229 rows/s); count matched 0.0 s after the last upsert; index step 1.0 s (status Green, 0 of 524 vectors in HNSW segments (2 segments), waited 1.0 s)
- Search: 22270 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 0.74 ms, 8 searchers 0.54 ms
- RAM: 37.95 MiB (docker stats); disk: 196.08 MiB (collection folder)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started qdrant.compose.yaml with every container created on CPUs 2-3,6-7: gvb-qdrant cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [qdrant.compose.yaml eb57e55e93c1]. This run created the engine from these files.
- Index after the load: ready, 0 of 524 indexed (status Green, optimizer ok, indexed_vectors_count 0 of 524 points, 2 segments, indexing_threshold_kb 10,000, full_scan_threshold_kb 10,000; no index used, exact scan by design (the builder's sink sends exact=true on every search); no graph was built; searches counted since the collection was created: unfiltered_hnsw 0, unfiltered_plain 0, unfiltered_exact 0). Durability: What was measured (strace -f on the Qdrant server process, 30 single-point REST upserts per setting, every sync call timed against the request that caused it): with wait=true each upsert had exactly one msync(MS_SYNC) of the write-ahead-log segment inside the request, before the reply (30 of 30); with wait=false the 30 upserts were acknowledged within 0.26 s with no sync call, and the first WAL msync (one call covering all 30 records) came 2.7 s after the last reply, followed by the segment-file flushes. So the log is flushed to disk under both settings, but with wait=false the flush comes after the acknowledgement: an operating-system crash or power cut in that gap loses acknowledged writes, while a crash of the Qdrant process alone should not, because the bytes are already in the kernel's page cache (inferred, not tested). Every upsert here is sent with wait=true in batches of 256 over gRPC; the strace used one point per request, so one flush per batch is inferred, not measured. Segment files are flushed every 5 s; log segments 32 MB, 0 created ahead. /qdrant/config/config.yaml in container gvb-qdrant (the image's own file; its storage keys are listed) sets: storage.collection.quantization = null, storage.collection.replication_factor = 1, storage.collection.vectors.on_disk = null, storage.collection.write_consistency_factor = 1, storage.hnsw_index.ef_construct = 100, storage.hnsw_index.full_scan_threshold_kb = 10000, storage.hnsw_index.m = 16, storage.hnsw_index.max_indexing_threads = 0, storage.hnsw_index.on_disk = false, storage.hnsw_index.payload_m = null, storage.max_collections = null, storage.node_type = Normal, storage.on_disk_payload = true, storage.optimizers.default_segment_number = 0, storage.optimizers.deleted_threshold = 0.2, storage.optimizers.flush_interval_sec = 5, storage.optimizers.indexing_threshold_kb = 10000, storage.optimizers.max_optimization_threads = null, storage.optimizers.max_segment_size_kb = null, storage.optimizers.vacuum_min_vector_number = 1000, storage.performance.max_search_threads = 0, storage.performance.optimizer_cpu_budget = 0, storage.performance.update_rate_limit = null, storage.shard_transfer_method = null, storage.snapshots_config.snapshots_storage = local, storage.snapshots_path = ./snapshots, storage.storage_path = ./storage, storage.temp_path = null, storage.update_concurrency = null, storage.wal.wal_capacity_mb = 32, storage.wal.wal_segments_ahead = 0; no QDRANT__ environment overrides in the server process. Not tested by cutting power; whether the disk's own write cache reaches the media was not checked..
- Outside load when searching began: 0.13 CPUs on average over the last 2 s and 0.13 over the last few seconds (limit 0.3), a quiet box. The load average 1.97 2.98 3.22 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@1 33,704 searches in 30.0 s, default@8 98,761 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. default@1: warm-up 15.0 s and 16,700 searches (0 failed) at 1 searcher; p50 of the last windows 0.889, 0.884, 0.886 ms (windows of at least 2 s and 100 searches); trial of 3,350 searches in 3.0 s at 1 searcher: p50 0.884 ms against the settled p50 0.886 ms, 0% apart (limit 10%); the timed pass p50 0.887 ms, 0% from the settled p50 0.886 ms (limit 10% or 0.1 ms; 0.001 ms); default@8: warm-up 15.0 s and 49,068 searches (0 failed) at 8 searchers; QPS of the last windows 3,253, 3,242, 3,294 (windows of at least 2 s and 100 searches); trial of 9,809 searches in 3.0 s at 8 searchers: 3,268 QPS against the settled 3,253 QPS, 0% apart (limit 10%); the timed pass 3,293 QPS, 1% from the settled 3,253 QPS (limit 10%).
- Pass default@1 after rehearsal: 2026-10-07T22:25:54.331Z to 2026-10-07T22:26:14.331Z (20.0 s), 22,270 searches, 0 failed, p50 0.887 ms, mean 0.896 ms, p99 1.405 ms, 1113.5 QPS (1000/QPS 0.898 ms).
- Pass default@8 after default@1: 2026-10-07T22:26:32.361Z to 2026-10-07T22:26:52.363Z (20.0 s), 65,873 searches, 0 failed, p50 2.343 ms, mean 2.427 ms, p99 4.194 ms, 3293.3 QPS (1000/QPS 0.304 ms).
- Passes in the order run: default@1, default@8; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 211,392 sent, 0 failed (warmupErrors). Index after the searches: ready, 0 of 524 indexed (status Green, optimizer ok, indexed_vectors_count 0 of 524 points, 2 segments, indexing_threshold_kb 10,000, full_scan_threshold_kb 10,000; no index used, exact scan by design (the builder's sink sends exact=true on every search); no graph was built; searches counted since the collection was created: unfiltered_hnsw 0, unfiltered_plain 599,070, unfiltered_exact 0).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-qdrant (093e2271cbc0) on network gvb-qdrant_default at container-ip:6334, not through docker-proxy localhost:16334; container gvb-qdrant (093e2271cbc0) on network gvb-qdrant_default at container-ip:6333, not through docker-proxy localhost:16333. Open after the passes: container-ip:6334 x1 (this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-qdrant: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-qdrant: 24 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): default@1 3492/3492 MHz, average 3492.0/3492.0, performance, outside load 0.13 before the warm-up, 0.09 during, client CPU 0.740 ms per search, engine CPU 0.797 ms per search; default@8 3492/3492 MHz, average 3492.0/3492.0, performance, outside load 0.11 before the warm-up, 0.11 during, client CPU 0.535 ms per search, engine CPU 1.004 ms per search.

### elasticsearch

- Engine: Elasticsearch 9.5.3 (dense_vector) (compose)
- Index: HNSW float32, no quantization, m=16, ef_construction=128, cosine; search k=top, num_candidates=100; 1 shard, 0 replicas; force-merged to one segment after the load (at 1,024 dimensions a segment under 1,043 vectors gets no graph)
- Search settings: ef_construction=128, k=top, m=16, num_candidates=100
- Engine settings: HostConfig.Memory = 6442450944 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit)); heap_init_in_bytes = 2147483648 (read: GET /_nodes/jvm (nodes.<id>.jvm.mem.heap_init_in_bytes and heap_max_in_bytes)); heap_max_in_bytes = 2147483648 (read: GET /_nodes/jvm (nodes.<id>.jvm.mem.heap_init_in_bytes and heap_max_in_bytes))
- Image: elasticsearch:9.5.3, id sha256:e23d4758358a4e356cc2ef3259a5a6f345cacc1722c10dffe6d7150a1a78e52f (the pinned id), tagged on this machine at 2026-10-03T14:07:11.337389556Z
- Data folder ~/gvb-data/engines/elasticsearch: 189.2 KiB at the start, 2.57 MiB at the end
- Load: 524 rows in batches of 1000, 1.1 s of upserts (463 rows/s); count matched 0.1 s after the last upsert; index step 0.4 s (FAILED after 0.4 s: NO HNSW GRAPH, searches scan all 524 vectors: 1 segment(s), 524 vectors, total_vex_size_bytes 0 (_stats dense_vector), index_options hnsw m=16 ef_construction=128; Lucene builds no graph for a segment this small (measured at 1,024 dimensions: 1,042 vectors none, 1,043 vectors a graph), so default search equals exact search)
- Search: 14950 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 0.46 ms, 8 searchers 0.50 ms
- Exact mode: 47,055 searches, p50 1.25 ms, p95 1.44 ms, recall 1.000
- RAM: 2.54 GiB (docker stats); disk: 2.39 MiB added by this load (2.57 MiB (whole engine data folder), 189.2 KiB before)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started elasticsearch.compose.yaml with every container created on CPUs 2-3,6-7: gvb-elasticsearch cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [elasticsearch.compose.yaml 582b9a97b8de]. This run created the engine from these files.
- WARNING: the index step after the load FAILED (NO HNSW GRAPH, searches scan all 524 vectors: 1 segment(s), 524 vectors, total_vex_size_bytes 0 (_stats dense_vector), index_options hnsw m=16 ef_construction=128; Lucene builds no graph for a segment this small (measured at 1,024 dimensions: 1,042 vectors none, 1,043 vectors a graph), so default search equals exact search). Searched anyway; see the index state.
- WARNING: index not ready after the load: NOT ready, 0 of 524 indexed (NO HNSW GRAPH, searches scan all 524 vectors: 1 segment(s), 524 vectors, total_vex_size_bytes 0 (_stats dense_vector), index_options hnsw m=16 ef_construction=128; Lucene builds no graph for a segment this small (measured at 1,024 dimensions: 1,042 vectors none, 1,043 vectors a graph), so default search equals exact search). Measured anyway, so its search numbers may come from a scan or a half-built index. Durability: Every acknowledged bulk request is fsynced to the translog before the answer: index.translog.durability=request, the Elasticsearch default, which neither this sink nor elasticsearch.compose.yaml overrides (ElasticsearchReadinessTests reads it back from the live index). By that setting a process crash or power loss loses no acknowledged write; this is read from the setting, not shown by pulling power. One node and no replicas, so a lost disk loses the data..
- Outside load when searching began: 0.16 CPUs on average over the last 2 s and 0.16 over the last few seconds (limit 0.3), a quiet box. The load average 3.45 3.35 3.32 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@8 36,417 searches in 30.0 s, exact 22,229 searches in 30.0 s, default@1 22,537 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. default@8: warm-up 15.0 s and 40,652 searches (0 failed) at 8 searchers; QPS of the last windows 2,718, 2,738, 2,748 (windows of at least 2 s and 100 searches); trial of 8,168 searches in 3.0 s at 8 searchers: 2,721 QPS against the settled 2,738 QPS, 1% apart (limit 10%); the timed pass 2,732 QPS, 0% from the settled 2,738 QPS (limit 10%); exact: warm-up 15.0 s and 11,748 searches (0 failed) at 1 searcher; p50 of the last windows 1.252, 1.252, 1.258 ms (windows of at least 2 s and 100 searches); trial of 2,335 searches in 3.0 s at 1 searcher: p50 1.260 ms against the settled p50 1.252 ms, 1% apart (limit 10%); the timed pass p50 1.254 ms, 0% from the settled p50 1.252 ms (limit 10% or 0.1 ms; 0.002 ms); default@1: warm-up 15.0 s and 11,268 searches (0 failed) at 1 searcher; p50 of the last windows 1.305, 1.318, 1.312 ms (windows of at least 2 s and 100 searches); trial of 2,243 searches in 3.0 s at 1 searcher: p50 1.316 ms against the settled p50 1.312 ms, 0% apart (limit 10%); the timed pass p50 1.312 ms, 0% from the settled p50 1.312 ms (limit 10% or 0.1 ms; 0.001 ms).
- Pass default@8 after rehearsal: 2026-10-07T22:29:15.492Z to 2026-10-07T22:29:35.495Z (20.0 s), 54,645 searches, 0 failed, p50 2.799 ms, mean 2.926 ms, p99 5.394 ms, 2731.9 QPS (1000/QPS 0.366 ms).
- Pass exact after default@8: 2026-10-07T22:29:53.518Z to 2026-10-07T22:30:53.518Z (60.0 s), 47,055 searches, 0 failed, p50 1.254 ms, mean 1.273 ms, p99 1.687 ms, 784.2 QPS (1000/QPS 1.275 ms).
- Pass default@1 after exact: 2026-10-07T22:31:11.573Z to 2026-10-07T22:31:31.575Z (20.0 s), 14,950 searches, 0 failed, p50 1.312 ms, mean 1.336 ms, p99 1.762 ms, 747.5 QPS (1000/QPS 1.338 ms).
- Passes in the order run: default@8, exact, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 157,597 sent, 0 failed (warmupErrors). Index after the searches: NOT ready, 0 of 524 indexed (NO HNSW GRAPH, searches scan all 524 vectors: 1 segment(s), 524 vectors, total_vex_size_bytes 0 (_stats dense_vector), index_options hnsw m=16 ef_construction=128; Lucene builds no graph for a segment this small (measured at 1,024 dimensions: 1,042 vectors none, 1,043 vectors a graph), so default search equals exact search).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-elasticsearch (f74ac9505b0e) on network engines_default at container-ip:9200, not through docker-proxy localhost:9200. Open after the passes: container-ip:9200 x1 (this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-elasticsearch: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-elasticsearch: 74 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): default@8 3492/3492 MHz, average 3492.0/3492.2, performance, outside load 0.14 before the warm-up, 0.17 during, client CPU 0.499 ms per search, engine CPU 1.370 ms per search; exact 3492/3492 MHz, average 3491.5/3490.6, performance, outside load 0.15 before the warm-up, 0.15 during, client CPU 0.459 ms per search, engine CPU 0.966 ms per search; default@1 3492/3492 MHz, average 3491.8/3493.0, performance, outside load 0.15 before the warm-up, 0.18 during, client CPU 0.457 ms per search, engine CPU 1.042 ms per search.

### weaviate

- Engine: Weaviate 1.39.8 (single node, REST + GraphQL) (compose)
- Index: HNSW maxConnections(M)=16 efConstruction=128, ef=-1 (dynamic: limit x 8 clamped 100..500), cosine, no quantization; approximate only (no exact mode)
- Search settings: ef=-1, efConstruction=128
- Engine settings: HostConfig.Memory = 8589934592 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit)); GOMEMLIMIT = 6GiB (read: docker inspect Config.Env, the name GOMEMLIMIT only)
- Image: semitechnologies/weaviate:1.39.8, id sha256:f6f4a5961f99e8718a02c822ca6a0d65ed3f9c567734419a7b2144933014f13a (the pinned id), tagged on this machine at 2026-10-03T14:05:40.948499656Z
- Data folder ~/gvb-data/engines/weaviate: 2.06 MiB at the start, 5.02 MiB at the end
- Load: 524 rows in batches of 1000, 0.5 s of upserts (979 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (HNSW is updated inside every batch, nothing to build; confirmed in 0.0 s: shard O3CCaueAwTBV vectorIndexingStatus READY, vectorQueueLength 0, node status objectCount 0 (refreshed only when the memtable is flushed, so it lags a minute and does not decide readiness), schema shard O3CCaueAwTBV status READY, Aggregate count 524, Aggregate nearVector with objectLimit 524 reached 524, Weaviate reports no count of vectors in the HNSW index, so indexed vectors are not reported)
- Search: 3348 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 0.56 ms, 8 searchers 0.61 ms
- RAM: 83.91 MiB (docker stats); disk: 2.96 MiB added by this load (5.02 MiB (whole engine data folder), 2.06 MiB before)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started weaviate.compose.yaml with every container created on CPUs 2-3,6-7: gvb-weaviate cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [weaviate.compose.yaml d96b7399662a]. This run created the engine from these files.
- Index after the load: ready, ? of 524 indexed (shard O3CCaueAwTBV vectorIndexingStatus READY, vectorQueueLength 0, node status objectCount 0 (refreshed only when the memtable is flushed, so it lags a minute and does not decide readiness), schema shard O3CCaueAwTBV status READY, Aggregate count 524, Aggregate nearVector with objectLimit 524 reached 524, Weaviate reports no count of vectors in the HNSW index, so indexed vectors are not reported). Durability: Weaviate 1.39.8 defaults, weaviate.compose.yaml sets no persistence variable: every object is appended to the LSM write-ahead log with a plain write and no fsync, and the log is fsynced only when its memtable is flushed, 60 seconds after the last write (PERSISTENCE_MEMTABLES_FLUSH_IDLE_AFTER_SECONDS default; measured with strace: 2,080 writes into the objects log during a 2,000-object load, the first fsync 60 s after the last write), so a power cut can lose the last minute of writes; the HNSW commit log is buffered inside the process (77 writes for 2,000 vectors), so a killed process loses the newest vectors from the vector index while their objects survive (measured with docker kill, which is SIGKILL, one second after the load and a restart, two runs each: 0 to 1 of 50, 264 to 267 of 300 and 1,992 to 1,997 of 2,000 stored objects were still reachable through a vector search).
- Outside load when searching began: 0.10 CPUs on average over the last 1 s and 0.10 over the last few seconds (limit 0.3), a quiet box. The load average 1.65 3.21 3.35 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@1 5,076 searches in 30.0 s, default@8 9,305 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. default@1: warm-up 15.0 s and 2,519 searches (0 failed) at 1 searcher; p50 of the last windows 5.310, 5.313, 5.317 ms (windows of at least 2 s and 100 searches); trial of 504 searches in 3.0 s at 1 searcher: p50 5.315 ms against the settled p50 5.313 ms, 0% apart (limit 10%); the timed pass p50 5.311 ms, 0% from the settled p50 5.313 ms (limit 10% or 0.1 ms; 0.001 ms); default@8: warm-up 15.0 s and 4,593 searches (0 failed) at 8 searchers; QPS of the last windows 305, 306, 305 (windows of at least 2 s and 100 searches); trial of 926 searches in 3.0 s at 8 searchers: 307 QPS against the settled 305 QPS, 1% apart (limit 10%); the timed pass 306 QPS, 0% from the settled 305 QPS (limit 10%).
- Pass default@1 after rehearsal: 2026-10-07T22:33:16.254Z to 2026-10-07T22:33:36.257Z (20.0 s), 3,348 searches, 0 failed, p50 5.311 ms, mean 5.972 ms, p99 11.628 ms, 167.4 QPS (1000/QPS 5.975 ms).
- Pass default@8 after default@1: 2026-10-07T22:33:54.297Z to 2026-10-07T22:34:14.321Z (20.0 s), 6,133 searches, 0 failed, p50 24.619 ms, mean 26.103 ms, p99 53.417 ms, 306.3 QPS (1000/QPS 3.265 ms).
- Passes in the order run: default@1, default@8; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 22,923 sent, 0 failed (warmupErrors). Index after the searches: ready, ? of 524 indexed (shard O3CCaueAwTBV vectorIndexingStatus READY, vectorQueueLength 0, node status objectCount 524 (refreshed only when the memtable is flushed, so it lags a minute and does not decide readiness), schema shard O3CCaueAwTBV status READY, Aggregate count 524, Aggregate nearVector with objectLimit 524 reached 524, Weaviate reports no count of vectors in the HNSW index, so indexed vectors are not reported).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-weaviate (9688bf464138) on network gvb-weaviate_default at container-ip:8080, not through docker-proxy localhost:8085. Open after the passes: container-ip:8080 x8 (this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-weaviate: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-weaviate: 9 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): default@1 3492/3492 MHz, average 3492.1/3493.5, performance, outside load 0.14 before the warm-up, 0.14 during, client CPU 0.563 ms per search, engine CPU 8.246 ms per search; default@8 3492/3492 MHz, average 3492.0/3492.7, performance, outside load 0.13 before the warm-up, 0.14 during, client CPU 0.608 ms per search, engine CPU 12.959 ms per search.

### pgvector

- Engine: PostgreSQL 17 + pgvector 0.8.7 (compose)
- Index: HNSW vector_cosine_ops m=16 ef_construction=128, hnsw.ef_search=100 per query, float32 vector(n), cosine; exact mode = same query with index scans off (sequential scan)
- Search settings: ef_construction=128, hnsw.ef_search=100, m=16
- Engine settings: HostConfig.Memory = 8589934592 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit)); shared_buffers = 2GB (read: docker exec gvb-pgvector psql current_setting); enable_seqscan = off (set: src/GenericVectorBuilder.Engines/Sinks/PgVectorSink.cs#SET LOCAL enable_seqscan = off; SET LOCAL jit = off;); jit = off (set: src/GenericVectorBuilder.Engines/Sinks/PgVectorSink.cs#SET LOCAL enable_seqscan = off; SET LOCAL jit = off;)
- Image: pgvector/pgvector:0.8.7-pg17-trixie, id sha256:7a7e9f22015b67edb4bef5c59daeebcd7e74bfa570df6ce60ae01237c8648a84 (the pinned id), tagged on this machine at 2026-10-03T14:04:52.665253039Z
- Data folder ~/gvb-data/engines/pgvector: 351 MiB at the start, 358.57 MiB at the end
- Load: 524 rows in batches of 1000, 0.9 s of upserts (594 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (HNSW is maintained inside every insert, nothing to build; confirmed in 0.0 s: pg_indexes lists gvb_gvbbench_eshoponweb_hnsw (USING hnsw (embedding vector_cosine_ops) WITH (m='16', ef_construction='128')), pg_index valid and ready = True, EXPLAIN of the default search uses Index Scan on it = True, that search returned 10 of 10 rows; PostgreSQL keeps no entry count for an HNSW index, so indexed vectors are not reported)
- Search: 22535 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 0.69 ms, 8 searchers 0.54 ms
- Exact mode: 26,816 searches, p50 2.19 ms, p95 2.43 ms, recall 1.000
- RAM: 100.4 MiB (docker stats); disk: 7.57 MiB added by this load (358.57 MiB (whole engine data folder), 351 MiB before)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started pgvector.compose.yaml with every container created on CPUs 2-3,6-7: gvb-pgvector cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [pgvector.compose.yaml 27549563c622]. This run created the engine from these files.
- Index after the load: ready, ? of 524 indexed (pg_indexes lists gvb_gvbbench_eshoponweb_hnsw (USING hnsw (embedding vector_cosine_ops) WITH (m='16', ef_construction='128')), pg_index valid and ready = True, EXPLAIN of the default search uses Index Scan on it = True, that search returned 10 of 10 rows; PostgreSQL keeps no entry count for an HNSW index, so indexed vectors are not reported). Durability: fsync on, synchronous_commit on, full_page_writes on, wal_sync_method fdatasync (PostgreSQL defaults; pgvector.compose.yaml sets only shared_buffers, maintenance_work_mem and max_wal_size): every commit is flushed to the write-ahead log before it returns and HNSW index changes are WAL-logged, so a crash loses no committed row (max_wal_size 4GB only spaces out checkpoints; measured with docker kill, which is SIGKILL, right after a 2,000-vector load and a restart: all 2,000 rows were there and the HNSW index was valid and used by the default search).
- Outside load when searching began: 0.06 CPUs on average over the last 2 s and 0.06 over the last few seconds (limit 0.3), a quiet box. The load average 3.82 3.38 3.38 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: exact 13,683 searches in 30.0 s, default@8 124,436 searches in 30.0 s, default@1 34,627 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. exact: warm-up 15.0 s and 6,763 searches (0 failed) at 1 searcher; p50 of the last windows 2.179, 2.188, 2.191 ms (windows of at least 2 s and 100 searches); trial of 1,339 searches in 3.0 s at 1 searcher: p50 2.224 ms against the settled p50 2.188 ms, 2% apart (limit 10%); the timed pass p50 2.193 ms, 0% from the settled p50 2.188 ms (limit 10% or 0.1 ms; 0.005 ms); default@8: warm-up 15.0 s and 59,991 searches (0 failed) at 8 searchers; QPS of the last windows 4,038, 4,004, 3,929 (windows of at least 2 s and 100 searches); trial of 12,035 searches in 3.0 s at 8 searchers: 4,011 QPS against the settled 4,004 QPS, 0% apart (limit 10%); the timed pass 4,006 QPS, 0% from the settled 4,004 QPS (limit 10%); default@1: warm-up 15.0 s and 17,021 searches (0 failed) at 1 searcher; p50 of the last windows 0.868, 0.862, 0.867 ms (windows of at least 2 s and 100 searches); trial of 3,331 searches in 3.0 s at 1 searcher: p50 0.883 ms against the settled p50 0.867 ms, 2% apart (limit 10%); the timed pass p50 0.869 ms, 0% from the settled p50 0.867 ms (limit 10% or 0.1 ms; 0.002 ms).
- Pass exact after rehearsal: 2026-10-07T22:36:12.109Z to 2026-10-07T22:37:12.109Z (60.0 s), 26,816 searches, 0 failed, p50 2.193 ms, mean 2.235 ms, p99 3.608 ms, 446.9 QPS (1000/QPS 2.237 ms).
- Pass default@8 after exact: 2026-10-07T22:37:30.145Z to 2026-10-07T22:37:50.146Z (20.0 s), 80,130 searches, 0 failed, p50 1.952 ms, mean 1.995 ms, p99 3.500 ms, 4006.3 QPS (1000/QPS 0.250 ms).
- Pass default@1 after default@8: 2026-10-07T22:38:08.180Z to 2026-10-07T22:38:28.181Z (20.0 s), 22,535 searches, 0 failed, p50 0.869 ms, mean 0.886 ms, p99 1.310 ms, 1126.7 QPS (1000/QPS 0.888 ms).
- Passes in the order run: exact, default@8, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 273,226 sent, 0 failed (warmupErrors). Index after the searches: ready, ? of 524 indexed (pg_indexes lists gvb_gvbbench_eshoponweb_hnsw (USING hnsw (embedding vector_cosine_ops) WITH (m='16', ef_construction='128')), pg_index valid and ready = True, EXPLAIN of the default search uses Index Scan on it = True, that search returned 10 of 10 rows; PostgreSQL keeps no entry count for an HNSW index, so indexed vectors are not reported).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-pgvector (0ed79a41bbac) on network gvb-pgvector_default at container-ip:5432, not through docker-proxy localhost:5432. Open after the passes: container-ip:5432 x8 (this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-pgvector: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-pgvector: 6 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): exact 3492/3492 MHz, average 3492.8/3492.7, performance, outside load 0.13 before the warm-up, 0.14 during, client CPU 0.685 ms per search, engine CPU 2.076 ms per search; default@8 3492/3492 MHz, average 3492.1/3492.1, performance, outside load 0.14 before the warm-up, 0.14 during, client CPU 0.542 ms per search, engine CPU 0.977 ms per search; default@1 3492/3492 MHz, average 3492.6/3492.2, performance, outside load 0.13 before the warm-up, 0.15 during, client CPU 0.693 ms per search, engine CPU 0.732 ms per search.

### opensearch

- Engine: OpenSearch 3.9.0 (k-NN plugin, faiss) (compose)
- Index: faiss HNSW float32, no compression, m=16, ef_construction=128, cosinesimil; search k=top, ef_search=100; 1 shard, 0 replicas; graph built at any segment size (approximate_threshold=0); force-merged to one segment after the load
- Search settings: approximate_threshold=0, ef_construction=128, ef_search=100, k=top, m=16
- Engine settings: HostConfig.Memory = 6442450944 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit)); heap_init_in_bytes = 2147483648 (read: GET /_nodes/jvm (nodes.<id>.jvm.mem.heap_init_in_bytes and heap_max_in_bytes)); heap_max_in_bytes = 2147483648 (read: GET /_nodes/jvm (nodes.<id>.jvm.mem.heap_init_in_bytes and heap_max_in_bytes))
- Image: opensearchproject/opensearch:3.9.0, id sha256:adfa61f85025d06b4aeb562e7e74fde7e31c437039c93c3862c17e9acebd6c7c (the pinned id), tagged on this machine at 2026-10-03T14:09:29.512875356Z
- Data folder ~/gvb-data/engines/opensearch: 3.92 MiB at the start, 13.49 MiB at the end
- Load: 524 rows in batches of 1000, 1.4 s of upserts (384 rows/s); count matched 0.1 s after the last upsert; index step 0.9 s (force-merged to one segment, graph ready after 0.9 s: every segment searched through its HNSW graph, covering 524 of 524 vectors: 1 segment(s), 0 merge(s) running, profiled probe search: ann_search_count 1 (segments tried through a graph), exact_search_count 0 (of those, scanned instead), approximate_threshold 0; force-merged to one segment on purpose, so one graph answers every search)
- Search: 9599 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 0.47 ms, 8 searchers 0.51 ms
- Exact mode: 22,308 searches, p50 2.66 ms, p95 2.96 ms, recall 1.000
- RAM: 2.56 GiB (docker stats); disk: 9.57 MiB added by this load (13.49 MiB (whole engine data folder), 3.92 MiB before)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started opensearch.compose.yaml with every container created on CPUs 2-3,6-7: gvb-opensearch cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [opensearch.compose.yaml 028b44c40129]. This run created the engine from these files.
- Index after the load: ready, 524 of 524 indexed (every segment searched through its HNSW graph, covering 524 of 524 vectors: 1 segment(s), 0 merge(s) running, profiled probe search: ann_search_count 1 (segments tried through a graph), exact_search_count 0 (of those, scanned instead), approximate_threshold 0; force-merged to one segment on purpose, so one graph answers every search). Durability: Every acknowledged bulk request is fsynced to the translog before the answer: index.translog.durability=request, the OpenSearch default, which neither this sink nor opensearch.compose.yaml overrides (OpenSearchReadinessTests reads it back from the live index). By that setting a process crash or power loss loses no acknowledged write; this is read from the setting, not shown by pulling power. One node and no replicas, so a lost disk loses the data..
- Outside load when searching began: 0.11 CPUs on average over the last 3 s and 0.11 over the last few seconds (limit 0.3), a quiet box. The load average 3.65 3.71 3.52 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: exact 8,102 searches in 30.0 s, default@8 36,743 searches in 30.0 s, default@1 14,268 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. exact: warm-up 15.0 s and 5,395 searches (0 failed) at 1 searcher; p50 of the last windows 2.693, 2.679, 2.687 ms (windows of at least 2 s and 100 searches); trial of 1,089 searches in 3.0 s at 1 searcher: p50 2.673 ms against the settled p50 2.687 ms, 1% apart (limit 10%); the timed pass p50 2.658 ms, 1% from the settled p50 2.687 ms (limit 10% or 0.1 ms; 0.029 ms); default@8: warm-up 15.0 s and 21,886 searches (0 failed) at 8 searchers; QPS of the last windows 1,458, 1,469, 1,466 (windows of at least 2 s and 100 searches); trial of 4,407 searches in 3.0 s at 8 searchers: 1,468 QPS against the settled 1,466 QPS, 0% apart (limit 10%); the timed pass 1,462 QPS, 0% from the settled 1,466 QPS (limit 10%); default@1: warm-up 15.0 s and 7,157 searches (0 failed) at 1 searcher; p50 of the last windows 2.068, 2.070, 2.072 ms (windows of at least 2 s and 100 searches); trial of 1,427 searches in 3.0 s at 1 searcher: p50 2.067 ms against the settled p50 2.070 ms, 0% apart (limit 10%); the timed pass p50 2.065 ms, 0% from the settled p50 2.070 ms (limit 10% or 0.1 ms; 0.006 ms).
- Pass exact after rehearsal: 2026-10-07T22:40:47.338Z to 2026-10-07T22:41:47.340Z (60.0 s), 22,308 searches, 0 failed, p50 2.658 ms, mean 2.688 ms, p99 3.665 ms, 371.8 QPS (1000/QPS 2.690 ms).
- Pass default@8 after exact: 2026-10-07T22:42:05.368Z to 2026-10-07T22:42:25.372Z (20.0 s), 29,250 searches, 0 failed, p50 5.257 ms, mean 5.468 ms, p99 10.651 ms, 1462.2 QPS (1000/QPS 0.684 ms).
- Pass default@1 after default@8: 2026-10-07T22:42:43.387Z to 2026-10-07T22:43:03.387Z (20.0 s), 9,599 searches, 0 failed, p50 2.065 ms, mean 2.082 ms, p99 2.826 ms, 479.9 QPS (1000/QPS 2.084 ms).
- Passes in the order run: exact, default@8, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 100,474 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (every segment searched through its HNSW graph, covering 524 of 524 vectors: 1 segment(s), 0 merge(s) running, profiled probe search: ann_search_count 1 (segments tried through a graph), exact_search_count 0 (of those, scanned instead), approximate_threshold 0; force-merged to one segment on purpose, so one graph answers every search).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-opensearch (c405cd3b80af) on network engines_default at container-ip:9200, not through docker-proxy localhost:9201. Open after the passes: container-ip:9200 x8 (this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-opensearch: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-opensearch: 55 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): exact 3492/3492 MHz, average 3492.2/3493.4, performance, outside load 0.12 before the warm-up, 0.15 during, client CPU 0.467 ms per search, engine CPU 2.412 ms per search; default@8 3492/3492 MHz, average 3492.0/3492.3, performance, outside load 0.15 before the warm-up, 0.11 during, client CPU 0.510 ms per search, engine CPU 2.645 ms per search; default@1 3492/3492 MHz, average 3492.2/3493.4, performance, outside load 0.13 before the warm-up, 0.13 during, client CPU 0.468 ms per search, engine CPU 1.789 ms per search.

### sql

- Engine: Microsoft SQL Server 2025 (RTM-CU9) (KB5122048) - 17.0.5005.3 (X64), Enterprise Developer Edition (64-bit), in container gvb-mssql (image mcr.microsoft.com/mssql/server:2025-CU9-ubuntu-24.04@sha256:2b5b581621126574f3d1f75e78d3eebe8d05aedb59ad0cfdf9aa42cb0634d726; cpuset 2-3,6-7 from its creation; SQL Server counts 4 CPU(s) and runs 4 visible scheduler(s), affinity AUTO) (compose)
- Index: exact VECTOR_DISTANCE cosine, no vector index (full scan)
- Search settings: description=exact VECTOR_DISTANCE cosine, no vector index (full scan)
- Engine settings: HostConfig.Memory = 8589934592 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit))
- Image: mcr.microsoft.com/mssql/server:2025-CU9-ubuntu-24.04@sha256:2b5b581621126574f3d1f75e78d3eebe8d05aedb59ad0cfdf9aa42cb0634d726, id sha256:2b5b581621126574f3d1f75e78d3eebe8d05aedb59ad0cfdf9aa42cb0634d726 (the pinned id), tagged on this machine at 2026-10-04T22:02:11.293843649Z
- Data folder ~/gvb-data/engines/mssql: 105.07 MiB at the start, 441.85 MiB at the end
- Load: 524 rows in batches of 1000, 0.9 s of upserts (605 rows/s); count matched 0.0 s after the last upsert; index step n/a
- Search: 4994 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 1.21 ms, 8 searchers 1.19 ms
- RAM: 531.1 MiB (docker stats); disk: 5.15 MiB (table and its indexes, reserved pages)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started mssql.compose.yaml with every container created on CPUs 2-3,6-7: gvb-mssql cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [mssql.compose.yaml b7359663cc15]. This run created the engine from these files.
- Index after the load: ready, 0 of 524 indexed (no index used, exact scan by design: dbo.gvb_gvbbench_eshoponweb in GvbBench holds 524 rows; sys.vector_indexes lists no index on the table; indexes: PK_gvb_gvbbench_eshoponweb CLUSTERED, IX_gvb_gvbbench_eshoponweb_DocKey NONCLUSTERED; a real search under SET STATISTICS XML ON ran: Clustered Index Scan of GvbBench.gvb_gvbbench_eshoponweb through PK_gvb_gvbbench_eshoponweb (IndexKind Clustered), 524 rows read, no vector index operator, 10 hits). Durability: A commit returns after its transaction-log records are written to disk (SQL Server write-ahead logging; delayed durability is DISABLED); a database created here copies the model database: recovery model FULL, page_verify CHECKSUM; no global trace flags are enabled; mssql.conf of container gvb-mssql (/var/opt/mssql/mssql.conf, on the host at ~/gvb-data/engines/mssql/mssql.conf) does not exist, so it sets: nothing; it has no [control] or [traceflag] entry, so SQL Server's own Linux defaults for flushing writes apply (Microsoft's Linux performance guide names trace flag 3982 as that default; read from the guide, not tested here). Not tested by cutting power; whether the disk's own write cache reaches the media was not checked. Container settings from its environment (names only): MSSQL_AGENT_ENABLED, MSSQL_MEMORY_LIMIT_MB, MSSQL_PID, MSSQL_RPC_PORT, MSSQL_SA_PASSWORD..
- Outside load when searching began: 0.17 CPUs on average over the last 3 s and 0.17 over the last few seconds (limit 0.3), a quiet box. The load average 2.80 3.39 3.43 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@8 16,810 searches in 30.0 s, default@1 7,505 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. default@8: warm-up 15.0 s and 9,314 searches (0 failed) at 8 searchers; QPS of the last windows 622, 603, 628 (windows of at least 2 s and 100 searches); trial of 1,841 searches in 3.0 s at 8 searchers: 612 QPS against the settled 622 QPS, 2% apart (limit 10%); the timed pass 607 QPS, 3% from the settled 622 QPS (limit 10%); default@1: warm-up 15.0 s and 3,742 searches (0 failed) at 1 searcher; p50 of the last windows 3.918, 3.914, 3.922 ms (windows of at least 2 s and 100 searches); trial of 755 searches in 3.0 s at 1 searcher: p50 3.914 ms against the settled p50 3.918 ms, 0% apart (limit 10%); the timed pass p50 3.926 ms, 0% from the settled p50 3.918 ms (limit 10% or 0.1 ms; 0.008 ms).
- Pass default@8 after rehearsal: 2026-10-07T22:44:33.027Z to 2026-10-07T22:44:53.033Z (20.0 s), 12,135 searches, 0 failed, p50 12.568 ms, mean 13.183 ms, p99 19.648 ms, 606.5 QPS (1000/QPS 1.649 ms).
- Pass default@1 after default@8: 2026-10-07T22:45:11.042Z to 2026-10-07T22:45:31.046Z (20.0 s), 4,994 searches, 0 failed, p50 3.926 ms, mean 4.002 ms, p99 4.944 ms, 249.7 QPS (1000/QPS 4.006 ms).
- Passes in the order run: default@8, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 39,967 sent, 0 failed (warmupErrors). Index after the searches: ready, 0 of 524 indexed (no index used, exact scan by design: dbo.gvb_gvbbench_eshoponweb in GvbBench holds 524 rows; sys.vector_indexes lists no index on the table; indexes: PK_gvb_gvbbench_eshoponweb CLUSTERED, IX_gvb_gvbbench_eshoponweb_DocKey NONCLUSTERED; a real search under SET STATISTICS XML ON ran: Clustered Index Scan of GvbBench.gvb_gvbbench_eshoponweb through PK_gvb_gvbbench_eshoponweb (IndexKind Clustered), 524 rows read, no vector index operator, 10 hits).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-mssql (784ab5129fa4) on network gvb-mssql_default at container-ip:1433, not through docker-proxy localhost:14330. Open after the passes: container-ip:1433 x9 (this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-mssql: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-mssql: 85 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): default@8 3492/3492 MHz, average 3492.0/3492.5, performance, outside load 0.16 before the warm-up, 0.16 during, client CPU 1.195 ms per search, engine CPU 6.099 ms per search; default@1 3492/3492 MHz, average 3493.2/3492.3, performance, outside load 0.15 before the warm-up, 0.14 during, client CPU 1.211 ms per search, engine CPU 3.864 ms per search.

### clickhouse

- Engine: ClickHouse 26.3.39.7 (MergeTree + vector_similarity index; image-default system logging, each log table with a TTL of 1 hour) (compose)
- Index: vector_similarity HNSW cosineDistance, quantization bf16, M=16 ef_construction=128, hnsw_candidate_list_size_for_search=256, rescoring off; exact mode = full scan with skip indexes off
- Search settings: M=16, ef_construction=128, hnsw_candidate_list_size_for_search=256
- Engine settings: HostConfig.Memory = 8589934592 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit)); max_threads = \'auto(4)\' (read: ClickHouse system.settings, name max_threads, over HTTP)
- Image: clickhouse/clickhouse-server:26.3.39.7, id sha256:3a91276f066905da0edbbd622d3fc2a2df632c87ea7fe32ef4e74fdf8c9567b0 (the pinned id), tagged on this machine at 2026-10-03T14:12:01.743437616Z
- Data folder ~/gvb-data/engines/clickhouse: 5.32 GiB at the start, 2.72 GiB after the reset, 6.9 GiB at the end; reset at the start: truncated 9 MergeTree log tables in database system (asynchronous_insert_log, asynchronous_metric_log, background_schedule_pool_log, metric_log, part_log, processors_profile_log, query_log, text_log, trace_log); ClickHouse's own active bytes of those tables 183.97 KiB before and 0 B after; data folder 5.32 GiB before and 2.72 GiB after (the folder was steady); not truncated, name does not end in _log: none
- Load: 524 rows in batches of 1000, 0.2 s of upserts (2,735 rows/s); count matched 0.0 s after the last upsert; index step 6.2 s (1 data part(s), index files on 1 of them, 524 of 524 live rows in indexed parts; 0 merge(s) running, 0 unfinished mutation(s); plan of the default search uses the Skip index vec_idx on 1 of 1 parts; wait took 6.2 s)
- Search: 3878 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 0.86 ms, 8 searchers 1.04 ms
- Exact mode: 8,569 searches, p50 6.63 ms, p95 10.0 ms, recall 1.000
- RAM: 797.7 MiB (docker stats); disk: 4.18 GiB added by this load (6.9 GiB (whole engine data folder), 2.72 GiB before)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started clickhouse.compose.yaml with every container created on CPUs 2-3,6-7: gvb-clickhouse cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [clickhouse-config/gvb-system-logs.xml 02cc53f0f171; clickhouse.compose.yaml 560e0ae487c6]. This run created the engine from these files.
- Index after the load: ready, 524 of 524 indexed (1 data part(s), index files on 1 of them, 524 of 524 live rows in indexed parts; 0 merge(s) running, 0 unfinished mutation(s); plan of the default search uses the Skip index vec_idx on 1 of 1 parts). Durability: Acknowledged inserts are not fsynced. The sink creates plain MergeTree tables, so the MergeTree settings are the defaults: fsync_after_insert 0 and fsync_part_directory 0 (system.merge_tree_settings), and async_insert 1 with wait_for_async_insert 1 is the server default for the gvb login (system.settings), so an insert is acknowledged once its part is written, not once it is on disk. Measured 2026-10-04 with strace over 30 acknowledged single-row inserts (30 Ok rows in system.asynchronous_insert_log): zero fsync, fdatasync or sync_file_range calls, while the control, a table created with fsync_after_insert=1, made 61 fdatasync calls for 5 inserts. A host power loss or kernel crash can lose acknowledged rows that are still in the OS page cache; a ClickHouse process crash alone should not (inferred, not tested)..
- Outside load when searching began: 0.16 CPUs on average over the last 28 s and 0.13 over the last few seconds (limit 0.3), a quiet box. The load average 1.51 2.85 3.24 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@1 5,990 searches in 30.0 s, default@8 13,063 searches in 30.0 s, exact 4,314 searches in 30.0 s.
- WARNING: latency had NOT settled when timing began for default@8 (each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it). default@1: warm-up 15.0 s and 2,959 searches (0 failed) at 1 searcher; p50 of the last windows 4.840, 4.828, 4.892 ms (windows of at least 2 s and 100 searches); trial of 611 searches in 3.0 s at 1 searcher: p50 4.814 ms against the settled p50 4.840 ms, 1% apart (limit 10%); the timed pass p50 4.869 ms, 1% from the settled p50 4.840 ms (limit 10% or 0.1 ms; 0.029 ms); default@8: warm-up 15.0 s and 6,659 searches (0 failed) at 8 searchers; QPS of the last windows 437, 469, 461 (windows of at least 2 s and 100 searches) (its last 3 windows differed by more than 5%); trial of 1,088 searches in 3.0 s at 8 searchers: 361 QPS against the settled 461 QPS, 28% apart (limit 10%), so the warm-up was EXTENDED once (30.0 s and 12,488 searches (0 failed) at 8 searchers; QPS of its older 6 windows 434 (windows 459, 453, 386, 410, 446, 452) and of its newer 6 432 (windows 460, 362, 440, 432, 454, 442), 0.6% apart (limit 5%) (windows of at least 2 s and 100 searches)); second trial of 985 searches in 3.0 s at 8 searchers: 327 QPS against the settled 432 QPS, 32% apart (limit 10%), outside its newer windows' range 362 to 460 QPS; still disagreeing after the extension; the timed pass 380 QPS, 13% from the settled 432 QPS (limit 10%), NOT HELD: the engine was still changing when it was timed; exact: warm-up 15.0 s and 1,913 searches (0 failed) at 1 searcher; p50 of the last windows 6.837, 6.787, 6.839 ms (windows of at least 2 s and 100 searches); trial of 400 searches in 3.0 s at 1 searcher: p50 6.790 ms against the settled p50 6.837 ms, 1% apart (limit 10%); the timed pass p50 6.629 ms, 3% from the settled p50 6.837 ms (limit 10% or 0.1 ms; 0.208 ms). Its numbers may still include warm-up or a change the engine was still going through; rerun before quoting them.
- Pass default@1 after rehearsal: 2026-10-07T22:47:55.060Z to 2026-10-07T22:48:15.061Z (20.0 s), 3,878 searches, 0 failed, p50 4.869 ms, mean 5.154 ms, p99 8.098 ms, 193.9 QPS (1000/QPS 5.158 ms).
- Pass default@8 after default@1: 2026-10-07T22:49:21.123Z to 2026-10-07T22:49:41.136Z (20.0 s), 7,611 searches, 0 failed, p50 20.408 ms, mean 21.026 ms, p99 38.988 ms, 380.3 QPS (1000/QPS 2.630 ms).
- Pass exact after default@8: 2026-10-07T22:49:59.149Z to 2026-10-07T22:50:59.152Z (60.0 s), 8,569 searches, 0 failed, p50 6.629 ms, mean 6.998 ms, p99 10.689 ms, 142.8 QPS (1000/QPS 7.002 ms).
- Passes in the order run: default@1, default@8, exact; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 56,904 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (1 data part(s), index files on 1 of them, 524 of 524 live rows in indexed parts; 0 merge(s) running, 0 unfinished mutation(s); plan of the default search uses the Skip index vec_idx on 1 of 1 parts).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-clickhouse (e4ab7ab19294) on network gvb-clickhouse_default at container-ip:8123, not through docker-proxy localhost:8123. Open after the passes: container-ip:8123 x1 (this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-clickhouse: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-clickhouse: 728 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): default@1 3492/3492 MHz, average 3492.0/3492.6, performance, outside load 0.17 before the warm-up, 0.20 during, client CPU 0.862 ms per search, engine CPU 7.411 ms per search; default@8 3492/3492 MHz, average 3492.0/3492.3, performance, outside load 0.15 before the warm-up, 0.19 during, client CPU 1.043 ms per search, engine CPU 10.439 ms per search; exact 3492/3492 MHz, average 3492.0/3492.7, performance, outside load 0.16 before the warm-up, 0.18 during, client CPU 0.895 ms per search, engine CPU 10.378 ms per search.

### typesense

- Engine: Typesense 30.2 (vector search) (compose)
- Index: HNSW float32 (hnswlib), m=16, ef_construction=128, cosine; search k=top, ef=100; exact mode = filter ordinal:>=0 with flat_search_cutoff; index held in memory
- Search settings: ef=100, ef_construction=128, k=top, m=16
- Engine settings: HostConfig.Memory = 6442450944 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit))
- Image: typesense/typesense:30.2, id sha256:610f2d34b1f93d00762869da2c67736775e5798d19a2c8b91b014b8a0cc1e110 (the pinned id), tagged on this machine at 2026-10-03T14:12:37.929033132Z
- Data folder ~/gvb-data/engines/typesense: 856.64 MiB at the start, 294.02 MiB at the end
- Load: 524 rows in batches of 1000, 0.5 s of upserts (1,006 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (nothing to build, Typesense inserts into the HNSW graph while writing; ready after 0.0 s: HNSW graph returns 524 of 524 vectors, no writes queued: num_documents 524, pending_write_batches 0, health ok, unfiltered vector query with k above the count returned 524; the graph is built in memory during each write, so there is no later build step)
- Search: 4748 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 0.52 ms, 8 searchers 0.58 ms
- Exact mode: 10,548 searches, p50 5.63 ms, p95 5.98 ms, recall 1.000
- RAM: 118.1 MiB (docker stats); disk: 8.1 MiB added by this load (294.02 MiB (whole engine data folder), 285.91 MiB before)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started typesense.compose.yaml with every container created on CPUs 2-3,6-7: gvb-typesense cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [typesense.compose.yaml 96a165f62566]. This run created the engine from these files.
- Index after the load: ready, 524 of 524 indexed (HNSW graph returns 524 of 524 vectors, no writes queued: num_documents 524, pending_write_batches 0, health ok, unfiltered vector query with k above the count returned 524; the graph is built in memory during each write, so there is no later build step). Durability: Every acknowledged write is appended to Typesense's raft log and fsynced before the answer: braft raft_sync=true with raft_sync_policy=0 (sync immediately), read from the running container's brpc /flags page on 2026-10-04. A restart replays the log from the last snapshot (typesense.compose.yaml sets TYPESENSE_SNAPSHOT_INTERVAL_SECONDS=300) and rebuilds the in-memory HNSW graph, so by those settings a crash loses no acknowledged write, but the restart is slow after a big load. This is read from the settings, not shown by pulling power. One node and no replicas, so a lost disk loses the data..
- Outside load when searching began: 0.18 CPUs on average over the last 7 s and 0.18 over the last few seconds (limit 0.3), a quiet box. The load average 4.34 4.90 4.09 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: exact 5,265 searches in 30.0 s, default@1 7,128 searches in 30.0 s, default@8 17,252 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. exact: warm-up 15.0 s and 2,637 searches (0 failed) at 1 searcher; p50 of the last windows 5.621, 5.644, 5.630 ms (windows of at least 2 s and 100 searches); trial of 529 searches in 3.0 s at 1 searcher: p50 5.633 ms against the settled p50 5.630 ms, 0% apart (limit 10%); the timed pass p50 5.630 ms, 0% from the settled p50 5.630 ms (limit 10% or 0.1 ms; 0.000 ms); default@1: warm-up 15.0 s and 3,560 searches (0 failed) at 1 searcher; p50 of the last windows 4.158, 4.157, 4.171 ms (windows of at least 2 s and 100 searches); trial of 707 searches in 3.0 s at 1 searcher: p50 4.162 ms against the settled p50 4.158 ms, 0% apart (limit 10%); the timed pass p50 4.163 ms, 0% from the settled p50 4.158 ms (limit 10% or 0.1 ms; 0.005 ms); default@8: warm-up 15.0 s and 8,634 searches (0 failed) at 8 searchers; QPS of the last windows 576, 576, 576 (windows of at least 2 s and 100 searches); trial of 1,726 searches in 3.0 s at 8 searchers: 574 QPS against the settled 576 QPS, 0% apart (limit 10%); the timed pass 576 QPS, 0% from the settled 576 QPS (limit 10%).
- Pass exact after rehearsal: 2026-10-07T22:53:07.878Z to 2026-10-07T22:54:07.882Z (60.0 s), 10,548 searches, 0 failed, p50 5.630 ms, mean 5.687 ms, p99 6.599 ms, 175.8 QPS (1000/QPS 5.689 ms).
- Pass default@1 after exact: 2026-10-07T22:54:25.898Z to 2026-10-07T22:54:45.900Z (20.0 s), 4,748 searches, 0 failed, p50 4.163 ms, mean 4.211 ms, p99 5.010 ms, 237.4 QPS (1000/QPS 4.213 ms).
- Pass default@8 after default@1: 2026-10-07T22:55:03.918Z to 2026-10-07T22:55:23.927Z (20.0 s), 11,531 searches, 0 failed, p50 13.367 ms, mean 13.877 ms, p99 25.585 ms, 576.3 QPS (1000/QPS 1.735 ms).
- Passes in the order run: exact, default@1, default@8; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 47,438 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (HNSW graph returns 524 of 524 vectors, no writes queued: num_documents 524, pending_write_batches 0, health ok, unfiltered vector query with k above the count returned 524; the graph is built in memory during each write, so there is no later build step).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-typesense (3ec2de30be8a) on network engines_default at container-ip:8108, not through docker-proxy localhost:8108. Open after the passes: container-ip:8108 x8 (this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-typesense: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-typesense: 303 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): exact 3492/3492 MHz, average 3493.0/3493.4, performance, outside load 0.05 before the warm-up, 0.15 during, client CPU 0.520 ms per search, engine CPU 5.334 ms per search; default@1 3492/3492 MHz, average 3492.7/3493.5, performance, outside load 0.15 before the warm-up, 0.14 during, client CPU 0.520 ms per search, engine CPU 3.868 ms per search; default@8 3492/3492 MHz, average 3492.0/3492.6, performance, outside load 0.14 before the warm-up, -0.03 during, client CPU 0.580 ms per search, engine CPU 6.704 ms per search.

### chroma

- Engine: Chroma 1.4.4 (single node, REST API v2) (compose)
- Index: HNSW M=16 ef_construction=128, ef_search=100 (Chroma default), cosine; approximate only (no exact mode)
- Search settings: M=16, ef_construction=128, ef_search=100
- Engine settings: HostConfig.Memory = 6442450944 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit)); chroma_version = 1.0.0 (read: GET /api/v2/version)
- Image: chromadb/chroma:latest, id sha256:1e0b73a187a28757c572acba508c46f48c9e8b0acaf5c20e6d95cdedce1acdf6 (the pinned id), tagged on this machine at 2026-10-03T14:09:56.392230738Z
- Data folder ~/gvb-data/engines/chroma: 2.01 GiB at the start, 2.01 GiB at the end
- Load: 524 rows in batches of 1000, 0.6 s of upserts (878 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (Chroma has no index build to wait for and exposes no index status; checked in 0.0 s: Chroma exposes no index status (indexing_status endpoint: HTTP 500 {"error":"InternalError","message":"Method scout_logs is not implemented"}), so indexed vectors are not reported and ready only means the two counts agree to within 1 percent: count endpoint 524, a vector search asking for 524 results returned 524)
- Search: 9353 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 0.48 ms, 8 searchers 0.47 ms
- RAM: 49 MiB (docker stats); disk: 414.16 KiB added by this load (2.01 GiB (whole engine data folder), 2.01 GiB before)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started chroma.compose.yaml with every container created on CPUs 2-3,6-7: gvb-chroma cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [chroma.compose.yaml f7ed9d3c8ff7]. This run created the engine from these files.
- Index after the load: ready, ? of 524 indexed (Chroma exposes no index status (indexing_status endpoint: HTTP 500 {"error":"InternalError","message":"Method scout_logs is not implemented"}), so indexed vectors are not reported and ready only means the two counts agree to within 1 percent: count endpoint 524, a vector search asking for 524 results returned 524). Durability: SQLite rollback journal with its default synchronous=FULL under the data directory (chroma.compose.yaml sets IS_PERSISTENT=1 and PERSIST_DIRECTORY=/data and no sync setting): every write is committed to chroma.sqlite3 with fsync of the journal, the directory and the database file before the call returns (strace: 15 database fsyncs and 38 journal fsyncs for one create, four 500-vector upserts and one delete), and the HNSW files are written every sync_threshold=1000 vectors and rebuilt from the SQLite log after a crash; measured with kill -9 and a restart: all 50, 300 and 2,000 vectors were still stored and searchable.
- Outside load when searching began: 0.08 CPUs on average over the last 1 s and 0.08 over the last few seconds (limit 0.3), a quiet box. The load average 3.87 3.85 3.81 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@8 23,472 searches in 30.0 s, default@1 14,158 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. default@8: warm-up 15.0 s and 11,742 searches (0 failed) at 8 searchers; QPS of the last windows 780, 782, 787 (windows of at least 2 s and 100 searches); trial of 2,354 searches in 3.0 s at 8 searchers: 782 QPS against the settled 782 QPS, 0% apart (limit 10%); the timed pass 779 QPS, 0% from the settled 782 QPS (limit 10%); default@1: warm-up 15.0 s and 7,003 searches (0 failed) at 1 searcher; p50 of the last windows 2.111, 2.122, 2.118 ms (windows of at least 2 s and 100 searches); trial of 1,408 searches in 3.0 s at 1 searcher: p50 2.115 ms against the settled p50 2.118 ms, 0% apart (limit 10%); the timed pass p50 2.117 ms, 0% from the settled p50 2.118 ms (limit 10% or 0.1 ms; 0.002 ms).
- Pass default@8 after rehearsal: 2026-10-07T22:56:51.752Z to 2026-10-07T22:57:11.760Z (20.0 s), 15,577 searches, 0 failed, p50 10.204 ms, mean 10.272 ms, p99 12.268 ms, 778.5 QPS (1000/QPS 1.284 ms).
- Pass default@1 after default@8: 2026-10-07T22:57:29.769Z to 2026-10-07T22:57:49.771Z (20.0 s), 9,353 searches, 0 failed, p50 2.117 ms, mean 2.137 ms, p99 2.616 ms, 467.6 QPS (1000/QPS 2.139 ms).
- Passes in the order run: default@8, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 60,137 sent, 0 failed (warmupErrors). Index after the searches: ready, ? of 524 indexed (Chroma exposes no index status (indexing_status endpoint: HTTP 500 {"error":"InternalError","message":"Method scout_logs is not implemented"}), so indexed vectors are not reported and ready only means the two counts agree to within 1 percent: count endpoint 524, a vector search asking for 524 results returned 524).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-chroma (4dbc2a779e9a) on network gvb-chroma_default at container-ip:8000, not through docker-proxy localhost:8000. Open after the passes: container-ip:8000 x8 (this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-chroma: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-chroma: 9 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): default@8 3492/3492 MHz, average 3492.0/3492.6, performance, outside load 0.10 before the warm-up, 0.08 during, client CPU 0.473 ms per search, engine CPU 2.743 ms per search; default@1 3492/3492 MHz, average 3492.0/3493.1, performance, outside load 0.10 before the warm-up, 0.09 during, client CPU 0.482 ms per search, engine CPU 2.496 ms per search.

### milvus

- Engine: Milvus 2.6.25 (standalone, embedded etcd, local storage, REST API v2) (compose)
- Index: HNSW M=16 efConstruction=128, ef=100, metric COSINE, Strong consistency searches; approximate only (no exact mode)
- Search settings: M=16, ef=100, efConstruction=128
- Engine settings: HostConfig.Memory = 8589934592 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit))
- Image: milvusdb/milvus:v2.6.25, id sha256:29f7668e64df1c6d5cdadbfbee99f38c72e4001054f149a298725acae57230ad (the pinned id), tagged on this machine at 2026-10-03T14:05:28.412151323Z
- Data folder ~/gvb-data/engines/milvus: 226.87 MiB at the start, 237.74 MiB at the end
- Load: 524 rows in batches of 1000, 0.4 s of upserts (1,182 rows/s); count matched 1.7 s after the last upsert; index step 138.5 s (index Finished, type HNSW (COSINE, {"M":16,"efConstruction":128}), indexedRows 524 of 524 sealed rows, pendingRows 0; stored rows 524; LoadStateLoaded; query node: 1 sealed segment(s) with the index loaded covering 524 rows against 524 stored rows, 0 sealed without it, 0 rows in growing segments; datacoord: 1 live segment(s), 1 flushed and indexed covering 524 rows, 0 delete-log (L0) segment(s) waiting for a compaction, 3 segment(s) compacted away so far; flush, index build and load took 14.6 s, then compaction: 1 job(s) accepted by Milvus, layout ready and unchanged for 75 s, 123.9 s for the whole step; search ledger started, later state reads count the searches against the query node's own counters)
- Search: 8542 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 0.55 ms, 8 searchers 0.63 ms
- RAM: 190.9 MiB (docker stats); disk: 10.86 MiB added by this load (237.74 MiB (whole engine data folder), 226.87 MiB before)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started milvus.compose.yaml with every container created on CPUs 2-3,6-7: gvb-milvus cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [milvus.compose.yaml eaefecdc3671]. This run created the engine from these files.
- Index after the load: ready, 524 of 524 indexed (index Finished, type HNSW (COSINE, {"M":16,"efConstruction":128}), indexedRows 524 of 524 sealed rows, pendingRows 0; stored rows 524; LoadStateLoaded; query node: 1 sealed segment(s) with the index loaded covering 524 rows against 524 stored rows, 0 sealed without it, 0 rows in growing segments; datacoord: 1 live segment(s), 1 flushed and indexed covering 524 rows, 0 delete-log (L0) segment(s) waiting for a compaction, 3 segment(s) compacted away so far; search ledger since the index finished: the sink sent 0 searches (searchParams.params.ef=100, Strong consistency), the query node counted 0 search requests for this collection, segment searches Sealed +0 and Growing +0, the same sealed segments with the same loaded index builds at both readings, segments compacted away +0). Durability: Writes are not fsynced. Standalone uses the default message queue rocksmq (RocksDB, /var/lib/milvus/rdb_data, mq.type default) and flushed segments go to local-disk object storage (COMMON_STORAGETYPE=local in milvus.compose.yaml). Measured 2026-10-04 with strace over 30 acknowledged single-row upserts and one flush: fdatasync ran only on the embedded etcd files (member/wal, member/snap/db), never on rdb_data or the segment files. A host power loss or kernel crash can lose acknowledged rows that are still in the OS page cache; a Milvus process crash alone should not (inferred, not tested). Metadata in embedded etcd is fdatasynced on every commit..
- Outside load when searching began: 0.14 CPUs on average over the last 60 s and 0.11 over the last few seconds (limit 0.3), a quiet box. The load average 0.48 2.05 3.08 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@1 12,783 searches in 30.0 s, default@8 38,171 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. default@1: warm-up 15.0 s and 6,383 searches (0 failed) at 1 searcher; p50 of the last windows 2.228, 2.222, 2.285 ms (windows of at least 2 s and 100 searches); trial of 1,268 searches in 3.0 s at 1 searcher: p50 2.244 ms against the settled p50 2.228 ms, 1% apart (limit 10%); the timed pass p50 2.230 ms, 0% from the settled p50 2.228 ms (limit 10% or 0.1 ms; 0.002 ms); default@8: warm-up 15.0 s and 19,038 searches (0 failed) at 8 searchers; QPS of the last windows 1,274, 1,278, 1,268 (windows of at least 2 s and 100 searches); trial of 3,820 searches in 3.0 s at 8 searchers: 1,272 QPS against the settled 1,274 QPS, 0% apart (limit 10%); the timed pass 1,274 QPS, 0% from the settled 1,274 QPS (limit 10%).
- Pass default@1 after rehearsal: 2026-10-07T23:01:38.150Z to 2026-10-07T23:01:58.152Z (20.0 s), 8,542 searches, 0 failed, p50 2.230 ms, mean 2.339 ms, p99 3.672 ms, 427.1 QPS (1000/QPS 2.342 ms).
- Pass default@8 after default@1: 2026-10-07T23:02:16.168Z to 2026-10-07T23:02:36.172Z (20.0 s), 25,485 searches, 0 failed, p50 5.725 ms, mean 6.276 ms, p99 14.465 ms, 1274.0 QPS (1000/QPS 0.785 ms).
- Passes in the order run: default@1, default@8; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 81,463 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (index Finished, type HNSW (COSINE, {"M":16,"efConstruction":128}), indexedRows 524 of 524 sealed rows, pendingRows 0; stored rows 524; LoadStateLoaded; query node: 1 sealed segment(s) with the index loaded covering 524 rows against 524 stored rows, 0 sealed without it, 0 rows in growing segments; datacoord: 1 live segment(s), 1 flushed and indexed covering 524 rows, 0 delete-log (L0) segment(s) waiting for a compaction, 3 segment(s) compacted away so far; search ledger since the index finished: the sink sent 115490 searches (searchParams.params.ef=100, Strong consistency), the query node counted 115490 search requests for this collection, segment searches Sealed +113526 and Growing +0, the same sealed segments with the same loaded index builds at both readings, segments compacted away +0).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-milvus (fb95a6c673c3) on network gvb-milvus_default at container-ip:19530, not through docker-proxy localhost:19530; container gvb-milvus (fb95a6c673c3) on network gvb-milvus_default at container-ip:9091, not through docker-proxy localhost:9091. Open after the passes: container-ip:19530 x8 (this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-milvus: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-milvus: 24 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): default@1 3492/3492 MHz, average 3492.0/3492.4, performance, outside load 0.15 before the warm-up, 0.12 during, client CPU 0.549 ms per search, engine CPU 2.718 ms per search; default@8 3492/3492 MHz, average 3492.0/3492.3, performance, outside load 0.15 before the warm-up, 0.16 during, client CPU 0.633 ms per search, engine CPU 2.953 ms per search.

### mariadb

- Engine: MariaDB 11.8.9 (InnoDB + VECTOR INDEX) (compose)
- Index: VECTOR INDEX (HNSW variant) DISTANCE=cosine, M=16 (no ef_construction setting exists), mhnsw_ef_search=100 per statement (the ef 100 most engines here use, so the search effort matches; MariaDB's own default is 20; recall@10 at ef 100 falls as the set grows (random 1024-dimension vectors, measured 2026-10-04: 0.99 at 524, 0.89 to 0.92 at 2,000; an earlier run gave about 0.09 at 100,000)), mhnsw_max_cache_size 4G; exact mode = IGNORE INDEX full scan
- Search settings: DISTANCE=cosine, M=16, mhnsw_ef_search=100
- Engine settings: HostConfig.Memory = 8589934592 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit)); innodb_buffer_pool_size = 2147483648 (read: docker exec gvbbench-mariadb mariadb SHOW GLOBAL VARIABLES); innodb_flush_log_at_trx_commit = 2 (read: docker exec gvbbench-mariadb mariadb SHOW GLOBAL VARIABLES); mhnsw_max_cache_size = 4294967296 (read: docker exec gvbbench-mariadb mariadb SHOW GLOBAL VARIABLES)
- Image: mariadb:11.8.9, id sha256:6422478cb8e159f080fb1d8ccf65101e26fe51385787fde7d16c3b165a331f15 (the pinned id), tagged on this machine at 2026-10-03T14:10:11.675870329Z
- Data folder ~/gvb-data/engines/mariadb-bench: 166.77 MiB at the start, 188.79 MiB at the end
- Load: 524 rows in batches of 1000, 0.5 s of upserts (1,006 rows/s); count matched 0.0 s after the last upsert; index step 0.5 s (VECTOR INDEX is maintained inside every INSERT, nothing to build; confirmed after 2 poll(s) in 0.5 s (poll 1 of 2 said: not ready: a search at mhnsw_ef_search 100 returned only 397 of 524 requested rows through the index (EXPLAIN key 'vec_idx') (information_schema lists vec_idx as INDEX_TYPE VECTOR, EXPLAIN of the default search uses key vec_idx, the server applied mhnsw_ef_search 100 (asked for 100, read back from the server), a search at that effort returned 397 of 524 requested rows through the index; MariaDB has no per-index row count (the graph is a hidden InnoDB table), so indexed vectors are not reported)): information_schema lists vec_idx as INDEX_TYPE VECTOR, EXPLAIN of the default search uses key vec_idx, the server applied mhnsw_ef_search 100 (asked for 100, read back from the server), a search at that effort returned 518 of 524 requested rows through the index; MariaDB has no per-index row count (the graph is a hidden InnoDB table), so indexed vectors are not reported)
- Search: 31265 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 0.48 ms, 8 searchers 0.37 ms
- Exact mode: 36,367 searches, p50 1.62 ms, p95 1.73 ms, recall 1.000
- RAM: 164.9 MiB (docker stats); disk: 22.01 MiB added by this load (188.79 MiB (whole engine data folder), 166.78 MiB before)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started mariadb-bench.compose.yaml with every container created on CPUs 2-3,6-7: gvbbench-mariadb cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [mariadb-bench.compose.yaml e9a2147e6f91]. This run created the engine from these files.
- Index after the load: ready, ? of 524 indexed (information_schema lists vec_idx as INDEX_TYPE VECTOR, EXPLAIN of the default search uses key vec_idx, the server applied mhnsw_ef_search 100 (asked for 100, read back from the server), a search at that effort returned 518 of 524 requested rows through the index; MariaDB has no per-index row count (the graph is a hidden InnoDB table), so indexed vectors are not reported). Durability: innodb_flush_log_at_trx_commit=2 (set in mariadb-bench.compose.yaml for the benchmark's own container gvbbench-mariadb, and in mariadb.compose.yaml for the daily gvb-mariadb, which has the same server settings): the InnoDB redo log is written to the operating system at every commit but fsynced about once a second, so a crash of the mariadbd process loses nothing, while an operating-system crash or power cut can lose the last second of commits; innodb_doublewrite is on (default), the binary log is off, and the vector graph is an InnoDB table under the same log (settings read from the running server with SHOW VARIABLES; the crash behaviour is InnoDB's documented behaviour for this setting and was not tested on either container).
- Outside load when searching began: 0.13 CPUs on average over the last 1 s and 0.13 over the last few seconds (limit 0.3), a quiet box. The load average 5.16 3.20 3.35 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@8 179,988 searches in 30.0 s, exact 18,148 searches in 30.0 s, default@1 46,978 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. default@8: warm-up 15.0 s and 89,342 searches (0 failed) at 8 searchers; QPS of the last windows 5,966, 5,988, 5,989 (windows of at least 2 s and 100 searches); trial of 17,763 searches in 3.0 s at 8 searchers: 5,919 QPS against the settled 5,988 QPS, 1% apart (limit 10%); the timed pass 5,926 QPS, 1% from the settled 5,988 QPS (limit 10%); exact: warm-up 15.0 s and 9,031 searches (0 failed) at 1 searcher; p50 of the last windows 1.658, 1.653, 1.620 ms (windows of at least 2 s and 100 searches); trial of 1,828 searches in 3.0 s at 1 searcher: p50 1.621 ms against the settled p50 1.653 ms, 2% apart (limit 10%); the timed pass p50 1.625 ms, 2% from the settled p50 1.653 ms (limit 10% or 0.1 ms; 0.029 ms); default@1: warm-up 15.0 s and 23,268 searches (0 failed) at 1 searcher; p50 of the last windows 0.631, 0.618, 0.636 ms (windows of at least 2 s and 100 searches); trial of 4,652 searches in 3.0 s at 1 searcher: p50 0.634 ms against the settled p50 0.631 ms, 0% apart (limit 10%); the timed pass p50 0.628 ms, 1% from the settled p50 0.631 ms (limit 10% or 0.1 ms; 0.003 ms).
- Pass default@8 after rehearsal: 2026-10-07T23:04:35.342Z to 2026-10-07T23:04:55.343Z (20.0 s), 118,529 searches, 0 failed, p50 1.304 ms, mean 1.348 ms, p99 2.491 ms, 5926.2 QPS (1000/QPS 0.169 ms).
- Pass exact after default@8: 2026-10-07T23:05:13.392Z to 2026-10-07T23:06:13.393Z (60.0 s), 36,367 searches, 0 failed, p50 1.625 ms, mean 1.648 ms, p99 2.082 ms, 606.1 QPS (1000/QPS 1.650 ms).
- Pass default@1 after exact: 2026-10-07T23:06:31.434Z to 2026-10-07T23:06:51.435Z (20.0 s), 31,265 searches, 0 failed, p50 0.628 ms, mean 0.638 ms, p99 0.893 ms, 1563.2 QPS (1000/QPS 0.640 ms).
- Passes in the order run: default@8, exact, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 390,998 sent, 0 failed (warmupErrors). Index after the searches: ready, ? of 524 indexed (information_schema lists vec_idx as INDEX_TYPE VECTOR, EXPLAIN of the default search uses key vec_idx, the server applied mhnsw_ef_search 100 (asked for 100, read back from the server), a search at that effort returned 518 of 524 requested rows through the index; MariaDB has no per-index row count (the graph is a hidden InnoDB table), so indexed vectors are not reported).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvbbench-mariadb (4dbb97b211ba) on network gvbbench-mariadb_default at container-ip:3306, not through docker-proxy localhost:13306. Open after the passes: container-ip:3306 x8 (this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvbbench-mariadb: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvbbench-mariadb: 14 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): default@8 3492/3492 MHz, average 3492.0/3492.0, performance, outside load 0.14 before the warm-up, 0.12 during, client CPU 0.374 ms per search, engine CPU 0.621 ms per search; exact 3492/3492 MHz, average 3494.9/3492.6, performance, outside load 0.12 before the warm-up, 0.13 during, client CPU 0.493 ms per search, engine CPU 1.448 ms per search; default@1 3492/3492 MHz, average 3494.9/3492.7, performance, outside load 0.13 before the warm-up, 0.19 during, client CPU 0.475 ms per search, engine CPU 0.454 ms per search.

### duckdb

- Engine: DuckDB 1.5.6 + vss b833341 (HNSW, embedded, in process) (embedded)
- Index: HNSW (vss extension) FLOAT[n] metric=cosine m=16 ef_construction=128, ef_search=100 per connection, persistent (hnsw_enable_experimental_persistence=true, checkpoint_threshold=256MB); exact mode = array_cosine_similarity sequential scan; score = 1 - cosine distance; searches run concurrently, one connection per searcher (opened as searchers arrive, at most 32), writes run one at a time and never overlap a search
- Search settings: checkpoint_threshold=256MB, ef_construction=128, ef_search=100, hnsw_enable_experimental_persistence=true, m=16, metric=cosine
- Engine settings: duckdb_version = v1.5.6 (read: SELECT version() on an in-memory DuckDB connection); threads = 8 (read: SELECT current_setting('threads') on an in-memory DuckDB connection; DuckDB's own thread count, not a CPU cap (it counts all CPUs and ignores affinity)); hnsw_ef_search = 100 (set: src/GenericVectorBuilder.Engines/Sinks/DuckDbSink.cs#SET hnsw_ef_search); hnsw_enable_experimental_persistence = true (set: src/GenericVectorBuilder.Engines/Sinks/DuckDbSink.cs#SET hnsw_enable_experimental_persistence = true); memory_limit = 7.4 GiB (read: SELECT current_setting('memory_limit') on a second connection to the bench file opened with the sink's own connection string); checkpoint_threshold = 244.1 MiB (read: SELECT current_setting('checkpoint_threshold') on a second connection to the bench file opened with the sink's own connection string)
- Data folder ~/gvb-data/engines/duckdb: 1.26 GiB at the start, 1.26 GiB at the end; the benchmark's own database file 2.9 MiB
- Load: 524 rows in batches of 1000, 0.3 s of upserts (1,563 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (HNSW is maintained inside every transaction, nothing to build; confirmed in 0.01 s: duckdb_indexes() lists gvb_gvbbench_eshoponweb_hnsw, pragma_hnsw_index_info() counts 524 vectors of 524 rows, EXPLAIN of the default search shows HNSW_INDEX_SCAN on it = True)
- Search: 5475 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 4.98 ms, 8 searchers 6.89 ms
- Exact mode: 14,230 searches, p50 4.14 ms, p95 4.88 ms, recall 1.000
- RAM: in this process, not measured; disk: 2.9 MiB added by this load (1.26 GiB (engine data folder), 1.26 GiB before)
- Index after the load: ready, 524 of 524 indexed (duckdb_indexes() lists gvb_gvbbench_eshoponweb_hnsw, pragma_hnsw_index_info() counts 524 vectors of 524 rows, EXPLAIN of the default search shows HNSW_INDEX_SCAN on it = True). Durability: DuckDB's write-ahead log is fsynced at every commit and no DuckDbSink setting changes that (checkpoint_threshold=256MB only spaces out the checkpoints that write the database file; measured with strace: 42 WAL fsyncs for 40 single-record commits plus 2 setup statements), so an operating-system crash or power cut loses no committed row; measured with kill -9 right after the last commit and a reopen: all 50, 300 and 2,000 rows and an HNSW index that counted the same number were recovered from the log; not tested and documented by DuckDB: the HNSW index is file-backed only through hnsw_enable_experimental_persistence = true, which this sink turns on, and WAL recovery for such custom indexes is not complete, so a crash during a checkpoint or a later commit can damage the index while the rows survive.
- Outside load when searching began: not known yet (too little sampled history). Load average 2.54 3.39 3.45 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@1 8,360 searches in 30.0 s, default@8 17,528 searches in 30.0 s, exact 7,152 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. default@1: warm-up 15.0 s and 4,113 searches (0 failed) at 1 searcher; p50 of the last windows 3.613, 3.610, 3.585 ms (windows of at least 2 s and 100 searches); trial of 822 searches in 3.0 s at 1 searcher: p50 3.613 ms against the settled p50 3.610 ms, 0% apart (limit 10%); the timed pass p50 3.616 ms, 0% from the settled p50 3.610 ms (limit 10% or 0.1 ms; 0.006 ms); default@8: warm-up 15.0 s and 8,709 searches (0 failed) at 8 searchers; QPS of the last windows 583, 579, 580 (windows of at least 2 s and 100 searches); trial of 1,734 searches in 3.0 s at 8 searchers: 577 QPS against the settled 580 QPS, 1% apart (limit 10%); the timed pass 578 QPS, 0% from the settled 580 QPS (limit 10%); exact: warm-up 15.0 s and 3,606 searches (0 failed) at 1 searcher; p50 of the last windows 4.120, 4.125, 4.133 ms (windows of at least 2 s and 100 searches); trial of 714 searches in 3.0 s at 1 searcher: p50 4.119 ms against the settled p50 4.125 ms, 0% apart (limit 10%); the timed pass p50 4.141 ms, 0% from the settled p50 4.125 ms (limit 10% or 0.1 ms; 0.016 ms).
- Pass default@1 after rehearsal: 2026-10-07T23:08:41.660Z to 2026-10-07T23:09:01.662Z (20.0 s), 5,475 searches, 0 failed, p50 3.616 ms, mean 3.651 ms, p99 5.167 ms, 273.7 QPS (1000/QPS 3.653 ms).
- Pass default@8 after default@1: 2026-10-07T23:09:19.685Z to 2026-10-07T23:09:39.695Z (20.0 s), 11,565 searches, 0 failed, p50 13.096 ms, mean 13.831 ms, p99 28.784 ms, 577.9 QPS (1000/QPS 1.730 ms).
- Pass exact after default@8: 2026-10-07T23:09:57.705Z to 2026-10-07T23:10:57.707Z (60.0 s), 14,230 searches, 0 failed, p50 4.141 ms, mean 4.213 ms, p99 5.689 ms, 237.2 QPS (1000/QPS 4.217 ms).
- Passes in the order run: default@1, default@8, exact; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 52,738 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (duckdb_indexes() lists gvb_gvbbench_eshoponweb_hnsw, pragma_hnsw_index_info() counts 524 vectors of 524 rows, EXPLAIN of the default search shows HNSW_INDEX_SCAN on it = True).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: in this process (embedded), no network: no network address. No open connection seen (no open TCP connection was found).
- CPU pinning: none: an embedded engine runs inside the benchmark client process, so it shares the client CPUs, CPUs 0-1,4-5.
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): default@1 3492/3492 MHz, average 3494.8/3492.1, performance, outside load 0.14 before the warm-up, 0.11 during, client CPU 4.985 ms per search; default@8 3494/3492 MHz, average 3495.0/3492.0, performance, outside load 0.13 before the warm-up, 0.14 during, client CPU 6.889 ms per search; exact 3493/3492 MHz, average 3495.0/3492.2, performance, outside load 0.13 before the warm-up, 0.14 during, client CPU 6.190 ms per search.

### redis

- Engine: Redis 8.10.2 (query engine, HASH + vector index) (compose)
- Index: HNSW TYPE FLOAT32 M=16 EF_CONSTRUCTION=128, EF_RUNTIME=100 per query, cosine; exact mode = FLAT index built on first exact query
- Search settings: EF_CONSTRUCTION=128, EF_RUNTIME=100, M=16
- Engine settings: HostConfig.Memory = 12884901888 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit)); search-workers = 8 (read: docker exec gvb-redis redis-cli CONFIG GET search-workers); save = 300 1 (read: docker exec gvb-redis redis-cli CONFIG GET save); appendonly = no (read: docker exec gvb-redis redis-cli CONFIG GET appendonly); maxmemory = 0 (read: docker exec gvb-redis redis-cli CONFIG GET maxmemory)
- Image: redis:8.10.2, id sha256:6f81e8915c60b065a524e6967e0ad1c639ba6efa84d669f823683ea04d9150ee (the pinned id), tagged on this machine at 2026-10-03T14:09:34.185138197Z
- Data folder ~/gvb-data/engines/redis: 94.22 MiB at the start, 96.65 MiB at the end
- Load: 524 rows in batches of 1000, 0.0 s of upserts (11,254 rows/s); count matched 1.0 s after the last upsert; index step 0.0 s (HNSW index finished after 0.0 s of waiting: FT.INFO indexing 0, percent_indexed 1, hash_indexing_failures 0, flat_buffer_size 0, num_docs 524, hashes counted with SCAN under gvb_gvbbench_eshoponweb: 524)
- Search: 51741 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 0.55 ms, 8 searchers 0.36 ms
- Exact mode: 164,245 searches, p50 0.37 ms, p95 0.46 ms, recall 1.000
- RAM: 253.1 MiB (docker stats); disk: 2.43 MiB added by this load (96.65 MiB (whole engine data folder), 94.22 MiB before)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started redis.compose.yaml with every container created on CPUs 2-3,6-7: gvb-redis cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [redis.compose.yaml 7448634e0d92]. This run created the engine from these files.
- Index after the load: ready, 524 of 524 indexed (FT.INFO indexing 0, percent_indexed 1, hash_indexing_failures 0, flat_buffer_size 0, num_docs 524, hashes counted with SCAN under gvb_gvbbench_eshoponweb: 524). Durability: save "300 1" and appendonly no (redis.compose.yaml): an RDB snapshot is written every 5 minutes if at least one key changed, and on a clean stop; there is no append-only log, so a crash loses every write since the last snapshot (up to 5 minutes plus the time a snapshot takes; measured with docker kill, which is SIGKILL, right after a 2,000-vector load: none of the 2,000 hashes were there after the restart, and the restart loaded an older snapshot that still held keys of a collection that had been dropped since).
- Outside load when searching began: 0.18 CPUs on average over the last 2 s and 0.18 over the last few seconds (limit 0.3), a quiet box. The load average 3.47 3.97 3.71 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@1 79,177 searches in 30.0 s, default@8 156,632 searches in 30.0 s, exact 78,857 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. default@1: warm-up 15.0 s and 39,014 searches (0 failed) at 1 searcher; p50 of the last windows 0.406, 0.405, 0.365 ms (windows of at least 2 s and 100 searches) (its last 3 windows differed by more than 5%); trial of 7,872 searches in 3.0 s at 1 searcher: p50 0.372 ms against the settled p50 0.405 ms, 9% apart (limit 10%), so the warm-up was EXTENDED once (30.0 s and 77,273 searches (0 failed) at 1 searcher; p50 of its older 6 windows 0.395 ms (windows 0.380, 0.388, 0.407, 0.399, 0.412, 0.358) and of its newer 6 0.382 ms (windows 0.368, 0.389, 0.370, 0.372, 0.395, 0.399), 3.2% apart (limit 5% or 0.1 ms; 0.012 ms) (windows of at least 2 s and 100 searches)); second trial of 7,730 searches in 3.0 s at 1 searcher: p50 0.390 ms against the settled p50 0.382 ms, 2% apart (limit 10%), inside its newer windows' range 0.368 to 0.399 ms; settled after the extension; the timed pass p50 0.387 ms, 1% from the settled p50 0.382 ms (limit 10% or 0.1 ms; 0.005 ms); default@8: warm-up 15.0 s and 78,755 searches (0 failed) at 8 searchers; QPS of the last windows 5,114, 5,284, 5,245 (windows of at least 2 s and 100 searches); trial of 15,547 searches in 3.0 s at 8 searchers: 5,180 QPS against the settled 5,245 QPS, 1% apart (limit 10%); the timed pass 5,253 QPS, 0% from the settled 5,245 QPS (limit 10%); exact: warm-up 15.0 s and 40,047 searches (0 failed) at 1 searcher; p50 of the last windows 0.373, 0.374, 0.380 ms (windows of at least 2 s and 100 searches); trial of 8,162 searches in 3.0 s at 1 searcher: p50 0.372 ms against the settled p50 0.374 ms, 1% apart (limit 10%); the timed pass p50 0.365 ms, 2% from the settled p50 0.374 ms (limit 10% or 0.1 ms; 0.009 ms).
- Pass default@1 after rehearsal: 2026-10-07T23:13:56.820Z to 2026-10-07T23:14:16.820Z (20.0 s), 51,741 searches, 0 failed, p50 0.387 ms, mean 0.385 ms, p99 0.663 ms, 2587.0 QPS (1000/QPS 0.387 ms).
- Pass default@8 after default@1: 2026-10-07T23:14:34.887Z to 2026-10-07T23:14:54.888Z (20.0 s), 105,076 searches, 0 failed, p50 1.504 ms, mean 1.521 ms, p99 2.150 ms, 5253.4 QPS (1000/QPS 0.190 ms).
- Pass exact after default@8: 2026-10-07T23:15:12.938Z to 2026-10-07T23:16:12.938Z (60.0 s), 164,245 searches, 0 failed, p50 0.365 ms, mean 0.364 ms, p99 0.659 ms, 2737.4 QPS (1000/QPS 0.365 ms).
- Passes in the order run: default@1, default@8, exact; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 627,780 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (FT.INFO indexing 0, percent_indexed 1, hash_indexing_failures 0, flat_buffer_size 0, num_docs 524, hashes counted with SCAN under gvb_gvbbench_eshoponweb: 524).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-redis (691c8e36ee45) on network gvb-redis_default at container-ip:6379, not through docker-proxy localhost:6379. Open after the passes: container-ip:6379 x1 (this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-redis: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-redis: 16 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): default@1 3492/3492 MHz, average 3492.4/3492.0, performance, outside load 0.13 before the warm-up, 0.13 during, client CPU 0.555 ms per search, engine CPU 0.256 ms per search; default@8 3492/3492 MHz, average 3492.6/3491.9, performance, outside load 0.14 before the warm-up, 0.13 during, client CPU 0.359 ms per search, engine CPU 0.231 ms per search; exact 3492/3492 MHz, average 3492.4/3492.1, performance, outside load 0.12 before the warm-up, 0.13 during, client CPU 0.557 ms per search, engine CPU 0.241 ms per search.

### qdrant-hnsw

- Engine: Qdrant 1.17.0 in container gvb-qdrant (image qdrant/qdrant:v1.17.0@sha256:f1c7272cdac52b38c1a0e89313922d940ba50afd90d593a1605dbbc214e66ffb; cpuset 2-3,6-7 from its creation; 3 search thread(s)) (compose)
- Index: HNSW m=16 ef_construct=100, hnsw_ef=server default, cosine; indexing_threshold_kb 1 and full_scan_threshold_kb 10 (server defaults are 10,000 each) so a small collection builds and walks its graph
- Search settings: ef_construct=100, hnsw_ef=server, m=16
- Engine settings: HostConfig.Memory = 8589934592 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit)); search_threads = 3 (read: threads named search-* of the qdrant process in container gvb-qdrant (/proc))
- Image: qdrant/qdrant:v1.17.0@sha256:f1c7272cdac52b38c1a0e89313922d940ba50afd90d593a1605dbbc214e66ffb, id sha256:f1c7272cdac52b38c1a0e89313922d940ba50afd90d593a1605dbbc214e66ffb (the pinned id), tagged on this machine at 2026-10-04T22:03:16.973657523Z
- Data folder ~/gvb-data/engines/qdrant: 357 B at the start, 164.1 MiB at the end
- Load: 524 rows in batches of 1000, 0.1 s of upserts (7,366 rows/s); count matched 0.0 s after the last upsert; index step 2.0 s (indexing_threshold_kb 1, full_scan_threshold_kb 10; status Green, 524 of 524 vectors in HNSW segments (2 segments), waited 2.0 s)
- Search: 21498 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 0.72 ms, 8 searchers 0.52 ms
- Exact mode: 66,068 searches, p50 0.90 ms, p95 1.01 ms, recall 1.000
- RAM: 36.01 MiB (docker stats); disk: 164.1 MiB (collection folder)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started qdrant.compose.yaml with every container created on CPUs 2-3,6-7: gvb-qdrant cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [qdrant.compose.yaml eb57e55e93c1]. This run created the engine from these files.
- Index after the load: ready, 524 of 524 indexed (status Green, optimizer ok, indexed_vectors_count 524 of 524 points, 2 segments, indexing_threshold_kb 1, full_scan_threshold_kb 10; every one of 1 non-empty segments has a complete HNSW graph above its full-scan threshold; no search has run yet, so the walk is proven by the segment settings only; searches counted since the collection was created: unfiltered_hnsw 0, unfiltered_plain 0, unfiltered_exact 0). Durability: What was measured (strace -f on the Qdrant server process, 30 single-point REST upserts per setting, every sync call timed against the request that caused it): with wait=true each upsert had exactly one msync(MS_SYNC) of the write-ahead-log segment inside the request, before the reply (30 of 30); with wait=false the 30 upserts were acknowledged within 0.26 s with no sync call, and the first WAL msync (one call covering all 30 records) came 2.7 s after the last reply, followed by the segment-file flushes. So the log is flushed to disk under both settings, but with wait=false the flush comes after the acknowledgement: an operating-system crash or power cut in that gap loses acknowledged writes, while a crash of the Qdrant process alone should not, because the bytes are already in the kernel's page cache (inferred, not tested). Every upsert here is sent with wait=true in batches of 256 over gRPC; the strace used one point per request, so one flush per batch is inferred, not measured. Segment files are flushed every 5 s; log segments 32 MB, 0 created ahead. /qdrant/config/config.yaml in container gvb-qdrant (the image's own file; its storage keys are listed) sets: storage.collection.quantization = null, storage.collection.replication_factor = 1, storage.collection.vectors.on_disk = null, storage.collection.write_consistency_factor = 1, storage.hnsw_index.ef_construct = 100, storage.hnsw_index.full_scan_threshold_kb = 10000, storage.hnsw_index.m = 16, storage.hnsw_index.max_indexing_threads = 0, storage.hnsw_index.on_disk = false, storage.hnsw_index.payload_m = null, storage.max_collections = null, storage.node_type = Normal, storage.on_disk_payload = true, storage.optimizers.default_segment_number = 0, storage.optimizers.deleted_threshold = 0.2, storage.optimizers.flush_interval_sec = 5, storage.optimizers.indexing_threshold_kb = 10000, storage.optimizers.max_optimization_threads = null, storage.optimizers.max_segment_size_kb = null, storage.optimizers.vacuum_min_vector_number = 1000, storage.performance.max_search_threads = 0, storage.performance.optimizer_cpu_budget = 0, storage.performance.update_rate_limit = null, storage.shard_transfer_method = null, storage.snapshots_config.snapshots_storage = local, storage.snapshots_path = ./snapshots, storage.storage_path = ./storage, storage.temp_path = null, storage.update_concurrency = null, storage.wal.wal_capacity_mb = 32, storage.wal.wal_segments_ahead = 0; no QDRANT__ environment overrides in the server process. Not tested by cutting power; whether the disk's own write cache reaches the media was not checked..
- Outside load when searching began: 0.15 CPUs on average over the last 2 s and 0.15 over the last few seconds (limit 0.3), a quiet box. The load average 2.56 3.16 3.43 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@8 104,222 searches in 30.0 s, default@1 32,269 searches in 30.0 s, exact 33,065 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. default@8: warm-up 15.0 s and 51,906 searches (0 failed) at 8 searchers; QPS of the last windows 3,456, 3,500, 3,454 (windows of at least 2 s and 100 searches); trial of 10,467 searches in 3.0 s at 8 searchers: 3,487 QPS against the settled 3,456 QPS, 1% apart (limit 10%); the timed pass 3,472 QPS, 0% from the settled 3,456 QPS (limit 10%); default@1: warm-up 15.0 s and 16,089 searches (0 failed) at 1 searcher; p50 of the last windows 0.918, 0.921, 0.921 ms (windows of at least 2 s and 100 searches); trial of 3,219 searches in 3.0 s at 1 searcher: p50 0.921 ms against the settled p50 0.921 ms, 0% apart (limit 10%); the timed pass p50 0.920 ms, 0% from the settled p50 0.921 ms (limit 10% or 0.1 ms; 0.001 ms); exact: warm-up 15.0 s and 16,522 searches (0 failed) at 1 searcher; p50 of the last windows 0.901, 0.894, 0.902 ms (windows of at least 2 s and 100 searches); trial of 3,301 searches in 3.0 s at 1 searcher: p50 0.895 ms against the settled p50 0.901 ms, 1% apart (limit 10%); the timed pass p50 0.897 ms, 1% from the settled p50 0.901 ms (limit 10% or 0.1 ms; 0.005 ms).
- Pass default@8 after rehearsal: 2026-10-07T23:18:12.051Z to 2026-10-07T23:18:32.051Z (20.0 s), 69,448 searches, 0 failed, p50 2.218 ms, mean 2.301 ms, p99 3.964 ms, 3472.3 QPS (1000/QPS 0.288 ms).
- Pass default@1 after default@8: 2026-10-07T23:18:50.079Z to 2026-10-07T23:19:10.079Z (20.0 s), 21,498 searches, 0 failed, p50 0.920 ms, mean 0.929 ms, p99 1.446 ms, 1074.9 QPS (1000/QPS 0.930 ms).
- Pass exact after default@1: 2026-10-07T23:19:28.106Z to 2026-10-07T23:20:28.106Z (60.0 s), 66,068 searches, 0 failed, p50 0.897 ms, mean 0.906 ms, p99 1.389 ms, 1101.1 QPS (1000/QPS 0.908 ms).
- Passes in the order run: default@8, default@1, exact; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 271,060 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (status Green, optimizer ok, indexed_vectors_count 524 of 524 points, 2 segments, indexing_threshold_kb 1, full_scan_threshold_kb 10; every one of 1 non-empty segments has a complete HNSW graph above its full-scan threshold; 309,118 segment searches walked the graph and none scanned; searches counted since the collection was created: unfiltered_hnsw 309,118, unfiltered_plain 0, unfiltered_exact 118,956).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-qdrant (47842ce980aa) on network gvb-qdrant_default at container-ip:6334, not through docker-proxy localhost:16334; container gvb-qdrant (47842ce980aa) on network gvb-qdrant_default at container-ip:6333, not through docker-proxy localhost:16333. Open after the passes: container-ip:6334 x1 (this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-qdrant: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-qdrant: 24 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): default@8 3492/3492 MHz, average 3492.0/3492.0, performance, outside load 0.09 before the warm-up, 0.13 during, client CPU 0.516 ms per search, engine CPU 0.933 ms per search; default@1 3492/3492 MHz, average 3492.0/3492.0, performance, outside load 0.12 before the warm-up, 0.10 during, client CPU 0.722 ms per search, engine CPU 0.733 ms per search; exact 3492/3492 MHz, average 3492.0/3492.0, performance, outside load 0.11 before the warm-up, 0.11 during, client CPU 0.732 ms per search, engine CPU 0.711 ms per search.

### sqlitevec

- Engine: SQLite 3.53.3 + sqlite-vec 0.1.7-alpha.2.1 (vec0, embedded, in process) (embedded)
- Index: vec0 brute-force scan, no ANN index (exact), float32, cosine distance, default chunk_size=1024; score = 1 - cosine distance; searches run concurrently, one WAL reader connection per searcher (opened as searchers arrive, at most 32), writes run one at a time and may overlap searches
- Search settings: chunk_size=1024
- Engine settings: sqlite_version = 3.53.3 (read: SELECT sqlite_version() on an in-memory SQLite connection); synchronous = NORMAL (set: src/GenericVectorBuilder.Engines/Sinks/SqliteVecSink.cs#PRAGMA synchronous=NORMAL); journal_mode = wal (read: PRAGMA journal_mode on a second, read-only connection to the bench file)
- Data folder ~/gvb-data/engines/sqlitevec: 292.93 MiB at the start, 302.65 MiB at the end; the benchmark's own database file 9.72 MiB
- Load: 524 rows in batches of 1000, 0.1 s of upserts (4,360 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (nothing to build, every search scans all vectors; checked in 0.00 s: no index, exact scan by design: gvb_gvbbench_eshoponweb is a vec0 virtual table holding 524 vectors and every search compares all of them (EXPLAIN QUERY PLAN: SCAN gvb_gvbbench_eshoponweb VIRTUAL TABLE INDEX 0:3{___}___ | USE TEMP B-TREE FOR ORDER BY))
- Search: 10412 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 2.02 ms, 8 searchers 3.51 ms
- Exact mode: 31,214 searches, p50 1.89 ms, p95 2.11 ms, recall 1.000
- RAM: in this process, not measured; disk: 9.72 MiB added by this load (302.65 MiB (engine data folder), 292.93 MiB before)
- Index after the load: ready, ? of 524 indexed (no index, exact scan by design: gvb_gvbbench_eshoponweb is a vec0 virtual table holding 524 vectors and every search compares all of them (EXPLAIN QUERY PLAN: SCAN gvb_gvbbench_eshoponweb VIRTUAL TABLE INDEX 0:3{___}___ | USE TEMP B-TREE FOR ORDER BY)). Durability: PRAGMA journal_mode=WAL and PRAGMA synchronous=NORMAL (set by this sink when it opens the file): each commit is appended to the -wal file and handed to the operating system, but the WAL is fsynced only when SQLite checkpoints it (measured with strace: 40 single-record commits caused 3 WAL fsyncs), so a crash of the process loses nothing (measured with kill -9 and a reopen: all 50, 300 and 2,000 rows were there), while an operating-system crash or power cut can lose the newest commits since the last checkpoint and leaves the database consistent.
- Outside load when searching began: not known yet (too little sampled history). Load average 2.60 3.17 3.40 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@8 34,046 searches in 30.0 s, default@1 15,721 searches in 30.0 s, exact 15,630 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. default@8: warm-up 15.0 s and 17,036 searches (0 failed) at 8 searchers; QPS of the last windows 1,129, 1,134, 1,144 (windows of at least 2 s and 100 searches); trial of 3,405 searches in 3.0 s at 8 searchers: 1,133 QPS against the settled 1,134 QPS, 0% apart (limit 10%); the timed pass 1,136 QPS, 0% from the settled 1,134 QPS (limit 10%); default@1: warm-up 15.0 s and 7,804 searches (0 failed) at 1 searcher; p50 of the last windows 1.904, 1.898, 1.887 ms (windows of at least 2 s and 100 searches); trial of 1,576 searches in 3.0 s at 1 searcher: p50 1.881 ms against the settled p50 1.898 ms, 1% apart (limit 10%); the timed pass p50 1.894 ms, 0% from the settled p50 1.898 ms (limit 10% or 0.1 ms; 0.004 ms); exact: warm-up 15.0 s and 7,793 searches (0 failed) at 1 searcher; p50 of the last windows 1.888, 1.893, 1.892 ms (windows of at least 2 s and 100 searches); trial of 1,571 searches in 3.0 s at 1 searcher: p50 1.885 ms against the settled p50 1.892 ms, 0% apart (limit 10%); the timed pass p50 1.895 ms, 0% from the settled p50 1.892 ms (limit 10% or 0.1 ms; 0.002 ms).
- Pass default@8 after rehearsal: 2026-10-07T23:22:18.004Z to 2026-10-07T23:22:38.009Z (20.0 s), 22,734 searches, 0 failed, p50 6.748 ms, mean 7.029 ms, p99 13.925 ms, 1136.4 QPS (1000/QPS 0.880 ms).
- Pass default@1 after default@8: 2026-10-07T23:22:56.018Z to 2026-10-07T23:23:16.019Z (20.0 s), 10,412 searches, 0 failed, p50 1.894 ms, mean 1.918 ms, p99 2.290 ms, 520.6 QPS (1000/QPS 1.921 ms).
- Pass exact after default@1: 2026-10-07T23:23:34.032Z to 2026-10-07T23:24:34.033Z (60.0 s), 31,214 searches, 0 failed, p50 1.895 ms, mean 1.920 ms, p99 2.308 ms, 520.2 QPS (1000/QPS 1.922 ms).
- Passes in the order run: default@8, default@1, exact; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 104,582 sent, 0 failed (warmupErrors). Index after the searches: ready, ? of 524 indexed (no index, exact scan by design: gvb_gvbbench_eshoponweb is a vec0 virtual table holding 524 vectors and every search compares all of them (EXPLAIN QUERY PLAN: SCAN gvb_gvbbench_eshoponweb VIRTUAL TABLE INDEX 0:3{___}___ | USE TEMP B-TREE FOR ORDER BY)).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: in this process (embedded), no network: no network address. No open connection seen (no open TCP connection was found).
- CPU pinning: none: an embedded engine runs inside the benchmark client process, so it shares the client CPUs, CPUs 0-1,4-5.
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): default@8 3493/3492 MHz, average 3494.9/3492.0, performance, outside load 0.15 before the warm-up, 0.13 during, client CPU 3.507 ms per search; default@1 3500/3492 MHz, average 3496.4/3494.4, performance, outside load 0.16 before the warm-up, 0.14 during, client CPU 2.017 ms per search; exact 3498/3492 MHz, average 3495.9/3494.3, performance, outside load 0.14 before the warm-up, 0.14 during, client CPU 2.015 ms per search.

### mongodb

- Engine: MongoDB 8.0.32 Atlas Local (mongod + mongot Vector Search) (compose)
- Index: vectorSearch index, HNSW maxEdges=16 numEdgeCandidates=128, float32 binData, cosine, numCandidates=20x hits (min 100); exact mode = $vectorSearch exact:true
- Search settings: maxEdges=16, numCandidates=20x, numEdgeCandidates=128
- Engine settings: HostConfig.Memory = 8589934592 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit))
- Image: mongodb/mongodb-atlas-local:8.0.32, id sha256:1985314b0ded756ba965e0f400d17406b7a89bec659d4763a490f42f006e4fbd (the pinned id), tagged on this machine at 2026-10-03T14:11:20.531944474Z
- Data folder ~/gvb-data/engines/mongodb: 1.38 GiB at the start, 1.38 GiB at the end
- Load: 524 rows in batches of 1000, 0.2 s of upserts (3,076 rows/s); count matched 1.0 s after the last upsert; index step 10.2 s (index status READY, queryable True; mongot holds 524 of 524 documents in 2 segment(s) (278 + 246 documents), 2 searched through the HNSW graph (Approximate); segment layout unchanged for 10.2 s over 11 readings; wait took 10.2 s)
- Search: 14350 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 0.30 ms, 8 searchers 0.29 ms
- Exact mode: 47,727 searches, p50 1.24 ms, p95 1.40 ms, recall 1.000
- RAM: 639.7 MiB (docker stats); disk: 2.21 MiB added by this load (1.38 GiB (whole engine data folder), 1.38 GiB before)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started mongodb.compose.yaml with every container created on CPUs 2-3,6-7: gvb-mongodb cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [mongodb.compose.yaml 4ee6f6411f1f]. This run created the engine from these files.
- Index after the load: ready, 524 of 524 indexed (index status READY, queryable True; mongot holds 524 of 524 documents in 2 segment(s) (278 + 246 documents), 2 searched through the HNSW graph (Approximate)). Durability: Acknowledged writes are journaled before the acknowledgement. The sink sets no write concern, so the server default applies: getDefaultRWConcern gives w majority and the one-member replica set (--replSet gvbmongo in the image) has writeConcernMajorityJournalDefault true, with WiredTiger journaling on (journalCommitInterval 100 ms is only the interval for unacknowledged work). Measured 2026-10-04: 30 acknowledged single-document inserts raised WiredTiger 'log sync operations' by 30 and strace showed fdatasync on /data/db/journal/WiredTigerLog files. A crash loses no acknowledged write. mongot's search index is not part of that promise: it follows the collection asynchronously and is rebuilt from it..
- Outside load when searching began: 0.14 CPUs on average over the last 13 s and 0.15 over the last few seconds (limit 0.3), a quiet box. The load average 1.52 2.89 3.30 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@8 49,885 searches in 30.0 s, default@1 21,252 searches in 30.0 s, exact 23,544 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. default@8: warm-up 15.0 s and 31,343 searches (0 failed) at 8 searchers; QPS of the last windows 2,148, 2,156, 2,160 (windows of at least 2 s and 100 searches); trial of 6,339 searches in 3.0 s at 8 searchers: 2,110 QPS against the settled 2,156 QPS, 2% apart (limit 10%); the timed pass 2,150 QPS, 0% from the settled 2,156 QPS (limit 10%); default@1: warm-up 15.0 s and 10,743 searches (0 failed) at 1 searcher; p50 of the last windows 1.377, 1.359, 1.378 ms (windows of at least 2 s and 100 searches); trial of 2,159 searches in 3.0 s at 1 searcher: p50 1.369 ms against the settled p50 1.377 ms, 1% apart (limit 10%); the timed pass p50 1.368 ms, 1% from the settled p50 1.377 ms (limit 10% or 0.1 ms; 0.009 ms); exact: warm-up 15.0 s and 11,894 searches (0 failed) at 1 searcher; p50 of the last windows 1.229, 1.232, 1.242 ms (windows of at least 2 s and 100 searches); trial of 2,389 searches in 3.0 s at 1 searcher: p50 1.231 ms against the settled p50 1.232 ms, 0% apart (limit 10%); the timed pass p50 1.235 ms, 0% from the settled p50 1.232 ms (limit 10% or 0.1 ms; 0.004 ms).
- Pass default@8 after rehearsal: 2026-10-07T23:26:41.071Z to 2026-10-07T23:27:01.073Z (20.0 s), 43,006 searches, 0 failed, p50 3.597 ms, mean 3.718 ms, p99 6.592 ms, 2150.0 QPS (1000/QPS 0.465 ms).
- Pass default@1 after default@8: 2026-10-07T23:27:19.091Z to 2026-10-07T23:27:39.092Z (20.0 s), 14,350 searches, 0 failed, p50 1.368 ms, mean 1.392 ms, p99 1.937 ms, 717.5 QPS (1000/QPS 1.394 ms).
- Pass exact after default@1: 2026-10-07T23:27:57.110Z to 2026-10-07T23:28:57.111Z (60.0 s), 47,727 searches, 0 failed, p50 1.235 ms, mean 1.255 ms, p99 1.768 ms, 795.4 QPS (1000/QPS 1.257 ms).
- Passes in the order run: default@8, default@1, exact; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 159,548 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (index status READY, queryable True; mongot holds 524 of 524 documents in 2 segment(s) (278 + 246 documents), 2 searched through the HNSW graph (Approximate)).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-mongodb (848f111dacc1) on network gvb-mongodb_default at container-ip:27017, not through docker-proxy localhost:27017. Open after the passes: container-ip:27017 x10 (this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-mongodb: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-mongodb: 209 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): default@8 3492/3492 MHz, average 3492.0/3492.2, performance, outside load 0.24 before the warm-up, 0.14 during, client CPU 0.286 ms per search, engine CPU 1.767 ms per search; default@1 3492/3492 MHz, average 3492.6/3492.5, performance, outside load 0.19 before the warm-up, 0.23 during, client CPU 0.299 ms per search, engine CPU 1.296 ms per search; exact 3492/3492 MHz, average 3492.6/3491.8, performance, outside load 0.21 before the warm-up, 0.23 during, client CPU 0.300 ms per search, engine CPU 1.159 ms per search.

### vespa

- Engine: Vespa 8.754.14 (tensor attribute + HNSW) (compose)
- Index: HNSW float32 tensor, prenormalized-angular (cosine), max-links-per-node=16, neighbors-to-explore-at-insert=128; search targetHits=top, ef=100 via exploreAdditionalHits; exact mode = approximate:false; vectors held in memory
- Search settings: ef=100, max-links-per-node=16, neighbors-to-explore-at-insert=128, targetHits=top
- Engine settings: HostConfig.Memory = 6442450944 (read: docker inspect HostConfig.Memory (0 = no limit)); HostConfig.CpusetCpus = 2-3,6-7 (read: docker inspect HostConfig.CpusetCpus (empty = every CPU)); HostConfig.NanoCpus = 0 (read: docker inspect HostConfig.NanoCpus (0 = no limit))
- Image: vespaengine/vespa:8.754.14, id sha256:5c30f5c41e7563498c4f925db6a837a3848f04726a3ed26aed4a7c8ab69f18fd (the pinned id), tagged on this machine at 2026-10-03T14:13:52.169232039Z
- Data folder ~/gvb-data/engines/vespa: 2.13 GiB at the start, 2.13 GiB at the end
- Load: 524 rows in batches of 1000, 1.8 s of upserts (296 rows/s); count matched 0.0 s after the last upsert; index step 0.1 s (nothing to build, Vespa inserts into the HNSW graph while writing; ready after 0.1 s: HNSW graph returns 524 of 524 vectors: totalCount 524, coverage full True, nearestNeighbor approximate:true with targetHits above the count: has_index True, algorithm "index top k", top_k_hits 524; the graph is updated while each write is applied, so there is no later build step)
- Search: 10062 latency samples, target held 524 rows, 0 errors
- Client CPU per search: 1 searcher 0.91 ms, 8 searchers 0.87 ms
- Exact mode: 30,342 searches, p50 1.92 ms, p95 2.38 ms, recall 1.000
- RAM: 2.69 GiB (docker stats); disk: 3.59 MiB added by this load (2.13 GiB (whole engine data folder), 2.13 GiB before)
- Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started vespa.compose.yaml with every container created on CPUs 2-3,6-7: gvb-vespa cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
- Engine files (SHA-256, first 12 hex digits): [vespa.compose.yaml 3cfb702cc29f]. This run created the engine from these files.
- Index after the load: ready, 524 of 524 indexed (HNSW graph returns 524 of 524 vectors: totalCount 524, coverage full True, nearestNeighbor approximate:true with targetHits above the count: has_index True, algorithm "index top k", top_k_hits 524; the graph is updated while each write is applied, so there is no later build step). Durability: Vespa's transaction log server fsyncs after each commit: searchlib.translogserver usefsync=true, and an operation is searchable when it is acknowledged: proton documentdb visibilitydelay=0. Neither is overridden by this sink or by the services.xml it deploys; VespaReadinessTests reads both from the live config server. After a crash the node replays the transaction log over its last flushed data and rebuilds the in-memory HNSW graph. By those settings an acknowledged write survives a crash; this is read from the settings, not shown by pulling power. One node and min-redundancy 1, so a lost disk loses the data..
- Outside load when searching began: 0.16 CPUs on average over the last 40 s and 0.15 over the last few seconds (limit 0.3), a quiet box. The load average 5.73 3.61 3.48 (1/5/15 min; it also counts this benchmark's own client and engines, so it is recorded, not judged).
- Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: exact 8,474 searches in 30.0 s, default@8 34,504 searches in 30.0 s, default@1 15,001 searches in 30.0 s.
- Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure, and the timed pass itself within 10% of it. exact: warm-up 15.0 s and 7,108 searches (0 failed) at 1 searcher; p50 of the last windows 1.970, 1.965, 1.943 ms (windows of at least 2 s and 100 searches); trial of 1,473 searches in 3.0 s at 1 searcher: p50 1.943 ms against the settled p50 1.965 ms, 1% apart (limit 10%); the timed pass p50 1.916 ms, 3% from the settled p50 1.965 ms (limit 10% or 0.1 ms; 0.049 ms); default@8: warm-up 15.0 s and 23,182 searches (0 failed) at 8 searchers; QPS of the last windows 1,550, 1,566, 1,550 (windows of at least 2 s and 100 searches); trial of 4,583 searches in 3.0 s at 8 searchers: 1,526 QPS against the settled 1,550 QPS, 2% apart (limit 10%); the timed pass 1,556 QPS, 0% from the settled 1,550 QPS (limit 10%); default@1: warm-up 15.0 s and 7,541 searches (0 failed) at 1 searcher; p50 of the last windows 1.918, 1.924, 1.922 ms (windows of at least 2 s and 100 searches); trial of 1,534 searches in 3.0 s at 1 searcher: p50 1.914 ms against the settled p50 1.922 ms, 0% apart (limit 10%); the timed pass p50 1.922 ms, 0% from the settled p50 1.922 ms (limit 10% or 0.1 ms; 0.000 ms).
- Pass exact after rehearsal: 2026-10-07T23:32:43.077Z to 2026-10-07T23:33:43.079Z (60.0 s), 30,342 searches, 0 failed, p50 1.916 ms, mean 1.975 ms, p99 2.691 ms, 505.7 QPS (1000/QPS 1.978 ms).
- Pass default@8 after exact: 2026-10-07T23:34:01.116Z to 2026-10-07T23:34:21.121Z (20.0 s), 31,120 searches, 0 failed, p50 5.023 ms, mean 5.139 ms, p99 8.357 ms, 1555.7 QPS (1000/QPS 0.643 ms).
- Pass default@1 after default@8: 2026-10-07T23:34:39.135Z to 2026-10-07T23:34:59.137Z (20.0 s), 10,062 searches, 0 failed, p50 1.922 ms, mean 1.985 ms, p99 2.784 ms, 503.1 QPS (1000/QPS 1.988 ms).
- Passes in the order run: exact, default@8, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 103,400 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (HNSW graph returns 524 of 524 vectors: totalCount 524, coverage full True, nearestNeighbor approximate:true with targetHits above the count: has_index True, algorithm "index top k", top_k_hits 524; the graph is updated while each write is applied, so there is no later build step).
- Benchmark copy gvbbench_eshoponweb dropped afterwards.
- Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-vespa (3e11cd75a7e0) on network engines_default at container-ip:8080, not through docker-proxy localhost:8090; container gvb-vespa (3e11cd75a7e0) on network engines_default at container-ip:19071, not through docker-proxy localhost:19071. Open after the passes: container-ip:8080 x1 (this target).
- CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-vespa: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-vespa: 214 thread(s) on CPUs 2-3,6-7 (every thread of the container).
- Clock per pass (median MHz of engine CPUs / client CPUs, each the median of its CPUs' medians and every CPU when the CPUs were not split, their average MHz, governor, CPUs busy outside the benchmark, client and engine CPU per search; the clock was pinned at 3500 MHz (turbo off), and a pass whose median on either group is more than 100 bp (1%) off it, or that has no reading on a group, is flagged): exact 3492/3492 MHz, average 3492.0/3492.3, performance, outside load 0.16 before the warm-up, 0.12 during, client CPU 0.925 ms per search, engine CPU 2.266 ms per search; default@8 3492/3492 MHz, average 3492.0/3492.1, performance, outside load 0.12 before the warm-up, 0.19 during, client CPU 0.868 ms per search, engine CPU 2.456 ms per search; default@1 3492/3492 MHz, average 3492.0/3492.2, performance, outside load 0.16 before the warm-up, 0.12 during, client CPU 0.913 ms per search, engine CPU 2.282 ms per search.


## Notes

- Load average at start: 2.66 3.30 3.46 on 8 logical CPUs (1/5/15 min). It counts this benchmark's own earlier work and the engines it started, so it is recorded here and never judged; each timed pass is judged by the outside load machine control measures (conditions.passes).
- Order: targets ran one at a time in a random order from seed 802 (runSeed; --seed 802 repeats it, targetOrder lists it). Inside each target the timed passes also ran in a random order from the same seed and the target's name (passOrder): default@1 is one searcher for 20 s (every search's latency gives p50/p95/p99, the completed searches give QPS@1, its first answer to each query gives recall and nDCG), default@N is N searchers for 20 s (QPS@N), exact is the engine's exact mode, one searcher for 60 s cycling the queries.
- Preparation, untimed, before any timed pass: a rehearsal of every pass type at its own concurrency for 30 s each, through the same code the passes use.
- Warm-up and settle check, untimed, right before every timed pass: the pass's own search at the pass's own number of searchers for at least 15 s and at least 20 searches (at most 120 s), read in windows of at least 2 s and 100 searches; then a 3 s trial of the same pass. The trial's figure (p50 with one searcher, QPS with several) must lie within 10% of the warm-up's settled figure (the median of its last 3 windows, which must agree within 5%); if not, the warm-up is extended once (at least 30 s, until its windows agree, at most 120 s), where an extension's windows agree when its level test passes: the older and the newer half of its latest windows, 5 to 10 windows a half, each half read as the pass reads it (the p50 of all its searches with one searcher, its searches per second with several), agree within 5%, judged on the windows that stopped it (an extension whose cap runs out with fewer than 10 windows is judged by its last 3 instead); then a second trial is taken, which must lie inside the range of the newer half's windows or within 10% of the newer half's figure, and the warm-up runs again before the pass. After the pass its own figure is held against the settled figure, within 10%. In the level test, the second trial and the hold, two one-searcher p50s no more than 0.1 ms apart agree whatever their percentage (below what this box resolves at one searcher). Each target's notes give every check, and a pass that still disagrees, or whose own figure did not hold, flags its target as unsettled. With machine control on, the check for a quiet box is made when the warm-up is announced, before it starts (a wait between the warm-up and the timed pass let the engine go cold), so by the time the clock opens that check is as old as the warm-up and the trial (about 18 s, more after an extension); each pass's conditions record that lead (quietCheckLeadSeconds). Failed untimed searches are counted per target (warmupErrors) and are not in the timed error counts.
- Load rows/s counts only time inside each target's upsert calls: one writer, batches of 1000, rows already in memory, the collection dropped and created fresh first. Engines that build or finish their index after the writes do it in a separate timed index step (the load's index seconds), and the run waits for it before searching.
- Index proof: each engine's own report of its index (indexState) is read after the load and again after the last pass. A target whose index was not ready after the load was still measured and carries a WARNING; an engine that reports nothing counts as not ready.
- Search settings: each searched target records the settings its own index description states (searchSettings: the build parameters and the effort per query, such as m, ef_construction and ef_search), read after the load; consolidating never averages runs whose settings differ.
- Durability: each target's crash-safety setting as configured here (durability). Engines that do not force writes to disk on every commit load faster for that reason.
- Engine record: right after each engine was bound, its settings (engineSettings: the container's limits and what the engine itself reports, each entry saying how it was obtained), its image against the id pinned in deploy/bench/image-pins.json (image; another id ends that target with an error) and the size of its data folder (dataFolder: at the start, after any reset, and at the end) were recorded. In run-all, ClickHouse's system log tables are truncated before its load, and the reset is written into its dataFolder.
- Latency is client-side wall time around each search (network and driver included), every search of the default@1 window, one searcher, the queries cycled; a window with fewer than 200 searches, a p50 above 1.25 x its mean, or a mean above its p99 is flagged.
- QPS: N searchers back to back for 20 s per level; completed searches divided by the window's elapsed time.
- Recall@10: share of the exact top 10 (brute force in memory) that the engine returned. A hit whose exact similarity ties the 10th best (within 1e-5) also counts, because duplicate rows embed to identical vectors.
- Routes: a container engine (the benchmark's own SQL Server and Qdrant containers included) is reached at its container's own address on its Docker network, never through the published localhost port (docker-proxy); the native comparison targets (sql-native, qdrant-native) are native services reached directly over loopback. Each target's addresses and the connections the client held open after its passes are in its notes and in conditions.connections.
- RAM of compose engines, the benchmark's own SQL Server and Qdrant containers (sql, sql-diskann, qdrant, qdrant-hnsw) included, is docker stats of the engine's containers; for the native comparison targets (sql-native, qdrant-native) it is the whole native process, including every other database or collection it serves. Disk is the table's reserved pages (SQL), the collection folder (Qdrant), or for other engines what the load added to the engine's data folder (engines that keep data in memory until a snapshot show almost nothing); when the folder was smaller after the load than before it, no figure is given.
- Engines: run-all starts an engine that is down and stops it afterwards only if it was not running when the run began; an engine that was already running is left running.
- WARNING: the engine did not report a ready index after the load for: elasticsearch. They were measured anyway; their numbers may come from a scan or a half-built index (see each target's index state).
- Machine control on: governor performance on every CPU during the run (before: schedutil; at the end: performance; after putting it back: schedutil). CPU clock pinned for the run: turbo off (intel_pstate/no_turbo 0 -> 1), so every CPU's clock is held at its ceiling of 3500 MHz whatever the engine runs; the uncore (L3 and memory) clock is held at 3000 MHz (min 3000 MHz, max 3000 MHz (MSR 0x620 = 0x1e1e); was min 1200 MHz, max 3000 MHz (MSR 0x620 = 0xc1e)) (before: turbo on (intel_pstate/no_turbo 0), ceiling 3600 MHz on CPUs 0-7, floor 1200 MHz on CPUs 0-7; while pinned: turbo off (intel_pstate/no_turbo 1), ceiling 3500 MHz on CPUs 0-7, floor 1200 MHz on CPUs 0-7; at the end: turbo off (intel_pstate/no_turbo 1), ceiling 3500 MHz on CPUs 0-7, floor 1200 MHz on CPUs 0-7; after putting it back: turbo on (intel_pstate/no_turbo 0), ceiling 3600 MHz on CPUs 0-7, floor 1200 MHz on CPUs 0-7). Uncore limit at the end: min 3000 MHz, max 3000 MHz (MSR 0x620 = 0x1e1e); after putting it back: min 1200 MHz, max 3000 MHz (MSR 0x620 = 0xc1e). Each target's clock note gives every pass's median MHz on the engine CPUs and on the client CPUs (the median of each CPU's median); a pass whose median on either is more than 100 bp (1%) off the pinned 3500 MHz, or that has no reading on one, is flagged (conditions.clock.rule). Figures from runs made with turbo on are not comparable with these in absolute terms. engine CPUs 2-3,6-7 (cores 2,6 and 3,7), client CPUs 0-1,4-5 (cores 0,4 and 1,5); the client process was pinned; each engine was pinned to the engine CPUs for its turn and put back after (conditions.engines); an engine run-all started (and stopped) was asked to be created on them, and its notes say whether the host did so or it was moved there after the start. Busy box: right before each pass's warm-up the run waits, up to 10 min, while processes outside the benchmark (everything but this client and the engine under test's cgroups) use more than 0.3 CPUs on average over the last 60 s or the last 5 s, neither window reaching back past the start of the target or of its engine (so the engine's own start-up is not outside work); if the box does not clear the pass runs anyway, flagged 'busy box', as is a pass whose own outside load is above the limit. Outside load counts busy = user + nice + system + irq + softirq + steal (guest time is already in user); CONFIG_IRQ_TIME_ACCOUNTING is not set, so task and cgroup run time include the interrupt and softirq time that hit them and it is added; CONFIG_PARAVIRT_TIME_ACCOUNTING is not set, so steal is added (/boot/config-6.8.0-142-generic) (conditions.cpuAccounting); each pass also records this client's own CPU time per search and the engine's (conditions.passes[].clientCpuMsPerSearch and engineCpuMsPerSearch; how in conditions.engineCpu.rule). CPU clocks were sampled every 250 ms; each pass's min/median/max per CPU is in conditions.passes. CPU idle states, recorded and left as found: driver intel_idle, governor menu, intel_idle max_cstate 9; POLL on, C1 on, C1E on, C3 OFF on CPUs 0-7 (default disabled), C6 on (conditions.cpuIdle). How each target was reached: conditions.connections. Client build Release, .NET 10.0.12.
- duckdb is embedded: it ran inside the client process on the client CPUs 0-1,4-5, sharing them with the client.
- sqlitevec is embedded: it ran inside the client process on the client CPUs 0-1,4-5, sharing them with the client.