# Vector engine benchmark: eshoponweb Run 2026-10-05T07:33:29Z (run-all). Command line: ``` ~/gvb-work/lanes/v6-final/src/GenericVectorBuilder.Bench/bin/Release/net10.0/GenericVectorBuilder.Bench.dll run-all --pipeline eshoponweb --queries golden --seed 601 --out ~/ForClaude/GenericVectorBuilder/bench-results ``` - Machine: bench-host, Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz (8 logical CPUs), 62.7 GiB RAM, GPU NVIDIA GeForce RTX 3060, 12288 MiB, Ubuntu 24.04.5 LTS, kernel 6.8.0-142-generic, .NET 10.0.12, load average at start 1.98 5.18 4.13 - Data: 524 vectors x 1024 dims, collection `gvbbench_eshoponweb`. Read READ-ONLY from GenericVectorBuilder.dbo.gvb_eshoponweb in ChunkId order, vectors as native SqlVector, 0.2 s. - Queries: golden: 20 labelled questions from ~/ForClaude/evalkit/questions_golden.json, embedded with qwen3-emb-0.6b (cached in ~/gvb-data/bench-cache/golden-eshoponweb-68df42ca69efa079.json, no embedding calls). Top 10. Throughput at concurrency 1, 8 for 20 s each. - Request speed on a small collection (524 vectors): Measured end to end through each engine's .NET client; at this size it reflects per-request cost including the client library, not index scaling. - Client CPU per search is the CPU time the test's .NET client itself used for each search, measured in the same pass as the figure beside it. Where it is close to the latency, the client library is a large part of what is measured. For an embedded engine (DuckDB, sqlite-vec) the engine runs inside the client process, so its figure is the engine's own CPU time, not client overhead. - Ground truth: brute force over every vector in memory, 0.0 s for all 20 queries. - nDCG@10 of the exact answer itself (the ceiling for this embedder): 0.518 | engine | index | load rows/s | p50 ms | p95 ms | p99 ms | QPS@1 | QPS@8 | client CPU ms/search@1 | client CPU ms/search@8 | recall@10 | nDCG@10 | RAM | disk | |---|---|---|---|---|---|---|---|---|---|---|---|---|---| | qdrant-hnsw | HNSW m=16 ef_construct=100, hnsw_ef=server default, cosine; indexing_threshold_kb 1 and full_scan_threshold_kb 10 (server defaults are 10,000 each) so a small collection builds and walks its graph | 4,928 | 0.90 | 1.01 | 1.39 | 1093.0 | 3473.2 | 0.72 | 0.51 | 1.000 | 0.518 | 36.4 MiB | 164.1 MiB | | milvus | HNSW M=16 efConstruction=128, ef=100, metric COSINE, Strong consistency searches; approximate only (no exact mode) | 1,073 | 2.16 | 2.67 | 3.53 | 440.8 | 1290.5 | 0.55 | 0.62 | 1.000 | 0.518 | 183.3 MiB | 10.86 MiB | | oracle | HNSW in-memory neighbor graph NEIGHBORS=16 EFCONSTRUCTION=128, EFSEARCH=100 per query, cosine; exact mode = FETCH EXACT FIRST (full scan); Oracle Free caps itself at 2 CPUs (cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit) | 1,938 | 0.88 | 1.05 | 1.75 | 1103.8 | 2879.2 | 0.85 | 0.64 | 1.000 | 0.518 | 2.12 GiB | 0 B | | weaviate | HNSW maxConnections(M)=16 efConstruction=128, ef=-1 (dynamic: limit x 8 clamped 100..500), cosine, no quantization; approximate only (no exact mode) | 1,007 | 5.19 | 10.4 | 11.4 | 171.2 | 312.2 | 0.57 | 0.60 | 1.000 | 0.518 | 96.28 MiB | 2.96 MiB | | clickhouse | vector_similarity HNSW cosineDistance, quantization bf16, M=16 ef_construction=128, hnsw_candidate_list_size_for_search=256, rescoring off; exact mode = full scan with skip indexes off | 2,520 | 4.50 | 5.55 | 6.88 | 215.2 | 495.8 | 0.83 | 1.02 | 0.995 | 0.528 | 231.1 MiB | 3.25 MiB | | sqlitevec | vec0 brute-force scan, no ANN index (exact), float32, cosine distance, default chunk_size=1024; score = 1 - cosine distance; searches run concurrently, one WAL reader connection per searcher (opened as searchers arrive, at most 32), writes run one at a time and may overlap searches | 5,046 | 1.82 | 2.04 | 2.27 | 542.2 | 1161.4 | 1.93 | 3.44 | 1.000 | 0.518 | - | 9.72 MiB | | mongodb | vectorSearch index, HNSW maxEdges=16 numEdgeCandidates=128, float32 binData, cosine, numCandidates=20x hits (min 100); exact mode = $vectorSearch exact:true | 3,084 | 1.39 | 1.61 | 1.97 | 705.0 | 2102.4 | 0.30 | 0.29 | 1.000 | 0.518 | 684.2 MiB | 0 B | | sql | exact VECTOR_DISTANCE cosine, no vector index (full scan) | 601 | 3.89 | 4.48 | 6.00 | 250.7 | 621.4 | 1.19 | 1.19 | 1.000 | 0.518 | 528.9 MiB | 5.15 MiB | | elasticsearch | HNSW float32, no quantization, m=16, ef_construction=128, cosine; search k=top, num_candidates=100; 1 shard, 0 replicas; force-merged to one segment after the load (at 1,024 dimensions a segment under 1,043 vectors gets no graph) | 442 | 1.30 | 1.48 | 1.74 | 754.5 | 2706.6 | 0.46 | 0.49 | 1.000 | 0.518 | 2.54 GiB | 2.4 MiB | | duckdb | HNSW (vss extension) FLOAT[n] metric=cosine m=16 ef_construction=128, ef_search=100 per connection, persistent (hnsw_enable_experimental_persistence=true, checkpoint_threshold=256MB); exact mode = array_cosine_similarity sequential scan; score = 1 - cosine distance; searches run concurrently, one connection per searcher (opened as searchers arrive, at most 32), writes run one at a time and never overlap a search | 1,574 | 3.53 | 3.92 | 5.04 | 280.6 | 596.6 | 4.84 | 6.66 | 1.000 | 0.518 | - | 2.9 MiB | | chroma | HNSW M=16 ef_construction=128, ef_search=100 (Chroma default), cosine; approximate only (no exact mode) | 841 | 2.03 | 2.30 | 2.53 | 487.6 | 804.4 | 0.47 | 0.45 | 1.000 | 0.518 | 56.51 MiB | 414.16 KiB | | opensearch | faiss HNSW float32, no compression, m=16, ef_construction=128, cosinesimil; search k=top, ef_search=100; 1 shard, 0 replicas; graph built at any segment size (approximate_threshold=0); force-merged to one segment after the load | 405 | 1.99 | 2.24 | 2.81 | 495.2 | 1528.2 | 0.46 | 0.51 | 1.000 | 0.518 | 2.58 GiB | 9 MiB | | vespa | HNSW float32 tensor, prenormalized-angular (cosine), max-links-per-node=16, neighbors-to-explore-at-insert=128; search targetHits=top, ef=100 via exploreAdditionalHits; exact mode = approximate:false; vectors held in memory | 276 | 1.85 | 2.27 | 2.56 | 524.0 | 1539.8 | 0.94 | 0.88 | 1.000 | 0.518 | 2.79 GiB | 0 B | | typesense | HNSW float32 (hnswlib), m=16, ef_construction=128, cosine; search k=top, ef=100; exact mode = filter ordinal:>=0 with flat_search_cutoff; index held in memory | 838 | 4.04 | 4.39 | 5.68 | 244.1 | 591.3 | 0.52 | 0.58 | 1.000 | 0.518 | 115 MiB | 0 B | | qdrant | exact scan: the builder's sink sends exact=true on every search, so no HNSW graph is used whether or not Qdrant has built one (see the index state) | 9,285 | 0.88 | 0.99 | 1.39 | 1125.4 | 3303.0 | 0.72 | 0.53 | 1.000 | 0.518 | 37.63 MiB | 196.08 MiB | | mariadb | VECTOR INDEX (HNSW variant) DISTANCE=cosine, M=16 (no ef_construction setting exists), mhnsw_ef_search=100 per statement (the ef 100 most engines here use, so the search effort matches; MariaDB's own default is 20; recall@10 at ef 100 falls as the set grows (random 1024-dimension vectors, measured 2026-10-04: 0.99 at 524, 0.89 to 0.92 at 2,000; an earlier run gave about 0.09 at 100,000)), mhnsw_max_cache_size 4G; exact mode = IGNORE INDEX full scan | 1,052 | 0.63 | 0.71 | 0.88 | 1569.4 | 5930.1 | 0.47 | 0.38 | 1.000 | 0.518 | 170.9 MiB | 22.01 MiB | | redis | HNSW TYPE FLOAT32 M=16 EF_CONSTRUCTION=128, EF_RUNTIME=100 per query, cosine; exact mode = FLAT index built on first exact query | 11,834 | 0.36 | 0.47 | 0.71 | 2653.9 | 5074.5 | 0.54 | 0.36 | 1.000 | 0.518 | 21.07 MiB | 2.43 MiB | | pgvector | HNSW vector_cosine_ops m=16 ef_construction=128, hnsw.ef_search=100 per query, float32 vector(n), cosine; exact mode = same query with index scans off (sequential scan) | 616 | 0.85 | 1.00 | 1.28 | 1153.4 | 4007.6 | 0.68 | 0.54 | 1.000 | 0.518 | 99.96 MiB | 7.57 MiB | | sql-diskann | DiskANN (preview) via VECTOR_SEARCH, cosine, build {"StartId":"306", "L":"48", "M":"8", "R":"48"}; the exact mode scans the same table | 637 | 3.60 | 4.06 | 4.86 | 272.5 | 690.4 | 1.03 | 1.10 | 0.965 | 0.526 | 527.2 MiB | 4.52 MiB | ## Details per target ### qdrant-hnsw - Engine: Qdrant 1.17.0 in container gvb-qdrant (image qdrant/qdrant:v1.17.0@sha256:f1c7272cdac52b38c1a0e89313922d940ba50afd90d593a1605dbbc214e66ffb; cpuset 2-3,6-7 from its creation; 3 search thread(s)) (compose) - Index: HNSW m=16 ef_construct=100, hnsw_ef=server default, cosine; indexing_threshold_kb 1 and full_scan_threshold_kb 10 (server defaults are 10,000 each) so a small collection builds and walks its graph - Search settings: ef_construct=100, hnsw_ef=server, m=16 - Load: 524 rows in batches of 1000, 0.1 s of upserts (4,928 rows/s); count matched 0.0 s after the last upsert; index step 2.0 s (indexing_threshold_kb 1, full_scan_threshold_kb 10; status Green, 524 of 524 vectors in HNSW segments (2 segments), waited 2.0 s) - Search: 21860 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 0.72 ms, 8 searchers 0.51 ms - Exact mode: 66,988 searches, p50 0.88 ms, p95 1.00 ms, recall 1.000 - RAM: 36.4 MiB (docker stats); disk: 164.1 MiB (collection folder) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started qdrant.compose.yaml with every container created on CPUs 2-3,6-7: gvb-qdrant cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - Index after the load: ready, 524 of 524 indexed (status Green, optimizer ok, indexed_vectors_count 524 of 524 points, 2 segments, indexing_threshold_kb 1, full_scan_threshold_kb 10; every one of 1 non-empty segments has a complete HNSW graph above its full-scan threshold; no search has run yet, so the walk is proven by the segment settings only; searches counted since the collection was created: unfiltered_hnsw 0, unfiltered_plain 0, unfiltered_exact 0). Durability: What was measured (strace -f on the Qdrant server process, 30 single-point REST upserts per setting, every sync call timed against the request that caused it): with wait=true each upsert had exactly one msync(MS_SYNC) of the write-ahead-log segment inside the request, before the reply (30 of 30); with wait=false the 30 upserts were acknowledged within 0.26 s with no sync call, and the first WAL msync (one call covering all 30 records) came 2.7 s after the last reply, followed by the segment-file flushes. So the log is flushed to disk under both settings, but with wait=false the flush comes after the acknowledgement: an operating-system crash or power cut in that gap loses acknowledged writes, while a crash of the Qdrant process alone should not, because the bytes are already in the kernel's page cache (inferred, not tested). Every upsert here is sent with wait=true in batches of 256 over gRPC; the strace used one point per request, so one flush per batch is inferred, not measured. Segment files are flushed every 5 s; log segments 32 MB, 0 created ahead. /qdrant/config/config.yaml in container gvb-qdrant (the image's own file; its storage keys are listed) sets: storage.collection.quantization = null, storage.collection.replication_factor = 1, storage.collection.vectors.on_disk = null, storage.collection.write_consistency_factor = 1, storage.hnsw_index.ef_construct = 100, storage.hnsw_index.full_scan_threshold_kb = 10000, storage.hnsw_index.m = 16, storage.hnsw_index.max_indexing_threads = 0, storage.hnsw_index.on_disk = false, storage.hnsw_index.payload_m = null, storage.max_collections = null, storage.node_type = Normal, storage.on_disk_payload = true, storage.optimizers.default_segment_number = 0, storage.optimizers.deleted_threshold = 0.2, storage.optimizers.flush_interval_sec = 5, storage.optimizers.indexing_threshold_kb = 10000, storage.optimizers.max_optimization_threads = null, storage.optimizers.max_segment_size_kb = null, storage.optimizers.vacuum_min_vector_number = 1000, storage.performance.max_search_threads = 0, storage.performance.optimizer_cpu_budget = 0, storage.performance.update_rate_limit = null, storage.shard_transfer_method = null, storage.snapshots_config.snapshots_storage = local, storage.snapshots_path = ./snapshots, storage.storage_path = ./storage, storage.temp_path = null, storage.update_concurrency = null, storage.wal.wal_capacity_mb = 32, storage.wal.wal_segments_ahead = 0; no QDRANT__ environment overrides in the server process. Not tested by cutting power; whether the disk's own write cache reaches the media was not checked.. - Load average 1.83 5.04 4.10 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@8 102,965 searches in 30.0 s, default@1 32,695 searches in 30.0 s, exact 33,539 searches in 30.0 s. - Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure. default@8: warm-up 15.0 s and 52,128 searches (0 failed) at 8 searchers; QPS of the last windows 3,508, 3,475, 3,438 (windows of at least 2 s and 100 searches); trial of 10,382 searches in 3.0 s at 8 searchers: 3,458 QPS against the settled 3,475 QPS, 0% apart (limit 10%); default@1: warm-up 15.0 s and 16,345 searches (0 failed) at 1 searcher; p50 of the last windows 0.908, 0.907, 0.906 ms (windows of at least 2 s and 100 searches); trial of 3,246 searches in 3.0 s at 1 searcher: p50 0.911 ms against the settled p50 0.907 ms, 0% apart (limit 10%); exact: warm-up 15.0 s and 16,730 searches (0 failed) at 1 searcher; p50 of the last windows 0.888, 0.887, 0.887 ms (windows of at least 2 s and 100 searches); trial of 3,340 searches in 3.0 s at 1 searcher: p50 0.882 ms against the settled p50 0.887 ms, 1% apart (limit 10%). - Pass default@8 after rehearsal: 2026-10-05T07:35:28.329Z to 2026-10-05T07:35:48.332Z (20.0 s), 69,472 searches, 0 failed, p50 2.214 ms, mean 2.301 ms, p99 4.029 ms, 3473.2 QPS (1000/QPS 0.288 ms). - Pass default@1 after default@8: 2026-10-05T07:36:06.416Z to 2026-10-05T07:36:26.416Z (20.0 s), 21,860 searches, 0 failed, p50 0.904 ms, mean 0.913 ms, p99 1.392 ms, 1093.0 QPS (1000/QPS 0.915 ms). - Pass exact after default@1: 2026-10-05T07:36:44.455Z to 2026-10-05T07:37:44.455Z (60.0 s), 66,988 searches, 0 failed, p50 0.884 ms, mean 0.894 ms, p99 1.374 ms, 1116.5 QPS (1000/QPS 0.896 ms). - Passes in the order run: default@8, default@1, exact; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 271,370 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (status Green, optimizer ok, indexed_vectors_count 524 of 524 points, 2 segments, indexing_threshold_kb 1, full_scan_threshold_kb 10; every one of 1 non-empty segments has a complete HNSW graph above its full-scan threshold; 309,093 segment searches walked the graph and none scanned; searches counted since the collection was created: unfiltered_hnsw 309,093, unfiltered_plain 0, unfiltered_exact 120,597). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-qdrant (7b3b8a9d21fb) on network gvb-qdrant_default at container-ip:6334, not through docker-proxy localhost:16334; container gvb-qdrant (7b3b8a9d21fb) on network gvb-qdrant_default at container-ip:6333, not through docker-proxy localhost:16333. Open after the passes: localhost:1433 x2 (not this target), container-ip:6334 x1 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-qdrant: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-qdrant: 24 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@8 3492/3492 MHz, performance, outside load 0.12 before the warm-up, 0.14 during, client CPU 0.513 ms per search; default@1 3492/3492 MHz, performance, outside load 0.13 before the warm-up, 0.10 during, client CPU 0.716 ms per search; exact 3492/3492 MHz, performance, outside load 0.12 before the warm-up, 0.11 during, client CPU 0.722 ms per search. ### milvus - Engine: Milvus 2.6.25 (standalone, embedded etcd, local storage, REST API v2) (compose) - Index: HNSW M=16 efConstruction=128, ef=100, metric COSINE, Strong consistency searches; approximate only (no exact mode) - Search settings: M=16, ef=100, efConstruction=128 - Load: 524 rows in batches of 1000, 0.5 s of upserts (1,073 rows/s); count matched 1.6 s after the last upsert; index step 138.8 s (index Finished, type HNSW (COSINE, {"M":16,"efConstruction":128}), indexedRows 524 of 524 sealed rows, pendingRows 0; stored rows 524; LoadStateLoaded; query node: 1 sealed segment(s) with the index loaded covering 524 rows against 524 stored rows, 0 sealed without it, 0 rows in growing segments; datacoord: 1 live segment(s), 1 flushed and indexed covering 524 rows, 0 delete-log (L0) segment(s) waiting for a compaction, 3 segment(s) compacted away so far; flush, index build and load took 14.7 s, then compaction: 1 job(s) accepted by Milvus, layout ready and unchanged for 75 s, 124.1 s for the whole step; search ledger started, later state reads count the searches against the query node's own counters) - Search: 8817 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 0.55 ms, 8 searchers 0.62 ms - RAM: 183.3 MiB (docker stats); disk: 10.86 MiB added by this load (175.66 MiB (whole engine data folder), 164.8 MiB before) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started milvus.compose.yaml with every container created on CPUs 2-3,6-7: gvb-milvus cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - Index after the load: ready, 524 of 524 indexed (index Finished, type HNSW (COSINE, {"M":16,"efConstruction":128}), indexedRows 524 of 524 sealed rows, pendingRows 0; stored rows 524; LoadStateLoaded; query node: 1 sealed segment(s) with the index loaded covering 524 rows against 524 stored rows, 0 sealed without it, 0 rows in growing segments; datacoord: 1 live segment(s), 1 flushed and indexed covering 524 rows, 0 delete-log (L0) segment(s) waiting for a compaction, 3 segment(s) compacted away so far; search ledger since the index finished: the sink sent 0 searches (searchParams.params.ef=100, Strong consistency), the query node counted 0 search requests for this collection, segment searches Sealed +0 and Growing +0, the same sealed segments with the same loaded index builds at both readings, segments compacted away +0). Durability: Writes are not fsynced. Standalone uses the default message queue rocksmq (RocksDB, /var/lib/milvus/rdb_data, mq.type default) and flushed segments go to local-disk object storage (COMMON_STORAGETYPE=local in milvus.compose.yaml). Measured 2026-10-04 with strace over 30 acknowledged single-row upserts and one flush: fdatasync ran only on the embedded etcd files (member/wal, member/snap/db), never on rdb_data or the segment files. A host power loss or kernel crash can lose acknowledged rows that are still in the OS page cache; a Milvus process crash alone should not (inferred, not tested). Metadata in embedded etcd is fdatasynced on every commit.. - Load average 0.45 2.47 3.34 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@8 38,466 searches in 30.0 s, default@1 13,214 searches in 30.0 s. - Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure. default@8: warm-up 15.0 s and 19,336 searches (0 failed) at 8 searchers; QPS of the last windows 1,283, 1,302, 1,298 (windows of at least 2 s and 100 searches); trial of 3,902 searches in 3.0 s at 8 searchers: 1,300 QPS against the settled 1,298 QPS, 0% apart (limit 10%); default@1: warm-up 15.0 s and 6,585 searches (0 failed) at 1 searcher; p50 of the last windows 2.161, 2.168, 2.160 ms (windows of at least 2 s and 100 searches); trial of 1,319 searches in 3.0 s at 1 searcher: p50 2.159 ms against the settled p50 2.161 ms, 0% apart (limit 10%). - Pass default@8 after rehearsal: 2026-10-05T07:41:33.646Z to 2026-10-05T07:41:53.649Z (20.0 s), 25,814 searches, 0 failed, p50 5.675 ms, mean 6.196 ms, p99 13.995 ms, 1290.5 QPS (1000/QPS 0.775 ms). - Pass default@1 after default@8: 2026-10-05T07:42:11.658Z to 2026-10-05T07:42:31.659Z (20.0 s), 8,817 searches, 0 failed, p50 2.159 ms, mean 2.266 ms, p99 3.531 ms, 440.8 QPS (1000/QPS 2.268 ms). - Passes in the order run: default@8, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 82,822 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (index Finished, type HNSW (COSINE, {"M":16,"efConstruction":128}), indexedRows 524 of 524 sealed rows, pendingRows 0; stored rows 524; LoadStateLoaded; query node: 1 sealed segment(s) with the index loaded covering 524 rows against 524 stored rows, 0 sealed without it, 0 rows in growing segments; datacoord: 1 live segment(s), 1 flushed and indexed covering 524 rows, 0 delete-log (L0) segment(s) waiting for a compaction, 3 segment(s) compacted away so far; search ledger since the index finished: the sink sent 117453 searches (searchParams.params.ef=100, Strong consistency), the query node counted 117453 search requests for this collection, segment searches Sealed +115475 and Growing +0, the same sealed segments with the same loaded index builds at both readings, segments compacted away +0). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-milvus (e6154ee62ada) on network gvb-milvus_default at container-ip:19530, not through docker-proxy localhost:19530; container gvb-milvus (e6154ee62ada) on network gvb-milvus_default at container-ip:9091, not through docker-proxy localhost:9091. Open after the passes: container-ip:19530 x8 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-milvus: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-milvus: 24 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@8 3493/3494 MHz, performance, outside load 0.15 before the warm-up, 0.17 during, client CPU 0.625 ms per search; default@1 3537/3498 MHz, performance, outside load 0.16 before the warm-up, 0.12 during, client CPU 0.546 ms per search. ### oracle - Engine: Oracle AI Database Free 23.26 (23ai line, VECTOR FLOAT32) (compose) - Index: HNSW in-memory neighbor graph NEIGHBORS=16 EFCONSTRUCTION=128, EFSEARCH=100 per query, cosine; exact mode = FETCH EXACT FIRST (full scan); Oracle Free caps itself at 2 CPUs (cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit) - Search settings: EFCONSTRUCTION=128, EFSEARCH=100, NEIGHBORS=16 - Load: 524 rows in batches of 1000, 0.3 s of upserts (1,938 rows/s); count matched 0.1 s after the last upsert; index step 3.4 s (HNSW graph holds 524 of 524 rows, change log waiting: 0 inserts and 0 deletes, USER_INDEXES status VALID, plan of the default search: VECTOR INDEX HNSW SCAN > TABLE ACCESS BY INDEX ROWID, index used by 0 queries so far; Oracle Free caps itself at 2 CPUs (cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit); graph rebuilt in 2.2 s) - Search: 22077 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 0.85 ms, 8 searchers 0.64 ms - Exact mode: 24,539 searches, p50 2.37 ms, p95 2.91 ms, recall 1.000 - RAM: 2.12 GiB (docker stats); disk: 0 B added by this load (6.88 GiB (whole engine data folder), 6.88 GiB before) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started oracle.compose.yaml with every container created on CPUs 2-3,6-7: gvb-oracle cpuset 2-3,6-7, gvb-oracle-seed cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - Index after the load: ready, 524 of 524 indexed (HNSW graph holds 524 of 524 rows, change log waiting: 0 inserts and 0 deletes, USER_INDEXES status VALID, plan of the default search: VECTOR INDEX HNSW SCAN > TABLE ACCESS BY INDEX ROWID, index used by 0 queries so far; Oracle Free caps itself at 2 CPUs (cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit)). Durability: By these settings a committed row should survive a crash or power loss. The sink uses a plain COMMIT and commit_logging, commit_wait and commit_write are unset in the database, so every commit waits for its redo to be written. Measured 2026-10-04: 50 separate client commits raised V$SYSSTAT 'redo synch writes' by 55 (the 50 commits plus the CREATE and DROP of the probe table), and the log writer and the datafile writer hold their files open with O_DSYNC (open flags 02110002, filesystemio_options none). The database runs NOARCHIVELOG (V$DATABASE.LOG_MODE), so redo serves crash recovery only and there is no point-in-time restore. The HNSW graph lives in the 768 MB vector memory pool (oracle-init/01-vector-memory.sh) and is not the durable copy; the table is. Not tested by cutting power; whether the disk's own write cache reaches the media was not checked. CPU: Oracle Free caps itself at 2 CPUs (measured 2026-10-04: cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit; this sink reads the live value when the index state is read).. - Load average 3.45 3.26 3.53 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: exact 12,352 searches in 30.0 s, default@8 89,522 searches in 30.0 s, default@1 33,035 searches in 30.0 s. - WARNING: latency had NOT settled when timing began for default@8 (each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure). exact: warm-up 15.0 s and 6,134 searches (0 failed) at 1 searcher; p50 of the last windows 2.383, 2.353, 2.360 ms (windows of at least 2 s and 100 searches); trial of 1,220 searches in 3.0 s at 1 searcher: p50 2.382 ms against the settled p50 2.360 ms, 1% apart (limit 10%); default@8: warm-up 15.0 s and 43,925 searches (0 failed) at 8 searchers; QPS of the last windows 2,416, 3,480, 2,581 (windows of at least 2 s and 100 searches) (its last 3 windows differed by more than 5%); trial of 9,374 searches in 3.0 s at 8 searchers: 3,124 QPS against the settled 2,581 QPS, 17% apart (limit 10%), so the warm-up was EXTENDED once (120.3 s and 343,792 searches (0 failed) at 8 searchers; QPS of the last windows 3,449, 2,488, 3,050 (windows of at least 2 s and 100 searches), the 120 s cap ran out); second trial of 9,654 searches in 3.0 s at 8 searchers: 3,217 QPS against the settled 3,050 QPS, 5% apart (limit 10%); still disagreeing after the extension; default@1: warm-up 15.0 s and 16,210 searches (0 failed) at 1 searcher; p50 of the last windows 0.894, 0.885, 0.895 ms (windows of at least 2 s and 100 searches); trial of 3,375 searches in 3.0 s at 1 searcher: p50 0.867 ms against the settled p50 0.894 ms, 3% apart (limit 10%). Its numbers may still include warm-up; rerun before quoting them. - Pass exact after rehearsal: 2026-10-05T07:44:41.053Z to 2026-10-05T07:45:41.054Z (60.0 s), 24,539 searches, 0 failed, p50 2.373 ms, mean 2.443 ms, p99 3.969 ms, 409.0 QPS (1000/QPS 2.445 ms). - Pass default@8 after exact: 2026-10-05T07:48:17.365Z to 2026-10-05T07:48:37.485Z (20.1 s), 57,927 searches, 0 failed, p50 1.581 ms, mean 2.776 ms, p99 3.454 ms, 2879.2 QPS (1000/QPS 0.347 ms). - Pass default@1 after default@8: 2026-10-05T07:48:55.504Z to 2026-10-05T07:49:15.505Z (20.0 s), 22,077 searches, 0 failed, p50 0.884 ms, mean 0.904 ms, p99 1.755 ms, 1103.8 QPS (1000/QPS 0.906 ms). - Passes in the order run: exact, default@8, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 611,684 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (HNSW graph holds 524 of 524 rows, change log waiting: 0 inserts and 0 deletes, USER_INDEXES status VALID, plan of the default search: VECTOR INDEX HNSW SCAN > TABLE ACCESS BY INDEX ROWID, index used by 671982 queries so far; Oracle Free caps itself at 2 CPUs (cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit)). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-oracle (5a3115443a16) on network gvb-oracle_default at container-ip:1521, not through docker-proxy localhost:1521. Open after the passes: container-ip:1521 x14 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-oracle: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-oracle: 86 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): exact 3492/3492 MHz, performance, outside load 0.17 before the warm-up, 0.16 during, client CPU 0.923 ms per search; default@8 3492/3492 MHz, performance, outside load 0.16 before the warm-up, 0.18 during, client CPU 0.637 ms per search; default@1 3492/3492 MHz, performance, outside load 0.17 before the warm-up, 0.16 during, client CPU 0.853 ms per search. ### weaviate - Engine: Weaviate 1.39.8 (single node, REST + GraphQL) (compose) - Index: HNSW maxConnections(M)=16 efConstruction=128, ef=-1 (dynamic: limit x 8 clamped 100..500), cosine, no quantization; approximate only (no exact mode) - Search settings: ef=-1, efConstruction=128 - Load: 524 rows in batches of 1000, 0.5 s of upserts (1,007 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (HNSW is updated inside every batch, nothing to build; confirmed in 0.0 s: shard JOIqxslXLYy9 vectorIndexingStatus READY, vectorQueueLength 0, node status objectCount 0 (refreshed only when the memtable is flushed, so it lags a minute and does not decide readiness), schema shard JOIqxslXLYy9 status READY, Aggregate count 524, Aggregate nearVector with objectLimit 524 reached 524, Weaviate reports no count of vectors in the HNSW index, so indexed vectors are not reported) - Search: 3425 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 0.57 ms, 8 searchers 0.60 ms - RAM: 96.28 MiB (docker stats); disk: 2.96 MiB added by this load (5.02 MiB (whole engine data folder), 2.06 MiB before) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started weaviate.compose.yaml with every container created on CPUs 2-3,6-7: gvb-weaviate cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - Index after the load: ready, ? of 524 indexed (shard JOIqxslXLYy9 vectorIndexingStatus READY, vectorQueueLength 0, node status objectCount 0 (refreshed only when the memtable is flushed, so it lags a minute and does not decide readiness), schema shard JOIqxslXLYy9 status READY, Aggregate count 524, Aggregate nearVector with objectLimit 524 reached 524, Weaviate reports no count of vectors in the HNSW index, so indexed vectors are not reported). Durability: Weaviate 1.39.8 defaults, weaviate.compose.yaml sets no persistence variable: every object is appended to the LSM write-ahead log with a plain write and no fsync, and the log is fsynced only when its memtable is flushed, 60 seconds after the last write (PERSISTENCE_MEMTABLES_FLUSH_IDLE_AFTER_SECONDS default; measured with strace: 2,080 writes into the objects log during a 2,000-object load, the first fsync 60 s after the last write), so a power cut can lose the last minute of writes; the HNSW commit log is buffered inside the process (77 writes for 2,000 vectors), so a killed process loses the newest vectors from the vector index while their objects survive (measured with docker kill, which is SIGKILL, one second after the load and a restart, two runs each: 0 to 1 of 50, 264 to 267 of 300 and 1,992 to 1,997 of 2,000 stored objects were still reachable through a vector search). - Load average 2.53 3.68 3.64 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@1 5,172 searches in 30.0 s, default@8 9,473 searches in 30.0 s. - Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure. default@1: warm-up 15.0 s and 2,581 searches (0 failed) at 1 searcher; p50 of the last windows 5.193, 5.180, 5.188 ms (windows of at least 2 s and 100 searches); trial of 514 searches in 3.0 s at 1 searcher: p50 5.182 ms against the settled p50 5.188 ms, 0% apart (limit 10%); default@8: warm-up 15.0 s and 4,682 searches (0 failed) at 8 searchers; QPS of the last windows 312, 310, 314 (windows of at least 2 s and 100 searches); trial of 944 searches in 3.0 s at 8 searchers: 312 QPS against the settled 312 QPS, 0% apart (limit 10%). - Pass default@1 after rehearsal: 2026-10-05T07:51:01.502Z to 2026-10-05T07:51:21.506Z (20.0 s), 3,425 searches, 0 failed, p50 5.186 ms, mean 5.838 ms, p99 11.403 ms, 171.2 QPS (1000/QPS 5.840 ms). - Pass default@8 after default@1: 2026-10-05T07:51:39.547Z to 2026-10-05T07:51:59.562Z (20.0 s), 6,248 searches, 0 failed, p50 24.367 ms, mean 25.620 ms, p99 52.339 ms, 312.2 QPS (1000/QPS 3.203 ms). - Passes in the order run: default@1, default@8; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 23,366 sent, 0 failed (warmupErrors). Index after the searches: ready, ? of 524 indexed (shard JOIqxslXLYy9 vectorIndexingStatus READY, vectorQueueLength 0, node status objectCount 524 (refreshed only when the memtable is flushed, so it lags a minute and does not decide readiness), schema shard JOIqxslXLYy9 status READY, Aggregate count 524, Aggregate nearVector with objectLimit 524 reached 524, Weaviate reports no count of vectors in the HNSW index, so indexed vectors are not reported). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-weaviate (3f3f05090b3b) on network gvb-weaviate_default at container-ip:8080, not through docker-proxy localhost:8085. Open after the passes: container-ip:8080 x8 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-weaviate: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-weaviate: 9 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@1 3587/3557 MHz, performance, outside load 0.15 before the warm-up, 0.14 during, client CPU 0.567 ms per search; default@8 3592/3573 MHz, performance, outside load 0.14 before the warm-up, 0.14 during, client CPU 0.604 ms per search. ### clickhouse - Engine: ClickHouse 26.3.39.7 (MergeTree + vector_similarity index) (compose) - Index: vector_similarity HNSW cosineDistance, quantization bf16, M=16 ef_construction=128, hnsw_candidate_list_size_for_search=256, rescoring off; exact mode = full scan with skip indexes off - Search settings: M=16, ef_construction=128, hnsw_candidate_list_size_for_search=256 - Load: 524 rows in batches of 1000, 0.2 s of upserts (2,520 rows/s); count matched 0.0 s after the last upsert; index step 6.2 s (1 data part(s), index files on 1 of them, 524 of 524 live rows in indexed parts; 0 merge(s) running, 0 unfinished mutation(s); plan of the default search uses the Skip index vec_idx on 1 of 1 parts; wait took 6.2 s) - Search: 4304 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 0.83 ms, 8 searchers 1.02 ms - Exact mode: 9,609 searches, p50 6.09 ms, p95 6.96 ms, recall 1.000 - RAM: 231.1 MiB (docker stats); disk: 3.25 MiB added by this load (3.38 MiB (whole engine data folder), 136.21 KiB before) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started clickhouse.compose.yaml with every container created on CPUs 2-3,6-7: gvb-clickhouse cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - Index after the load: ready, 524 of 524 indexed (1 data part(s), index files on 1 of them, 524 of 524 live rows in indexed parts; 0 merge(s) running, 0 unfinished mutation(s); plan of the default search uses the Skip index vec_idx on 1 of 1 parts). Durability: Acknowledged inserts are not fsynced. The sink creates plain MergeTree tables, so the MergeTree settings are the defaults: fsync_after_insert 0 and fsync_part_directory 0 (system.merge_tree_settings), and async_insert 1 with wait_for_async_insert 1 is the server default for the gvb login (system.settings), so an insert is acknowledged once its part is written, not once it is on disk. Measured 2026-10-04 with strace over 30 acknowledged single-row inserts (30 Ok rows in system.asynchronous_insert_log): zero fsync, fdatasync or sync_file_range calls, while the control, a table created with fsync_after_insert=1, made 61 fdatasync calls for 5 inserts. A host power loss or kernel crash can lose acknowledged rows that are still in the OS page cache; a ClickHouse process crash alone should not (inferred, not tested).. - Load average 2.69 3.28 3.49 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: exact 4,874 searches in 30.0 s, default@1 6,464 searches in 30.0 s, default@8 14,539 searches in 30.0 s. - WARNING: latency had NOT settled when timing began for default@8 (each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure). exact: warm-up 15.0 s and 2,395 searches (0 failed) at 1 searcher; p50 of the last windows 6.058, 6.063, 6.057 ms (windows of at least 2 s and 100 searches); trial of 488 searches in 3.0 s at 1 searcher: p50 6.032 ms against the settled p50 6.058 ms, 0% apart (limit 10%); default@1: warm-up 15.0 s and 3,279 searches (0 failed) at 1 searcher; p50 of the last windows 4.472, 4.478, 4.471 ms (windows of at least 2 s and 100 searches); trial of 654 searches in 3.0 s at 1 searcher: p50 4.486 ms against the settled p50 4.472 ms, 0% apart (limit 10%); default@8: warm-up 15.0 s and 7,362 searches (0 failed) at 8 searchers; QPS of the last windows 434, 504, 522 (windows of at least 2 s and 100 searches) (its last 3 windows differed by more than 5%); trial of 1,512 searches in 3.0 s at 8 searchers: 502 QPS against the settled 504 QPS, 0% apart (limit 10%), so the warm-up was EXTENDED once (38.1 s and 18,460 searches (0 failed) at 8 searchers; QPS of the last windows 525, 524, 488 (windows of at least 2 s and 100 searches), its last 3 windows differed by more than 5%); second trial of 1,302 searches in 3.0 s at 8 searchers: 432 QPS against the settled 524 QPS, 21% apart (limit 10%); still disagreeing after the extension. Its numbers may still include warm-up; rerun before quoting them. - Pass exact after rehearsal: 2026-10-05T07:54:02.179Z to 2026-10-05T07:55:02.182Z (60.0 s), 9,609 searches, 0 failed, p50 6.088 ms, mean 6.241 ms, p99 9.109 ms, 160.1 QPS (1000/QPS 6.244 ms). - Pass default@1 after exact: 2026-10-05T07:55:20.193Z to 2026-10-05T07:55:40.197Z (20.0 s), 4,304 searches, 0 failed, p50 4.501 ms, mean 4.645 ms, p99 6.884 ms, 215.2 QPS (1000/QPS 4.648 ms). - Pass default@8 after default@1: 2026-10-05T07:56:54.305Z to 2026-10-05T07:57:14.316Z (20.0 s), 9,921 searches, 0 failed, p50 15.604 ms, mean 16.130 ms, p99 28.788 ms, 495.8 QPS (1000/QPS 2.017 ms). - Passes in the order run: exact, default@1, default@8; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 68,576 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (1 data part(s), index files on 1 of them, 524 of 524 live rows in indexed parts; 0 merge(s) running, 0 unfinished mutation(s); plan of the default search uses the Skip index vec_idx on 1 of 1 parts). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-clickhouse (64730ab84aad) on network gvb-clickhouse_default at container-ip:8123, not through docker-proxy localhost:8123. Open after the passes: container-ip:8123 x8 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-clickhouse: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-clickhouse: 662 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): exact 3592/3592 MHz, performance, outside load 0.15 before the warm-up, 0.17 during, client CPU 0.846 ms per search; default@1 3574/3585 MHz, performance, outside load 0.17 before the warm-up, 0.17 during, client CPU 0.826 ms per search; default@8 3542/3552 MHz, performance, outside load 0.13 before the warm-up, 0.14 during, client CPU 1.022 ms per search. ### sqlitevec - Engine: SQLite 3.53.3 + sqlite-vec 0.1.7-alpha.2.1 (vec0, embedded, in process) (embedded) - Index: vec0 brute-force scan, no ANN index (exact), float32, cosine distance, default chunk_size=1024; score = 1 - cosine distance; searches run concurrently, one WAL reader connection per searcher (opened as searchers arrive, at most 32), writes run one at a time and may overlap searches - Search settings: chunk_size=1024 - Load: 524 rows in batches of 1000, 0.1 s of upserts (5,046 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (nothing to build, every search scans all vectors; checked in 0.00 s: no index, exact scan by design: gvb_gvbbench_eshoponweb is a vec0 virtual table holding 524 vectors and every search compares all of them (EXPLAIN QUERY PLAN: SCAN gvb_gvbbench_eshoponweb VIRTUAL TABLE INDEX 0:3{___}___ | USE TEMP B-TREE FOR ORDER BY)) - Search: 10845 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 1.93 ms, 8 searchers 3.44 ms - Exact mode: 32,307 searches, p50 1.83 ms, p95 2.04 ms, recall 1.000 - RAM: in this process, not measured; disk: 9.72 MiB added by this load (9.72 MiB (engine data folder), 0 B before) - Index after the load: ready, ? of 524 indexed (no index, exact scan by design: gvb_gvbbench_eshoponweb is a vec0 virtual table holding 524 vectors and every search compares all of them (EXPLAIN QUERY PLAN: SCAN gvb_gvbbench_eshoponweb VIRTUAL TABLE INDEX 0:3{___}___ | USE TEMP B-TREE FOR ORDER BY)). Durability: PRAGMA journal_mode=WAL and PRAGMA synchronous=NORMAL (set by this sink when it opens the file): each commit is appended to the -wal file and handed to the operating system, but the WAL is fsynced only when SQLite checkpoints it (measured with strace: 40 single-record commits caused 3 WAL fsyncs), so a crash of the process loses nothing (measured with kill -9 and a reopen: all 50, 300 and 2,000 rows were there), while an operating-system crash or power cut can lose the newest commits since the last checkpoint and leaves the database consistent. - WARNING: load average 10.04 5.74 4.35 (1/5/15 min) on 8 logical CPUs when searching began; the box was busy, so latency and QPS are inflated by other work. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@8 34,725 searches in 30.0 s, default@1 16,117 searches in 30.0 s, exact 16,108 searches in 30.0 s. - Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure. default@8: warm-up 15.0 s and 17,387 searches (0 failed) at 8 searchers; QPS of the last windows 1,162, 1,158, 1,158 (windows of at least 2 s and 100 searches); trial of 3,477 searches in 3.0 s at 8 searchers: 1,158 QPS against the settled 1,158 QPS, 0% apart (limit 10%); default@1: warm-up 15.0 s and 8,077 searches (0 failed) at 1 searcher; p50 of the last windows 1.838, 1.826, 1.823 ms (windows of at least 2 s and 100 searches); trial of 1,617 searches in 3.0 s at 1 searcher: p50 1.831 ms against the settled p50 1.826 ms, 0% apart (limit 10%); exact: warm-up 15.0 s and 8,078 searches (0 failed) at 1 searcher; p50 of the last windows 1.820, 1.837, 1.830 ms (windows of at least 2 s and 100 searches); trial of 1,612 searches in 3.0 s at 1 searcher: p50 1.832 ms against the settled p50 1.830 ms, 0% apart (limit 10%). - Pass default@8 after rehearsal: 2026-10-05T07:59:09.161Z to 2026-10-05T07:59:29.164Z (20.0 s), 23,231 searches, 0 failed, p50 6.663 ms, mean 6.882 ms, p99 13.384 ms, 1161.4 QPS (1000/QPS 0.861 ms). - Pass default@1 after default@8: 2026-10-05T07:59:47.172Z to 2026-10-05T08:00:07.173Z (20.0 s), 10,845 searches, 0 failed, p50 1.816 ms, mean 1.842 ms, p99 2.267 ms, 542.2 QPS (1000/QPS 1.844 ms). - Pass exact after default@1: 2026-10-05T08:00:25.185Z to 2026-10-05T08:01:25.186Z (60.0 s), 32,307 searches, 0 failed, p50 1.832 ms, mean 1.855 ms, p99 2.244 ms, 538.4 QPS (1000/QPS 1.857 ms). - Passes in the order run: default@8, default@1, exact; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 107,198 sent, 0 failed (warmupErrors). Index after the searches: ready, ? of 524 indexed (no index, exact scan by design: gvb_gvbbench_eshoponweb is a vec0 virtual table holding 524 vectors and every search compares all of them (EXPLAIN QUERY PLAN: SCAN gvb_gvbbench_eshoponweb VIRTUAL TABLE INDEX 0:3{___}___ | USE TEMP B-TREE FOR ORDER BY)). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: in this process (embedded), no network: no network address. No open connection seen (no open TCP connection was found). - CPU pinning: none: an embedded engine runs inside the benchmark client process, so it shares the client CPUs, CPUs 0-1,4-5. - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@8 3592/3592 MHz, performance, outside load 0.15 before the warm-up, 0.16 during, client CPU 3.438 ms per search; default@1 3593/3592 MHz, performance, outside load 0.16 before the warm-up, 0.15 during, client CPU 1.932 ms per search; exact 3598/3592 MHz, performance, outside load 0.16 before the warm-up, 0.15 during, client CPU 1.948 ms per search. ### mongodb - Engine: MongoDB 8.0.32 Atlas Local (mongod + mongot Vector Search) (compose) - Index: vectorSearch index, HNSW maxEdges=16 numEdgeCandidates=128, float32 binData, cosine, numCandidates=20x hits (min 100); exact mode = $vectorSearch exact:true - Search settings: maxEdges=16, numCandidates=20x, numEdgeCandidates=128 - Load: 524 rows in batches of 1000, 0.2 s of upserts (3,084 rows/s); count matched 1.7 s after the last upsert; index step 10.2 s (index status READY, queryable True; mongot holds 524 of 524 documents in 2 segment(s) (269 + 255 documents), 2 searched through the HNSW graph (Approximate); segment layout unchanged for 10.2 s over 11 readings; wait took 10.2 s) - Search: 14100 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 0.30 ms, 8 searchers 0.29 ms - Exact mode: 48,332 searches, p50 1.22 ms, p95 1.40 ms, recall 1.000 - RAM: 684.2 MiB (docker stats); disk: 0 B added by this load (1.36 GiB (whole engine data folder), 1.41 GiB before) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started mongodb.compose.yaml with every container created on CPUs 2-3,6-7: gvb-mongodb cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - Index after the load: ready, 524 of 524 indexed (index status READY, queryable True; mongot holds 524 of 524 documents in 2 segment(s) (269 + 255 documents), 2 searched through the HNSW graph (Approximate)). Durability: Acknowledged writes are journaled before the acknowledgement. The sink sets no write concern, so the server default applies: getDefaultRWConcern gives w majority and the one-member replica set (--replSet gvbmongo in the image) has writeConcernMajorityJournalDefault true, with WiredTiger journaling on (journalCommitInterval 100 ms is only the interval for unacknowledged work). Measured 2026-10-04: 30 acknowledged single-document inserts raised WiredTiger 'log sync operations' by 30 and strace showed fdatasync on /data/db/journal/WiredTigerLog files. A crash loses no acknowledged write. mongot's search index is not part of that promise: it follows the collection asynchronously and is rebuilt from it.. - Load average 2.17 4.25 4.13 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@1 17,985 searches in 30.0 s, exact 23,442 searches in 30.0 s, default@8 61,056 searches in 30.0 s. - Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure. default@1: warm-up 15.0 s and 10,587 searches (0 failed) at 1 searcher; p50 of the last windows 1.396, 1.392, 1.382 ms (windows of at least 2 s and 100 searches); trial of 2,096 searches in 3.0 s at 1 searcher: p50 1.381 ms against the settled p50 1.392 ms, 1% apart (limit 10%); exact: warm-up 15.0 s and 12,070 searches (0 failed) at 1 searcher; p50 of the last windows 1.205, 1.222, 1.205 ms (windows of at least 2 s and 100 searches); trial of 2,442 searches in 3.0 s at 1 searcher: p50 1.206 ms against the settled p50 1.205 ms, 0% apart (limit 10%); default@8: warm-up 15.0 s and 31,487 searches (0 failed) at 8 searchers; QPS of the last windows 2,100, 2,094, 2,108 (windows of at least 2 s and 100 searches); trial of 6,302 searches in 3.0 s at 8 searchers: 2,099 QPS against the settled 2,100 QPS, 0% apart (limit 10%). - Pass default@1 after rehearsal: 2026-10-05T08:03:33.286Z to 2026-10-05T08:03:53.287Z (20.0 s), 14,100 searches, 0 failed, p50 1.389 ms, mean 1.416 ms, p99 1.966 ms, 705.0 QPS (1000/QPS 1.419 ms). - Pass exact after default@1: 2026-10-05T08:04:11.303Z to 2026-10-05T08:05:11.303Z (60.0 s), 48,332 searches, 0 failed, p50 1.217 ms, mean 1.239 ms, p99 1.747 ms, 805.5 QPS (1000/QPS 1.241 ms). - Pass default@8 after exact: 2026-10-05T08:05:29.348Z to 2026-10-05T08:05:49.351Z (20.0 s), 42,054 searches, 0 failed, p50 3.686 ms, mean 3.802 ms, p99 6.771 ms, 2102.4 QPS (1000/QPS 0.476 ms). - Passes in the order run: default@1, exact, default@8; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 167,467 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (index status READY, queryable True; mongot holds 524 of 524 documents in 2 segment(s) (269 + 255 documents), 2 searched through the HNSW graph (Approximate)). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-mongodb (388ee5b2e82f) on network gvb-mongodb_default at container-ip:27017, not through docker-proxy localhost:27017. Open after the passes: container-ip:27017 x10 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-mongodb: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-mongodb: 209 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@1 3492/3492 MHz, performance, outside load 0.20 before the warm-up, 0.25 during, client CPU 0.305 ms per search; exact 3492/3492 MHz, performance, outside load 0.22 before the warm-up, 0.24 during, client CPU 0.298 ms per search; default@8 3492/3492 MHz, performance, outside load 0.24 before the warm-up, 0.17 during, client CPU 0.292 ms per search. ### sql - Engine: Microsoft SQL Server 2025 (RTM-CU9) (KB5122048) - 17.0.5005.3 (X64), Enterprise Developer Edition (64-bit), in container gvb-mssql (image mcr.microsoft.com/mssql/server:2025-CU9-ubuntu-24.04@sha256:2b5b581621126574f3d1f75e78d3eebe8d05aedb59ad0cfdf9aa42cb0634d726; cpuset 2-3,6-7 from its creation; SQL Server counts 4 CPU(s) and runs 4 visible scheduler(s), affinity AUTO) (compose) - Index: exact VECTOR_DISTANCE cosine, no vector index (full scan) - Search settings: description=exact VECTOR_DISTANCE cosine, no vector index (full scan) - Load: 524 rows in batches of 1000, 0.9 s of upserts (601 rows/s); count matched 0.0 s after the last upsert; index step n/a - Search: 5015 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 1.19 ms, 8 searchers 1.19 ms - RAM: 528.9 MiB (docker stats); disk: 5.15 MiB (table and its indexes, reserved pages) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started mssql.compose.yaml with every container created on CPUs 2-3,6-7: gvb-mssql cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - Index after the load: ready, 0 of 524 indexed (no index used, exact scan by design: dbo.gvb_gvbbench_eshoponweb in GvbBench holds 524 rows; sys.vector_indexes lists no index on the table; indexes: PK_gvb_gvbbench_eshoponweb CLUSTERED, IX_gvb_gvbbench_eshoponweb_DocKey NONCLUSTERED; a real search under SET STATISTICS XML ON ran: Clustered Index Scan of GvbBench.gvb_gvbbench_eshoponweb through PK_gvb_gvbbench_eshoponweb (IndexKind Clustered), 524 rows read, no vector index operator, 10 hits). Durability: A commit returns after its transaction-log records are written to disk (SQL Server write-ahead logging; delayed durability is DISABLED); a database created here copies the model database: recovery model FULL, page_verify CHECKSUM; no global trace flags are enabled; mssql.conf of container gvb-mssql (/var/opt/mssql/mssql.conf, on the host at ~/gvb-data/engines/mssql/mssql.conf) does not exist, so it sets: nothing; it has no [control] or [traceflag] entry, so SQL Server's own Linux defaults for flushing writes apply (Microsoft's Linux performance guide names trace flag 3982 as that default; read from the guide, not tested here). Not tested by cutting power; whether the disk's own write cache reaches the media was not checked. Container settings from its environment (names only): MSSQL_AGENT_ENABLED, MSSQL_MEMORY_LIMIT_MB, MSSQL_PID, MSSQL_RPC_PORT, MSSQL_SA_PASSWORD.. - Load average 1.86 3.26 3.76 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@8 18,024 searches in 30.0 s, default@1 7,566 searches in 30.0 s. - Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure. default@8: warm-up 15.0 s and 9,299 searches (0 failed) at 8 searchers; QPS of the last windows 626, 606, 631 (windows of at least 2 s and 100 searches); trial of 1,853 searches in 3.0 s at 8 searchers: 616 QPS against the settled 626 QPS, 2% apart (limit 10%); default@1: warm-up 15.0 s and 3,713 searches (0 failed) at 1 searcher; p50 of the last windows 3.861, 3.860, 3.837 ms (windows of at least 2 s and 100 searches); trial of 766 searches in 3.0 s at 1 searcher: p50 3.861 ms against the settled p50 3.860 ms, 0% apart (limit 10%). - Pass default@8 after rehearsal: 2026-10-05T08:08:18.631Z to 2026-10-05T08:08:38.642Z (20.0 s), 12,435 searches, 0 failed, p50 12.361 ms, mean 12.866 ms, p99 19.789 ms, 621.4 QPS (1000/QPS 1.609 ms). - Pass default@1 after default@8: 2026-10-05T08:08:56.651Z to 2026-10-05T08:09:16.653Z (20.0 s), 5,015 searches, 0 failed, p50 3.890 ms, mean 3.985 ms, p99 6.004 ms, 250.7 QPS (1000/QPS 3.988 ms). - Passes in the order run: default@8, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 41,221 sent, 0 failed (warmupErrors). Index after the searches: ready, 0 of 524 indexed (no index used, exact scan by design: dbo.gvb_gvbbench_eshoponweb in GvbBench holds 524 rows; sys.vector_indexes lists no index on the table; indexes: PK_gvb_gvbbench_eshoponweb CLUSTERED, IX_gvb_gvbbench_eshoponweb_DocKey NONCLUSTERED; a real search under SET STATISTICS XML ON ran: Clustered Index Scan of GvbBench.gvb_gvbbench_eshoponweb through PK_gvb_gvbbench_eshoponweb (IndexKind Clustered), 524 rows read, no vector index operator, 10 hits). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-mssql (2932955cf2f4) on network gvb-mssql_default at container-ip:1433, not through docker-proxy localhost:14330. Open after the passes: container-ip:1433 x9 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-mssql: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-mssql: 85 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@8 3492/3492 MHz, performance, outside load 0.16 before the warm-up, 0.17 during, client CPU 1.186 ms per search; default@1 3503/3494 MHz, performance, outside load 0.16 before the warm-up, 0.16 during, client CPU 1.189 ms per search. ### elasticsearch - Engine: Elasticsearch 9.5.3 (dense_vector) (compose) - Index: HNSW float32, no quantization, m=16, ef_construction=128, cosine; search k=top, num_candidates=100; 1 shard, 0 replicas; force-merged to one segment after the load (at 1,024 dimensions a segment under 1,043 vectors gets no graph) - Search settings: ef_construction=128, k=top, m=16, num_candidates=100 - Load: 524 rows in batches of 1000, 1.2 s of upserts (442 rows/s); count matched 0.1 s after the last upsert; index step 0.4 s (FAILED after 0.4 s: NO HNSW GRAPH, searches scan all 524 vectors: 1 segment(s), 524 vectors, total_vex_size_bytes 0 (_stats dense_vector), index_options hnsw m=16 ef_construction=128; Lucene builds no graph for a segment this small (measured at 1,024 dimensions: 1,042 vectors none, 1,043 vectors a graph), so default search equals exact search) - Search: 15090 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 0.46 ms, 8 searchers 0.49 ms - Exact mode: 47,120 searches, p50 1.25 ms, p95 1.46 ms, recall 1.000 - RAM: 2.54 GiB (docker stats); disk: 2.4 MiB added by this load (2.61 MiB (whole engine data folder), 218.79 KiB before) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started elasticsearch.compose.yaml with every container created on CPUs 2-3,6-7: gvb-elasticsearch cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - WARNING: the index step after the load FAILED (NO HNSW GRAPH, searches scan all 524 vectors: 1 segment(s), 524 vectors, total_vex_size_bytes 0 (_stats dense_vector), index_options hnsw m=16 ef_construction=128; Lucene builds no graph for a segment this small (measured at 1,024 dimensions: 1,042 vectors none, 1,043 vectors a graph), so default search equals exact search). Searched anyway; see the index state. - WARNING: index not ready after the load: NOT ready, 0 of 524 indexed (NO HNSW GRAPH, searches scan all 524 vectors: 1 segment(s), 524 vectors, total_vex_size_bytes 0 (_stats dense_vector), index_options hnsw m=16 ef_construction=128; Lucene builds no graph for a segment this small (measured at 1,024 dimensions: 1,042 vectors none, 1,043 vectors a graph), so default search equals exact search). Measured anyway, so its search numbers may come from a scan or a half-built index. Durability: Every acknowledged bulk request is fsynced to the translog before the answer: index.translog.durability=request, the Elasticsearch default, which neither this sink nor elasticsearch.compose.yaml overrides (ElasticsearchReadinessTests reads it back from the live index). By that setting a process crash or power loss loses no acknowledged write; this is read from the setting, not shown by pulling power. One node and no replicas, so a lost disk loses the data.. - Load average 2.20 3.07 3.61 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@8 33,203 searches in 30.0 s, exact 22,172 searches in 30.0 s, default@1 22,634 searches in 30.0 s. - Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure. default@8: warm-up 15.0 s and 39,728 searches (0 failed) at 8 searchers; QPS of the last windows 2,678, 2,678, 2,719 (windows of at least 2 s and 100 searches); trial of 8,147 searches in 3.0 s at 8 searchers: 2,714 QPS against the settled 2,678 QPS, 1% apart (limit 10%); exact: warm-up 15.0 s and 11,764 searches (0 failed) at 1 searcher; p50 of the last windows 1.246, 1.255, 1.255 ms (windows of at least 2 s and 100 searches); trial of 2,356 searches in 3.0 s at 1 searcher: p50 1.252 ms against the settled p50 1.255 ms, 0% apart (limit 10%); default@1: warm-up 15.0 s and 11,295 searches (0 failed) at 1 searcher; p50 of the last windows 1.314, 1.305, 1.298 ms (windows of at least 2 s and 100 searches); trial of 2,266 searches in 3.0 s at 1 searcher: p50 1.304 ms against the settled p50 1.305 ms, 0% apart (limit 10%). - Pass default@8 after rehearsal: 2026-10-05T08:11:39.734Z to 2026-10-05T08:11:59.736Z (20.0 s), 54,138 searches, 0 failed, p50 2.830 ms, mean 2.953 ms, p99 5.326 ms, 2706.6 QPS (1000/QPS 0.369 ms). - Pass exact after default@8: 2026-10-05T08:12:17.753Z to 2026-10-05T08:13:17.754Z (60.0 s), 47,120 searches, 0 failed, p50 1.250 ms, mean 1.271 ms, p99 1.705 ms, 785.3 QPS (1000/QPS 1.273 ms). - Pass default@1 after exact: 2026-10-05T08:13:35.794Z to 2026-10-05T08:13:55.795Z (20.0 s), 15,090 searches, 0 failed, p50 1.304 ms, mean 1.323 ms, p99 1.742 ms, 754.5 QPS (1000/QPS 1.325 ms). - Passes in the order run: default@8, exact, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 153,565 sent, 0 failed (warmupErrors). Index after the searches: NOT ready, 0 of 524 indexed (NO HNSW GRAPH, searches scan all 524 vectors: 1 segment(s), 524 vectors, total_vex_size_bytes 0 (_stats dense_vector), index_options hnsw m=16 ef_construction=128; Lucene builds no graph for a segment this small (measured at 1,024 dimensions: 1,042 vectors none, 1,043 vectors a graph), so default search equals exact search). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-elasticsearch (52532159ceb5) on network engines_default at container-ip:9200, not through docker-proxy localhost:9200. Open after the passes: container-ip:9200 x1 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-elasticsearch: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-elasticsearch: 73 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@8 3492/3492 MHz, performance, outside load 0.15 before the warm-up, 0.17 during, client CPU 0.491 ms per search; exact 3492/3492 MHz, performance, outside load 0.16 before the warm-up, 0.16 during, client CPU 0.458 ms per search; default@1 3492/3492 MHz, performance, outside load 0.16 before the warm-up, 0.14 during, client CPU 0.459 ms per search. ### duckdb - Engine: DuckDB 1.5.6 + vss b833341 (HNSW, embedded, in process) (embedded) - Index: HNSW (vss extension) FLOAT[n] metric=cosine m=16 ef_construction=128, ef_search=100 per connection, persistent (hnsw_enable_experimental_persistence=true, checkpoint_threshold=256MB); exact mode = array_cosine_similarity sequential scan; score = 1 - cosine distance; searches run concurrently, one connection per searcher (opened as searchers arrive, at most 32), writes run one at a time and never overlap a search - Search settings: checkpoint_threshold=256MB, ef_construction=128, ef_search=100, hnsw_enable_experimental_persistence=true, m=16, metric=cosine - Load: 524 rows in batches of 1000, 0.3 s of upserts (1,574 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (HNSW is maintained inside every transaction, nothing to build; confirmed in 0.01 s: duckdb_indexes() lists gvb_gvbbench_eshoponweb_hnsw, pragma_hnsw_index_info() counts 524 vectors of 524 rows, EXPLAIN of the default search shows HNSW_INDEX_SCAN on it = True) - Search: 5612 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 4.84 ms, 8 searchers 6.66 ms - Exact mode: 14,598 searches, p50 4.04 ms, p95 4.76 ms, recall 1.000 - RAM: in this process, not measured; disk: 2.9 MiB added by this load (413.83 MiB (engine data folder), 410.93 MiB before) - Index after the load: ready, 524 of 524 indexed (duckdb_indexes() lists gvb_gvbbench_eshoponweb_hnsw, pragma_hnsw_index_info() counts 524 vectors of 524 rows, EXPLAIN of the default search shows HNSW_INDEX_SCAN on it = True). Durability: DuckDB's write-ahead log is fsynced at every commit and no DuckDbSink setting changes that (checkpoint_threshold=256MB only spaces out the checkpoints that write the database file; measured with strace: 42 WAL fsyncs for 40 single-record commits plus 2 setup statements), so an operating-system crash or power cut loses no committed row; measured with kill -9 right after the last commit and a reopen: all 50, 300 and 2,000 rows and an HNSW index that counted the same number were recovered from the log; not tested and documented by DuckDB: the HNSW index is file-backed only through hnsw_enable_experimental_persistence = true, which this sink turns on, and WAL recovery for such custom indexes is not complete, so a crash during a checkpoint or a later commit can damage the index while the rows survive. - Load average 2.06 3.12 3.57 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: exact 7,485 searches in 30.0 s, default@8 18,030 searches in 30.0 s, default@1 8,461 searches in 30.0 s. - Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure. exact: warm-up 15.0 s and 3,647 searches (0 failed) at 1 searcher; p50 of the last windows 4.068, 4.036, 4.007 ms (windows of at least 2 s and 100 searches); trial of 735 searches in 3.0 s at 1 searcher: p50 3.996 ms against the settled p50 4.036 ms, 1% apart (limit 10%); default@8: warm-up 15.0 s and 8,958 searches (0 failed) at 8 searchers; QPS of the last windows 596, 604, 604 (windows of at least 2 s and 100 searches); trial of 1,794 searches in 3.0 s at 8 searchers: 597 QPS against the settled 604 QPS, 1% apart (limit 10%); default@1: warm-up 15.0 s and 4,219 searches (0 failed) at 1 searcher; p50 of the last windows 3.512, 3.520, 3.543 ms (windows of at least 2 s and 100 searches); trial of 847 searches in 3.0 s at 1 searcher: p50 3.508 ms against the settled p50 3.520 ms, 0% apart (limit 10%). - Pass exact after rehearsal: 2026-10-05T08:15:48.410Z to 2026-10-05T08:16:48.411Z (60.0 s), 14,598 searches, 0 failed, p50 4.044 ms, mean 4.107 ms, p99 5.562 ms, 243.3 QPS (1000/QPS 4.110 ms). - Pass default@8 after exact: 2026-10-05T08:17:06.436Z to 2026-10-05T08:17:26.440Z (20.0 s), 11,935 searches, 0 failed, p50 12.786 ms, mean 13.398 ms, p99 27.554 ms, 596.6 QPS (1000/QPS 1.676 ms). - Pass default@1 after default@8: 2026-10-05T08:17:44.447Z to 2026-10-05T08:18:04.448Z (20.0 s), 5,612 searches, 0 failed, p50 3.527 ms, mean 3.562 ms, p99 5.044 ms, 280.6 QPS (1000/QPS 3.564 ms). - Passes in the order run: exact, default@8, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 54,176 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (duckdb_indexes() lists gvb_gvbbench_eshoponweb_hnsw, pragma_hnsw_index_info() counts 524 vectors of 524 rows, EXPLAIN of the default search shows HNSW_INDEX_SCAN on it = True). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: in this process (embedded), no network: no network address. No open connection seen (no open TCP connection was found). - CPU pinning: none: an embedded engine runs inside the benchmark client process, so it shares the client CPUs, CPUs 0-1,4-5. - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): exact 3593/3592 MHz, performance, outside load 0.14 before the warm-up, 0.15 during, client CPU 6.047 ms per search; default@8 3593/3592 MHz, performance, outside load 0.15 before the warm-up, 0.15 during, client CPU 6.665 ms per search; default@1 3594/3592 MHz, performance, outside load 0.15 before the warm-up, 0.13 during, client CPU 4.838 ms per search. ### chroma - Engine: Chroma 1.4.4 (single node, REST API v2) (compose) - Index: HNSW M=16 ef_construction=128, ef_search=100 (Chroma default), cosine; approximate only (no exact mode) - Search settings: M=16, ef_construction=128, ef_search=100 - Load: 524 rows in batches of 1000, 0.6 s of upserts (841 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (Chroma has no index build to wait for and exposes no index status; checked in 0.0 s: Chroma exposes no index status (indexing_status endpoint: HTTP 500 {"error":"InternalError","message":"Method scout_logs is not implemented"}), so indexed vectors are not reported and ready only means the two counts agree to within 1 percent: count endpoint 524, a vector search asking for 524 results returned 524) - Search: 9752 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 0.47 ms, 8 searchers 0.45 ms - RAM: 56.51 MiB (docker stats); disk: 414.16 KiB added by this load (2.01 GiB (whole engine data folder), 2.01 GiB before) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started chroma.compose.yaml with every container created on CPUs 2-3,6-7: gvb-chroma cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - Index after the load: ready, ? of 524 indexed (Chroma exposes no index status (indexing_status endpoint: HTTP 500 {"error":"InternalError","message":"Method scout_logs is not implemented"}), so indexed vectors are not reported and ready only means the two counts agree to within 1 percent: count endpoint 524, a vector search asking for 524 results returned 524). Durability: SQLite rollback journal with its default synchronous=FULL under the data directory (chroma.compose.yaml sets IS_PERSISTENT=1 and PERSIST_DIRECTORY=/data and no sync setting): every write is committed to chroma.sqlite3 with fsync of the journal, the directory and the database file before the call returns (strace: 15 database fsyncs and 38 journal fsyncs for one create, four 500-vector upserts and one delete), and the HNSW files are written every sync_threshold=1000 vectors and rebuilt from the SQLite log after a crash; measured with kill -9 and a restart: all 50, 300 and 2,000 vectors were still stored and searchable. - Load average 4.81 4.30 3.95 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@8 24,139 searches in 30.0 s, default@1 14,622 searches in 30.0 s. - Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure. default@8: warm-up 15.0 s and 12,108 searches (0 failed) at 8 searchers; QPS of the last windows 807, 806, 809 (windows of at least 2 s and 100 searches); trial of 2,414 searches in 3.0 s at 8 searchers: 803 QPS against the settled 807 QPS, 1% apart (limit 10%); default@1: warm-up 15.0 s and 7,308 searches (0 failed) at 1 searcher; p50 of the last windows 2.028, 2.040, 2.033 ms (windows of at least 2 s and 100 searches); trial of 1,462 searches in 3.0 s at 1 searcher: p50 2.024 ms against the settled p50 2.033 ms, 0% apart (limit 10%). - Pass default@8 after rehearsal: 2026-10-05T08:19:29.580Z to 2026-10-05T08:19:49.589Z (20.0 s), 16,095 searches, 0 failed, p50 9.864 ms, mean 9.941 ms, p99 11.931 ms, 804.4 QPS (1000/QPS 1.243 ms). - Pass default@1 after default@8: 2026-10-05T08:20:07.596Z to 2026-10-05T08:20:27.596Z (20.0 s), 9,752 searches, 0 failed, p50 2.027 ms, mean 2.049 ms, p99 2.534 ms, 487.6 QPS (1000/QPS 2.051 ms). - Passes in the order run: default@8, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 62,053 sent, 0 failed (warmupErrors). Index after the searches: ready, ? of 524 indexed (Chroma exposes no index status (indexing_status endpoint: HTTP 500 {"error":"InternalError","message":"Method scout_logs is not implemented"}), so indexed vectors are not reported and ready only means the two counts agree to within 1 percent: count endpoint 524, a vector search asking for 524 results returned 524). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-chroma (9e8efaae5b9b) on network gvb-chroma_default at container-ip:8000, not through docker-proxy localhost:8000. Open after the passes: container-ip:8000 x8 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-chroma: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-chroma: 9 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@8 3592/3592 MHz, performance, outside load 0.08 before the warm-up, 0.08 during, client CPU 0.453 ms per search; default@1 3592/3592 MHz, performance, outside load 0.08 before the warm-up, 0.09 during, client CPU 0.466 ms per search. ### opensearch - Engine: OpenSearch 3.9.0 (k-NN plugin, faiss) (compose) - Index: faiss HNSW float32, no compression, m=16, ef_construction=128, cosinesimil; search k=top, ef_search=100; 1 shard, 0 replicas; graph built at any segment size (approximate_threshold=0); force-merged to one segment after the load - Search settings: approximate_threshold=0, ef_construction=128, ef_search=100, k=top, m=16 - Load: 524 rows in batches of 1000, 1.3 s of upserts (405 rows/s); count matched 0.1 s after the last upsert; index step 0.9 s (force-merged to one segment, graph ready after 0.9 s: every segment searched through its HNSW graph, covering 524 of 524 vectors: 1 segment(s), 0 merge(s) running, profiled probe search: ann_search_count 1 (segments tried through a graph), exact_search_count 0 (of those, scanned instead), approximate_threshold 0; force-merged to one segment on purpose, so one graph answers every search) - Search: 9905 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 0.46 ms, 8 searchers 0.51 ms - Exact mode: 22,952 searches, p50 2.58 ms, p95 2.92 ms, recall 1.000 - RAM: 2.58 GiB (docker stats); disk: 9 MiB added by this load (11.16 MiB (whole engine data folder), 2.16 MiB before) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started opensearch.compose.yaml with every container created on CPUs 2-3,6-7: gvb-opensearch cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - Index after the load: ready, 524 of 524 indexed (every segment searched through its HNSW graph, covering 524 of 524 vectors: 1 segment(s), 0 merge(s) running, profiled probe search: ann_search_count 1 (segments tried through a graph), exact_search_count 0 (of those, scanned instead), approximate_threshold 0; force-merged to one segment on purpose, so one graph answers every search). Durability: Every acknowledged bulk request is fsynced to the translog before the answer: index.translog.durability=request, the OpenSearch default, which neither this sink nor opensearch.compose.yaml overrides (OpenSearchReadinessTests reads it back from the live index). By that setting a process crash or power loss loses no acknowledged write; this is read from the setting, not shown by pulling power. One node and no replicas, so a lost disk loses the data.. - Load average 2.50 3.53 3.72 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: exact 8,399 searches in 30.0 s, default@8 37,323 searches in 30.0 s, default@1 14,763 searches in 30.0 s. - Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure. exact: warm-up 15.0 s and 5,540 searches (0 failed) at 1 searcher; p50 of the last windows 2.582, 2.591, 2.598 ms (windows of at least 2 s and 100 searches); trial of 1,135 searches in 3.0 s at 1 searcher: p50 2.584 ms against the settled p50 2.591 ms, 0% apart (limit 10%); default@8: warm-up 15.0 s and 22,730 searches (0 failed) at 8 searchers; QPS of the last windows 1,526, 1,509, 1,514 (windows of at least 2 s and 100 searches); trial of 4,564 searches in 3.0 s at 8 searchers: 1,519 QPS against the settled 1,514 QPS, 0% apart (limit 10%); default@1: warm-up 15.0 s and 7,415 searches (0 failed) at 1 searcher; p50 of the last windows 1.993, 1.993, 1.991 ms (windows of at least 2 s and 100 searches); trial of 1,478 searches in 3.0 s at 1 searcher: p50 1.999 ms against the settled p50 1.993 ms, 0% apart (limit 10%). - Pass exact after rehearsal: 2026-10-05T08:22:47.126Z to 2026-10-05T08:23:47.129Z (60.0 s), 22,952 searches, 0 failed, p50 2.576 ms, mean 2.612 ms, p99 3.549 ms, 382.5 QPS (1000/QPS 2.614 ms). - Pass default@8 after exact: 2026-10-05T08:24:05.156Z to 2026-10-05T08:24:25.160Z (20.0 s), 30,570 searches, 0 failed, p50 5.031 ms, mean 5.232 ms, p99 10.185 ms, 1528.2 QPS (1000/QPS 0.654 ms). - Pass default@1 after default@8: 2026-10-05T08:24:43.171Z to 2026-10-05T08:25:03.172Z (20.0 s), 9,905 searches, 0 failed, p50 1.991 ms, mean 2.017 ms, p99 2.812 ms, 495.2 QPS (1000/QPS 2.019 ms). - Passes in the order run: exact, default@8, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 103,347 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (every segment searched through its HNSW graph, covering 524 of 524 vectors: 1 segment(s), 0 merge(s) running, profiled probe search: ann_search_count 1 (segments tried through a graph), exact_search_count 0 (of those, scanned instead), approximate_threshold 0; force-merged to one segment on purpose, so one graph answers every search). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-opensearch (161ab46a291e) on network engines_default at container-ip:9200, not through docker-proxy localhost:9201. Open after the passes: container-ip:9200 x8 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-opensearch: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-opensearch: 54 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): exact 3492/3492 MHz, performance, outside load 0.14 before the warm-up, 0.15 during, client CPU 0.462 ms per search; default@8 3492/3492 MHz, performance, outside load 0.15 before the warm-up, 0.12 during, client CPU 0.508 ms per search; default@1 3492/3492 MHz, performance, outside load 0.14 before the warm-up, 0.15 during, client CPU 0.458 ms per search. ### vespa - Engine: Vespa 8.754.14 (tensor attribute + HNSW) (compose) - Index: HNSW float32 tensor, prenormalized-angular (cosine), max-links-per-node=16, neighbors-to-explore-at-insert=128; search targetHits=top, ef=100 via exploreAdditionalHits; exact mode = approximate:false; vectors held in memory - Search settings: ef=100, max-links-per-node=16, neighbors-to-explore-at-insert=128, targetHits=top - Load: 524 rows in batches of 1000, 1.9 s of upserts (276 rows/s); count matched 0.0 s after the last upsert; index step 0.1 s (nothing to build, Vespa inserts into the HNSW graph while writing; ready after 0.0 s: HNSW graph returns 524 of 524 vectors: totalCount 524, coverage full True, nearestNeighbor approximate:true with targetHits above the count: has_index True, algorithm "index top k", top_k_hits 524; the graph is updated while each write is applied, so there is no later build step) - Search: 10481 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 0.94 ms, 8 searchers 0.88 ms - Exact mode: 31,525 searches, p50 1.84 ms, p95 2.28 ms, recall 1.000 - RAM: 2.79 GiB (docker stats); disk: 0 B added by this load (2.11 GiB (whole engine data folder), 2.12 GiB before) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started vespa.compose.yaml with every container created on CPUs 2-3,6-7: gvb-vespa cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - Index after the load: ready, 524 of 524 indexed (HNSW graph returns 524 of 524 vectors: totalCount 524, coverage full True, nearestNeighbor approximate:true with targetHits above the count: has_index True, algorithm "index top k", top_k_hits 524; the graph is updated while each write is applied, so there is no later build step). Durability: Vespa's transaction log server fsyncs after each commit: searchlib.translogserver usefsync=true, and an operation is searchable when it is acknowledged: proton documentdb visibilitydelay=0. Neither is overridden by this sink or by the services.xml it deploys; VespaReadinessTests reads both from the live config server. After a crash the node replays the transaction log over its last flushed data and rebuilds the in-memory HNSW graph. By those settings an acknowledged write survives a crash; this is read from the settings, not shown by pulling power. One node and min-redundancy 1, so a lost disk loses the data.. - Load average 6.33 4.14 3.86 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@8 18,443 searches in 30.0 s, exact 14,932 searches in 30.0 s, default@1 15,501 searches in 30.0 s. - Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure. default@8: warm-up 15.0 s and 20,828 searches (0 failed) at 8 searchers; QPS of the last windows 1,560, 1,488, 1,528 (windows of at least 2 s and 100 searches); trial of 4,633 searches in 3.0 s at 8 searchers: 1,541 QPS against the settled 1,528 QPS, 1% apart (limit 10%); exact: warm-up 15.0 s and 7,864 searches (0 failed) at 1 searcher; p50 of the last windows 1.845, 1.842, 1.845 ms (windows of at least 2 s and 100 searches); trial of 1,575 searches in 3.0 s at 1 searcher: p50 1.839 ms against the settled p50 1.845 ms, 0% apart (limit 10%); default@1: warm-up 15.0 s and 7,814 searches (0 failed) at 1 searcher; p50 of the last windows 1.851, 1.867, 1.857 ms (windows of at least 2 s and 100 searches); trial of 1,562 searches in 3.0 s at 1 searcher: p50 1.857 ms against the settled p50 1.857 ms, 0% apart (limit 10%). - Pass default@8 after rehearsal: 2026-10-05T08:27:48.915Z to 2026-10-05T08:28:08.917Z (20.0 s), 30,799 searches, 0 failed, p50 5.037 ms, mean 5.192 ms, p99 8.806 ms, 1539.8 QPS (1000/QPS 0.649 ms). - Pass exact after default@8: 2026-10-05T08:28:26.927Z to 2026-10-05T08:29:26.929Z (60.0 s), 31,525 searches, 0 failed, p50 1.844 ms, mean 1.900 ms, p99 2.612 ms, 525.4 QPS (1000/QPS 1.903 ms). - Pass default@1 after exact: 2026-10-05T08:29:44.957Z to 2026-10-05T08:30:04.957Z (20.0 s), 10,481 searches, 0 failed, p50 1.853 ms, mean 1.905 ms, p99 2.563 ms, 524.0 QPS (1000/QPS 1.908 ms). - Passes in the order run: default@8, exact, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 93,152 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (HNSW graph returns 524 of 524 vectors: totalCount 524, coverage full True, nearestNeighbor approximate:true with targetHits above the count: has_index True, algorithm "index top k", top_k_hits 524; the graph is updated while each write is applied, so there is no later build step). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-vespa (770bf4d66102) on network engines_default at container-ip:8080, not through docker-proxy localhost:8090; container gvb-vespa (770bf4d66102) on network engines_default at container-ip:19071, not through docker-proxy localhost:19071. Open after the passes: container-ip:8080 x1 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-vespa: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-vespa: 212 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@8 3492/3493 MHz, performance, outside load 0.12 before the warm-up, 0.20 during, client CPU 0.882 ms per search; exact 3518/3496 MHz, performance, outside load 0.18 before the warm-up, 0.12 during, client CPU 0.943 ms per search; default@1 3518/3494 MHz, performance, outside load 0.12 before the warm-up, 0.12 during, client CPU 0.941 ms per search. ### typesense - Engine: Typesense 30.2 (vector search) (compose) - Index: HNSW float32 (hnswlib), m=16, ef_construction=128, cosine; search k=top, ef=100; exact mode = filter ordinal:>=0 with flat_search_cutoff; index held in memory - Search settings: ef=100, ef_construction=128, k=top, m=16 - Load: 524 rows in batches of 1000, 0.6 s of upserts (838 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (nothing to build, Typesense inserts into the HNSW graph while writing; ready after 0.0 s: HNSW graph returns 524 of 524 vectors, no writes queued: num_documents 524, pending_write_batches 0, health ok, unfiltered vector query with k above the count returned 524; the graph is built in memory during each write, so there is no later build step) - Search: 4883 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 0.52 ms, 8 searchers 0.58 ms - Exact mode: 10,873 searches, p50 5.46 ms, p95 5.84 ms, recall 1.000 - RAM: 115 MiB (docker stats); disk: 0 B added by this load (228.45 MiB (whole engine data folder), 383.45 MiB before) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started typesense.compose.yaml with every container created on CPUs 2-3,6-7: gvb-typesense cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - Index after the load: ready, 524 of 524 indexed (HNSW graph returns 524 of 524 vectors, no writes queued: num_documents 524, pending_write_batches 0, health ok, unfiltered vector query with k above the count returned 524; the graph is built in memory during each write, so there is no later build step). Durability: Every acknowledged write is appended to Typesense's raft log and fsynced before the answer: braft raft_sync=true with raft_sync_policy=0 (sync immediately), read from the running container's brpc /flags page on 2026-10-04. A restart replays the log from the last snapshot (typesense.compose.yaml sets TYPESENSE_SNAPSHOT_INTERVAL_SECONDS=300) and rebuilds the in-memory HNSW graph, so by those settings a crash loses no acknowledged write, but the restart is slow after a big load. This is read from the settings, not shown by pulling power. One node and no replicas, so a lost disk loses the data.. - Load average 2.83 4.12 4.02 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@8 17,688 searches in 30.0 s, default@1 7,331 searches in 30.0 s, exact 5,424 searches in 30.0 s. - Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure. default@8: warm-up 15.0 s and 8,858 searches (0 failed) at 8 searchers; QPS of the last windows 589, 594, 590 (windows of at least 2 s and 100 searches); trial of 1,776 searches in 3.0 s at 8 searchers: 591 QPS against the settled 590 QPS, 0% apart (limit 10%); default@1: warm-up 15.0 s and 3,666 searches (0 failed) at 1 searcher; p50 of the last windows 4.027, 4.037, 4.031 ms (windows of at least 2 s and 100 searches); trial of 729 searches in 3.0 s at 1 searcher: p50 4.038 ms against the settled p50 4.031 ms, 0% apart (limit 10%); exact: warm-up 15.0 s and 2,715 searches (0 failed) at 1 searcher; p50 of the last windows 5.467, 5.473, 5.461 ms (windows of at least 2 s and 100 searches); trial of 543 searches in 3.0 s at 1 searcher: p50 5.463 ms against the settled p50 5.467 ms, 0% apart (limit 10%). - Pass default@8 after rehearsal: 2026-10-05T08:32:11.748Z to 2026-10-05T08:32:31.758Z (20.0 s), 11,833 searches, 0 failed, p50 13.021 ms, mean 13.524 ms, p99 25.098 ms, 591.3 QPS (1000/QPS 1.691 ms). - Pass default@1 after default@8: 2026-10-05T08:32:49.766Z to 2026-10-05T08:33:09.769Z (20.0 s), 4,883 searches, 0 failed, p50 4.036 ms, mean 4.094 ms, p99 5.684 ms, 244.1 QPS (1000/QPS 4.096 ms). - Pass exact after default@1: 2026-10-05T08:33:27.779Z to 2026-10-05T08:34:27.782Z (60.0 s), 10,873 searches, 0 failed, p50 5.460 ms, mean 5.517 ms, p99 6.368 ms, 181.2 QPS (1000/QPS 5.519 ms). - Passes in the order run: default@8, default@1, exact; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 48,730 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (HNSW graph returns 524 of 524 vectors, no writes queued: num_documents 524, pending_write_batches 0, health ok, unfiltered vector query with k above the count returned 524; the graph is built in memory during each write, so there is no later build step). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-typesense (1ebdbb711c65) on network engines_default at container-ip:8108, not through docker-proxy localhost:8108. Open after the passes: container-ip:8108 x1 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-typesense: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-typesense: 303 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@8 3592/3592 MHz, performance, outside load 0.15 before the warm-up, -0.02 during, client CPU 0.577 ms per search; default@1 3592/3592 MHz, performance, outside load 0.04 before the warm-up, 0.15 during, client CPU 0.515 ms per search; exact 3592/3592 MHz, performance, outside load 0.09 before the warm-up, 0.16 during, client CPU 0.525 ms per search. ### qdrant - Engine: Qdrant 1.17.0 in container gvb-qdrant (image qdrant/qdrant:v1.17.0@sha256:f1c7272cdac52b38c1a0e89313922d940ba50afd90d593a1605dbbc214e66ffb; cpuset 2-3,6-7 from its creation; 3 search thread(s)) (compose) - Index: exact scan: the builder's sink sends exact=true on every search, so no HNSW graph is used whether or not Qdrant has built one (see the index state) - Search settings: exact=true - Load: 524 rows in batches of 1000, 0.1 s of upserts (9,285 rows/s); count matched 0.0 s after the last upsert; index step 1.0 s (status Green, 0 of 524 vectors in HNSW segments (2 segments), waited 1.0 s) - Search: 22509 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 0.72 ms, 8 searchers 0.53 ms - RAM: 37.63 MiB (docker stats); disk: 196.08 MiB (collection folder) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started qdrant.compose.yaml with every container created on CPUs 2-3,6-7: gvb-qdrant cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - Index after the load: ready, 0 of 524 indexed (status Green, optimizer ok, indexed_vectors_count 0 of 524 points, 2 segments, indexing_threshold_kb 10,000, full_scan_threshold_kb 10,000; no index used, exact scan by design (the builder's sink sends exact=true on every search); no graph was built; searches counted since the collection was created: unfiltered_hnsw 0, unfiltered_plain 0, unfiltered_exact 0). Durability: What was measured (strace -f on the Qdrant server process, 30 single-point REST upserts per setting, every sync call timed against the request that caused it): with wait=true each upsert had exactly one msync(MS_SYNC) of the write-ahead-log segment inside the request, before the reply (30 of 30); with wait=false the 30 upserts were acknowledged within 0.26 s with no sync call, and the first WAL msync (one call covering all 30 records) came 2.7 s after the last reply, followed by the segment-file flushes. So the log is flushed to disk under both settings, but with wait=false the flush comes after the acknowledgement: an operating-system crash or power cut in that gap loses acknowledged writes, while a crash of the Qdrant process alone should not, because the bytes are already in the kernel's page cache (inferred, not tested). Every upsert here is sent with wait=true in batches of 256 over gRPC; the strace used one point per request, so one flush per batch is inferred, not measured. Segment files are flushed every 5 s; log segments 32 MB, 0 created ahead. /qdrant/config/config.yaml in container gvb-qdrant (the image's own file; its storage keys are listed) sets: storage.collection.quantization = null, storage.collection.replication_factor = 1, storage.collection.vectors.on_disk = null, storage.collection.write_consistency_factor = 1, storage.hnsw_index.ef_construct = 100, storage.hnsw_index.full_scan_threshold_kb = 10000, storage.hnsw_index.m = 16, storage.hnsw_index.max_indexing_threads = 0, storage.hnsw_index.on_disk = false, storage.hnsw_index.payload_m = null, storage.max_collections = null, storage.node_type = Normal, storage.on_disk_payload = true, storage.optimizers.default_segment_number = 0, storage.optimizers.deleted_threshold = 0.2, storage.optimizers.flush_interval_sec = 5, storage.optimizers.indexing_threshold_kb = 10000, storage.optimizers.max_optimization_threads = null, storage.optimizers.max_segment_size_kb = null, storage.optimizers.vacuum_min_vector_number = 1000, storage.performance.max_search_threads = 0, storage.performance.optimizer_cpu_budget = 0, storage.performance.update_rate_limit = null, storage.shard_transfer_method = null, storage.snapshots_config.snapshots_storage = local, storage.snapshots_path = ./snapshots, storage.storage_path = ./storage, storage.temp_path = null, storage.update_concurrency = null, storage.wal.wal_capacity_mb = 32, storage.wal.wal_segments_ahead = 0; no QDRANT__ environment overrides in the server process. Not tested by cutting power; whether the disk's own write cache reaches the media was not checked.. - Load average 1.53 3.26 3.74 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@1 34,234 searches in 30.0 s, default@8 99,040 searches in 30.0 s. - Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure. default@1: warm-up 15.0 s and 16,887 searches (0 failed) at 1 searcher; p50 of the last windows 0.874, 0.880, 0.874 ms (windows of at least 2 s and 100 searches); trial of 3,382 searches in 3.0 s at 1 searcher: p50 0.873 ms against the settled p50 0.874 ms, 0% apart (limit 10%); default@8: warm-up 15.0 s and 49,577 searches (0 failed) at 8 searchers; QPS of the last windows 3,330, 3,312, 3,250 (windows of at least 2 s and 100 searches); trial of 9,935 searches in 3.0 s at 8 searchers: 3,310 QPS against the settled 3,312 QPS, 0% apart (limit 10%). - Pass default@1 after rehearsal: 2026-10-05T08:35:56.175Z to 2026-10-05T08:36:16.176Z (20.0 s), 22,509 searches, 0 failed, p50 0.877 ms, mean 0.887 ms, p99 1.394 ms, 1125.4 QPS (1000/QPS 0.889 ms). - Pass default@8 after default@1: 2026-10-05T08:36:34.200Z to 2026-10-05T08:36:54.202Z (20.0 s), 66,067 searches, 0 failed, p50 2.335 ms, mean 2.419 ms, p99 4.236 ms, 3303.0 QPS (1000/QPS 0.303 ms). - Passes in the order run: default@1, default@8; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 213,055 sent, 0 failed (warmupErrors). Index after the searches: ready, 0 of 524 indexed (status Green, optimizer ok, indexed_vectors_count 0 of 524 points, 2 segments, indexing_threshold_kb 10,000, full_scan_threshold_kb 10,000; no index used, exact scan by design (the builder's sink sends exact=true on every search); no graph was built; searches counted since the collection was created: unfiltered_hnsw 0, unfiltered_plain 603,262, unfiltered_exact 0). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-qdrant (9764fc8cf51b) on network gvb-qdrant_default at container-ip:6334, not through docker-proxy localhost:16334; container gvb-qdrant (9764fc8cf51b) on network gvb-qdrant_default at container-ip:6333, not through docker-proxy localhost:16333. Open after the passes: container-ip:6334 x1 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-qdrant: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-qdrant: 24 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@1 3492/3492 MHz, performance, outside load 0.14 before the warm-up, 0.12 during, client CPU 0.724 ms per search; default@8 3492/3492 MHz, performance, outside load 0.14 before the warm-up, 0.15 during, client CPU 0.525 ms per search. ### mariadb - Engine: MariaDB 11.8.9 (InnoDB + VECTOR INDEX) (compose) - Index: VECTOR INDEX (HNSW variant) DISTANCE=cosine, M=16 (no ef_construction setting exists), mhnsw_ef_search=100 per statement (the ef 100 most engines here use, so the search effort matches; MariaDB's own default is 20; recall@10 at ef 100 falls as the set grows (random 1024-dimension vectors, measured 2026-10-04: 0.99 at 524, 0.89 to 0.92 at 2,000; an earlier run gave about 0.09 at 100,000)), mhnsw_max_cache_size 4G; exact mode = IGNORE INDEX full scan - Search settings: DISTANCE=cosine, M=16, mhnsw_ef_search=100 - Load: 524 rows in batches of 1000, 0.5 s of upserts (1,052 rows/s); count matched 0.0 s after the last upsert; index step 0.5 s (VECTOR INDEX is maintained inside every INSERT, nothing to build; confirmed after 2 poll(s) in 0.5 s (poll 1 of 2 said: not ready: a search at mhnsw_ef_search 100 returned only 405 of 524 requested rows through the index (EXPLAIN key 'vec_idx') (information_schema lists vec_idx as INDEX_TYPE VECTOR, EXPLAIN of the default search uses key vec_idx, the server applied mhnsw_ef_search 100 (asked for 100, read back from the server), a search at that effort returned 405 of 524 requested rows through the index; MariaDB has no per-index row count (the graph is a hidden InnoDB table), so indexed vectors are not reported)): information_schema lists vec_idx as INDEX_TYPE VECTOR, EXPLAIN of the default search uses key vec_idx, the server applied mhnsw_ef_search 100 (asked for 100, read back from the server), a search at that effort returned 518 of 524 requested rows through the index; MariaDB has no per-index row count (the graph is a hidden InnoDB table), so indexed vectors are not reported) - Search: 31389 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 0.47 ms, 8 searchers 0.38 ms - Exact mode: 37,311 searches, p50 1.58 ms, p95 1.69 ms, recall 1.000 - RAM: 170.9 MiB (docker stats); disk: 22.01 MiB added by this load (182.79 MiB (whole engine data folder), 160.78 MiB before) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started mariadb-bench.compose.yaml with every container created on CPUs 2-3,6-7: gvbbench-mariadb cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - Index after the load: ready, ? of 524 indexed (information_schema lists vec_idx as INDEX_TYPE VECTOR, EXPLAIN of the default search uses key vec_idx, the server applied mhnsw_ef_search 100 (asked for 100, read back from the server), a search at that effort returned 518 of 524 requested rows through the index; MariaDB has no per-index row count (the graph is a hidden InnoDB table), so indexed vectors are not reported). Durability: innodb_flush_log_at_trx_commit=2 (set in mariadb-bench.compose.yaml for the benchmark's own container gvbbench-mariadb, and in mariadb.compose.yaml for the daily gvb-mariadb, which has the same server settings): the InnoDB redo log is written to the operating system at every commit but fsynced about once a second, so a crash of the mariadbd process loses nothing, while an operating-system crash or power cut can lose the last second of commits; innodb_doublewrite is on (default), the binary log is off, and the vector graph is an InnoDB table under the same log (settings read from the running server with SHOW VARIABLES; the crash behaviour is InnoDB's documented behaviour for this setting and was not tested on either container). - Load average 4.09 3.62 3.80 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: exact 18,686 searches in 30.0 s, default@8 180,320 searches in 30.0 s, default@1 47,492 searches in 30.0 s. - Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure. exact: warm-up 15.0 s and 9,334 searches (0 failed) at 1 searcher; p50 of the last windows 1.568, 1.576, 1.579 ms (windows of at least 2 s and 100 searches); trial of 1,887 searches in 3.0 s at 1 searcher: p50 1.566 ms against the settled p50 1.576 ms, 1% apart (limit 10%); default@8: warm-up 15.0 s and 88,919 searches (0 failed) at 8 searchers; QPS of the last windows 5,960, 5,890, 5,883 (windows of at least 2 s and 100 searches); trial of 17,710 searches in 3.0 s at 8 searchers: 5,901 QPS against the settled 5,890 QPS, 0% apart (limit 10%); default@1: warm-up 15.0 s and 23,423 searches (0 failed) at 1 searcher; p50 of the last windows 0.638, 0.636, 0.633 ms (windows of at least 2 s and 100 searches); trial of 4,755 searches in 3.0 s at 1 searcher: p50 0.620 ms against the settled p50 0.636 ms, 3% apart (limit 10%). - Pass exact after rehearsal: 2026-10-05T08:38:51.385Z to 2026-10-05T08:39:51.386Z (60.0 s), 37,311 searches, 0 failed, p50 1.578 ms, mean 1.606 ms, p99 2.435 ms, 621.8 QPS (1000/QPS 1.608 ms). - Pass default@8 after exact: 2026-10-05T08:40:09.416Z to 2026-10-05T08:40:29.417Z (20.0 s), 118,608 searches, 0 failed, p50 1.300 ms, mean 1.347 ms, p99 2.494 ms, 5930.1 QPS (1000/QPS 0.169 ms). - Pass default@1 after default@8: 2026-10-05T08:40:47.455Z to 2026-10-05T08:41:07.455Z (20.0 s), 31,389 searches, 0 failed, p50 0.625 ms, mean 0.635 ms, p99 0.879 ms, 1569.4 QPS (1000/QPS 0.637 ms). - Passes in the order run: exact, default@8, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 392,526 sent, 0 failed (warmupErrors). Index after the searches: ready, ? of 524 indexed (information_schema lists vec_idx as INDEX_TYPE VECTOR, EXPLAIN of the default search uses key vec_idx, the server applied mhnsw_ef_search 100 (asked for 100, read back from the server), a search at that effort returned 518 of 524 requested rows through the index; MariaDB has no per-index row count (the graph is a hidden InnoDB table), so indexed vectors are not reported). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvbbench-mariadb (bc10d7d6071d) on network gvbbench-mariadb_default at container-ip:3306, not through docker-proxy localhost:13306. Open after the passes: container-ip:3306 x8 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvbbench-mariadb: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvbbench-mariadb: 12 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): exact 3592/3592 MHz, performance, outside load 0.11 before the warm-up, 0.14 during, client CPU 0.493 ms per search; default@8 3492/3492 MHz, performance, outside load 0.14 before the warm-up, 0.10 during, client CPU 0.381 ms per search; default@1 3492/3492 MHz, performance, outside load 0.12 before the warm-up, 0.13 during, client CPU 0.470 ms per search. ### redis - Engine: Redis 8.10.2 (query engine, HASH + vector index) (compose) - Index: HNSW TYPE FLOAT32 M=16 EF_CONSTRUCTION=128, EF_RUNTIME=100 per query, cosine; exact mode = FLAT index built on first exact query - Search settings: EF_CONSTRUCTION=128, EF_RUNTIME=100, M=16 - Load: 524 rows in batches of 1000, 0.0 s of upserts (11,834 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (HNSW index finished after 0.0 s of waiting: FT.INFO indexing 0, percent_indexed 1, hash_indexing_failures 0, flat_buffer_size 0, num_docs 524, hashes counted with SCAN under gvb_gvbbench_eshoponweb: 524) - Search: 53078 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 0.54 ms, 8 searchers 0.36 ms - Exact mode: 168,524 searches, p50 0.35 ms, p95 0.45 ms, recall 1.000 - RAM: 21.07 MiB (docker stats); disk: 2.43 MiB added by this load (2.43 MiB (whole engine data folder), 89 B before) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started redis.compose.yaml with every container created on CPUs 2-3,6-7: gvb-redis cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - Index after the load: ready, 524 of 524 indexed (FT.INFO indexing 0, percent_indexed 1, hash_indexing_failures 0, flat_buffer_size 0, num_docs 524, hashes counted with SCAN under gvb_gvbbench_eshoponweb: 524). Durability: save "300 1" and appendonly no (redis.compose.yaml): an RDB snapshot is written every 5 minutes if at least one key changed, and on a clean stop; there is no append-only log, so a crash loses every write since the last snapshot (up to 5 minutes plus the time a snapshot takes; measured with docker kill, which is SIGKILL, right after a 2,000-vector load: none of the 2,000 hashes were there after the restart, and the restart loaded an older snapshot that still held keys of a collection that had been dropped since). - Load average 3.85 3.89 3.87 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@1 78,747 searches in 30.0 s, default@8 151,474 searches in 30.0 s, exact 83,217 searches in 30.0 s. - WARNING: latency had NOT settled when timing began for exact (each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure). default@1: warm-up 15.0 s and 39,275 searches (0 failed) at 1 searcher; p50 of the last windows 0.383, 0.380, 0.382 ms (windows of at least 2 s and 100 searches); trial of 8,168 searches in 3.0 s at 1 searcher: p50 0.349 ms against the settled p50 0.382 ms, 10% apart (limit 10%); default@8: warm-up 15.0 s and 76,531 searches (0 failed) at 8 searchers; QPS of the last windows 5,217, 5,302, 5,056 (windows of at least 2 s and 100 searches); trial of 15,028 searches in 3.0 s at 8 searchers: 5,007 QPS against the settled 5,217 QPS, 4% apart (limit 10%); exact: warm-up 15.0 s and 41,741 searches (0 failed) at 1 searcher; p50 of the last windows 0.364, 0.362, 0.324 ms (windows of at least 2 s and 100 searches) (its last 3 windows differed by more than 5%); trial of 7,959 searches in 3.0 s at 1 searcher: p50 0.380 ms against the settled p50 0.362 ms, 5% apart (limit 10%), so the warm-up was EXTENDED once (36.0 s and 99,849 searches (0 failed) at 1 searcher; p50 of the last windows 0.362, 0.353, 0.342 ms (windows of at least 2 s and 100 searches), its last 3 windows differed by more than 5%); second trial of 8,351 searches in 3.0 s at 1 searcher: p50 0.359 ms against the settled p50 0.353 ms, 2% apart (limit 10%); still disagreeing after the extension. Its numbers may still include warm-up; rerun before quoting them. - Pass default@1 after rehearsal: 2026-10-05T08:43:03.688Z to 2026-10-05T08:43:23.688Z (20.0 s), 53,078 searches, 0 failed, p50 0.358 ms, mean 0.375 ms, p99 0.715 ms, 2653.9 QPS (1000/QPS 0.377 ms). - Pass default@8 after default@1: 2026-10-05T08:43:41.738Z to 2026-10-05T08:44:01.739Z (20.0 s), 101,499 searches, 0 failed, p50 1.546 ms, mean 1.575 ms, p99 2.243 ms, 5074.5 QPS (1000/QPS 0.197 ms). - Pass exact after default@8: 2026-10-05T08:45:13.821Z to 2026-10-05T08:46:13.821Z (60.0 s), 168,524 searches, 0 failed, p50 0.350 ms, mean 0.354 ms, p99 0.690 ms, 2808.7 QPS (1000/QPS 0.356 ms). - Passes in the order run: default@1, default@8, exact; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 652,079 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (FT.INFO indexing 0, percent_indexed 1, hash_indexing_failures 0, flat_buffer_size 0, num_docs 524, hashes counted with SCAN under gvb_gvbbench_eshoponweb: 524). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-redis (33d79628d41b) on network gvb-redis_default at container-ip:6379, not through docker-proxy localhost:6379. Open after the passes: container-ip:6379 x1 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-redis: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-redis: 6 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@1 3492/3492 MHz, performance, outside load 0.13 before the warm-up, 0.13 during, client CPU 0.538 ms per search; default@8 3492/3492 MHz, performance, outside load 0.13 before the warm-up, 0.13 during, client CPU 0.364 ms per search; exact 3492/3492 MHz, performance, outside load 0.13 before the warm-up, 0.13 during, client CPU 0.547 ms per search. ### pgvector - Engine: PostgreSQL 17 + pgvector 0.8.7 (compose) - Index: HNSW vector_cosine_ops m=16 ef_construction=128, hnsw.ef_search=100 per query, float32 vector(n), cosine; exact mode = same query with index scans off (sequential scan) - Search settings: ef_construction=128, hnsw.ef_search=100, m=16 - Load: 524 rows in batches of 1000, 0.9 s of upserts (616 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (HNSW is maintained inside every insert, nothing to build; confirmed in 0.0 s: pg_indexes lists gvb_gvbbench_eshoponweb_hnsw (USING hnsw (embedding vector_cosine_ops) WITH (m='16', ef_construction='128')), pg_index valid and ready = True, EXPLAIN of the default search uses Index Scan on it = True, that search returned 10 of 10 rows; PostgreSQL keeps no entry count for an HNSW index, so indexed vectors are not reported) - Search: 23068 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 0.68 ms, 8 searchers 0.54 ms - Exact mode: 26,899 searches, p50 2.18 ms, p95 2.46 ms, recall 1.000 - RAM: 99.96 MiB (docker stats); disk: 7.57 MiB added by this load (438.64 MiB (whole engine data folder), 431.07 MiB before) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started pgvector.compose.yaml with every container created on CPUs 2-3,6-7: gvb-pgvector cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - Index after the load: ready, ? of 524 indexed (pg_indexes lists gvb_gvbbench_eshoponweb_hnsw (USING hnsw (embedding vector_cosine_ops) WITH (m='16', ef_construction='128')), pg_index valid and ready = True, EXPLAIN of the default search uses Index Scan on it = True, that search returned 10 of 10 rows; PostgreSQL keeps no entry count for an HNSW index, so indexed vectors are not reported). Durability: fsync on, synchronous_commit on, full_page_writes on, wal_sync_method fdatasync (PostgreSQL defaults; pgvector.compose.yaml sets only shared_buffers, maintenance_work_mem and max_wal_size): every commit is flushed to the write-ahead log before it returns and HNSW index changes are WAL-logged, so a crash loses no committed row (max_wal_size 4GB only spaces out checkpoints; measured with docker kill, which is SIGKILL, right after a 2,000-vector load and a restart: all 2,000 rows were there and the HNSW index was valid and used by the default search). - Load average 2.44 3.23 3.62 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: exact 13,945 searches in 30.0 s, default@8 125,241 searches in 30.0 s, default@1 35,007 searches in 30.0 s. - Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure. exact: warm-up 15.0 s and 6,732 searches (0 failed) at 1 searcher; p50 of the last windows 2.200, 2.163, 2.193 ms (windows of at least 2 s and 100 searches); trial of 1,335 searches in 3.0 s at 1 searcher: p50 2.195 ms against the settled p50 2.193 ms, 0% apart (limit 10%); default@8: warm-up 15.0 s and 60,119 searches (0 failed) at 8 searchers; QPS of the last windows 3,986, 4,026, 3,960 (windows of at least 2 s and 100 searches); trial of 12,158 searches in 3.0 s at 8 searchers: 4,051 QPS against the settled 3,986 QPS, 2% apart (limit 10%); default@1: warm-up 15.0 s and 17,251 searches (0 failed) at 1 searcher; p50 of the last windows 0.872, 0.852, 0.853 ms (windows of at least 2 s and 100 searches); trial of 3,401 searches in 3.0 s at 1 searcher: p50 0.860 ms against the settled p50 0.853 ms, 1% apart (limit 10%). - Pass exact after rehearsal: 2026-10-05T08:48:11.201Z to 2026-10-05T08:49:11.202Z (60.0 s), 26,899 searches, 0 failed, p50 2.183 ms, mean 2.228 ms, p99 3.561 ms, 448.3 QPS (1000/QPS 2.231 ms). - Pass default@8 after exact: 2026-10-05T08:49:29.224Z to 2026-10-05T08:49:49.225Z (20.0 s), 80,158 searches, 0 failed, p50 1.954 ms, mean 1.994 ms, p99 3.509 ms, 4007.6 QPS (1000/QPS 0.250 ms). - Pass default@1 after default@8: 2026-10-05T08:50:07.252Z to 2026-10-05T08:50:27.253Z (20.0 s), 23,068 searches, 0 failed, p50 0.849 ms, mean 0.865 ms, p99 1.276 ms, 1153.4 QPS (1000/QPS 0.867 ms). - Passes in the order run: exact, default@8, default@1; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 275,189 sent, 0 failed (warmupErrors). Index after the searches: ready, ? of 524 indexed (pg_indexes lists gvb_gvbbench_eshoponweb_hnsw (USING hnsw (embedding vector_cosine_ops) WITH (m='16', ef_construction='128')), pg_index valid and ready = True, EXPLAIN of the default search uses Index Scan on it = True, that search returned 10 of 10 rows; PostgreSQL keeps no entry count for an HNSW index, so indexed vectors are not reported). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-pgvector (429c002c4862) on network gvb-pgvector_default at container-ip:5432, not through docker-proxy localhost:5432. Open after the passes: container-ip:5432 x8 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-pgvector: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-pgvector: 6 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): exact 3492/3492 MHz, performance, outside load 0.14 before the warm-up, 0.15 during, client CPU 0.684 ms per search; default@8 3492/3492 MHz, performance, outside load 0.15 before the warm-up, 0.16 during, client CPU 0.540 ms per search; default@1 3492/3492 MHz, performance, outside load 0.14 before the warm-up, 0.14 during, client CPU 0.681 ms per search. ### sql-diskann - Engine: Microsoft SQL Server 2025 (RTM-CU9) (KB5122048) - 17.0.5005.3 (X64), Enterprise Developer Edition (64-bit), in container gvb-mssql (image mcr.microsoft.com/mssql/server:2025-CU9-ubuntu-24.04@sha256:2b5b581621126574f3d1f75e78d3eebe8d05aedb59ad0cfdf9aa42cb0634d726; cpuset 2-3,6-7 from its creation; SQL Server counts 4 CPU(s) and runs 4 visible scheduler(s), affinity AUTO) (compose) - Index: DiskANN (preview) via VECTOR_SEARCH, cosine, build {"StartId":"306", "L":"48", "M":"8", "R":"48"}; the exact mode scans the same table - Search settings: L=48, M=8, R=48, StartId=306 - Load: 524 rows in batches of 1000, 0.8 s of upserts (637 rows/s); count matched 0.0 s after the last upsert; index step 4.0 s (copied 524 rows to dbo.gvb_gvbbench_eshoponweb_ann (INT key) and built DiskANN, parameters {"StartId":"306", "L":"48", "M":"8", "R":"48"}, graph covers 524 rows) - Search: 5450 latency samples, target held 524 rows, 0 errors - Client CPU per search: 1 searcher 1.03 ms, 8 searchers 1.10 ms - Exact mode: 16,555 searches, p50 3.55 ms, p95 4.06 ms, recall 1.000 - RAM: 527.2 MiB (docker stats); disk: 4.52 MiB (table and its indexes, reserved pages) - Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started mssql.compose.yaml with every container created on CPUs 2-3,6-7: gvb-mssql cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards. - Index after the load: ready, 524 of 524 indexed (DiskANN index built and used. sys.vector_indexes: vix_gvb_gvbbench_eshoponweb on dbo.gvb_gvbbench_eshoponweb_ann in GvbBenchDiskAnn, DiskANN, COSINE, enabled, build {"StartId":"306", "L":"48", "M":"8", "R":"48"}; graph table rows 524 of 524 table rows. Real VECTOR_SEARCH plan: Vector Index Seek on index vix_gvb_gvbbench_eshoponweb of GvbBenchDiskAnn.gvb_gvbbench_eshoponweb_ann (IndexKind DiskANN), 10 rows returned, 10 hits. The exact mode searches the SAME table (dbo.gvb_gvbbench_eshoponweb_ann, database GvbBenchDiskAnn) and its real plan is: Clustered Index Scan of GvbBenchDiskAnn.gvb_gvbbench_eshoponweb_ann through PK_gvb_gvbbench_eshoponweb_ann (IndexKind Clustered), 524 rows read, no vector index operator, 10 hits). Durability: A commit returns after its transaction-log records are written to disk (SQL Server write-ahead logging; delayed durability is DISABLED); a database created here copies the model database: recovery model FULL, page_verify CHECKSUM; no global trace flags are enabled; mssql.conf of container gvb-mssql (/var/opt/mssql/mssql.conf, on the host at ~/gvb-data/engines/mssql/mssql.conf) does not exist, so it sets: nothing; it has no [control] or [traceflag] entry, so SQL Server's own Linux defaults for flushing writes apply (Microsoft's Linux performance guide names trace flag 3982 as that default; read from the guide, not tested here). Not tested by cutting power; whether the disk's own write cache reaches the media was not checked. Container settings from its environment (names only): MSSQL_AGENT_ENABLED, MSSQL_MEMORY_LIMIT_MB, MSSQL_PID, MSSQL_RPC_PORT, MSSQL_SA_PASSWORD.. - Load average 3.04 3.35 3.57 (1/5/15 min) when searching began. - Rehearsal before any timed pass, untimed, every pass type at its own concurrency for at least 30 s through the same code the timed passes use: default@1 8,091 searches in 30.0 s, exact 8,172 searches in 30.0 s, default@8 20,015 searches in 30.0 s. - Settle: settled before every timed pass, each by its own warm-up at its own concurrency and a trial of the same pass within 10% of the warm-up's settled figure. default@1: warm-up 15.0 s and 4,136 searches (0 failed) at 1 searcher; p50 of the last windows 3.590, 3.581, 3.588 ms (windows of at least 2 s and 100 searches); trial of 828 searches in 3.0 s at 1 searcher: p50 3.595 ms against the settled p50 3.588 ms, 0% apart (limit 10%); exact: warm-up 15.0 s and 4,121 searches (0 failed) at 1 searcher; p50 of the last windows 3.518, 3.521, 3.561 ms (windows of at least 2 s and 100 searches); trial of 816 searches in 3.0 s at 1 searcher: p50 3.551 ms against the settled p50 3.521 ms, 1% apart (limit 10%); default@8: warm-up 15.0 s and 9,834 searches (0 failed) at 8 searchers; QPS of the last windows 676, 695, 674 (windows of at least 2 s and 100 searches); trial of 2,058 searches in 3.0 s at 8 searchers: 684 QPS against the settled 676 QPS, 1% apart (limit 10%). - Pass default@1 after rehearsal: 2026-10-05T08:52:28.692Z to 2026-10-05T08:52:48.693Z (20.0 s), 5,450 searches, 0 failed, p50 3.604 ms, mean 3.667 ms, p99 4.857 ms, 272.5 QPS (1000/QPS 3.670 ms). - Pass exact after default@1: 2026-10-05T08:53:06.700Z to 2026-10-05T08:54:06.701Z (60.0 s), 16,555 searches, 0 failed, p50 3.548 ms, mean 3.621 ms, p99 4.878 ms, 275.9 QPS (1000/QPS 3.624 ms). - Pass default@8 after exact: 2026-10-05T08:54:24.728Z to 2026-10-05T08:54:44.735Z (20.0 s), 13,813 searches, 0 failed, p50 11.409 ms, mean 11.583 ms, p99 19.571 ms, 690.4 QPS (1000/QPS 1.448 ms). - Passes in the order run: default@1, exact, default@8; each after its own warm-up. Untimed searches (rehearsals, per-pass warm-ups, settle checks and extensions): 58,071 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (DiskANN index built and used. sys.vector_indexes: vix_gvb_gvbbench_eshoponweb on dbo.gvb_gvbbench_eshoponweb_ann in GvbBenchDiskAnn, DiskANN, COSINE, enabled, build {"StartId":"306", "L":"48", "M":"8", "R":"48"}; graph table rows 524 of 524 table rows. Real VECTOR_SEARCH plan: Vector Index Seek on index vix_gvb_gvbbench_eshoponweb of GvbBenchDiskAnn.gvb_gvbbench_eshoponweb_ann (IndexKind DiskANN), 10 rows returned, 10 hits. The exact mode searches the SAME table (dbo.gvb_gvbbench_eshoponweb_ann, database GvbBenchDiskAnn) and its real plan is: Clustered Index Scan of GvbBenchDiskAnn.gvb_gvbbench_eshoponweb_ann through PK_gvb_gvbbench_eshoponweb_ann (IndexKind Clustered), 524 rows read, no vector index operator, 10 hits). - Benchmark copy gvbbench_eshoponweb dropped afterwards. - Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-mssql (2fac522fca10) on network gvb-mssql_default at container-ip:1433, not through docker-proxy localhost:14330. Open after the passes: container-ip:1433 x10 (this target). - CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-mssql: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-mssql: 85 thread(s) on CPUs 2-3,6-7 (every thread of the container). - Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@1 3500/3494 MHz, performance, outside load 0.14 before the warm-up, 0.16 during, client CPU 1.026 ms per search; exact 3500/3494 MHz, performance, outside load 0.14 before the warm-up, 0.16 during, client CPU 1.146 ms per search; default@8 3492/3492 MHz, performance, outside load 0.16 before the warm-up, 0.14 during, client CPU 1.095 ms per search. ## Notes - Order: targets ran one at a time in a random order from seed 601 (runSeed; --seed 601 repeats it, targetOrder lists it). Inside each target the timed passes also ran in a random order from the same seed and the target's name (passOrder): default@1 is one searcher for 20 s (every search's latency gives p50/p95/p99, the completed searches give QPS@1, its first answer to each query gives recall and nDCG), default@N is N searchers for 20 s (QPS@N), exact is the engine's exact mode, one searcher for 60 s cycling the queries. - Preparation, untimed, before any timed pass: a rehearsal of every pass type at its own concurrency for 30 s each, through the same code the passes use. - Warm-up and settle check, untimed, right before every timed pass: the pass's own search at the pass's own number of searchers for at least 15 s and at least 20 searches (at most 120 s), read in windows of at least 2 s and 100 searches; then a 3 s trial of the same pass. The trial's figure (p50 with one searcher, QPS with several) must lie within 10% of the warm-up's settled figure (the median of its last 3 windows, which must agree within 5%); if not, the warm-up is extended once (at least 30 s, until its windows agree, at most 120 s), a second trial is taken and the warm-up runs again before the pass. Each target's notes give every check, and a pass that still disagrees flags its target as unsettled. With machine control on, the check for a quiet box is made when the warm-up is announced, before it starts (a wait between the warm-up and the timed pass let the engine go cold), so by the time the clock opens that check is as old as the warm-up and the trial (about 18 s, more after an extension); each pass's conditions record that lead (quietCheckLeadSeconds). Failed untimed searches are counted per target (warmupErrors) and are not in the timed error counts. - Load rows/s counts only time inside each target's upsert calls: one writer, batches of 1000, rows already in memory, the collection dropped and created fresh first. Engines that build or finish their index after the writes do it in a separate timed index step (the load's index seconds), and the run waits for it before searching. - Index proof: each engine's own report of its index (indexState) is read after the load and again after the last pass. A target whose index was not ready after the load was still measured and carries a WARNING; an engine that reports nothing counts as not ready. - Search settings: each searched target records the settings its own index description states (searchSettings: the build parameters and the effort per query, such as m, ef_construction and ef_search), read after the load; consolidating never averages runs whose settings differ. - Durability: each target's crash-safety setting as configured here (durability). Engines that do not force writes to disk on every commit load faster for that reason. - Latency is client-side wall time around each search (network and driver included), every search of the default@1 window, one searcher, the queries cycled; a window with fewer than 200 searches, a p50 above 1.25 x its mean, or a mean above its p99 is flagged. - QPS: N searchers back to back for 20 s per level; completed searches divided by the window's elapsed time. - Recall@10: share of the exact top 10 (brute force in memory) that the engine returned. A hit whose exact similarity ties the 10th best (within 1e-5) also counts, because duplicate rows embed to identical vectors. - Routes: a container engine (the benchmark's own SQL Server and Qdrant containers included) is reached at its container's own address on its Docker network, never through the published localhost port (docker-proxy); the native comparison targets (sql-native, qdrant-native) are native services reached directly over loopback. Each target's addresses and the connections the client held open after its passes are in its notes and in conditions.connections. - RAM of compose engines, the benchmark's own SQL Server and Qdrant containers (sql, sql-diskann, qdrant, qdrant-hnsw) included, is docker stats of the engine's containers; for the native comparison targets (sql-native, qdrant-native) it is the whole native process, including every other database or collection it serves. Disk is the table's reserved pages (SQL), the collection folder (Qdrant), or for other engines what the load added to the engine's data folder (engines that keep data in memory until a snapshot show almost nothing). - Engines: run-all starts an engine that is down and stops it afterwards only if it was not running when the run began; an engine that was already running is left running. - WARNING: the engine did not report a ready index after the load for: elasticsearch. They were measured anyway; their numbers may come from a scan or a half-built index (see each target's index state). - Machine control on: governor performance on every CPU during the run (before: schedutil; at the end: performance; after putting it back: schedutil). engine CPUs 2-3,6-7 (cores 2,6 and 3,7), client CPUs 0-1,4-5 (cores 0,4 and 1,5); the client process was pinned; each engine was pinned to the engine CPUs for its turn and put back after (conditions.engines); an engine run-all started (and stopped) was asked to be created on them, and its notes say whether the host did so or it was moved there after the start. Busy box: right before each pass's warm-up the run waits, up to 10 min, while processes outside the benchmark (everything but this client and the engine under test's cgroups) use more than 0.3 CPUs on average over the last 60 s or the last 5 s, neither window reaching back past the start of the target or of its engine (so the engine's own start-up is not outside work); if the box does not clear the pass runs anyway, flagged 'busy box', as is a pass whose own outside load is above the limit. Outside load counts busy = user + nice + system + irq + softirq + steal (guest time is already in user); CONFIG_IRQ_TIME_ACCOUNTING is not set, so task and cgroup run time include the interrupt and softirq time that hit them and it is added; CONFIG_PARAVIRT_TIME_ACCOUNTING is not set, so steal is added (/boot/config-6.8.0-142-generic) (conditions.cpuAccounting); each pass also records this client's own CPU time per search (conditions.passes[].clientCpuMsPerSearch). CPU clocks were sampled every 250 ms; each pass's min/median/max per CPU is in conditions.passes. CPU idle states, recorded and left as found: driver intel_idle, governor menu, intel_idle max_cstate 9; POLL on, C1 on, C1E on, C3 OFF on CPUs 0-7 (default disabled), C6 on (conditions.cpuIdle). How each target was reached: conditions.connections. Client build Release, .NET 10.0.12. - sqlitevec is embedded: it ran inside the client process on the client CPUs 0-1,4-5, sharing them with the client. - duckdb is embedded: it ran inside the client process on the client CPUs 0-1,4-5, sharing them with the client.