Vector engine benchmark: eshoponweb

Run started 5 Oct 2026, 02:34 UTC. Data: eShopOnWeb, 524 vectors. Queries: 20 labelled questions.

Used by: published-2026-10-08 (basis v5)

The runs of v5 were also used by the set blocked-2026-10-05-v5, which was not published; its verdict, design/verdicts/v5-verdict.txt, holds the word BLOCK.

sources

file design/verdicts/v5-verdict.txt#BLOCK = BLOCK
consolidated consolidated:reuse[session=v5].verdict = design/verdicts/v5-verdict.txt
consolidated consolidated:reuse[session=v5].blockedFolders[0] = blocked-2026-10-05-v5

Corrections to statements recorded in this run

This run's report is kept as it was written. A set that uses this run states that the recorded statements below are false, misleading or unbacked. Each is marked in the report text.

False: Weaviate, durability. Recorded: weaviate.compose.yaml sets no persistence variable deploy/engines/weaviate.compose.yaml sets PERSISTENCE_DATA_PATH to /var/lib/weaviate. The compose file does set a persistence variable. Listed in published-2026-10-08. Marked in the report text, lines: 1.
sources
- file deploy/engines/weaviate.compose.yaml#PERSISTENCE_DATA_PATH: /var/lib/weaviate = PERSISTENCE_DATA_PATH: /var/lib/weaviate
- results results@20261005-023459-eshoponweb:targets[weaviate].durability#sets no persistence variable = sets no persistence variable
Misleading: ClickHouse, disk.text. Recorded: 1.27 GiB added by this load The figure is the growth of the whole data folder over the engine's turn, as the same line says (2.63 GiB at the end, 1.36 GiB before). The load itself is 524 vectors of 1024 dimensions, 2,146,304 bytes as 4-byte floats (2.05 MiB); the figure is 1,362,545,922 bytes. Listed in published-2026-10-08. Marked in the report text, lines: 1.
sources
- results results@20261005-023459-eshoponweb:targets[clickhouse].disk.text = 1.27 GiB added by this load (2.63 GiB (whole engine data folder), 1.36 GiB before)
- results results@20261005-023459-eshoponweb:rows = 524
- results results@20261005-023459-eshoponweb:dimension = 1024
False: framing line. Recorded: Measured end to end through each engine's .NET client The statement is false for 8 of the 19 engines of this run: chroma, clickhouse, elasticsearch, milvus, opensearch, typesense, vespa and weaviate are reached through the benchmark's own HttpClient REST code, not through a .NET client of the engine. For those engines the figure is the cost of the benchmark's own request code. A code line of each is cited. Listed in published-2026-10-08. Marked in the report text, lines: 1.
sources
- file src/GenericVectorBuilder.Engines/Sinks/ChromaSink.cs#new HttpClient { BaseAddress = new Uri( options.BaseUrl = new HttpClient { BaseAddress = new Uri( options.BaseUrl
- file src/GenericVectorBuilder.Engines/Sinks/ClickHouseSink.cs#new HttpClient { BaseAddress = new Uri( $"http://{options.Host}:{options.Port}/" = new HttpClient { BaseAddress = new Uri( $"http://{options.Host}:{options.Port}/"
- file src/GenericVectorBuilder.Engines/Sinks/ElasticsearchRest.cs#new HttpClient { BaseAddress = new Uri( _baseUrl + "/" ) = new HttpClient { BaseAddress = new Uri( _baseUrl + "/" )
- file src/GenericVectorBuilder.Engines/Sinks/MilvusSink.cs#new HttpClient { BaseAddress = new Uri( options.BaseUrl = new HttpClient { BaseAddress = new Uri( options.BaseUrl
- file src/GenericVectorBuilder.Engines/Sinks/OpenSearchRest.cs#new HttpClient { BaseAddress = new Uri( _baseUrl + "/" ) = new HttpClient { BaseAddress = new Uri( _baseUrl + "/" )
- file src/GenericVectorBuilder.Engines/Sinks/TypesenseRest.cs#new HttpClient { BaseAddress = new Uri( _baseUrl + "/" ) = new HttpClient { BaseAddress = new Uri( _baseUrl + "/" )
- file src/GenericVectorBuilder.Engines/Sinks/VespaRest.cs#_data = new HttpClient( handler ) = _data = new HttpClient( handler )
- file src/GenericVectorBuilder.Engines/Sinks/WeaviateSink.cs#new HttpClient { BaseAddress = new Uri( options.BaseUrl = new HttpClient { BaseAddress = new Uri( options.BaseUrl
Not backed by a saved source: index. Recorded: an earlier run gave about 0.09 at 100,000 No log was saved for this figure, and the summary does not print the clause. A saved re-run of the MariaDB effort test (10 repeats, 524 and 2,000 random 1024-dimension vectors) read recall@10 at ef 100 of 0.988 to 0.996 at 524 vectors and 0.894 to 0.932 at 2,000. The 100,000-row figure was not re-measured the same way and is dropped. Listed in published-2026-10-08. Marked in the report text, lines: 2.
sources
- file design/bench-inputs/mariadb-effort-2026-10-07/SOURCES.txt#for which no log was saved = for which no log was saved
- file design/bench-inputs/mariadb-effort-2026-10-07/SOURCES.txt#The 100,000-row figure was not re-measured the same way, so the report drops it. = The 100,000-row figure was not re-measured the same way, so the report drops it.
- file design/bench-inputs/mariadb-effort-2026-10-07/run.log#RECALL@10 on 524 random 1024-dim vectors, 50 queries: ef 20 0.872, ef 100 0.988, ef 3200 0.988 = RECALL@10 on 524 random 1024-dim vectors, 50 queries: ef 20 0.872, ef 100 0.988, ef 3200 0.988
- file design/bench-inputs/mariadb-effort-2026-10-07/run.log#RECALL@10 on 524 random 1024-dim vectors, 50 queries: ef 20 0.886, ef 100 0.996, ef 3200 0.996 = RECALL@10 on 524 random 1024-dim vectors, 50 queries: ef 20 0.886, ef 100 0.996, ef 3200 0.996
- file design/bench-inputs/mariadb-effort-2026-10-07/run.log#RECALL@10 on 2000 random 1024-dim vectors, 50 queries: ef 20 0.468, ef 100 0.894, ef 3200 0.994 = RECALL@10 on 2000 random 1024-dim vectors, 50 queries: ef 20 0.468, ef 100 0.894, ef 3200 0.994
- file design/bench-inputs/mariadb-effort-2026-10-07/run.log#RECALL@10 on 2000 random 1024-dim vectors, 50 queries: ef 20 0.482, ef 100 0.932, ef 3200 0.992 = RECALL@10 on 2000 random 1024-dim vectors, 50 queries: ef 20 0.482, ef 100 0.932, ef 3200 0.992
- file deploy/bench/recorded-text-classes.json#"drop": "mariadb-recall" = "drop": "mariadb-recall"

Machine: CPU Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz; Logical CPUs 8; RAM (GiB) 62.7; OS Ubuntu 24.04.5 LTS, kernel 6.8.0-142-generic; Governor performance; Partition client 0-1,4-5; engines 2-3,6-7; Release build; Exact mode seconds 60.

Warm-up, as this run's notes record it: Warm-up: every timed pass started with its own untimed warm-up of 20 searches in the same search mode and with the same number of searchers, stopped early after 60 s; with machine control on, the check for a quiet box comes right before the warm-up, so warm-up and timed pass run back to back.

No CPU clock pin is recorded in this run's notes.

Engine	p50 (ms)	Searches per second, one searcher	Searches per second, eight searchers at once	Exact mode p50 (ms)	Client CPU per search, one searcher (ms)	Recall in hits	Flags
Chroma	2.04	485	806	-	0.47	200 of 200
ClickHouse	4.74	203	423	6.39	0.89	197 of 200
DuckDB	3.55	279	280	4.04	4.87	200 of 200
Elasticsearch	1.32	747	2,633	1.27	0.47	200 of 200	Ix
MariaDB	0.63	1,558	5,909	1.58	0.49	200 of 200
Milvus	2.22	431	1,282	-	0.55	200 of 200
MongoDB Atlas Local	1.35	722	2,095	1.24	0.31	200 of 200
OpenSearch	1.98	496	1,422	2.58	0.47	200 of 200
Oracle 23ai Free	0.88	1,103	3,007	2.38	0.87	200 of 200
Qdrant (HNSW)	0.91	1,080	3,467	0.89	0.73	200 of 200
Qdrant (exact)	0.87	1,126	3,303	-	0.74	200 of 200
Redis redis holds its data in memory: its saved docs page says 'Redis is an in-memory but persistent on disk database', and its compose file sets save "300 1" and appendonly no. sources doc `doc:design/engine-docs/redis-faq-2026-10-07.html#Redis is an in-memory but persistent on disk database` = Redis is an in-memory but persistent on disk database file `deploy/engines/redis.compose.yaml#--save "300 1"` = --save "300 1" file `deploy/engines/redis.compose.yaml#--appendonly no` = --appendonly no	0.35	2,699	5,027	0.33	0.53	200 of 200
SQL Server 2025	3.85	253	626	-	1.22	200 of 200
SQL Server 2025 + DiskANN	3.58	275	690	3.59	1.03	193 of 200
Typesense	4.04	243	587	5.47	0.51	200 of 200
Vespa	1.87	520	1,550	1.93	0.96	200 of 200
Weaviate	5.18	174	316	-	0.59	200 of 200
pgvector	0.85	1,157	4,116	2.15	0.69	200 of 200
sqlite-vec	1.82	541	532	1.82	1.94	200 of 200

Engines are listed in alphabetical order. The table does not rank them.

A dash means the table has no figure there.

The small markers after an engine name are flags. Hover a marker for its evidence, or read the list below the tables.

Ix index-not-ready The engine did not report a finished index after the load or after the searches.

Evidence behind the flags

Elasticsearch index-not-ready afterLoad: the engine's index state read not ready, 0 of 524 vectors indexed
Elasticsearch index-not-ready afterSearch: the engine's index state read not ready, 0 of 524 vectors indexed

Full results

This report is printed as the run wrote it, except for any sentence a note above it says was left out. This page does not check the engine texts in it. The summary page's facts table gives the label of each fact it uses.

Run 2026-10-05T02:34:59Z (run-all). Command line:

~/gvb-work/lanes/v5-final/src/GenericVectorBuilder.Bench/bin/Release/net10.0/GenericVectorBuilder.Bench.dll run-all --pipeline eshoponweb --queries golden --seed 501 --out ~/ForClaude/GenericVectorBuilder/bench-results

Machine: bench-host, Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz (8 logical CPUs), 62.7 GiB RAM, GPU NVIDIA GeForce RTX 3060, 12288 MiB, Ubuntu 24.04.5 LTS, kernel 6.8.0-142-generic, .NET 10.0.12, load average at start 0.68 1.55 2.18
Data: 524 vectors x 1024 dims, collection gvbbench_eshoponweb. Read READ-ONLY from GenericVectorBuilder.dbo.gvb_eshoponweb in ChunkId order, vectors as native SqlVector<float>, 0.2 s.
Queries: golden: 20 labelled questions from ~/ForClaude/evalkit/questions_golden.json, embedded with qwen3-emb-0.6b (cached in ~/gvb-data/bench-cache/golden-eshoponweb-68df42ca69efa079.json, no embedding calls). Top 10. Throughput at concurrency 1, 8 for 20 s each.
Request speed on a small collection (524 vectors): Measured end to end through each engine's .NET client [Correction 3]; at this size it reflects per-request cost including the client library, not index scaling.
Client CPU per search is the CPU time the test's .NET client itself used for each search, measured in the same pass as the figure beside it.
Ground truth: brute force over every vector in memory, 0.0 s for all 20 queries.
nDCG@10 of the exact answer itself (the ceiling for this embedder): 0.518

engine	index	load rows/s	p50 ms	p95 ms	p99 ms	QPS@1	QPS@8	client CPU ms/search@1	client CPU ms/search@8	recall@10	nDCG@10	RAM	disk
clickhouse	vector_similarity HNSW cosineDistance, quantization bf16, M=16 ef_construction=128, hnsw_candidate_list_size_for_search=256, rescoring off; exact mode = full scan with skip indexes off	2,390	4.74	6.06	7.52	202.9	423.2	0.89	1.10	0.985	0.528	1.9 GiB	1.27 GiB
oracle	HNSW in-memory neighbor graph NEIGHBORS=16 EFCONSTRUCTION=128, EFSEARCH=100 per query, cosine; exact mode = FETCH EXACT FIRST (full scan); Oracle Free caps itself at 2 CPUs (cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit)	1,781	0.88	1.05	1.79	1102.6	3006.9	0.87	0.64	1.000	0.518	2.15 GiB	0 B
milvus	HNSW M=16 efConstruction=128, ef=100, metric COSINE, Strong consistency searches; approximate only (no exact mode)	1,084	2.22	2.72	3.61	431.3	1282.0	0.55	0.64	1.000	0.518	195 MiB	10.86 MiB
sqlitevec	vec0 brute-force scan, no ANN index (exact), float32, cosine distance, default chunk_size=1024; score = 1 - cosine distance	4,302	1.82	2.03	2.27	541.0	532.1	1.94	1.98	1.000	0.518	-	9.72 MiB
redis	HNSW TYPE FLOAT32 M=16 EF_CONSTRUCTION=128, EF_RUNTIME=100 per query, cosine; exact mode = FLAT index built on first exact query	11,357	0.35	0.46	0.67	2699.3	5027.4	0.53	0.36	1.000	0.518	24.61 MiB	0 B
weaviate	HNSW maxConnections(M)=16 efConstruction=128, ef=-1 (dynamic: limit x 8 clamped 100..500), cosine, no quantization; approximate only (no exact mode)	1,025	5.18	10.3	11.4	173.6	315.6	0.59	0.61	1.000	0.518	156.9 MiB	2.88 MiB
sql-diskann	DiskANN (preview) via VECTOR_SEARCH, cosine, build {"StartId":"306", "L":"48", "M":"8", "R":"48"}; the exact mode scans the same table	604	3.58	4.03	5.19	275.4	690.0	1.03	1.09	0.965	0.526	531.8 MiB	4.52 MiB
mongodb	vectorSearch index, HNSW maxEdges=16 numEdgeCandidates=128, float32 binData, cosine, numCandidates=20x hits (min 100); exact mode = $vectorSearch exact:true	3,275	1.35	1.63	1.95	722.3	2095.5	0.31	0.30	1.000	0.518	911 MiB	0 B
duckdb	HNSW (vss extension) FLOAT[n] metric=cosine m=16 ef_construction=128, ef_search=100 per connection, persistent (hnsw_enable_experimental_persistence=true, checkpoint_threshold=256MB); exact mode = array_cosine_similarity sequential scan; score = 1 - cosine distance	1,543	3.55	3.93	5.11	278.8	280.1	4.87	4.94	1.000	0.518	-	2.9 MiB
qdrant	exact scan: the builder's sink sends exact=true on every search, so no HNSW graph is used whether or not Qdrant has built one (see the index state)	4,666	0.87	0.99	1.46	1125.8	3303.1	0.74	0.54	1.000	0.518	39.43 MiB	196.08 MiB
sql	exact VECTOR_DISTANCE cosine, no vector index (full scan)	641	3.85	4.39	5.09	253.0	625.6	1.22	1.19	1.000	0.518	529.9 MiB	5.15 MiB
opensearch	faiss HNSW float32, no compression, m=16, ef_construction=128, cosinesimil; search k=top, ef_search=100; 1 shard, 0 replicas; graph built at any segment size (approximate_threshold=0); force-merged to one segment after the load	388	1.98	2.19	2.73	496.1	1422.1	0.47	0.52	1.000	0.518	2.58 GiB	9.64 MiB
pgvector	HNSW vector_cosine_ops m=16 ef_construction=128, hnsw.ef_search=100 per query, float32 vector(n), cosine; exact mode = same query with index scans off (sequential scan)	616	0.85	0.99	1.29	1157.5	4116.0	0.69	0.54	1.000	0.518	100.2 MiB	7.59 MiB
chroma	HNSW M=16 ef_construction=128, ef_search=100 (Chroma default), cosine; approximate only (no exact mode)	882	2.04	2.29	2.55	484.5	805.9	0.47	0.47	1.000	0.518	90.18 MiB	414.16 KiB
vespa	HNSW float32 tensor, prenormalized-angular (cosine), max-links-per-node=16, neighbors-to-explore-at-insert=128; search targetHits=top, ef=100 via exploreAdditionalHits; exact mode = approximate:false; vectors held in memory	321	1.87	2.28	2.60	520.2	1550.0	0.96	0.90	1.000	0.518	2.93 GiB	31.23 MiB
qdrant-hnsw	HNSW m=16 ef_construct=100, hnsw_ef=server default, cosine; indexing_threshold_kb 1 and full_scan_threshold_kb 10 (server defaults are 10,000 each) so a small collection builds and walks its graph	9,152	0.91	1.02	1.49	1080.1	3467.1	0.73	0.53	1.000	0.518	36.52 MiB	164.1 MiB
elasticsearch	HNSW float32, no quantization, m=16, ef_construction=128, cosine; search k=top, num_candidates=100; 1 shard, 0 replicas; force-merged to one segment after the load (at 1,024 dimensions a segment under 1,043 vectors gets no graph)	499	1.32	1.48	1.78	746.8	2632.6	0.47	0.51	1.000	0.518	2.53 GiB	2.38 MiB
typesense	HNSW float32 (hnswlib), m=16, ef_construction=128, cosine; search k=top, ef=100; exact mode = filter ordinal:>=0 with flat_search_cutoff; index held in memory	354	4.04	4.44	6.68	243.2	586.8	0.51	0.58	1.000	0.518	171.7 MiB	0 B
mariadb	VECTOR INDEX (HNSW variant) DISTANCE=cosine, M=16 (no ef_construction setting exists), mhnsw_ef_search=100 per statement (the ef 100 most engines here use, so the search effort matches; MariaDB's own default is 20; recall@10 at ef 100 falls as the set grows (random 1024-dimension vectors, measured 2026-10-04: 0.99 at 524, 0.89 to 0.92 at 2,000; an earlier run gave about 0.09 at 100,000 [Correction 4])), mhnsw_max_cache_size 4G; exact mode = IGNORE INDEX full scan	1,089	0.63	0.71	0.89	1557.7	5909.4	0.49	0.38	1.000	0.518	156.3 MiB	22.01 MiB

Details per target

ClickHouse clickhouse

Engine: ClickHouse 26.3.39.7 (MergeTree + vector_similarity index) (compose)
Index: vector_similarity HNSW cosineDistance, quantization bf16, M=16 ef_construction=128, hnsw_candidate_list_size_for_search=256, rescoring off; exact mode = full scan with skip indexes off
Load: 524 rows in batches of 1000, 0.2 s of upserts (2,390 rows/s); count matched 0.0 s after the last upsert; index step 6.2 s (1 data part(s), index files on 1 of them, 524 of 524 live rows in indexed parts; 0 merge(s) running, 0 unfinished mutation(s); plan of the default search uses the Skip index vec_idx on 1 of 1 parts; wait took 6.2 s)
Search: 4059 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 0.89 ms, 8 searchers 1.10 ms
Exact mode: 9,056 searches, p50 6.39 ms, p95 8.76 ms, recall 1.000
RAM: 1.9 GiB (docker stats); disk: 1.27 GiB added by this load [Correction 2] (2.63 GiB (whole engine data folder), 1.36 GiB before)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started clickhouse.compose.yaml with every container created on CPUs 2-3,6-7: gvb-clickhouse cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
Index after the load: ready, 524 of 524 indexed (1 data part(s), index files on 1 of them, 524 of 524 live rows in indexed parts; 0 merge(s) running, 0 unfinished mutation(s); plan of the default search uses the Skip index vec_idx on 1 of 1 parts). Durability: Acknowledged inserts are not fsynced. The sink creates plain MergeTree tables, so the MergeTree settings are the defaults: fsync_after_insert 0 and fsync_part_directory 0 (system.merge_tree_settings), and async_insert 1 with wait_for_async_insert 1 is the server default for the gvb login (system.settings), so an insert is acknowledged once its part is written, not once it is on disk. Measured 2026-10-04 with strace over 30 acknowledged single-row inserts (30 Ok rows in system.asynchronous_insert_log): zero fsync, fdatasync or sync_file_range calls, while the control, a table created with fsync_after_insert=1, made 61 fdatasync calls for 5 inserts. A host power loss or kernel crash can lose acknowledged rows that are still in the OS page cache; a ClickHouse process crash alone should not (inferred, not tested)..
Load average 0.96 1.57 2.17 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: exact 765 searches in 5.0 s, default@1 1,011 searches in 5.0 s, default@8 2,163 searches in 5.0 s.
Settle: settled after 1.7 s and 300 searches (0 failed); p50 of the last windows of 100: 5.17, 5.27, 5.34 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 386 searches in 2.0 s: p50 4.802 ms against the settled p50 5.274 ms, 10% apart (limit 15%).
Pass exact after settle: 2026-10-05T02:35:33.151Z to 2026-10-05T02:36:33.152Z (60.0 s), 9,056 searches, 0 failed, p50 6.388 ms, mean 6.622 ms, p99 10.084 ms, 150.9 QPS (1000/QPS 6.626 ms).
Pass default@1 after exact: 2026-10-05T02:36:33.291Z to 2026-10-05T02:36:53.292Z (20.0 s), 4,059 searches, 0 failed, p50 4.736 ms, mean 4.925 ms, p99 7.515 ms, 202.9 QPS (1000/QPS 4.928 ms).
Pass default@8 after default@1: 2026-10-05T02:36:53.344Z to 2026-10-05T02:37:13.356Z (20.0 s), 8,470 searches, 0 failed, p50 18.293 ms, mean 18.893 ms, p99 35.706 ms, 423.2 QPS (1000/QPS 2.363 ms).
Passes in the order run: exact, default@1, default@8; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 4,685 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (1 data part(s), index files on 1 of them, 524 of 524 live rows in indexed parts; 0 merge(s) running, 0 unfinished mutation(s); plan of the default search uses the Skip index vec_idx on 1 of 1 parts).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-clickhouse (30f63ad4fb06) on network gvb-clickhouse_default at container-ip:8123, not through docker-proxy localhost:8123. Open after the passes: container-ip:8123 x8 (this target), localhost:1433 x2 (not this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-clickhouse: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-clickhouse: 738 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): exact 3592/3592 MHz, performance, outside load 0.16 before the warm-up, 0.14 during, client CPU 0.895 ms per search; default@1 3589/3591 MHz, performance, outside load 0.14 before the warm-up, 0.13 during, client CPU 0.886 ms per search; default@8 3553/3562 MHz, performance, outside load 0.13 before the warm-up, 0.11 during, client CPU 1.104 ms per search.

Oracle 23ai Free oracle

Engine: Oracle AI Database Free 23.26 (23ai line, VECTOR FLOAT32) (compose)
Index: HNSW in-memory neighbor graph NEIGHBORS=16 EFCONSTRUCTION=128, EFSEARCH=100 per query, cosine; exact mode = FETCH EXACT FIRST (full scan); Oracle Free caps itself at 2 CPUs (cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit)
Load: 524 rows in batches of 1000, 0.3 s of upserts (1,781 rows/s); count matched 0.1 s after the last upsert; index step 3.5 s (HNSW graph holds 524 of 524 rows, change log waiting: 0 inserts and 0 deletes, USER_INDEXES status VALID, plan of the default search: VECTOR INDEX HNSW SCAN > TABLE ACCESS BY INDEX ROWID, index used by 0 queries so far; Oracle Free caps itself at 2 CPUs (cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit); graph rebuilt in 2.3 s)
Search: 22053 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 0.87 ms, 8 searchers 0.64 ms
Exact mode: 24,606 searches, p50 2.38 ms, p95 2.78 ms, recall 1.000
RAM: 2.15 GiB (docker stats); disk: 0 B added by this load (6.88 GiB (whole engine data folder), 6.88 GiB before)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started oracle.compose.yaml with every container created on CPUs 2-3,6-7: gvb-oracle cpuset 2-3,6-7, gvb-oracle-seed cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
Index after the load: ready, 524 of 524 indexed (HNSW graph holds 524 of 524 rows, change log waiting: 0 inserts and 0 deletes, USER_INDEXES status VALID, plan of the default search: VECTOR INDEX HNSW SCAN > TABLE ACCESS BY INDEX ROWID, index used by 0 queries so far; Oracle Free caps itself at 2 CPUs (cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit)). Durability: By these settings a committed row should survive a crash or power loss. The sink uses a plain COMMIT and commit_logging, commit_wait and commit_write are unset in the database, so every commit waits for its redo to be written. Measured 2026-10-04: 50 separate client commits raised V$SYSSTAT 'redo synch writes' by 55 (the 50 commits plus the CREATE and DROP of the probe table), and the log writer and the datafile writer hold their files open with O_DSYNC (open flags 02110002, filesystemio_options none). The database runs NOARCHIVELOG (V$DATABASE.LOG_MODE), so redo serves crash recovery only and there is no point-in-time restore. The HNSW graph lives in the 768 MB vector memory pool (oracle-init/01-vector-memory.sh) and is not the durable copy; the table is. Not tested by cutting power; whether the disk's own write cache reaches the media was not checked. CPU: Oracle Free caps itself at 2 CPUs (measured 2026-10-04: cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit; this sink reads the live value when the index state is read)..
Load average 4.84 2.65 2.46 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: exact 1,819 searches in 5.0 s, default@8 19,198 searches in 5.2 s, default@1 5,401 searches in 5.0 s.
Settle: settled after 0.6 s and 600 searches (0 failed); p50 of the last windows of 100: 0.88, 0.91, 0.86 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 2,238 searches in 2.0 s: p50 0.871 ms against the settled p50 0.881 ms, 1% apart (limit 15%).
Pass exact after settle: 2026-10-05T02:37:56.711Z to 2026-10-05T02:38:56.713Z (60.0 s), 24,606 searches, 0 failed, p50 2.384 ms, mean 2.437 ms, p99 3.516 ms, 410.1 QPS (1000/QPS 2.439 ms).
Pass default@8 after exact: 2026-10-05T02:38:56.756Z to 2026-10-05T02:39:16.758Z (20.0 s), 60,143 searches, 0 failed, p50 1.568 ms, mean 2.658 ms, p99 3.357 ms, 3006.9 QPS (1000/QPS 0.333 ms).
Pass default@1 after default@8: 2026-10-05T02:39:16.806Z to 2026-10-05T02:39:36.806Z (20.0 s), 22,053 searches, 0 failed, p50 0.884 ms, mean 0.905 ms, p99 1.790 ms, 1102.6 QPS (1000/QPS 0.907 ms).
Passes in the order run: exact, default@8, default@1; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 29,316 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (HNSW graph holds 524 of 524 rows, change log waiting: 0 inserts and 0 deletes, USER_INDEXES status VALID, plan of the default search: VECTOR INDEX HNSW SCAN > TABLE ACCESS BY INDEX ROWID, index used by 109673 queries so far; Oracle Free caps itself at 2 CPUs (cpu_count 2 in V$PARAMETER, edition FREE in V$INSTANCE, 8 host CPUs in V$OSSTAT NUM_CPUS; the 2 CPU thread limit is Oracle's documented Free edition limit)).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-oracle (ca4c1282a7a0) on network gvb-oracle_default at container-ip:1521, not through docker-proxy localhost:1521. Open after the passes: container-ip:1521 x14 (this target), localhost:1433 x2 (not this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-oracle: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-oracle: 86 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): exact 3492/3492 MHz, performance, outside load 0.12 before the warm-up, 0.12 during, client CPU 0.924 ms per search; default@8 3492/3492 MHz, performance, outside load 0.12 before the warm-up, 0.11 during, client CPU 0.641 ms per search; default@1 3492/3492 MHz, performance, outside load 0.12 before the warm-up, 0.12 during, client CPU 0.874 ms per search.

Milvus milvus

Engine: Milvus 2.6.25 (standalone, embedded etcd, local storage, REST API v2) (compose)
Index: HNSW M=16 efConstruction=128, ef=100, metric COSINE, Strong consistency searches; approximate only (no exact mode)
Load: 524 rows in batches of 1000, 0.5 s of upserts (1,084 rows/s); count matched 1.7 s after the last upsert; index step 138.9 s (index Finished, type HNSW (COSINE, {"M":16,"efConstruction":128}), indexedRows 524 of 524 sealed rows, pendingRows 0; stored rows 524; LoadStateLoaded; query node: 1 sealed segment(s) with the index loaded covering 524 rows against 524 stored rows, 0 sealed without it, 0 rows in growing segments; datacoord: 1 live segment(s), 1 flushed and indexed covering 524 rows, 0 delete-log (L0) segment(s) waiting for a compaction, 3 segment(s) compacted away so far; flush, index build and load took 14.7 s, then compaction: 1 job(s) accepted by Milvus, layout ready and unchanged for 75 s, 124.1 s for the whole step; search ledger started, later state reads count the searches against the query node's own counters)
Search: 8627 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 0.55 ms, 8 searchers 0.64 ms
RAM: 195 MiB (docker stats); disk: 10.86 MiB added by this load (149.03 MiB (whole engine data folder), 138.18 MiB before)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started milvus.compose.yaml with every container created on CPUs 2-3,6-7: gvb-milvus cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
Index after the load: ready, 524 of 524 indexed (index Finished, type HNSW (COSINE, {"M":16,"efConstruction":128}), indexedRows 524 of 524 sealed rows, pendingRows 0; stored rows 524; LoadStateLoaded; query node: 1 sealed segment(s) with the index loaded covering 524 rows against 524 stored rows, 0 sealed without it, 0 rows in growing segments; datacoord: 1 live segment(s), 1 flushed and indexed covering 524 rows, 0 delete-log (L0) segment(s) waiting for a compaction, 3 segment(s) compacted away so far; search ledger since the index finished: the sink sent 0 searches (searchParams.params.ef=100, Strong consistency), the query node counted 0 search requests for this collection, segment searches Sealed +0 and Growing +0, the same sealed segments with the same loaded index builds at both readings, segments compacted away +0). Durability: Writes are not fsynced. Standalone uses the default message queue rocksmq (RocksDB, /var/lib/milvus/rdb_data, mq.type default) and flushed segments go to local-disk object storage (COMMON_STORAGETYPE=local in milvus.compose.yaml). Measured 2026-10-04 with strace over 30 acknowledged single-row upserts and one flush: fdatasync ran only on the embedded etcd files (member/wal, member/snap/db), never on rdb_data or the segment files. A host power loss or kernel crash can lose acknowledged rows that are still in the OS page cache; a Milvus process crash alone should not (inferred, not tested). Metadata in embedded etcd is fdatasynced on every commit..
Load average 0.53 1.64 2.12 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: default@1 2,089 searches in 5.0 s, default@8 6,356 searches in 5.0 s.
Settle: settled after 0.7 s and 300 searches (0 failed); p50 of the last windows of 100: 2.24, 2.22, 2.21 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 858 searches in 2.0 s: p50 2.221 ms against the settled p50 2.222 ms, 0% apart (limit 15%).
Pass default@1 after settle: 2026-10-05T02:42:21.709Z to 2026-10-05T02:42:41.710Z (20.0 s), 8,627 searches, 0 failed, p50 2.216 ms, mean 2.316 ms, p99 3.608 ms, 431.3 QPS (1000/QPS 2.318 ms).
Pass default@8 after default@1: 2026-10-05T02:42:41.735Z to 2026-10-05T02:43:01.739Z (20.0 s), 25,645 searches, 0 failed, p50 5.702 ms, mean 6.237 ms, p99 14.062 ms, 1282.0 QPS (1000/QPS 0.780 ms).
Passes in the order run: default@1, default@8; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 9,643 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (index Finished, type HNSW (COSINE, {"M":16,"efConstruction":128}), indexedRows 524 of 524 sealed rows, pendingRows 0; stored rows 524; LoadStateLoaded; query node: 1 sealed segment(s) with the index loaded covering 524 rows against 524 stored rows, 0 sealed without it, 0 rows in growing segments; datacoord: 1 live segment(s), 1 flushed and indexed covering 524 rows, 0 delete-log (L0) segment(s) waiting for a compaction, 3 segment(s) compacted away so far; search ledger since the index finished: the sink sent 43915 searches (searchParams.params.ef=100, Strong consistency), the query node counted 43915 search requests for this collection, segment searches Sealed +43247 and Growing +0, the same sealed segments with the same loaded index builds at both readings, segments compacted away +0).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-milvus (d756e733f234) on network gvb-milvus_default at container-ip:19530, not through docker-proxy localhost:19530; container gvb-milvus (d756e733f234) on network gvb-milvus_default at container-ip:9091, not through docker-proxy localhost:9091. Open after the passes: container-ip:19530 x8 (this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-milvus: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-milvus: 24 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@1 3545/3504 MHz, performance, outside load 0.14 before the warm-up, 0.10 during, client CPU 0.547 ms per search; default@8 3492/3494 MHz, performance, outside load 0.12 before the warm-up, 0.15 during, client CPU 0.640 ms per search.

sqlite-vec sqlitevec

Engine: SQLite 3.53.3 + sqlite-vec 0.1.7-alpha.2.1 (vec0, embedded, in process) (embedded)
Index: vec0 brute-force scan, no ANN index (exact), float32, cosine distance, default chunk_size=1024; score = 1 - cosine distance
Load: 524 rows in batches of 1000, 0.1 s of upserts (4,302 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (nothing to build, every search scans all vectors; checked in 0.00 s: no index, exact scan by design: gvb_gvbbench_eshoponweb is a vec0 virtual table holding 524 vectors and every search compares all of them (EXPLAIN QUERY PLAN: SCAN gvb_gvbbench_eshoponweb VIRTUAL TABLE INDEX 0:3{___}___ | USE TEMP B-TREE FOR ORDER BY))
Search: 10821 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 1.94 ms, 8 searchers 1.98 ms
Exact mode: 32,484 searches, p50 1.82 ms, p95 2.03 ms, recall 1.000
RAM: in this process, not measured; disk: 9.72 MiB added by this load (9.72 MiB (engine data folder), 0 B before)
Index after the load: ready, ? of 524 indexed (no index, exact scan by design: gvb_gvbbench_eshoponweb is a vec0 virtual table holding 524 vectors and every search compares all of them (EXPLAIN QUERY PLAN: SCAN gvb_gvbbench_eshoponweb VIRTUAL TABLE INDEX 0:3{___}___ | USE TEMP B-TREE FOR ORDER BY)). Durability: PRAGMA journal_mode=WAL and PRAGMA synchronous=NORMAL (set by this sink when it opens the file): each commit is appended to the -wal file and handed to the operating system, but the WAL is fsynced only when SQLite checkpoints it (measured with strace: 40 single-record commits caused 3 WAL fsyncs), so a crash of the process loses nothing (measured with kill -9 and a reopen: all 50, 300 and 2,000 rows were there), while an operating-system crash or power cut can lose the newest commits since the last checkpoint and leaves the database consistent.
Load average 3.69 2.31 2.31 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: default@8 2,611 searches in 5.0 s, default@1 2,723 searches in 5.0 s, exact 2,709 searches in 5.0 s.
Settle: settled after 0.6 s and 300 searches (0 failed); p50 of the last windows of 100: 1.82, 1.82, 1.82 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 1,090 searches in 2.0 s: p50 1.817 ms against the settled p50 1.816 ms, 0% apart (limit 15%).
Pass default@8 after settle: 2026-10-05T02:43:26.209Z to 2026-10-05T02:43:46.223Z (20.0 s), 10,649 searches, 0 failed, p50 14.831 ms, mean 15.028 ms, p99 17.377 ms, 532.1 QPS (1000/QPS 1.879 ms).
Pass default@1 after default@8: 2026-10-05T02:43:46.265Z to 2026-10-05T02:44:06.266Z (20.0 s), 10,821 searches, 0 failed, p50 1.823 ms, mean 1.846 ms, p99 2.268 ms, 541.0 QPS (1000/QPS 1.848 ms).
Pass exact after default@1: 2026-10-05T02:44:06.317Z to 2026-10-05T02:45:06.318Z (60.0 s), 32,484 searches, 0 failed, p50 1.823 ms, mean 1.845 ms, p99 2.249 ms, 541.4 QPS (1000/QPS 1.847 ms).
Passes in the order run: default@8, default@1, exact; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 9,493 sent, 0 failed (warmupErrors). Index after the searches: ready, ? of 524 indexed (no index, exact scan by design: gvb_gvbbench_eshoponweb is a vec0 virtual table holding 524 vectors and every search compares all of them (EXPLAIN QUERY PLAN: SCAN gvb_gvbbench_eshoponweb VIRTUAL TABLE INDEX 0:3{___}___ | USE TEMP B-TREE FOR ORDER BY)).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: in this process (embedded), no network: no network address. No open connection seen (no open TCP connection was found).
CPU pinning: none: an embedded engine runs inside the benchmark client process, so it shares the client CPUs, CPUs 0-1,4-5.
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@8 3593/3592 MHz, performance, outside load 0.12 before the warm-up, 0.12 during, client CPU 1.975 ms per search; default@1 3600/3592 MHz, performance, outside load 0.12 before the warm-up, 0.13 during, client CPU 1.940 ms per search; exact 3593/3592 MHz, performance, outside load 0.12 before the warm-up, 0.12 during, client CPU 1.939 ms per search.

Redis redis

Engine: Redis 8.10.2 (query engine, HASH + vector index) (compose)
Index: HNSW TYPE FLOAT32 M=16 EF_CONSTRUCTION=128, EF_RUNTIME=100 per query, cosine; exact mode = FLAT index built on first exact query
Load: 524 rows in batches of 1000, 0.0 s of upserts (11,357 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (HNSW index finished after 0.0 s of waiting: FT.INFO indexing 0, percent_indexed 1, hash_indexing_failures 0, flat_buffer_size 0, num_docs 524, hashes counted with SCAN under gvb_gvbbench_eshoponweb: 524)
Search: 53987 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 0.53 ms, 8 searchers 0.36 ms
Exact mode: 169,665 searches, p50 0.33 ms, p95 0.45 ms, recall 1.000
RAM: 24.61 MiB (docker stats); disk: 0 B added by this load (89 B (whole engine data folder), 89 B before)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started redis.compose.yaml with every container created on CPUs 2-3,6-7: gvb-redis cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
Index after the load: ready, 524 of 524 indexed (FT.INFO indexing 0, percent_indexed 1, hash_indexing_failures 0, flat_buffer_size 0, num_docs 524, hashes counted with SCAN under gvb_gvbbench_eshoponweb: 524). Durability: save "300 1" and appendonly no (redis.compose.yaml): an RDB snapshot is written every 5 minutes if at least one key changed, and on a clean stop; there is no append-only log, so a crash loses every write since the last snapshot (up to 5 minutes plus the time a snapshot takes; measured with docker kill, which is SIGKILL, right after a 2,000-vector load: none of the 2,000 hashes were there after the restart, and the restart loaded an older snapshot that still held keys of a collection that had been dropped since).
Load average 1.41 1.90 2.16 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: exact 12,924 searches in 5.0 s, default@8 25,244 searches in 5.0 s, default@1 13,330 searches in 5.0 s.
Settle: settled after 0.4 s and 1,100 searches (0 failed); p50 of the last windows of 100: 0.34, 0.35, 0.35 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 5,500 searches in 2.0 s: p50 0.343 ms against the settled p50 0.345 ms, 1% apart (limit 15%).
Pass exact after settle: 2026-10-05T02:45:30.347Z to 2026-10-05T02:46:30.347Z (60.0 s), 169,665 searches, 0 failed, p50 0.334 ms, mean 0.352 ms, p99 0.651 ms, 2827.7 QPS (1000/QPS 0.354 ms).
Pass default@8 after exact: 2026-10-05T02:46:30.549Z to 2026-10-05T02:46:50.550Z (20.0 s), 100,556 searches, 0 failed, p50 1.576 ms, mean 1.590 ms, p99 2.213 ms, 5027.4 QPS (1000/QPS 0.199 ms).
Pass default@1 after default@8: 2026-10-05T02:46:50.598Z to 2026-10-05T02:47:10.598Z (20.0 s), 53,987 searches, 0 failed, p50 0.350 ms, mean 0.369 ms, p99 0.674 ms, 2699.3 QPS (1000/QPS 0.370 ms).
Passes in the order run: exact, default@8, default@1; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 58,158 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (FT.INFO indexing 0, percent_indexed 1, hash_indexing_failures 0, flat_buffer_size 0, num_docs 524, hashes counted with SCAN under gvb_gvbbench_eshoponweb: 524).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-redis (5d14ad35a658) on network gvb-redis_default at container-ip:6379, not through docker-proxy localhost:6379. Open after the passes: container-ip:6379 x1 (this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-redis: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-redis: 6 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): exact 3492/3492 MHz, performance, outside load 0.16 before the warm-up, 0.11 during, client CPU 0.542 ms per search; default@8 3492/3492 MHz, performance, outside load 0.11 before the warm-up, 0.09 during, client CPU 0.356 ms per search; default@1 3492/3492 MHz, performance, outside load 0.10 before the warm-up, 0.10 during, client CPU 0.526 ms per search.

Weaviate weaviate

Engine: Weaviate 1.39.8 (single node, REST + GraphQL) (compose)
Index: HNSW maxConnections(M)=16 efConstruction=128, ef=-1 (dynamic: limit x 8 clamped 100..500), cosine, no quantization; approximate only (no exact mode)
Load: 524 rows in batches of 1000, 0.5 s of upserts (1,025 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (HNSW is updated inside every batch, nothing to build; confirmed in 0.0 s: shard qYqOggOKd2kN vectorIndexingStatus READY, vectorQueueLength 0, node status objectCount 0 (refreshed only when the memtable is flushed, so it lags a minute and does not decide readiness), schema shard qYqOggOKd2kN status READY, Aggregate count 524, Aggregate nearVector with objectLimit 524 reached 524, Weaviate reports no count of vectors in the HNSW index, so indexed vectors are not reported)
Search: 3472 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 0.59 ms, 8 searchers 0.61 ms
RAM: 156.9 MiB (docker stats); disk: 2.88 MiB added by this load (4.94 MiB (whole engine data folder), 2.06 MiB before)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started weaviate.compose.yaml with every container created on CPUs 2-3,6-7: gvb-weaviate cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
Index after the load: ready, ? of 524 indexed (shard qYqOggOKd2kN vectorIndexingStatus READY, vectorQueueLength 0, node status objectCount 0 (refreshed only when the memtable is flushed, so it lags a minute and does not decide readiness), schema shard qYqOggOKd2kN status READY, Aggregate count 524, Aggregate nearVector with objectLimit 524 reached 524, Weaviate reports no count of vectors in the HNSW index, so indexed vectors are not reported). Durability: Weaviate 1.39.8 defaults, weaviate.compose.yaml sets no persistence variable [Correction 1]: every object is appended to the LSM write-ahead log with a plain write and no fsync, and the log is fsynced only when its memtable is flushed, 60 seconds after the last write (PERSISTENCE_MEMTABLES_FLUSH_IDLE_AFTER_SECONDS default; measured with strace: 2,080 writes into the objects log during a 2,000-object load, the first fsync 60 s after the last write), so a power cut can lose the last minute of writes; the HNSW commit log is buffered inside the process (77 writes for 2,000 vectors), so a killed process loses the newest vectors from the vector index while their objects survive (measured with docker kill, which is SIGKILL, one second after the load and a restart, two runs each: 0 to 1 of 50, 264 to 267 of 300 and 1,992 to 1,997 of 2,000 stored objects were still reachable through a vector search).
Load average 1.88 2.01 2.17 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: default@8 1,582 searches in 5.0 s, default@1 867 searches in 5.0 s.
Settle: settled after 1.7 s and 300 searches (0 failed); p50 of the last windows of 100: 5.20, 5.22, 5.19 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 347 searches in 2.0 s: p50 5.184 ms against the settled p50 5.198 ms, 0% apart (limit 15%).
Pass default@8 after settle: 2026-10-05T02:47:48.443Z to 2026-10-05T02:48:08.460Z (20.0 s), 6,318 searches, 0 failed, p50 23.816 ms, mean 25.331 ms, p99 53.319 ms, 315.6 QPS (1000/QPS 3.168 ms).
Pass default@1 after default@8: 2026-10-05T02:48:08.580Z to 2026-10-05T02:48:28.582Z (20.0 s), 3,472 searches, 0 failed, p50 5.182 ms, mean 5.758 ms, p99 11.359 ms, 173.6 QPS (1000/QPS 5.761 ms).
Passes in the order run: default@8, default@1; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 3,136 sent, 0 failed (warmupErrors). Index after the searches: ready, ? of 524 indexed (shard qYqOggOKd2kN vectorIndexingStatus READY, vectorQueueLength 0, node status objectCount 0 (refreshed only when the memtable is flushed, so it lags a minute and does not decide readiness), schema shard qYqOggOKd2kN status READY, Aggregate count 524, Aggregate nearVector with objectLimit 524 reached 524, Weaviate reports no count of vectors in the HNSW index, so indexed vectors are not reported).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-weaviate (f80067834af1) on network gvb-weaviate_default at container-ip:8080, not through docker-proxy localhost:8085. Open after the passes: container-ip:8080 x1 (this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-weaviate: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-weaviate: 9 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@8 3577/3574 MHz, performance, outside load 0.12 before the warm-up, 0.12 during, client CPU 0.615 ms per search; default@1 3586/3564 MHz, performance, outside load 0.12 before the warm-up, 0.11 during, client CPU 0.593 ms per search.

SQL Server 2025 + DiskANN sql-diskann

Engine: Microsoft SQL Server 2025 (RTM-CU9) (KB5122048) - 17.0.5005.3 (X64), Enterprise Developer Edition (64-bit), in container gvb-mssql (image mcr.microsoft.com/mssql/server:2025-CU9-ubuntu-24.04@sha256:2b5b581621126574f3d1f75e78d3eebe8d05aedb59ad0cfdf9aa42cb0634d726; cpuset 2-3,6-7 from its creation; SQL Server counts 4 CPU(s) and runs 4 visible scheduler(s), affinity AUTO) (compose)
Index: DiskANN (preview) via VECTOR_SEARCH, cosine, build {"StartId":"306", "L":"48", "M":"8", "R":"48"}; the exact mode scans the same table
Load: 524 rows in batches of 1000, 0.9 s of upserts (604 rows/s); count matched 0.0 s after the last upsert; index step 4.0 s (copied 524 rows to dbo.gvb_gvbbench_eshoponweb_ann (INT key) and built DiskANN, parameters {"StartId":"306", "L":"48", "M":"8", "R":"48"}, graph covers 524 rows)
Search: 5509 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 1.03 ms, 8 searchers 1.09 ms
Exact mode: 16,339 searches, p50 3.59 ms, p95 4.10 ms, recall 1.000
RAM: 531.8 MiB (docker stats); disk: 4.52 MiB (table and its indexes, reserved pages)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started mssql.compose.yaml with every container created on CPUs 2-3,6-7: gvb-mssql cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
Index after the load: ready, 524 of 524 indexed (DiskANN index built and used. sys.vector_indexes: vix_gvb_gvbbench_eshoponweb on dbo.gvb_gvbbench_eshoponweb_ann in GvbBenchDiskAnn, DiskANN, COSINE, enabled, build {"StartId":"306", "L":"48", "M":"8", "R":"48"}; graph table rows 524 of 524 table rows. Real VECTOR_SEARCH plan: Vector Index Seek on index vix_gvb_gvbbench_eshoponweb of GvbBenchDiskAnn.gvb_gvbbench_eshoponweb_ann (IndexKind DiskANN), 10 rows returned, 10 hits. The exact mode searches the SAME table (dbo.gvb_gvbbench_eshoponweb_ann, database GvbBenchDiskAnn) and its real plan is: Clustered Index Scan of GvbBenchDiskAnn.gvb_gvbbench_eshoponweb_ann through PK_gvb_gvbbench_eshoponweb_ann (IndexKind Clustered), 524 rows read, no vector index operator, 10 hits). Durability: A commit returns after its transaction-log records are written to disk (SQL Server write-ahead logging; delayed durability is DISABLED); a database created here copies the model database: recovery model FULL, page_verify CHECKSUM; no global trace flags are enabled; mssql.conf of container gvb-mssql (/var/opt/mssql/mssql.conf, on the host at ~/gvb-data/engines/mssql/mssql.conf) does not exist, so it sets: nothing; it has no [control] or [traceflag] entry, so SQL Server's own Linux defaults for flushing writes apply (Microsoft's Linux performance guide names trace flag 3982 as that default; read from the guide, not tested here). Not tested by cutting power; whether the disk's own write cache reaches the media was not checked. Container settings from its environment (names only): MSSQL_AGENT_ENABLED, MSSQL_MEMORY_LIMIT_MB, MSSQL_PID, MSSQL_RPC_PORT, MSSQL_SA_PASSWORD..
Load average 2.27 2.16 2.21 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: default@1 1,332 searches in 5.0 s, default@8 3,090 searches in 5.0 s, exact 1,358 searches in 5.0 s.
Settle: settled after 1.1 s and 300 searches (0 failed); p50 of the last windows of 100: 3.57, 3.59, 3.60 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 555 searches in 2.0 s: p50 3.585 ms against the settled p50 3.588 ms, 0% apart (limit 15%).
Pass default@1 after settle: 2026-10-05T02:49:00.993Z to 2026-10-05T02:49:20.993Z (20.0 s), 5,509 searches, 0 failed, p50 3.583 ms, mean 3.628 ms, p99 5.192 ms, 275.4 QPS (1000/QPS 3.630 ms).
Pass default@8 after default@1: 2026-10-05T02:49:21.030Z to 2026-10-05T02:49:41.040Z (20.0 s), 13,806 searches, 0 failed, p50 11.384 ms, mean 11.590 ms, p99 20.156 ms, 690.0 QPS (1000/QPS 1.449 ms).
Pass exact after default@8: 2026-10-05T02:49:41.122Z to 2026-10-05T02:50:41.124Z (60.0 s), 16,339 searches, 0 failed, p50 3.594 ms, mean 3.669 ms, p99 5.402 ms, 272.3 QPS (1000/QPS 3.672 ms).
Passes in the order run: default@1, default@8, exact; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 6,695 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (DiskANN index built and used. sys.vector_indexes: vix_gvb_gvbbench_eshoponweb on dbo.gvb_gvbbench_eshoponweb_ann in GvbBenchDiskAnn, DiskANN, COSINE, enabled, build {"StartId":"306", "L":"48", "M":"8", "R":"48"}; graph table rows 524 of 524 table rows. Real VECTOR_SEARCH plan: Vector Index Seek on index vix_gvb_gvbbench_eshoponweb of GvbBenchDiskAnn.gvb_gvbbench_eshoponweb_ann (IndexKind DiskANN), 10 rows returned, 10 hits. The exact mode searches the SAME table (dbo.gvb_gvbbench_eshoponweb_ann, database GvbBenchDiskAnn) and its real plan is: Clustered Index Scan of GvbBenchDiskAnn.gvb_gvbbench_eshoponweb_ann through PK_gvb_gvbbench_eshoponweb_ann (IndexKind Clustered), 524 rows read, no vector index operator, 10 hits).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-mssql (acd8727a701c) on network gvb-mssql_default at container-ip:1433, not through docker-proxy localhost:14330. Open after the passes: container-ip:1433 x9 (this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-mssql: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-mssql: 85 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@1 3502/3493 MHz, performance, outside load 0.13 before the warm-up, 0.12 during, client CPU 1.029 ms per search; default@8 3492/3492 MHz, performance, outside load 0.12 before the warm-up, 0.08 during, client CPU 1.090 ms per search; exact 3501/3494 MHz, performance, outside load 0.11 before the warm-up, 0.13 during, client CPU 1.150 ms per search.

MongoDB Atlas Local mongodb

Engine: MongoDB 8.0.32 Atlas Local (mongod + mongot Vector Search) (compose)
Index: vectorSearch index, HNSW maxEdges=16 numEdgeCandidates=128, float32 binData, cosine, numCandidates=20x hits (min 100); exact mode = $vectorSearch exact:true
Load: 524 rows in batches of 1000, 0.2 s of upserts (3,275 rows/s); count matched 2.0 s after the last upsert; index step 1.1 s (index status READY, queryable True; mongot holds 524 of 524 documents in 4 segment(s), 4 searched through the HNSW graph (Approximate); wait took 1.1 s)
Search: 14446 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 0.31 ms, 8 searchers 0.30 ms
Exact mode: 47,429 searches, p50 1.24 ms, p95 1.44 ms, recall 1.000
RAM: 911 MiB (docker stats); disk: 0 B added by this load (1.29 GiB (whole engine data folder), 1.39 GiB before)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started mongodb.compose.yaml with every container created on CPUs 2-3,6-7: gvb-mongodb cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
Index after the load: ready, 524 of 524 indexed (index status READY, queryable True; mongot holds 524 of 524 documents in 4 segment(s), 4 searched through the HNSW graph (Approximate)). Durability: Acknowledged writes are journaled before the acknowledgement. The sink sets no write concern, so the server default applies: getDefaultRWConcern gives w majority and the one-member replica set (--replSet gvbmongo in the image) has writeConcernMajorityJournalDefault true, with WiredTiger journaling on (journalCommitInterval 100 ms is only the interval for unacknowledged work). Measured 2026-10-04: 30 acknowledged single-document inserts raised WiredTiger 'log sync operations' by 30 and strace showed fdatasync on /data/db/journal/WiredTigerLog files. A crash loses no acknowledged write. mongot's search index is not part of that promise: it follows the collection asynchronously and is rebuilt from it..
Load average 2.16 2.16 2.21 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: default@1 1,901 searches in 5.0 s, default@8 5,750 searches in 5.0 s, exact 3,407 searches in 5.0 s.
Settle: settled after 0.4 s and 300 searches (0 failed); p50 of the last windows of 100: 1.41, 1.38, 1.39 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 1,432 searches in 2.0 s: p50 1.355 ms against the settled p50 1.387 ms, 2% apart (limit 15%).
Pass default@1 after settle: 2026-10-05T02:51:12.885Z to 2026-10-05T02:51:32.885Z (20.0 s), 14,446 searches, 0 failed, p50 1.346 ms, mean 1.382 ms, p99 1.949 ms, 722.3 QPS (1000/QPS 1.384 ms).
Pass default@8 after default@1: 2026-10-05T02:51:32.910Z to 2026-10-05T02:51:52.913Z (20.0 s), 41,916 searches, 0 failed, p50 3.638 ms, mean 3.815 ms, p99 7.511 ms, 2095.5 QPS (1000/QPS 0.477 ms).
Pass exact after default@8: 2026-10-05T02:51:52.952Z to 2026-10-05T02:52:52.952Z (60.0 s), 47,429 searches, 0 failed, p50 1.239 ms, mean 1.263 ms, p99 1.810 ms, 790.5 QPS (1000/QPS 1.265 ms).
Passes in the order run: default@1, default@8, exact; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 12,850 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (index status READY, queryable True; mongot holds 524 of 524 documents in 4 segment(s), 4 searched through the HNSW graph (Approximate)).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-mongodb (b276bbc1f572) on network gvb-mongodb_default at container-ip:27017, not through docker-proxy localhost:27017. Open after the passes: container-ip:27017 x10 (this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-mongodb: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-mongodb: 211 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@1 3492/3492 MHz, performance, outside load 0.16 before the warm-up, 0.20 during, client CPU 0.315 ms per search; default@8 3492/3492 MHz, performance, outside load 0.18 before the warm-up, 0.12 during, client CPU 0.296 ms per search; exact 3492/3492 MHz, performance, outside load 0.16 before the warm-up, 0.21 during, client CPU 0.308 ms per search.

DuckDB duckdb

Engine: DuckDB 1.5.6 + vss b833341 (HNSW, embedded, in process) (embedded)
Index: HNSW (vss extension) FLOAT[n] metric=cosine m=16 ef_construction=128, ef_search=100 per connection, persistent (hnsw_enable_experimental_persistence=true, checkpoint_threshold=256MB); exact mode = array_cosine_similarity sequential scan; score = 1 - cosine distance
Load: 524 rows in batches of 1000, 0.3 s of upserts (1,543 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (HNSW is maintained inside every transaction, nothing to build; confirmed in 0.01 s: duckdb_indexes() lists gvb_gvbbench_eshoponweb_hnsw, pragma_hnsw_index_info() counts 524 vectors of 524 rows, EXPLAIN of the default search shows HNSW_INDEX_SCAN on it = True)
Search: 5576 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 4.87 ms, 8 searchers 4.94 ms
Exact mode: 14,581 searches, p50 4.04 ms, p95 4.73 ms, recall 1.000
RAM: in this process, not measured; disk: 2.9 MiB added by this load (413.83 MiB (engine data folder), 410.93 MiB before)
Index after the load: ready, 524 of 524 indexed (duckdb_indexes() lists gvb_gvbbench_eshoponweb_hnsw, pragma_hnsw_index_info() counts 524 vectors of 524 rows, EXPLAIN of the default search shows HNSW_INDEX_SCAN on it = True). Durability: DuckDB's write-ahead log is fsynced at every commit and no DuckDbSink setting changes that (checkpoint_threshold=256MB only spaces out the checkpoints that write the database file; measured with strace: 42 WAL fsyncs for 40 single-record commits plus 2 setup statements), so an operating-system crash or power cut loses no committed row; measured with kill -9 right after the last commit and a reopen: all 50, 300 and 2,000 rows and an HNSW index that counted the same number were recovered from the log; not tested and documented by DuckDB: the HNSW index is file-backed only through hnsw_enable_experimental_persistence = true, which this sink turns on, and WAL recovery for such custom indexes is not complete, so a crash during a checkpoint or a later commit can damage the index while the rows survive.
Load average 1.10 2.08 2.20 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: default@8 1,432 searches in 5.0 s, exact 1,256 searches in 5.0 s, default@1 1,426 searches in 5.0 s.
Settle: settled after 1.0 s and 300 searches (0 failed); p50 of the last windows of 100: 3.46, 3.47, 3.50 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 570 searches in 2.0 s: p50 3.484 ms against the settled p50 3.474 ms, 0% apart (limit 15%).
Pass default@8 after settle: 2026-10-05T02:54:13.293Z to 2026-10-05T02:54:33.320Z (20.0 s), 5,610 searches, 0 failed, p50 28.327 ms, mean 28.538 ms, p99 31.810 ms, 280.1 QPS (1000/QPS 3.570 ms).
Pass exact after default@8: 2026-10-05T02:54:33.420Z to 2026-10-05T02:55:33.420Z (60.0 s), 14,581 searches, 0 failed, p50 4.036 ms, mean 4.111 ms, p99 5.733 ms, 243.0 QPS (1000/QPS 4.115 ms).
Pass default@1 after exact: 2026-10-05T02:55:33.515Z to 2026-10-05T02:55:53.515Z (20.0 s), 5,576 searches, 0 failed, p50 3.552 ms, mean 3.584 ms, p99 5.106 ms, 278.8 QPS (1000/QPS 3.587 ms).
Passes in the order run: default@8, exact, default@1; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 5,044 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (duckdb_indexes() lists gvb_gvbbench_eshoponweb_hnsw, pragma_hnsw_index_info() counts 524 vectors of 524 rows, EXPLAIN of the default search shows HNSW_INDEX_SCAN on it = True).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: in this process (embedded), no network: no network address. No open connection seen (no open TCP connection was found).
CPU pinning: none: an embedded engine runs inside the benchmark client process, so it shares the client CPUs, CPUs 0-1,4-5.
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@8 3593/3592 MHz, performance, outside load 0.10 before the warm-up, 0.09 during, client CPU 4.944 ms per search; exact 3592/3592 MHz, performance, outside load 0.10 before the warm-up, 0.13 during, client CPU 6.064 ms per search; default@1 3593/3592 MHz, performance, outside load 0.13 before the warm-up, 0.10 during, client CPU 4.868 ms per search.

Qdrant (exact) qdrant

Engine: Qdrant 1.17.0 in container gvb-qdrant (image qdrant/qdrant:v1.17.0@sha256:f1c7272cdac52b38c1a0e89313922d940ba50afd90d593a1605dbbc214e66ffb; cpuset 2-3,6-7 from its creation; 3 search thread(s)) (compose)
Index: exact scan: the builder's sink sends exact=true on every search, so no HNSW graph is used whether or not Qdrant has built one (see the index state)
Load: 524 rows in batches of 1000, 0.1 s of upserts (4,666 rows/s); count matched 0.0 s after the last upsert; index step 1.0 s (status Green, 0 of 524 vectors in HNSW segments (2 segments), waited 1.0 s)
Search: 22516 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 0.74 ms, 8 searchers 0.54 ms
RAM: 39.43 MiB (docker stats); disk: 196.08 MiB (collection folder)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started qdrant.compose.yaml with every container created on CPUs 2-3,6-7: gvb-qdrant cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
Index after the load: ready, 0 of 524 indexed (status Green, optimizer ok, indexed_vectors_count 0 of 524 points, 2 segments, indexing_threshold_kb 10,000, full_scan_threshold_kb 10,000; no index used, exact scan by design (the builder's sink sends exact=true on every search); no graph was built; searches counted since the collection was created: unfiltered_hnsw 0, unfiltered_plain 0, unfiltered_exact 0). Durability: What was measured (strace -f on the Qdrant server process, 30 single-point REST upserts per setting, every sync call timed against the request that caused it): with wait=true each upsert had exactly one msync(MS_SYNC) of the write-ahead-log segment inside the request, before the reply (30 of 30); with wait=false the 30 upserts were acknowledged within 0.26 s with no sync call, and the first WAL msync (one call covering all 30 records) came 2.7 s after the last reply, followed by the segment-file flushes. So the log is flushed to disk under both settings, but with wait=false the flush comes after the acknowledgement: an operating-system crash or power cut in that gap loses acknowledged writes, while a crash of the Qdrant process alone should not, because the bytes are already in the kernel's page cache (inferred, not tested). Every upsert here is sent with wait=true in batches of 256 over gRPC; the strace used one point per request, so one flush per batch is inferred, not measured. Segment files are flushed every 5 s; log segments 32 MB, 0 created ahead. /qdrant/config/config.yaml in container gvb-qdrant (the image's own file; its storage keys are listed) sets: storage.collection.quantization = null, storage.collection.replication_factor = 1, storage.collection.vectors.on_disk = null, storage.collection.write_consistency_factor = 1, storage.hnsw_index.ef_construct = 100, storage.hnsw_index.full_scan_threshold_kb = 10000, storage.hnsw_index.m = 16, storage.hnsw_index.max_indexing_threads = 0, storage.hnsw_index.on_disk = false, storage.hnsw_index.payload_m = null, storage.max_collections = null, storage.node_type = Normal, storage.on_disk_payload = true, storage.optimizers.default_segment_number = 0, storage.optimizers.deleted_threshold = 0.2, storage.optimizers.flush_interval_sec = 5, storage.optimizers.indexing_threshold_kb = 10000, storage.optimizers.max_optimization_threads = null, storage.optimizers.max_segment_size_kb = null, storage.optimizers.vacuum_min_vector_number = 1000, storage.performance.max_search_threads = 0, storage.performance.optimizer_cpu_budget = 0, storage.performance.update_rate_limit = null, storage.shard_transfer_method = null, storage.snapshots_config.snapshots_storage = local, storage.snapshots_path = ./snapshots, storage.storage_path = ./storage, storage.temp_path = null, storage.update_concurrency = null, storage.wal.wal_capacity_mb = 32, storage.wal.wal_segments_ahead = 0; no QDRANT__ environment overrides in the server process. Not tested by cutting power; whether the disk's own write cache reaches the media was not checked..
Load average 1.13 1.79 2.07 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: default@1 5,462 searches in 5.0 s, default@8 16,594 searches in 5.0 s.
Settle: settled after 0.3 s and 300 searches (0 failed); p50 of the last windows of 100: 0.91, 0.90, 0.89 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 2,242 searches in 2.0 s: p50 0.875 ms against the settled p50 0.900 ms, 3% apart (limit 15%).
Pass default@1 after settle: 2026-10-05T02:56:13.683Z to 2026-10-05T02:56:33.683Z (20.0 s), 22,516 searches, 0 failed, p50 0.875 ms, mean 0.887 ms, p99 1.458 ms, 1125.8 QPS (1000/QPS 0.888 ms).
Pass default@8 after default@1: 2026-10-05T02:56:33.715Z to 2026-10-05T02:56:53.717Z (20.0 s), 66,068 searches, 0 failed, p50 2.335 ms, mean 2.420 ms, p99 4.204 ms, 3303.1 QPS (1000/QPS 0.303 ms).
Passes in the order run: default@1, default@8; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 24,638 sent, 0 failed (warmupErrors). Index after the searches: ready, 0 of 524 indexed (status Green, optimizer ok, indexed_vectors_count 0 of 524 points, 2 segments, indexing_threshold_kb 10,000, full_scan_threshold_kb 10,000; no index used, exact scan by design (the builder's sink sends exact=true on every search); no graph was built; searches counted since the collection was created: unfiltered_hnsw 0, unfiltered_plain 226,444, unfiltered_exact 0).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-qdrant (eddff31bf1a6) on network gvb-qdrant_default at container-ip:6334, not through docker-proxy localhost:16334; container gvb-qdrant (eddff31bf1a6) on network gvb-qdrant_default at container-ip:6333, not through docker-proxy localhost:16333. Open after the passes: container-ip:6334 x1 (this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-qdrant: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-qdrant: 24 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@1 3492/3492 MHz, performance, outside load 0.10 before the warm-up, 0.09 during, client CPU 0.740 ms per search; default@8 3492/3492 MHz, performance, outside load 0.10 before the warm-up, 0.12 during, client CPU 0.539 ms per search.

SQL Server 2025 sql

Engine: Microsoft SQL Server 2025 (RTM-CU9) (KB5122048) - 17.0.5005.3 (X64), Enterprise Developer Edition (64-bit), in container gvb-mssql (image mcr.microsoft.com/mssql/server:2025-CU9-ubuntu-24.04@sha256:2b5b581621126574f3d1f75e78d3eebe8d05aedb59ad0cfdf9aa42cb0634d726; cpuset 2-3,6-7 from its creation; SQL Server counts 4 CPU(s) and runs 4 visible scheduler(s), affinity AUTO) (compose)
Index: exact VECTOR_DISTANCE cosine, no vector index (full scan)
Load: 524 rows in batches of 1000, 0.8 s of upserts (641 rows/s); count matched 0.0 s after the last upsert; index step n/a
Search: 5061 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 1.22 ms, 8 searchers 1.19 ms
RAM: 529.9 MiB (docker stats); disk: 5.15 MiB (table and its indexes, reserved pages)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started mssql.compose.yaml with every container created on CPUs 2-3,6-7: gvb-mssql cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
Index after the load: ready, 0 of 524 indexed (no index used, exact scan by design: dbo.gvb_gvbbench_eshoponweb in GvbBench holds 524 rows; sys.vector_indexes lists no index on the table; indexes: PK_gvb_gvbbench_eshoponweb CLUSTERED, IX_gvb_gvbbench_eshoponweb_DocKey NONCLUSTERED; a real search under SET STATISTICS XML ON ran: Clustered Index Scan of GvbBench.gvb_gvbbench_eshoponweb through PK_gvb_gvbbench_eshoponweb (IndexKind Clustered), 524 rows read, no vector index operator, 10 hits). Durability: A commit returns after its transaction-log records are written to disk (SQL Server write-ahead logging; delayed durability is DISABLED); a database created here copies the model database: recovery model FULL, page_verify CHECKSUM; no global trace flags are enabled; mssql.conf of container gvb-mssql (/var/opt/mssql/mssql.conf, on the host at ~/gvb-data/engines/mssql/mssql.conf) does not exist, so it sets: nothing; it has no [control] or [traceflag] entry, so SQL Server's own Linux defaults for flushing writes apply (Microsoft's Linux performance guide names trace flag 3982 as that default; read from the guide, not tested here). Not tested by cutting power; whether the disk's own write cache reaches the media was not checked. Container settings from its environment (names only): MSSQL_AGENT_ENABLED, MSSQL_MEMORY_LIMIT_MB, MSSQL_PID, MSSQL_RPC_PORT, MSSQL_SA_PASSWORD..
Load average 3.07 2.25 2.21 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: default@8 2,789 searches in 5.0 s, default@1 1,244 searches in 5.0 s.
Settle: settled after 1.2 s and 300 searches (0 failed); p50 of the last windows of 100: 3.84, 3.82, 3.84 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 511 searches in 2.0 s: p50 3.847 ms against the settled p50 3.839 ms, 0% apart (limit 15%).
Pass default@8 after settle: 2026-10-05T02:57:18.073Z to 2026-10-05T02:57:38.081Z (20.0 s), 12,518 searches, 0 failed, p50 12.366 ms, mean 12.782 ms, p99 19.533 ms, 625.6 QPS (1000/QPS 1.598 ms).
Pass default@1 after default@8: 2026-10-05T02:57:38.162Z to 2026-10-05T02:57:58.165Z (20.0 s), 5,061 searches, 0 failed, p50 3.850 ms, mean 3.949 ms, p99 5.091 ms, 253.0 QPS (1000/QPS 3.952 ms).
Passes in the order run: default@8, default@1; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 4,884 sent, 0 failed (warmupErrors). Index after the searches: ready, 0 of 524 indexed (no index used, exact scan by design: dbo.gvb_gvbbench_eshoponweb in GvbBench holds 524 rows; sys.vector_indexes lists no index on the table; indexes: PK_gvb_gvbbench_eshoponweb CLUSTERED, IX_gvb_gvbbench_eshoponweb_DocKey NONCLUSTERED; a real search under SET STATISTICS XML ON ran: Clustered Index Scan of GvbBench.gvb_gvbbench_eshoponweb through PK_gvb_gvbbench_eshoponweb (IndexKind Clustered), 524 rows read, no vector index operator, 10 hits).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-mssql (74504bf10c36) on network gvb-mssql_default at container-ip:1433, not through docker-proxy localhost:14330. Open after the passes: container-ip:1433 x9 (this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-mssql: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-mssql: 85 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@8 3492/3492 MHz, performance, outside load 0.11 before the warm-up, 0.14 during, client CPU 1.195 ms per search; default@1 3510/3496 MHz, performance, outside load 0.13 before the warm-up, 0.13 during, client CPU 1.219 ms per search.

OpenSearch opensearch

Engine: OpenSearch 3.9.0 (k-NN plugin, faiss) (compose)
Index: faiss HNSW float32, no compression, m=16, ef_construction=128, cosinesimil; search k=top, ef_search=100; 1 shard, 0 replicas; graph built at any segment size (approximate_threshold=0); force-merged to one segment after the load
Load: 524 rows in batches of 1000, 1.4 s of upserts (388 rows/s); count matched 0.1 s after the last upsert; index step 0.9 s (force-merged to one segment, graph ready after 0.9 s: every segment searched through its HNSW graph, covering 524 of 524 vectors: 1 segment(s), 0 merge(s) running, profiled probe search: ann_search_count 1 (segments tried through a graph), exact_search_count 0 (of those, scanned instead), approximate_threshold 0; force-merged to one segment on purpose, so one graph answers every search)
Search: 9923 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 0.47 ms, 8 searchers 0.52 ms
Exact mode: 22,549 searches, p50 2.58 ms, p95 3.03 ms, recall 1.000
RAM: 2.58 GiB (docker stats); disk: 9.64 MiB added by this load (10.12 MiB (whole engine data folder), 489.11 KiB before)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started opensearch.compose.yaml with every container created on CPUs 2-3,6-7: gvb-opensearch cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
Index after the load: ready, 524 of 524 indexed (every segment searched through its HNSW graph, covering 524 of 524 vectors: 1 segment(s), 0 merge(s) running, profiled probe search: ann_search_count 1 (segments tried through a graph), exact_search_count 0 (of those, scanned instead), approximate_threshold 0; force-merged to one segment on purpose, so one graph answers every search). Durability: Every acknowledged bulk request is fsynced to the translog before the answer: index.translog.durability=request, the OpenSearch default, which neither this sink nor opensearch.compose.yaml overrides (OpenSearchReadinessTests reads it back from the live index). By that setting a process crash or power loss loses no acknowledged write; this is read from the setting, not shown by pulling power. One node and no replicas, so a lost disk loses the data..
Load average 3.77 2.73 2.39 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: exact 937 searches in 5.0 s, default@8 3,648 searches in 5.1 s, default@1 1,606 searches in 5.0 s.
Settle: settled after 2.9 s and 1,100 searches (0 failed); p50 of the last windows of 100: 2.05, 2.03, 2.06 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 823 searches in 2.0 s: p50 2.213 ms against the settled p50 2.051 ms, 7% apart (limit 15%).
Pass exact after settle: 2026-10-05T02:58:49.993Z to 2026-10-05T02:59:49.994Z (60.0 s), 22,549 searches, 0 failed, p50 2.575 ms, mean 2.659 ms, p99 3.772 ms, 375.8 QPS (1000/QPS 2.661 ms).
Pass default@8 after exact: 2026-10-05T02:59:50.033Z to 2026-10-05T03:00:10.035Z (20.0 s), 28,444 searches, 0 failed, p50 5.310 ms, mean 5.623 ms, p99 12.036 ms, 1422.1 QPS (1000/QPS 0.703 ms).
Pass default@1 after default@8: 2026-10-05T03:00:10.086Z to 2026-10-05T03:00:30.087Z (20.0 s), 9,923 searches, 0 failed, p50 1.980 ms, mean 2.014 ms, p99 2.730 ms, 496.1 QPS (1000/QPS 2.016 ms).
Passes in the order run: exact, default@8, default@1; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 8,174 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (every segment searched through its HNSW graph, covering 524 of 524 vectors: 1 segment(s), 0 merge(s) running, profiled probe search: ann_search_count 1 (segments tried through a graph), exact_search_count 0 (of those, scanned instead), approximate_threshold 0; force-merged to one segment on purpose, so one graph answers every search).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-opensearch (98e11fa74cb8) on network engines_default at container-ip:9200, not through docker-proxy localhost:9201. Open after the passes: container-ip:9200 x8 (this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-opensearch: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-opensearch: 55 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): exact 3492/3492 MHz, performance, outside load 0.11 before the warm-up, 0.12 during, client CPU 0.484 ms per search; default@8 3492/3492 MHz, performance, outside load 0.12 before the warm-up, 0.09 during, client CPU 0.518 ms per search; default@1 3492/3492 MHz, performance, outside load 0.11 before the warm-up, 0.13 during, client CPU 0.474 ms per search.

pgvector

Engine: PostgreSQL 17 + pgvector 0.8.7 (compose)
Index: HNSW vector_cosine_ops m=16 ef_construction=128, hnsw.ef_search=100 per query, float32 vector(n), cosine; exact mode = same query with index scans off (sequential scan)
Load: 524 rows in batches of 1000, 0.9 s of upserts (616 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (HNSW is maintained inside every insert, nothing to build; confirmed in 0.0 s: pg_indexes lists gvb_gvbbench_eshoponweb_hnsw (USING hnsw (embedding vector_cosine_ops) WITH (m='16', ef_construction='128')), pg_index valid and ready = True, EXPLAIN of the default search uses Index Scan on it = True, that search returned 10 of 10 rows; PostgreSQL keeps no entry count for an HNSW index, so indexed vectors are not reported)
Search: 23150 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 0.69 ms, 8 searchers 0.54 ms
Exact mode: 27,345 searches, p50 2.15 ms, p95 2.38 ms, recall 1.000
RAM: 100.2 MiB (docker stats); disk: 7.59 MiB added by this load (454.57 MiB (whole engine data folder), 446.98 MiB before)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started pgvector.compose.yaml with every container created on CPUs 2-3,6-7: gvb-pgvector cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
Index after the load: ready, ? of 524 indexed (pg_indexes lists gvb_gvbbench_eshoponweb_hnsw (USING hnsw (embedding vector_cosine_ops) WITH (m='16', ef_construction='128')), pg_index valid and ready = True, EXPLAIN of the default search uses Index Scan on it = True, that search returned 10 of 10 rows; PostgreSQL keeps no entry count for an HNSW index, so indexed vectors are not reported). Durability: fsync on, synchronous_commit on, full_page_writes on, wal_sync_method fdatasync (PostgreSQL defaults; pgvector.compose.yaml sets only shared_buffers, maintenance_work_mem and max_wal_size): every commit is flushed to the write-ahead log before it returns and HNSW index changes are WAL-logged, so a crash loses no committed row (max_wal_size 4GB only spaces out checkpoints; measured with docker kill, which is SIGKILL, right after a 2,000-vector load and a restart: all 2,000 rows were there and the HNSW index was valid and used by the default search).
Load average 3.25 2.95 2.52 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: default@8 20,171 searches in 5.0 s, exact 2,286 searches in 5.0 s, default@1 5,821 searches in 5.0 s.
Settle: settled after 0.2 s and 300 searches (0 failed); p50 of the last windows of 100: 0.80, 0.82, 0.82 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 2,351 searches in 2.0 s: p50 0.836 ms against the settled p50 0.819 ms, 2% apart (limit 15%).
Pass default@8 after settle: 2026-10-05T03:00:56.732Z to 2026-10-05T03:01:16.733Z (20.0 s), 82,325 searches, 0 failed, p50 1.899 ms, mean 1.942 ms, p99 3.400 ms, 4116.0 QPS (1000/QPS 0.243 ms).
Pass exact after default@8: 2026-10-05T03:01:16.807Z to 2026-10-05T03:02:16.809Z (60.0 s), 27,345 searches, 0 failed, p50 2.155 ms, mean 2.192 ms, p99 2.938 ms, 455.7 QPS (1000/QPS 2.194 ms).
Pass default@1 after exact: 2026-10-05T03:02:16.852Z to 2026-10-05T03:02:36.853Z (20.0 s), 23,150 searches, 0 failed, p50 0.847 ms, mean 0.862 ms, p99 1.285 ms, 1157.5 QPS (1000/QPS 0.864 ms).
Passes in the order run: default@8, exact, default@1; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 30,989 sent, 0 failed (warmupErrors). Index after the searches: ready, ? of 524 indexed (pg_indexes lists gvb_gvbbench_eshoponweb_hnsw (USING hnsw (embedding vector_cosine_ops) WITH (m='16', ef_construction='128')), pg_index valid and ready = True, EXPLAIN of the default search uses Index Scan on it = True, that search returned 10 of 10 rows; PostgreSQL keeps no entry count for an HNSW index, so indexed vectors are not reported).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-pgvector (43a7aabee907) on network gvb-pgvector_default at container-ip:5432, not through docker-proxy localhost:5432. Open after the passes: container-ip:5432 x8 (this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-pgvector: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-pgvector: 6 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@8 3492/3492 MHz, performance, outside load 0.12 before the warm-up, 0.09 during, client CPU 0.544 ms per search; exact 3492/3492 MHz, performance, outside load 0.11 before the warm-up, 0.10 during, client CPU 0.692 ms per search; default@1 3492/3492 MHz, performance, outside load 0.10 before the warm-up, 0.11 during, client CPU 0.688 ms per search.

Chroma chroma

Engine: Chroma 1.4.4 (single node, REST API v2) (compose)
Index: HNSW M=16 ef_construction=128, ef_search=100 (Chroma default), cosine; approximate only (no exact mode)
Load: 524 rows in batches of 1000, 0.6 s of upserts (882 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (Chroma has no index build to wait for and exposes no index status; checked in 0.0 s: Chroma exposes no index status (indexing_status endpoint: HTTP 500 {"error":"InternalError","message":"Method scout_logs is not implemented"}), so indexed vectors are not reported and ready only means the two counts agree to within 1 percent: count endpoint 524, a vector search asking for 524 results returned 524)
Search: 9691 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 0.47 ms, 8 searchers 0.47 ms
RAM: 90.18 MiB (docker stats); disk: 414.16 KiB added by this load (2 GiB (whole engine data folder), 2 GiB before)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started chroma.compose.yaml with every container created on CPUs 2-3,6-7: gvb-chroma cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
Index after the load: ready, ? of 524 indexed (Chroma exposes no index status (indexing_status endpoint: HTTP 500 {"error":"InternalError","message":"Method scout_logs is not implemented"}), so indexed vectors are not reported and ready only means the two counts agree to within 1 percent: count endpoint 524, a vector search asking for 524 results returned 524). Durability: SQLite rollback journal with its default synchronous=FULL under the data directory (chroma.compose.yaml sets IS_PERSISTENT=1 and PERSIST_DIRECTORY=/data and no sync setting): every write is committed to chroma.sqlite3 with fsync of the journal, the directory and the database file before the call returns (strace: 15 database fsyncs and 38 journal fsyncs for one create, four 500-vector upserts and one delete), and the HNSW files are written every sync_threshold=1000 vectors and rebuilt from the SQLite log after a crash; measured with kill -9 and a restart: all 50, 300 and 2,000 vectors were still stored and searchable.
Load average 2.70 3.09 2.64 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: default@8 4,010 searches in 5.0 s, default@1 2,413 searches in 5.0 s.
Settle: settled after 0.6 s and 300 searches (0 failed); p50 of the last windows of 100: 2.05, 2.05, 2.04 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 973 searches in 2.0 s: p50 2.033 ms against the settled p50 2.047 ms, 1% apart (limit 15%).
Pass default@8 after settle: 2026-10-05T03:02:58.086Z to 2026-10-05T03:03:18.093Z (20.0 s), 16,123 searches, 0 failed, p50 9.859 ms, mean 9.924 ms, p99 11.662 ms, 805.9 QPS (1000/QPS 1.241 ms).
Pass default@1 after default@8: 2026-10-05T03:03:18.140Z to 2026-10-05T03:03:38.141Z (20.0 s), 9,691 searches, 0 failed, p50 2.042 ms, mean 2.062 ms, p99 2.553 ms, 484.5 QPS (1000/QPS 2.064 ms).
Passes in the order run: default@8, default@1; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 7,736 sent, 0 failed (warmupErrors). Index after the searches: ready, ? of 524 indexed (Chroma exposes no index status (indexing_status endpoint: HTTP 500 {"error":"InternalError","message":"Method scout_logs is not implemented"}), so indexed vectors are not reported and ready only means the two counts agree to within 1 percent: count endpoint 524, a vector search asking for 524 results returned 524).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-chroma (d3e98e059806) on network gvb-chroma_default at container-ip:8000, not through docker-proxy localhost:8000. Open after the passes: container-ip:8000 x8 (this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-chroma: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-chroma: 9 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@8 3592/3592 MHz, performance, outside load 0.07 before the warm-up, 0.05 during, client CPU 0.471 ms per search; default@1 3592/3592 MHz, performance, outside load 0.06 before the warm-up, 0.06 during, client CPU 0.474 ms per search.

Vespa vespa

Engine: Vespa 8.754.14 (tensor attribute + HNSW) (compose)
Index: HNSW float32 tensor, prenormalized-angular (cosine), max-links-per-node=16, neighbors-to-explore-at-insert=128; search targetHits=top, ef=100 via exploreAdditionalHits; exact mode = approximate:false; vectors held in memory
Load: 524 rows in batches of 1000, 1.6 s of upserts (321 rows/s); count matched 0.0 s after the last upsert; index step 0.0 s (nothing to build, Vespa inserts into the HNSW graph while writing; ready after 0.0 s: HNSW graph returns 524 of 524 vectors: totalCount 524, coverage full True, nearestNeighbor approximate:true with targetHits above the count: has_index True, algorithm "index top k", top_k_hits 524; the graph is updated while each write is applied, so there is no later build step)
Search: 10404 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 0.96 ms, 8 searchers 0.90 ms
Exact mode: 28,119 searches, p50 1.93 ms, p95 3.23 ms, recall 1.000
RAM: 2.93 GiB (docker stats); disk: 31.23 MiB added by this load (1.7 GiB (whole engine data folder), 1.67 GiB before)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started vespa.compose.yaml with every container created on CPUs 2-3,6-7: gvb-vespa cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
Index after the load: ready, 524 of 524 indexed (HNSW graph returns 524 of 524 vectors: totalCount 524, coverage full True, nearestNeighbor approximate:true with targetHits above the count: has_index True, algorithm "index top k", top_k_hits 524; the graph is updated while each write is applied, so there is no later build step). Durability: Vespa's transaction log server fsyncs after each commit: searchlib.translogserver usefsync=true, and an operation is searchable when it is acknowledged: proton documentdb visibilitydelay=0. Neither is overridden by this sink or by the services.xml it deploys; VespaReadinessTests reads both from the live config server. After a crash the node replays the transaction log over its last flushed data and rebuilds the in-memory HNSW graph. By those settings an acknowledged write survives a crash; this is read from the settings, not shown by pulling power. One node and min-redundancy 1, so a lost disk loses the data..
Load average 4.52 3.35 2.77 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: exact 832 searches in 5.0 s, default@8 2,773 searches in 5.0 s, default@1 1,371 searches in 5.0 s.
Settle: settled after 1.9 s and 600 searches (0 failed); p50 of the last windows of 100: 3.21, 3.20, 3.20 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 626 searches in 2.0 s: p50 2.935 ms against the settled p50 3.197 ms, 9% apart (limit 15%).
Pass exact after settle: 2026-10-05T03:04:53.482Z to 2026-10-05T03:05:53.483Z (60.0 s), 28,119 searches, 0 failed, p50 1.926 ms, mean 2.131 ms, p99 3.826 ms, 468.6 QPS (1000/QPS 2.134 ms).
Pass default@8 after exact: 2026-10-05T03:05:53.524Z to 2026-10-05T03:06:13.526Z (20.0 s), 31,003 searches, 0 failed, p50 4.988 ms, mean 5.158 ms, p99 9.020 ms, 1550.0 QPS (1000/QPS 0.645 ms).
Pass default@1 after default@8: 2026-10-05T03:06:13.574Z to 2026-10-05T03:06:33.574Z (20.0 s), 10,404 searches, 0 failed, p50 1.870 ms, mean 1.920 ms, p99 2.605 ms, 520.2 QPS (1000/QPS 1.922 ms).
Passes in the order run: exact, default@8, default@1; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 6,262 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (HNSW graph returns 524 of 524 vectors: totalCount 524, coverage full True, nearestNeighbor approximate:true with targetHits above the count: has_index True, algorithm "index top k", top_k_hits 524; the graph is updated while each write is applied, so there is no later build step).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-vespa (c8fd62476fb5) on network engines_default at container-ip:8080, not through docker-proxy localhost:8090; container gvb-vespa (c8fd62476fb5) on network engines_default at container-ip:19071, not through docker-proxy localhost:19071. Open after the passes: container-ip:8080 x1 (this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-vespa: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-vespa: 215 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): exact 3524/3504 MHz, performance, outside load 0.13 before the warm-up, 0.10 during, client CPU 0.982 ms per search; default@8 3492/3492 MHz, performance, outside load 0.10 before the warm-up, 0.16 during, client CPU 0.897 ms per search; default@1 3515/3500 MHz, performance, outside load 0.12 before the warm-up, 0.08 during, client CPU 0.960 ms per search.

Qdrant (HNSW) qdrant-hnsw

Engine: Qdrant 1.17.0 in container gvb-qdrant (image qdrant/qdrant:v1.17.0@sha256:f1c7272cdac52b38c1a0e89313922d940ba50afd90d593a1605dbbc214e66ffb; cpuset 2-3,6-7 from its creation; 3 search thread(s)) (compose)
Index: HNSW m=16 ef_construct=100, hnsw_ef=server default, cosine; indexing_threshold_kb 1 and full_scan_threshold_kb 10 (server defaults are 10,000 each) so a small collection builds and walks its graph
Load: 524 rows in batches of 1000, 0.1 s of upserts (9,152 rows/s); count matched 0.0 s after the last upsert; index step 2.0 s (indexing_threshold_kb 1, full_scan_threshold_kb 10; status Green, 524 of 524 vectors in HNSW segments (2 segments), waited 2.0 s)
Search: 21603 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 0.73 ms, 8 searchers 0.53 ms
Exact mode: 66,418 searches, p50 0.89 ms, p95 1.00 ms, recall 1.000
RAM: 36.52 MiB (docker stats); disk: 164.1 MiB (collection folder)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started qdrant.compose.yaml with every container created on CPUs 2-3,6-7: gvb-qdrant cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
Index after the load: ready, 524 of 524 indexed (status Green, optimizer ok, indexed_vectors_count 524 of 524 points, 2 segments, indexing_threshold_kb 1, full_scan_threshold_kb 10; every one of 1 non-empty segments has a complete HNSW graph above its full-scan threshold; no search has run yet, so the walk is proven by the segment settings only; searches counted since the collection was created: unfiltered_hnsw 0, unfiltered_plain 0, unfiltered_exact 0). Durability: What was measured (strace -f on the Qdrant server process, 30 single-point REST upserts per setting, every sync call timed against the request that caused it): with wait=true each upsert had exactly one msync(MS_SYNC) of the write-ahead-log segment inside the request, before the reply (30 of 30); with wait=false the 30 upserts were acknowledged within 0.26 s with no sync call, and the first WAL msync (one call covering all 30 records) came 2.7 s after the last reply, followed by the segment-file flushes. So the log is flushed to disk under both settings, but with wait=false the flush comes after the acknowledgement: an operating-system crash or power cut in that gap loses acknowledged writes, while a crash of the Qdrant process alone should not, because the bytes are already in the kernel's page cache (inferred, not tested). Every upsert here is sent with wait=true in batches of 256 over gRPC; the strace used one point per request, so one flush per batch is inferred, not measured. Segment files are flushed every 5 s; log segments 32 MB, 0 created ahead. /qdrant/config/config.yaml in container gvb-qdrant (the image's own file; its storage keys are listed) sets: storage.collection.quantization = null, storage.collection.replication_factor = 1, storage.collection.vectors.on_disk = null, storage.collection.write_consistency_factor = 1, storage.hnsw_index.ef_construct = 100, storage.hnsw_index.full_scan_threshold_kb = 10000, storage.hnsw_index.m = 16, storage.hnsw_index.max_indexing_threads = 0, storage.hnsw_index.on_disk = false, storage.hnsw_index.payload_m = null, storage.max_collections = null, storage.node_type = Normal, storage.on_disk_payload = true, storage.optimizers.default_segment_number = 0, storage.optimizers.deleted_threshold = 0.2, storage.optimizers.flush_interval_sec = 5, storage.optimizers.indexing_threshold_kb = 10000, storage.optimizers.max_optimization_threads = null, storage.optimizers.max_segment_size_kb = null, storage.optimizers.vacuum_min_vector_number = 1000, storage.performance.max_search_threads = 0, storage.performance.optimizer_cpu_budget = 0, storage.performance.update_rate_limit = null, storage.shard_transfer_method = null, storage.snapshots_config.snapshots_storage = local, storage.snapshots_path = ./snapshots, storage.storage_path = ./storage, storage.temp_path = null, storage.update_concurrency = null, storage.wal.wal_capacity_mb = 32, storage.wal.wal_segments_ahead = 0; no QDRANT__ environment overrides in the server process. Not tested by cutting power; whether the disk's own write cache reaches the media was not checked..
Load average 4.83 3.82 3.02 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: default@8 17,280 searches in 5.0 s, default@1 5,407 searches in 5.0 s, exact 5,524 searches in 5.0 s.
Settle: settled after 0.3 s and 300 searches (0 failed); p50 of the last windows of 100: 0.89, 0.91, 0.92 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 2,150 searches in 2.0 s: p50 0.920 ms against the settled p50 0.912 ms, 1% apart (limit 15%).
Pass default@8 after settle: 2026-10-05T03:07:08.692Z to 2026-10-05T03:07:28.694Z (20.0 s), 69,349 searches, 0 failed, p50 2.220 ms, mean 2.305 ms, p99 4.001 ms, 3467.1 QPS (1000/QPS 0.288 ms).
Pass default@1 after default@8: 2026-10-05T03:07:28.738Z to 2026-10-05T03:07:48.739Z (20.0 s), 21,603 searches, 0 failed, p50 0.915 ms, mean 0.924 ms, p99 1.487 ms, 1080.1 QPS (1000/QPS 0.926 ms).
Pass exact after default@1: 2026-10-05T03:07:48.783Z to 2026-10-05T03:08:48.784Z (60.0 s), 66,418 searches, 0 failed, p50 0.892 ms, mean 0.902 ms, p99 1.420 ms, 1107.0 QPS (1000/QPS 0.903 ms).
Passes in the order run: default@8, default@1, exact; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 30,721 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (status Green, optimizer ok, indexed_vectors_count 524 of 524 points, 2 segments, indexing_threshold_kb 1, full_scan_threshold_kb 10; every one of 1 non-empty segments has a complete HNSW graph above its full-scan threshold; 116,129 segment searches walked the graph and none scanned; searches counted since the collection was created: unfiltered_hnsw 116,129, unfiltered_plain 0, unfiltered_exact 71,962).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-qdrant (27df0280636d) on network gvb-qdrant_default at container-ip:6334, not through docker-proxy localhost:16334; container gvb-qdrant (27df0280636d) on network gvb-qdrant_default at container-ip:6333, not through docker-proxy localhost:16333. Open after the passes: container-ip:6334 x1 (this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-qdrant: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-qdrant: 24 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@8 3492/3492 MHz, performance, outside load 0.08 before the warm-up, 0.11 during, client CPU 0.528 ms per search; default@1 3492/3492 MHz, performance, outside load 0.09 before the warm-up, 0.06 during, client CPU 0.731 ms per search; exact 3492/3492 MHz, performance, outside load 0.08 before the warm-up, 0.08 during, client CPU 0.735 ms per search.

Elasticsearch elasticsearch

Engine: Elasticsearch 9.5.3 (dense_vector) (compose)
Index: HNSW float32, no quantization, m=16, ef_construction=128, cosine; search k=top, num_candidates=100; 1 shard, 0 replicas; force-merged to one segment after the load (at 1,024 dimensions a segment under 1,043 vectors gets no graph)
Load: 524 rows in batches of 1000, 1.1 s of upserts (499 rows/s); count matched 0.1 s after the last upsert; index step 0.4 s (FAILED after 0.4 s: NO HNSW GRAPH, searches scan all 524 vectors: 1 segment(s), 524 vectors, total_vex_size_bytes 0 (_stats dense_vector), index_options hnsw m=16 ef_construction=128; Lucene builds no graph for a segment this small (measured at 1,024 dimensions: 1,042 vectors none, 1,043 vectors a graph), so default search equals exact search)
Search: 14936 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 0.47 ms, 8 searchers 0.51 ms
Exact mode: 45,915 searches, p50 1.27 ms, p95 1.52 ms, recall 1.000
RAM: 2.53 GiB (docker stats); disk: 2.38 MiB added by this load (2.59 MiB (whole engine data folder), 209.51 KiB before)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started elasticsearch.compose.yaml with every container created on CPUs 2-3,6-7: gvb-elasticsearch cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
WARNING: the index step after the load FAILED (NO HNSW GRAPH, searches scan all 524 vectors: 1 segment(s), 524 vectors, total_vex_size_bytes 0 (_stats dense_vector), index_options hnsw m=16 ef_construction=128; Lucene builds no graph for a segment this small (measured at 1,024 dimensions: 1,042 vectors none, 1,043 vectors a graph), so default search equals exact search). Searched anyway; see the index state.
WARNING: index not ready after the load: NOT ready, 0 of 524 indexed (NO HNSW GRAPH, searches scan all 524 vectors: 1 segment(s), 524 vectors, total_vex_size_bytes 0 (_stats dense_vector), index_options hnsw m=16 ef_construction=128; Lucene builds no graph for a segment this small (measured at 1,024 dimensions: 1,042 vectors none, 1,043 vectors a graph), so default search equals exact search). Measured anyway, so its search numbers may come from a scan or a half-built index. Durability: Every acknowledged bulk request is fsynced to the translog before the answer: index.translog.durability=request, the Elasticsearch default, which neither this sink nor elasticsearch.compose.yaml overrides (ElasticsearchReadinessTests reads it back from the live index). By that setting a process crash or power loss loses no acknowledged write; this is read from the setting, not shown by pulling power. One node and no replicas, so a lost disk loses the data..
Load average 2.66 3.54 3.06 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: exact 1,231 searches in 5.0 s, default@8 3,798 searches in 5.0 s, default@1 1,765 searches in 5.0 s.
Settle: settled after 3.1 s and 1,700 searches (0 failed); p50 of the last windows of 100: 1.81, 1.76, 1.74 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 1,178 searches in 2.0 s: p50 1.603 ms against the settled p50 1.760 ms, 10% apart (limit 15%).
Pass exact after settle: 2026-10-05T03:09:43.757Z to 2026-10-05T03:10:43.757Z (60.0 s), 45,915 searches, 0 failed, p50 1.270 ms, mean 1.305 ms, p99 1.769 ms, 765.2 QPS (1000/QPS 1.307 ms).
Pass default@8 after exact: 2026-10-05T03:10:43.813Z to 2026-10-05T03:11:03.814Z (20.0 s), 52,657 searches, 0 failed, p50 2.883 ms, mean 3.037 ms, p99 6.022 ms, 2632.6 QPS (1000/QPS 0.380 ms).
Pass default@1 after default@8: 2026-10-05T03:11:03.859Z to 2026-10-05T03:11:23.860Z (20.0 s), 14,936 searches, 0 failed, p50 1.318 ms, mean 1.337 ms, p99 1.778 ms, 746.8 QPS (1000/QPS 1.339 ms).
Passes in the order run: exact, default@8, default@1; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 9,732 sent, 0 failed (warmupErrors). Index after the searches: NOT ready, 0 of 524 indexed (NO HNSW GRAPH, searches scan all 524 vectors: 1 segment(s), 524 vectors, total_vex_size_bytes 0 (_stats dense_vector), index_options hnsw m=16 ef_construction=128; Lucene builds no graph for a segment this small (measured at 1,024 dimensions: 1,042 vectors none, 1,043 vectors a graph), so default search equals exact search).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-elasticsearch (d609f4f8f363) on network engines_default at container-ip:9200, not through docker-proxy localhost:9200. Open after the passes: container-ip:9200 x8 (this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-elasticsearch: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-elasticsearch: 74 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): exact 3492/3492 MHz, performance, outside load 0.11 before the warm-up, 0.12 during, client CPU 0.470 ms per search; default@8 3492/3492 MHz, performance, outside load 0.12 before the warm-up, 0.14 during, client CPU 0.507 ms per search; default@1 3492/3492 MHz, performance, outside load 0.13 before the warm-up, 0.10 during, client CPU 0.469 ms per search.

Typesense typesense

Engine: Typesense 30.2 (vector search) (compose)
Index: HNSW float32 (hnswlib), m=16, ef_construction=128, cosine; search k=top, ef=100; exact mode = filter ordinal:>=0 with flat_search_cutoff; index held in memory
Load: 524 rows in batches of 1000, 1.5 s of upserts (354 rows/s); count matched 0.0 s after the last upsert; index step 0.8 s (nothing to build, Typesense inserts into the HNSW graph while writing; ready after 0.8 s: HNSW graph returns 524 of 524 vectors, no writes queued: num_documents 524, pending_write_batches 0, health ok, unfiltered vector query with k above the count returned 524; the graph is built in memory during each write, so there is no later build step)
Search: 4864 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 0.51 ms, 8 searchers 0.58 ms
Exact mode: 10,881 searches, p50 5.47 ms, p95 5.83 ms, recall 1.000
RAM: 171.7 MiB (docker stats); disk: 0 B added by this load (260.67 MiB (whole engine data folder), 711.48 MiB before)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started typesense.compose.yaml with every container created on CPUs 2-3,6-7: gvb-typesense cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
Index after the load: ready, 524 of 524 indexed (HNSW graph returns 524 of 524 vectors, no writes queued: num_documents 524, pending_write_batches 0, health ok, unfiltered vector query with k above the count returned 524; the graph is built in memory during each write, so there is no later build step). Durability: Every acknowledged write is appended to Typesense's raft log and fsynced before the answer: braft raft_sync=true with raft_sync_policy=0 (sync immediately), read from the running container's brpc /flags page on 2026-10-04. A restart replays the log from the last snapshot (typesense.compose.yaml sets TYPESENSE_SNAPSHOT_INTERVAL_SECONDS=300) and rebuilds the in-memory HNSW graph, so by those settings a crash loses no acknowledged write, but the restart is slow after a big load. This is read from the settings, not shown by pulling power. One node and no replicas, so a lost disk loses the data..
Load average 3.00 3.30 3.03 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: default@1 1,219 searches in 5.0 s, default@8 2,937 searches in 5.0 s, exact 901 searches in 5.0 s.
Settle: settled after 1.2 s and 300 searches (0 failed); p50 of the last windows of 100: 4.06, 4.05, 4.05 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 483 searches in 2.0 s: p50 4.038 ms against the settled p50 4.050 ms, 0% apart (limit 15%).
Pass default@1 after settle: 2026-10-05T03:12:00.448Z to 2026-10-05T03:12:20.448Z (20.0 s), 4,864 searches, 0 failed, p50 4.044 ms, mean 4.110 ms, p99 6.676 ms, 243.2 QPS (1000/QPS 4.112 ms).
Pass default@8 after default@1: 2026-10-05T03:12:20.489Z to 2026-10-05T03:12:40.495Z (20.0 s), 11,740 searches, 0 failed, p50 13.113 ms, mean 13.630 ms, p99 25.851 ms, 586.8 QPS (1000/QPS 1.704 ms).
Pass exact after default@8: 2026-10-05T03:12:40.608Z to 2026-10-05T03:13:40.613Z (60.0 s), 10,881 searches, 0 failed, p50 5.469 ms, mean 5.513 ms, p99 6.123 ms, 181.3 QPS (1000/QPS 5.515 ms).
Passes in the order run: default@1, default@8, exact; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 5,900 sent, 0 failed (warmupErrors). Index after the searches: ready, 524 of 524 indexed (HNSW graph returns 524 of 524 vectors, no writes queued: num_documents 524, pending_write_batches 0, health ok, unfiltered vector query with k above the count returned 524; the graph is built in memory during each write, so there is no later build step).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvb-typesense (a15714d02dc3) on network engines_default at container-ip:8108, not through docker-proxy localhost:8108. Open after the passes: container-ip:8108 x1 (this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvb-typesense: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvb-typesense: 303 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): default@1 3592/3592 MHz, performance, outside load 0.10 before the warm-up, 0.12 during, client CPU 0.510 ms per search; default@8 3592/3592 MHz, performance, outside load 0.11 before the warm-up, -0.07 during, client CPU 0.578 ms per search; exact 3592/3592 MHz, performance, outside load 0.04 before the warm-up, 0.12 during, client CPU 0.529 ms per search.

MariaDB mariadb

Engine: MariaDB 11.8.9 (InnoDB + VECTOR INDEX) (compose)
Index: VECTOR INDEX (HNSW variant) DISTANCE=cosine, M=16 (no ef_construction setting exists), mhnsw_ef_search=100 per statement (the ef 100 most engines here use, so the search effort matches; MariaDB's own default is 20; recall@10 at ef 100 falls as the set grows (random 1024-dimension vectors, measured 2026-10-04: 0.99 at 524, 0.89 to 0.92 at 2,000; an earlier run gave about 0.09 at 100,000 [Correction 4])), mhnsw_max_cache_size 4G; exact mode = IGNORE INDEX full scan
Load: 524 rows in batches of 1000, 0.5 s of upserts (1,089 rows/s); count matched 0.0 s after the last upsert; index step 0.5 s (VECTOR INDEX is maintained inside every INSERT, nothing to build; confirmed after 2 poll(s) in 0.5 s (poll 1 of 2 said: not ready: a search at mhnsw_ef_search 100 returned only 376 of 524 requested rows through the index (EXPLAIN key 'vec_idx') (information_schema lists vec_idx as INDEX_TYPE VECTOR, EXPLAIN of the default search uses key vec_idx, the server applied mhnsw_ef_search 100 (asked for 100, read back from the server), a search at that effort returned 376 of 524 requested rows through the index; MariaDB has no per-index row count (the graph is a hidden InnoDB table), so indexed vectors are not reported)): information_schema lists vec_idx as INDEX_TYPE VECTOR, EXPLAIN of the default search uses key vec_idx, the server applied mhnsw_ef_search 100 (asked for 100, read back from the server), a search at that effort returned 518 of 524 requested rows through the index; MariaDB has no per-index row count (the graph is a hidden InnoDB table), so indexed vectors are not reported)
Search: 31155 latency samples, target held 524 rows, 0 errors
Client CPU per search: 1 searcher 0.49 ms, 8 searchers 0.38 ms
Exact mode: 37,277 searches, p50 1.58 ms, p95 1.69 ms, recall 1.000
RAM: 156.3 MiB (docker stats); disk: 22.01 MiB added by this load (182.79 MiB (whole engine data folder), 160.78 MiB before)
Started by run-all for this measurement with every container created on CPUs 2-3,6-7 (started mariadb-bench.compose.yaml with every container created on CPUs 2-3,6-7: gvbbench-mariadb cpuset 2-3,6-7 (read back from docker inspect)); stopped afterwards.
Index after the load: ready, ? of 524 indexed (information_schema lists vec_idx as INDEX_TYPE VECTOR, EXPLAIN of the default search uses key vec_idx, the server applied mhnsw_ef_search 100 (asked for 100, read back from the server), a search at that effort returned 518 of 524 requested rows through the index; MariaDB has no per-index row count (the graph is a hidden InnoDB table), so indexed vectors are not reported). Durability: innodb_flush_log_at_trx_commit=2 (mariadb.compose.yaml): the InnoDB redo log is written to the operating system at every commit but fsynced about once a second, so a crash of the mariadbd process loses nothing, while an operating-system crash or power cut can lose the last second of commits; innodb_doublewrite is on (default), the binary log is off, and the vector graph is an InnoDB table under the same log (settings read from the running server with SHOW VARIABLES; the crash behaviour is InnoDB's documented behaviour for this setting and was not tested, because gvb-mariadb is shared and stays up).
Load average 2.09 3.00 2.96 (1/5/15 min) when searching began.
Rehearsal before any timed pass, untimed, every pass type for at least 5 s through the same code the timed passes use: exact 3,093 searches in 5.0 s, default@1 7,809 searches in 5.0 s, default@8 29,720 searches in 5.0 s.
Settle: settled after 0.4 s and 600 searches (0 failed); p50 of the last windows of 100: 0.65, 0.67, 0.67 ms, within 5% across 3 windows; confirmed by a trial right before the first timed pass of 3,075 searches in 2.0 s: p50 0.644 ms against the settled p50 0.668 ms, 4% apart (limit 15%).
Pass exact after settle: 2026-10-05T03:14:12.991Z to 2026-10-05T03:15:12.992Z (60.0 s), 37,277 searches, 0 failed, p50 1.581 ms, mean 1.608 ms, p99 2.283 ms, 621.3 QPS (1000/QPS 1.610 ms).
Pass default@1 after exact: 2026-10-05T03:15:13.041Z to 2026-10-05T03:15:33.041Z (20.0 s), 31,155 searches, 0 failed, p50 0.630 ms, mean 0.640 ms, p99 0.885 ms, 1557.7 QPS (1000/QPS 0.642 ms).
Pass default@8 after default@1: 2026-10-05T03:15:33.082Z to 2026-10-05T03:15:53.083Z (20.0 s), 118,196 searches, 0 failed, p50 1.308 ms, mean 1.352 ms, p99 2.478 ms, 5909.4 QPS (1000/QPS 0.169 ms).
Passes in the order run: exact, default@1, default@8; each after its own warm-up. Untimed searches (rehearsal, settle, settle check and per-pass warm-ups): 44,357 sent, 0 failed (warmupErrors). Index after the searches: ready, ? of 524 indexed (information_schema lists vec_idx as INDEX_TYPE VECTOR, EXPLAIN of the default search uses key vec_idx, the server applied mhnsw_ef_search 100 (asked for 100, read back from the server), a search at that effort returned 518 of 524 requested rows through the index; MariaDB has no per-index row count (the graph is a hidden InnoDB table), so indexed vectors are not reported).
Benchmark copy gvbbench_eshoponweb dropped afterwards.
Connection: container address on its Docker network, not the published port (no docker-proxy): container gvbbench-mariadb (dad6f8489a43) on network gvbbench-mariadb_default at container-ip:3306, not through docker-proxy localhost:13306. Open after the passes: container-ip:3306 x8 (this target).
CPU pinning: cpuset from the container's creation when run-all started it on the engine CPUs, else docker update --cpuset-cpus (each container's change line says which), CPUs 2-3,6-7. Changed: container gvbbench-mariadb: already on CPUs 2-3,6-7 (its cpuset since creation), not changed. Read back: container gvbbench-mariadb: 11 thread(s) on CPUs 2-3,6-7 (every thread of the container).
Clock per pass (median MHz of engine CPUs / client CPUs, governor, CPUs busy outside the benchmark, client CPU per search): exact 3593/3592 MHz, performance, outside load 0.07 before the warm-up, 0.11 during, client CPU 0.497 ms per search; default@1 3492/3492 MHz, performance, outside load 0.11 before the warm-up, 0.12 during, client CPU 0.487 ms per search; default@8 3492/3492 MHz, performance, outside load 0.11 before the warm-up, 0.06 during, client CPU 0.383 ms per search.

Notes

Order: targets ran one at a time in a random order from seed 501 (runSeed; --seed 501 repeats it, targetOrder lists it). Inside each target the timed passes also ran in a random order from the same seed and the target's name (passOrder): default@1 is one searcher for 20 s (every search's latency gives p50/p95/p99, the completed searches give QPS@1, its first answer to each query gives recall and nDCG), default@N is N searchers for 20 s (QPS@N), exact is the engine's exact mode, one searcher for 60 s cycling the queries.
Preparation, untimed, before any timed pass: a rehearsal of every pass type for 5 s each through the same code the passes use, then a settle (default searches one at a time until the p50s of 3 windows of 100 agree within 5%, at most 120 s). Right before the first timed pass a 2 s trial of default searches must agree with the settled p50 within 15%; if it does not, or the settle did not settle, the settle is extended once (at least 30 s, at most 120 s more) and a second trial taken. Each target's notes give the result; a target still disagreeing is marked NOT settled.
Warm-up: every timed pass started with its own untimed warm-up of 20 searches in the same search mode and with the same number of searchers, stopped early after 60 s; with machine control on, the check for a quiet box comes right before the warm-up, so warm-up and timed pass run back to back. Failed untimed searches are counted per target (warmupErrors) and are not in the timed error counts.
Load rows/s counts only time inside each target's upsert calls: one writer, batches of 1000, rows already in memory, the collection dropped and created fresh first. Engines that build or finish their index after the writes do it in a separate timed index step (the load's index seconds), and the run waits for it before searching.
Index proof: each engine's own report of its index (indexState) is read after the load and again after the last pass. A target whose index was not ready after the load was still measured and carries a WARNING; an engine that reports nothing counts as not ready.
Durability: each target's crash-safety setting as configured here (durability). Engines that do not force writes to disk on every commit load faster for that reason.
Latency is client-side wall time around each search (network and driver included), every search of the default@1 window, one searcher, the queries cycled; a window with fewer than 200 searches, a p50 above 1.25 x its mean, or a mean above its p99 is flagged.
QPS: N searchers back to back for 20 s per level; completed searches divided by the window's elapsed time.
Recall@10: share of the exact top 10 (brute force in memory) that the engine returned. A hit whose exact similarity ties the 10th best (within 1e-5) also counts, because duplicate rows embed to identical vectors.
Routes: a container engine (the benchmark's own SQL Server and Qdrant containers included) is reached at its container's own address on its Docker network, never through the published localhost port (docker-proxy); the native comparison targets (sql-native, qdrant-native) are native services reached directly over loopback. Each target's addresses and the connections the client held open after its passes are in its notes and in conditions.connections.
RAM of compose engines, the benchmark's own SQL Server and Qdrant containers (sql, sql-diskann, qdrant, qdrant-hnsw) included, is docker stats of the engine's containers; for the native comparison targets (sql-native, qdrant-native) it is the whole native process, including every other database or collection it serves. Disk is the table's reserved pages (SQL), the collection folder (Qdrant), or for other engines what the load added to the engine's data folder (engines that keep data in memory until a snapshot show almost nothing).
Engines: run-all starts an engine that is down and stops it afterwards only if it was not running when the run began; an engine that was already running is left running.
WARNING: the engine did not report a ready index after the load for: elasticsearch. They were measured anyway; their numbers may come from a scan or a half-built index (see each target's index state).
Machine control on: governor performance on every CPU during the run (before: schedutil; at the end: performance; after putting it back: schedutil). engine CPUs 2-3,6-7 (cores 2,6 and 3,7), client CPUs 0-1,4-5 (cores 0,4 and 1,5); the client process was pinned; each engine was pinned to the engine CPUs for its turn and put back after (conditions.engines); an engine run-all started (and stopped) was asked to be created on them, and its notes say whether the host did so or it was moved there after the start. Busy box: right before each pass's warm-up the run waits, up to 10 min, while processes outside the benchmark (everything but this client and the engine under test's cgroups) use more than 0.3 CPUs on average over the last 60 s or the last 5 s, neither window reaching back past the start of the target or of its engine (so the engine's own start-up is not outside work); if the box does not clear the pass runs anyway, flagged 'busy box', as is a pass whose own outside load is above the limit. Outside load counts busy = user + nice + system + irq + softirq + steal (guest time is already in user); CONFIG_IRQ_TIME_ACCOUNTING is not set, so task and cgroup run time include the interrupt and softirq time that hit them and it is added; CONFIG_PARAVIRT_TIME_ACCOUNTING is not set, so steal is added (/boot/config-6.8.0-142-generic) (conditions.cpuAccounting); each pass also records this client's own CPU time per search (conditions.passes[].clientCpuMsPerSearch). CPU clocks were sampled every 250 ms; each pass's min/median/max per CPU is in conditions.passes. CPU idle states, recorded and left as found: driver intel_idle, governor menu, intel_idle max_cstate 9; POLL on, C1 on, C1E on, C3 OFF on CPUs 0-7 (default disabled), C6 on (conditions.cpuIdle). How each target was reached: conditions.connections. Client build Release, .NET 10.0.12.
sqlitevec is embedded: it ran inside the client process on the client CPUs 0-1,4-5, sharing them with the client.
duckdb is embedded: it ran inside the client process on the client CPUs 0-1,4-5, sharing them with the client.
WARNING: accounting mismatch in typesense default@8: outside load during the pass was -0.071 CPUs, below -0.05; the kernel's busy time minus this process minus the engine's cgroups came out negative, so the outside-load figures of this pass cannot be trusted.
WARNING: accounting mismatch in typesense exact: outside load over the last few seconds before the warm-up was -0.096 CPUs, below -0.05; the kernel's busy time minus this process minus the engine's cgroups came out negative, so the outside-load figures of this pass cannot be trusted.

Raw files

results.md is kept unedited. It still holds the sentences this page leaves out of its copy. Count: 1

The raw files below hold the recorded statements that the corrections above refer to, as written.