1. Overview

This page contains some tests of the vec1 extension run on publicly available datasets distributed for testing ANN systems. The following datasets are used:

"sift-128-euclidean" (also known as "sift1m"), from corpus-texmex.irisa.fr,
"imagenet-clip-512-normalized", from the VIBE project, and
"landmark-dino-768-cosine", also from the VIBE project, and
"agnews-mxbai-1024-euclidean", from VIBE again.

All four datasets are distributed with sample query vectors and results. "sift-128-euclidean" is distributed with dedicated training data (separate from the dataset itself), and the others are trained by sampling the main dataset. In all cases training and building the index are done with 16 threads. The "Training time" reported below is the wall-clock time the system spent running vec1_train() to create the model. The "Build time" is the wall-clock time spent running the 'rebuild' command to build the index.

Queries are run with a variety of values for parameters K and nprobe and the recall and throughput (queries/second) reported. The reported throughputs are for a single thread only.

Two flavors of recall are reported - recall@1 and recall@10. Recall@1 is the proportion of queries for the single nearest neighbor that do, in fact, return the true nearest neighbor. So a recall@1 of .916 means that if 1000 queries for the nearest neighbor are run, 916 of them return the true nearest neighbor.

Recall@10 is the average proportion of the true 10 nearest neighbors actually returned when the index is queried for the best 10 matches. i.e. if of the 10 results the query returns 7 of them are actually in the best 10 matches, the recall@10 is 0.7. The order of results returned does not matter - only the proportion that are part of the actual 10 best matches.

The SQL used for each query is:

SELECT *
FROM vec1tbl($query, '{K: $K, nprobe: $nprobe}')
ORDER BY vec1_l2_distance($query, vec1tbl.vector)
LIMIT $recall

Where $query is the query vector, $K is replaced by query parameter K, $nprobe by query parameter nprobe and $recall with the desired recall measure (1 or 10). For the "landmark-dino-768-cosine" index, vec1_cos_distance() is used instead of vec1_l2_distance(). In cases where $K==$recall the ORDER BY and LIMIT clauses are omitted from the query.

The results below were obtained on an AMD 5950X CPU based Linux workstation.

2. Dataset: sift-128-euclidean (1,000,000 128d vectors)

Parameter	Value
Vec1 Version	version 0.7 (AVX2, multi-threaded)
Index Parameters	`{nbucket:1024, quantizer:"pq", codesize:16}`
Training time	1.28s (100,000 samples, 16 threads)
Build time	622ms (16 threads)

Recall@10 results:

	K=10		K=50		K=100		K=200		K=300
nprobe	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS
16	0.556	7991	0.882	5265	0.919	4088	0.927	2875	0.929	2224
32	0.565	4683	0.916	3537	0.962	2937	0.975	2223	0.977	1797
48	0.567	3345	0.924	2678	0.973	2308	0.988	1833	0.990	1526
64	0.568	2615	0.927	2175	0.977	1915	0.992	1585	0.995	1337

Recall@1 results:

	K=1		K=10		K=50		K=100		K=200		K=300
nprobe	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS
16	0.469	8572	0.880	7166	0.947	5552	0.950	4253	0.951	2909	0.951	2250
32	0.474	5055	0.903	4440	0.981	3586	0.987	3004	0.987	2252	0.988	1820
48	0.475	3593	0.907	3221	0.987	2716	0.994	2341	0.995	1854	0.995	1541
64	0.476	2800	0.909	2536	0.990	2201	0.997	1954	0.998	1616	0.998	1351

3. Dataset: imagenet-clip-512-normalized (1,281,167 512d vectors)

Parameter	Value
Vec1 Version	version 0.7 (AVX2, multi-threaded)
Index Parameters	`{nbucket:1024, quantizer:"opq", codesize:32, residual:0}`
Training time	11.7s (100,000 samples, 16 threads)
Build time	3.45s (16 threads)

Recall@10 results:

	K=10		K=50		K=100		K=200		K=300
nprobe	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS
16	0.526	4417	0.896	3261	0.955	2684	0.975	2005	0.979	1611
32	0.529	2711	0.904	2181	0.966	1887	0.987	1515	0.992	1270
48	0.529	1980	0.907	1659	0.970	1476	0.990	1216	0.995	1059
64	0.529	1568	0.907	1346	0.970	1215	0.991	1039	0.996	913

Recall@1 results:

	K=1		K=10		K=50		K=100		K=200		K=300
nprobe	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS
16	0.413	4521	0.871	3563	0.976	2597	0.989	2437	0.991	1933	0.993	1610
32	0.410	2935	0.874	2615	0.981	2202	0.994	1901	0.996	1523	0.998	1278
48	0.411	2135	0.875	1931	0.982	1676	0.995	1484	0.997	1231	0.999	1064
64	0.411	1685	0.875	1537	0.982	1357	0.995	1226	0.997	1045	0.999	918

4. Dataset: landmark-dino-768-cosine (760,757 768d vectors)

Parameter	Value
Vec1 Version	version 0.7 (AVX2, multi-threaded)
Index Parameters	`{nbucket:1024, quantizer:"opq", codesize:48, residual:0}`
Training time	18.6s (100,000 samples, 16 threads)
Build time	2.88s (16 threads)

Recall@10 results:

	K=10		K=50		K=100		K=200		K=300
nprobe	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS
16	0.585	3747	0.903	2636	0.937	2226	0.948	1683	0.950	1370
32	0.591	2451	0.926	1856	0.964	1579	0.977	1312	0.980	1111
48	0.593	1869	0.931	1472	0.971	1271	0.985	1048	0.988	905
64	0.593	1522	0.933	1230	0.974	1078	0.988	906	0.992	792

Recall@1 results:

	K=1		K=10		K=50		K=100		K=200		K=300
nprobe	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS
16	0.520	3952	0.914	3041	0.968	2196	0.970	2057	0.972	1578	0.972	1371
32	0.525	2629	0.926	2203	0.986	1825	0.988	1579	0.990	1298	0.990	1115
48	0.525	2095	0.930	1800	0.991	1473	0.993	1245	0.995	1043	0.995	902
64	0.525	1707	0.930	1475	0.992	1236	0.994	1084	0.996	917	0.996	824

5. Dataset: agnews-mxbai-1024-euclidean (769,382 1024d vectors)

Parameter	Value
Vec1 Version	version 0.7 (AVX2, multi-threaded)
Index Parameters	`{nbucket:1024, quantizer:"opq", codesize:32, residual:0}`
Training time	25.9s (100,000 samples, 16 threads)
Build time	4.75s (16 threads)

Recall@10 results:

	K=10		K=50		K=100		K=200		K=300
nprobe	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS
16	0.618	3911	0.889	2783	0.930	2251	0.944	1643	0.949	1299
32	0.623	2826	0.904	2127	0.949	1792	0.965	1369	0.970	1124
48	0.625	2193	0.911	1730	0.958	1497	0.974	1179	0.980	985
64	0.626	1804	0.914	1466	0.961	1290	0.978	1044	0.983	883

Recall@1 results:

	K=1		K=10		K=50		K=100		K=200		K=300
nprobe	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS	Recall	QPS
16	0.611	4025	0.920	2922	0.960	2129	0.969	2088	0.969	1626	0.969	1291
32	0.609	3062	0.927	2637	0.971	2148	0.980	1803	0.980	1380	0.980	1131
48	0.613	2414	0.933	2081	0.980	1740	0.991	1505	0.991	1184	0.991	991
64	0.612	1997	0.934	1732	0.982	1471	0.993	1296	0.993	1047	0.993	887

This page was last updated on 2026-10-01 11:21:38Z