Copy of BEIR Leaderboards: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models

	A	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P	Q	R	S	T	U	V	W	X	Y	Z	AA	AB
1		Zero-shot Evaluation on BEIR Benchmark (NDCG@10 scores) [Code] [Paper]
2	Model	Model Name	Model Link	Trained on	Pooling Strategy	Sim Func.	Model Size	Queries (GPU/CPU)	Docs (GPU/ CPU)	In-domain	Zero-shot		Zero-Shot Datasets
3	Model	Model Name	Model Link	Trained on	Pooling Strategy	Sim Func.	Model Size	Queries (GPU/CPU)	Docs (GPU/ CPU)	MSMARCO	Rank	Average	TREC-COVID	BIOASQ	NFCorpus	NQ	HotpotQA	FiQA	Signal-1M	TREC-NEWS	ArguAna	Touche-2020	DBPedia	SCIDOCS	FEVER	Climate-FEVER	SciFact	Robust04
4	BM25	Elasticsearch (Default Settings)	Elastic	-	-	-	-	-	-	0.218	-	0.404	0.616	0.514	0.297	0.31	0.601	0.239	0.388	0.371	0.441		0.288	0.156	0.648	0.179	0.62	0.387
5	BM25-Anserini	Anserini (Default Lucene, k=0.9, b=0.4)	Pyserini	-	-	-	-	-	-	0.228	-	0.415	0.656	0.465	0.325	0.329	0.603	0.236	0.33	0.398	0.414	0.367	0.313	0.158	0.753	0.213	0.665	0.408
6	BM25-Anserini-RM3	Anserini (k=0.9, b=0.4) (RM3: 10,10,0.5)	Pyserini	-	-	-	-	-	-	0.388	-	0.371	0.654	0.43	0.348	0.255	0.456	0.193	0.321	0.433	0.25		0.273	0.147	0.547	0.184	0.642	0.426
7	BM25f	Manticore Search	Manticore							0.253		0.417	0.719		0.342	0.324	0.629	0.251			0.314	0.346	0.336	0.163	0.71	0.198	0.667
8
9	Contriever	bert-base-uncased (to be released!)	(Izacard et al. 2021)	MSMARCO	[CLS]	dot	-	-	-				0.596		0.328	0.498	0.638	0.329			0.446	0.23	0.413	0.165	0.758	0.237	0.677
10	MoDIR	RoBERTa-base (to be released!)	(Xin et al. 2021)	MSMARCO	[CLS]	dot	500 MB	4,000 / 170	540 / 30				0.676	0.32	0.244	0.442	0.462	0.296			0..418		0.284	0.124	0.68	0.206	0.502
11	DistilBERT (TAS-B)	msmarco-distilbert-base-tas-b	Hugging Face (HF)	MSMARCO	[CLS]	dot	250 MB	7,000 / 350	1,100 / 70	0.408	1	0.395	0.481	0.383	0.319	0.463	0.584	0.3	0.289	0.377	0.427	0.173	0.384	0.149	0.7	0.228	0.643	0.427
12	ANCE	msmarco-roberta-base-ance-fristp	Hugging Face (HF)	MSMARCO	[CLS]	dot	500 MB	4,000 / 170	540 / 30	0.388	2	0.368	0.654	0.306	0.237	0.446	0.456	0.295	0.249	0.382	0.415	0.284	0.281	0.122	0.669	0.198	0.507	0.392
13	DistilBERT	msmarco-distilbert-base-v3	Hugging Face (HF)	MSMARCO	mean	cosine	250 MB	7,000 / 350	1,100 / 70	0.389	3	0.361	0.482	0.295	0.257	0.45	0.513	0.258	0.261	0.367	0.429	0.249	0.339	0.133	0.67	0.205	0.531	0.337
14	DistilBERT (dot)	msmarco-distilbert-base-dot-prod-v3	Hugging Face (HF)	MSMARCO	mean	dot	250 MB	7,000 / 350	1,100 / 70	0.389	4	0.354	0.635	0.28	0.267	0.442	0.477	0.255	0.243	0.357	0.341	0.26	0.315	0.113	0.685	0.169	0.511	0.311
15	MiniLM-L-12	msmarco-MiniLM-L-12-v3	Hugging Face (HF)	MSMARCO	mean	cosine	130 MB	11,000 / 400	1,500 / 90	0.385	5	0.335	0.473	0.279	0.251	0.422	0.456	0.24	0.271	0.36	0.407	0.225	0.307	0.113	0.571	0.179	0.503	0.295
16	MiniLM-L-6	msmarco-MiniLM-L-6-v3	Hugging Face (HF)	MSMARCO	mean	cosine	90 MB	18,000 / 750	2,800 / 180	0.379	6	0.327	0.479	0.256	0.255	0.394	0.448	0.231	0.259	0.342	0.394	0.223	0.292	0.116	0.595	0.165	0.495	0.293
17	DistilBERT (NQ)	nq-distilbert-base-v1	Hugging Face (HF)	NQ	mean	cosine	250 MB	7,000 / 350	1,100 / 70	0.241	7	0.251	0.393	0.153	0.176	0.584	0.305	0.153	0.161	0.239	0.31	0.258	0.207	0.065	0.299	0.124	0.388	0.206
18	DPR (Multi)	facebook-dpr-ctx_encoder-multiset-base	Hugging Face (HF)	NQ, CuratedTREC WebQ, TriviaQA	[CLS]	dot	840 MB			0.177	8	0.241	0.332	0.127	0.189	0.474	0.391	0.112	0.155	0.161	0.175	0.127	0.263	0.077	0.562	0.148	0.318	0.252
19	DPR (KILT)	to be released by facebook	Hugging Face (HF)	KILT	[CLS]	dot	840 MB			0.15	9	0.203	0.177	0.068	0.131	0.46	0.518	0.039	0.127	0.11	0.025		0.286	0.05	0.686	0.196	0.119	0.051
20
21	ColBERT-v2	bert-base-uncased (to be released!)	(Santhanam et al. 2021)	MSMARCO						0.408			0.738		0.338	0.562	0.667	0.356			0.463	0.263	0.446	0.154	0.785	0.176	0.693
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100