FARM Inference Speed Benchmarks

	A	B	C	D	E	F	G	H	I	J	K	L	M	N	O
1		model name	question	preproc	language model	prediction head	total	batch_size	document_size	num_processes	max_seq_len	doc_stride	gpu	sample_file	error
2	0	deepset/bert-base-cased-squad2	When were the first traces of Human life found in France?	0.083	0.156	0.832	1.070	16	10000	1	384	128	TRUE	samples/question_answering_sample.txt
3	1	deepset/bert-base-cased-squad2	How many pretrained models are available in Transformers?	0.083	0.160	0.005	0.247	16	10000	1	384	128	TRUE	samples/question_answering_sample.txt
4	2	deepset/minilm-uncased-squad2	When were the first traces of Human life found in France?	0.109	0.066	0.593	0.768	16	10000	1	384	128	TRUE	samples/question_answering_sample.txt
5	3	deepset/minilm-uncased-squad2	How many pretrained models are available in Transformers?	0.109	0.066	0.011	0.186	16	10000	1	384	128	TRUE	samples/question_answering_sample.txt
6	4	deepset/roberta-base-squad2	When were the first traces of Human life found in France?	0.114	0.154	0.000	0.261	16	10000	1	384	128	TRUE	samples/question_answering_sample.txt
7	5	deepset/roberta-base-squad2	How many pretrained models are available in Transformers?	0.113	0.134	0.022	0.270	16	10000	1	384	128	TRUE	samples/question_answering_sample.txt
8	6	deepset/bert-large-uncased-whole-word-masking-squad2	When were the first traces of Human life found in France?	0.083	0.430	0.010	0.523	16	10000	1	384	128	TRUE	samples/question_answering_sample.txt
9	7	deepset/bert-large-uncased-whole-word-masking-squad2	How many pretrained models are available in Transformers?	0.083	0.430	0.011	0.524	16	10000	1	384	128	TRUE	samples/question_answering_sample.txt
10
11
12
13
14
15
16
17
18
19	Total inference time (in sec) for QA with deepset/bert-base-cased-squad2
20	document_size (chars)	batch_size	max_seq_len	PyTorch GPU	ONNXRuntime (w/o optimizations)	ONNXRuntime w/ V100 optimization	ONNX vs PyTorch(V100)	PyTorch CPU	ONNXRuntime CPU
21	10000	1	128	0.37
22	10000	1	256	0.37
23	10000	1	384	0.37
24	10000	4	128	0.20
25	10000	4	256	0.20
26	10000	4	384	0.20
27	10000	16	128	0.19
28	10000	16	256	0.19
29	10000	16	384	0.19
30	10000	64	128	0.18
31	10000	64	256	0.18
32	10000	64	384	0.18
33	100000	1	128	4.20
34	100000	1	256	4.24
35	100000	1	384	4.24
36	100000	4	128	2.42
37	100000	4	256	2.43
38	100000	4	384	2.43
39	100000	16	128	2.26
40	100000	16	256	2.31
41	100000	16	384	2.27
42	100000	64	128	2.25
43	100000	64	256	2.24
44	100000	64	384	2.30
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100