ABCDEFGHIJKLMNOPQRSTUVWXYZ
1
model namequestionpreproclanguage modelprediction headtotalbatch_sizedocument_sizenum_processesmax_seq_lendoc_stridegpusample_fileerror
2
0
deepset/bert-base-cased-squad2
When were the first traces of Human life found in France?
0.0830.1560.8321.07016100001384128TRUE
samples/question_answering_sample.txt
3
1
deepset/bert-base-cased-squad2
How many pretrained models are available in Transformers?
0.0830.1600.0050.24716100001384128TRUE
samples/question_answering_sample.txt
4
2
deepset/minilm-uncased-squad2
When were the first traces of Human life found in France?
0.1090.0660.5930.76816100001384128TRUE
samples/question_answering_sample.txt
5
3
deepset/minilm-uncased-squad2
How many pretrained models are available in Transformers?
0.1090.0660.0110.18616100001384128TRUE
samples/question_answering_sample.txt
6
4
deepset/roberta-base-squad2
When were the first traces of Human life found in France?
0.1140.1540.0000.26116100001384128TRUE
samples/question_answering_sample.txt
7
5
deepset/roberta-base-squad2
How many pretrained models are available in Transformers?
0.1130.1340.0220.27016100001384128TRUE
samples/question_answering_sample.txt
8
6
deepset/bert-large-uncased-whole-word-masking-squad2
When were the first traces of Human life found in France?
0.0830.4300.0100.52316100001384128TRUE
samples/question_answering_sample.txt
9
7
deepset/bert-large-uncased-whole-word-masking-squad2
How many pretrained models are available in Transformers?
0.0830.4300.0110.52416100001384128TRUE
samples/question_answering_sample.txt
10
11
12
13
14
15
16
17
18
19
Total inference time (in sec) for QA with deepset/bert-base-cased-squad2
20
document_size (chars)
batch_sizemax_seq_lenPyTorch
GPU
ONNXRuntime
(w/o optimizations)
ONNXRuntime
w/ V100 optimization
ONNX vs
PyTorch(V100)
PyTorch
CPU
ONNXRuntime
CPU
21
1000011280.37
22
1000012560.37
23
1000013840.37
24
1000041280.20
25
1000042560.20
26
1000043840.20
27
10000161280.19
28
10000162560.19
29
10000163840.19
30
10000641280.18
31
10000642560.18
32
10000643840.18
33
10000011284.20
34
10000012564.24
35
10000013844.24
36
10000041282.42
37
10000042562.43
38
10000043842.43
39
100000161282.26
40
100000162562.31
41
100000163842.27
42
100000641282.25
43
100000642562.24
44
100000643842.30
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100