Llama2 Benchmarks IREE vs. llama.cpp

	H	I	J	K
1
2	Latencies (ms) vs Context Lengths			**llama.cpp only gives average times, so the data is infered from several averages based on an assumption that the latency grows linearly similar to IREE. This is meant to be a crude approximation to show the rough difference between IREE and llama.cpp.
3	Context Length	IREE	llama.cpp
4	1	81	73.0
5	100	82.5	73.8
6	200	85	74.6
7	300	86.5	75.3
8	400	89.5	76.1
9	500	91.5	76.9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100