KIP-98 perf testing
 Share
The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss

 
View only
 
 
ABCDEFGHIJKLMNOPQRSTUVWXYZAAABAC
1
In all the tests below, record level crc checking on the new format is disabled. Batch level crc checking is enabled.
2
Java version -- OpenJDK 1.8.0_121
3
4
AWS Ubuntu 16.04 server, d2.xlarge, quad core Xeon processor, 32 GB RAM, 6TB local storage with spinning disk -- linger.ms=10000, produce, then consume.
5
Message size (bytes)
Produce MB/s (trunk)Produce MB/s (EOS)Produce Delta
producer Batch size EOS / Trunk (bytes)
Total size produced/consumed (MB
Consume MB/s (Trunk)
Consume MB/s (EOS)Consume Delta
On disk size GB (trunk)
On disk size GB (EOS)
Disk size delta
6
6443.3350.8717.40%16351 / 163669765141.17218.9755.11%14.6211.26-22.98%
7
25667.6681.1619.95%16226 / 162409765288.31320.511.17%10.829.93-8.23%
8
51283.5588.375.77%16212 / 163809765347.17366.375.53%10.179.75-4.13%
9
10
AWS Ubuntu 16.04 server, d2.xlarge, quad core Xeon processor, 32 GB RAM, 6TB local storage with spinning disk -- drop page caches before read, linger.ms=10000, produce, then consume.
11
Message size (bytes)
Produce MB/s (trunk)Produce MB/s (EOS)Produce Delta
producer Batch size EOS / Trunk (bytes)
Total size produced/consumed (MB
Consume MB/s (Trunk)
Consume MB/s (EOS)Consume Delta
On disk size GB (trunk)
On disk size GB (EOS)
Disk size delta
12
6443.3350.8717.40%16351 / 163669765104.49107.582.96%14.6211.26-22.98%
13
25667.6681.1619.95%16226 / 162409765109.47161.6847.69%10.829.93-8.23%
14
51283.5588.375.77%16212 / 163809765140.87161.0314.31%10.179.75-4.13%
15
16
AWS Ubuntu 16.04 server, m1.large, dual core Xeon processor, 7.5 GB RAM, 420GB local storage with spinning disk -- linger.ms=10000, produce, then consume.
17
Message size (bytes)
Produce MB/s (trunk)Produce MB/s (EOS)Produce Delta
producer Batch size EOS / Trunk (bytes)
Total size produced/consumed (MB
Consume MB/s (Trunk)
Consume MB/s (EOS)Consume Delta
On disk size GB (trunk)
On disk size GB (EOS)
Disk size delta
18
6412.7512.6-1.18%16351 / 16362976547.9655.215.10%15.612.2-21.79%
19
25626.0127.054.00% 16226 / 16240976568.2678.7615.38%10.89.9-8.33%
20
51230.8131.743.02%16212 / 16380976577.2180.233.91%10.179.76-4.03%
21
22
AWS Ubuntu 16.04 server, m1.large, dual core Xeon processor, 7.5 GB RAM, 420GB local storage with spinning disk -- 2.5GB log (fits in memory), linger.ms=10000, produce, then consume.
23
Message size (bytes)
Produce MB/s (trunk)Produce MB/s (EOS)Produce Delta
producer Batch size EOS / Trunk (bytes)
Total size produced/consumed (MB
Consume MB/s (Trunk)
Consume MB/s (EOS)Consume Delta
On disk size GB (trunk)
On disk size GB (EOS)
Disk size delta
24
25623.1523.953.46%16226 / 162402441115.33130.6513.28%2.722.48-8.82%
25
51227.827.52-1.01%16212 / 163802441134.35144.727.72%2.552.44-4.31%
26
27
Ubuntu 16.04 server (7200 rpm Spinning disk, dual core i3, 8GB ram) -- produce first, then consume from the beginning
28
Message size (bytes)
Produce MB/s (trunk)Produce MB/s (EOS)Produce DeltaBatch size EOS (bytes)
Total size produced/consumed (MB
Consume MB/s (Trunk)
Consume MB/s (EOS)Consume Delta
On disk size GB (trunk)
On disk size GB (EOS)
Disk size delta
29
6430.1147.3957.39%16351976541.149.8721.34%
30
1284049.4123.53%16218976547.965.4536.64%13.310.28-22.71%
31
25648.8148.16-1.33%16226976554.7655.090.60%
32
51252.8555.765.51%16212976558.361.174.92%
33
102456.5659.044.38%15556976563.1367.657.16%
34
204868.4364.08-6.36%14460976562.3764.182.90%
35
36
Ubuntu 16.04 server (7200 rpm disk, dual core i3, 8GB ram) -- simultaneous produce and consume
37
Message size (bytes)
Produce MB/s (trunk)Produce MB/s (EOS)Produce DeltaBatch size EOS (bytes)
Total size produced/consumed (MB
Consume MB/s (Trunk)
Consume MB/s (EOS)Consume Delta
38
12827.5132.6918.83%16281976527.0832.9321.60%
39
25635.5638.859.25%976535.8939.189.17%
40
51236.0138.747.58%16212976536.5739.668.45%
41
42
Ubuntu 16.04 server (7200rpm disk, dual core i3, 8GB RAM) -- using english dictionary of 400k words and snappy compression. Simultaneous produce and consume
43
Num RecordsProduce MB/s (trunk)Produce MB/s (EOS)Produce DeltaBatch Size EOS (bytes)
Total size produced/consumed MB
Consumed MB/s (trunk)
Consumeb MB/s (EOS)
Consume Delta
On disk size GB (trunk)
On disk size GB (EOS)
Disk size Delta
44
1000000004.345.7933.41%9034.495.9332.07%2.51.55-38.00%
45
46
Ubuntu 16.04 server (7200rpm disk, dual core i3, 8GB RAM) -- With linger.ms set to 10000ms, produce first and then consume
47
Message size (bytes)
Produce MB/s (trunk)Produce MB/s (EOS)Produce Delta
Batch size EOS / Trunk(bytes)
Total size produced/consumed (MB
Consume MB/s (Trunk)
Consume MB/s (EOS)Consume Delta
On disk size GB (trunk)
On disk size GB (EOS)
Disk size delta
48
6428.144.5958.68% 41.2648.918.52%14.611-24.66%
49
12839.748.8122.95%16281976556.7859.935.55%13.0710.28-21.35%
50
25650.5456.0910.98%16226976562.6566.516.16%
51
51255.9960.898.75%16212 / 16380976560.217016.26%10.179.75-4.13%
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Loading...
 
 
 
Message format changes
Transactions Perf
Idempotent producer
Streams
IGNORE -- NOT USED