Warc Compression Comparison
 Share
The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss

 
$
%
123
 
 
 
 
 
 
 
 
 
ABCDEFGHIJKLMNOPQRSTUVWXYZ
1
test.warc.gz:
2
levelelapsedrssorigsize% of sizeMB/s
3
xz-111.589,879,55240,079,90129,371,29673.28%3.300767288
4
xz-210.6617,711,10440,079,90123,581,48858.84%3.586122343
5
xz-38.8433,423,36040,079,90118,150,12845.28%4.322606707
6
xz-410.9950,245,63240,079,90117,817,62044.46%3.47852608
7
xz-510.6398,484,22440,079,90114,005,57634.94%3.59529419
8
xz-611.3898,504,70440,079,90114,000,54434.93%3.359067263
9
xz-710.56194,998,27240,079,90111,339,25228.29%3.619749521
10
xz-810.03377,171,96840,079,90110,403,26825.96%3.812564252
11
xz-99.99429,404,16040,079,90110,373,30825.88%3.826765488
12
13
Megabytes saved per minute per core
14
test.warc.gz:IanQ machine
time zstd -z -10 -o /dev/null gp.warc.gz
megabits saved per second per core
15
levelelapsedorigsize% of sizeMB/sdays to crunch0.05TB with1cores
16
zstd 10.46415,099,705406,467,94197.92%858.72001611071.395874142.85278320.0007066503674editable ^ editable ^
17
zstd 20.62415,099,705397,542,93995.77%639.53138511622.950257216.39336750.0009488429011
18
xz 070.958415,099,705389,667,96093.87%5.57893299420.508133292.7344177710.1087689735
19
zstd 31.18415,099,705353,678,16785.20%334.34960082968.385638395.78475180.001814911139
20
zstd 42.01415,099,705340,609,36082.05%197.34293492124.811431283.30819080.00307492546
21
zstd 56.07415,099,705329,228,76979.31%65.26045622810.0188545108.00251390.009298353858
22
xz 1102.67415,099,705304,452,71673.34%3.8557507361.666226718.2221635610.157379161
time xz -c1 gp.warc.gz >/dev/null
23
zstd 67.09415,099,705264,954,78963.83%55.850723391212.099465161.6132620.01086494101
24
zstd 79.01415,099,705264,924,17363.82%43.931853953.6247774127.14997030.01381263874
25
zstd 815.20415,099,705263,131,66563.39%26.03893491571.971387276.262851630.02330413348
26
zstd 913.07415,099,705261,074,05562.89%30.29771372674.530720189.937429340.02002840282
27
zstd 1019.23415,099,705260,202,41762.68%20.58498921460.885752961.451433730.02947851023
28
zstd 1118.97415,099,705260,036,64562.64%20.86600925467.67760262.35701360.02908149842
29
xz 298.856415,099,705232,632,57256.04%4.004510879105.61678714.082238270.1515328172
30
zstd 1218.05415,099,705189,254,58545.59%21.93549772716.072556495.476340860.02766359909
31
zstd 1326.58415,099,705189,238,74745.59%14.89128526486.152109764.820281290.04074966023
32
zstd 1426.25415,099,705189,238,67345.59%15.08248285492.394251465.652566860.04023308502
33
zstd 1532.11415,099,705188,910,64445.51%12.32817189403.059446553.741259540.04922180032
34
zstd 1640.61415,099,705189,714,34745.70%9.748329863317.581167642.344155670.06224807976
35
xz 372.244415,099,705158,589,80438.21%5.479623601203.167242527.088965670.1107402367
36
xz 4104.193415,099,705155,540,12837.47%3.799390817142.544298819.005906510.1597137131
37
zstd 1747.40415,099,705138,537,63433.37%8.351686232333.860941844.514792240.07265776011
38
zstd 1860.68415,099,705134,843,09632.48%6.523464627264.260955135.234794020.09302032732
39
zstd 1988.36415,099,705133,355,94232.13%4.480345953182.458746224.327832820.1354392766
40
xz 5104.53415,099,705111,430,10426.84%3.787069294166.227700122.163693350.160233354
41
xz 6125.449415,099,705111,274,05226.81%3.155624416138.58255818.47767440.1922962732
42
xz 7107.796415,099,70590,521,58821.81%3.672399045172.293116922.972415590.1652366226
43
xz 8111.431415,099,70579,192,58019.08%3.552601407172.490239422.998698590.1708085837
44
xz 9110.726415,099,70572,281,89217.41%3.575221063177.159769223.621302560.1697279145
45
46
47
test.warc.gz:resorted by
48
labelrealusersysrssosizecsizeMB/s
49
xz-012.411004,853,76040,079,9013748546093.53%3.080029991
50
zstd-31.24003,784,70440,079,9013349397983.57%30.82513886
51
xz-115.8514010,919,93640,079,9012937129673.28%2.411556605
52
zstd-1911.6610061,497,34440,079,9011510487937.69%3.278145128
above suggests zstd -17 is twice the speed for most of the compression
53
xz-616.6215099,688,44840,079,9011400054434.93%2.299829855
54
xz-914.74131430,514,17640,079,9011037330825.88%2.593159579
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Loading...
Main menu