100M Encoding Crawl

	A	B	C	D	E
1	Encoding	Count	Percent	Notes

2	Total	100276107	100		This data is taken from a crawl of a 100M pages randomly sampled from Google's index of active/popular webpages on November 11.15.2013. Encodings were determined using a custom encoding detection library which uses a variety of factors including the HTTP headers, <meta> tag as well as the textual content of the page. In the case of Visual Hebrew <space> preceeding a <final hebrew letter> is also taken as an indication of that encoding. The fact that Visual Hebrew was not detected even once in 100M pages suggests that it's not used really at all in the modern web.
3	ASCII	1754081	1.749	actually Latin1
4	ASCII-7-bit	5355620	5.341	subset of UTF-8
5	Arabic	375	0
6	BIG5	277010	0.276
7	BIG5-CP950	1206	0.001
8	BIG5_HKSCS	1494	0.001
9	CP1250	204694	0.204
10	CP1251	2068582	2.063	MS Cyrillic
11	CP1252	6896831	6.878	MS Latin
12	CP1253	17185	0.017
13	CP1254	452298	0.451
14	CP1255	60871	0.061
15	CP1256	613119	0.611	Arabic
16	CP1257	24137	0.024
17	CP852	3402	0.003
18	CP866	5529	0.006
19	CP874	170611	0.17
20	CP932	47374	0.047
21	CSN_369103	79	0
22	EUC	78	0
23	EUC-CN	194	0
24	EUC-JP	596864	0.595	Japanese
25	GB	3299997	3.291	Chinese
26	GB18030	6328	0.006
27	GBK	3266016	3.257	Chinese
28	Greek	20768	0.021
29	HZ-GB-2312	14	0
30	ISO-2022-KR	36	0
31	ISO-8859-11	134931	0.135
32	ISO-8859-13	1711	0.002
33	ISO-8859-15	346878	0.346
34	ISO-8859-5	1009	0.001
35	ISO-8859-8-I	1068	0.001	Logical Hebrew
36	ISO-8859-8	0	0	Visual Hebrew
37	ISO_2022_CN	7	0
38	JIS	4125	0.004
39	KOI8R	17473	0.017
40	KOI8U	2416	0.002
41	KSC	836953	0.835	Korean
42	Latin2	563444	0.562	Eastern European
43	Latin3	1920	0.002
44	Latin4	3778	0.004
45	Latin5	118449	0.118
46	Latin6	7663	0.008
47	MACINTOSH	10367	0.01
48	SJS	1378745	1.375	Japanese
49	UTF-16BE	129	0
50	UTF-16LE	4917	0.005
51	UTF-32BE	14	0
52	UTF-32LE	3	0
53	UTF7	378	0
54	UTF8	69865897	69.674	wow!
55	X-BINARYENC	1122	0.001
56	invalid_encoding	1827917	1.823
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100