Open_lm Training Data Token Counts

	A	B	C	E	F	G
1	Dataset	Source	Tokens
2	RPJ	RedPajama	1157.3B
3	S2ORC	S2ORC	48.9B
4	Pile of Law	Law	27.1B
5	DM Math	Math	4.4B		Tokens	DataMix
6	Pile	Arxiv	33.9B	RPJ	1157.3B	0.7258069201
7	Pile	PubMed Central	42.1B	Not RPJ	437.2B	0.2741930799
8	Pile	PhilPapers	1.2B	Total	1594.5B	1
9	Pile	Pubmed Abstracts	8.1B
10	Pile	NIH Exporter	721.2M
11	Pile	Pile-CC	53.2B
12	Pile	Books3	37.9B
13	Pile	Github	32.6B
14	Pile	OpenWebText2	29.9B
15	Pile	BookCorpus2	2.3B
16	Pile	Wikipedia	12.1B
17	Pile	Stackexchange	18.5B
18	Pile	FreeLaw	18.4B
19	Pile	Deepmind Mathematics	7.1B
20	Pile	HackerNews	2.0B
21	Pile	Ubuntu IRC	3.9B
22	Pile	Enron Emails	473.3M
23	Pile	EuroParl	2.9B
24	Pile	Gutenberg	6.9B
25	Pile	OpenSubtitles	5.5B
26	Pile	USPTO Backgrounds	9.5B
27	Pile	Youtube Subtitles	2.7B
28	RealNews	RealNews	25.0B
29		Total	1594.5B
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100