Common LLM Settings

	A	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P	Q	R	S	T	U	V	W
1	Model Name	Dataset	Tokenizer	Training Library	Pos Embed	Normalization	Norm H Params	Parallel Layers	Biases	Act Func	d_attn / d_ff	Optimizer	Optimizer H Params	LR Warm-Up	LR Decay	Precision	Clipping	Dropout	Weight Decay	Misc.	Date	Source
2	Recommended	Your favorite	GPT-NeoX-20B	GPT-NeoX	Rotary w/ ctx extension	LayerNorm	n/a	Yes	No	GeLU	4	AdamW	0.9, 0.95	Linear	Cosine to 10%	fp32 / bf16	1.0	0.0	0.10		1/1/2024	Stella Biderman

3	GPT-1	Unreleased	GPT-1	Unreleased	Learned	LayerNorm	n/a	No	Yes	GeLU	4	Adam	Not disclosed		Cosine to 0	Not disclosed	Not disclosed	0.1	Not disclosed		June 11, 2018	Paper
4	GPT-2	Unreleased	GPT-2	Unreleased	Learned	LayerNorm	n/a	No	Yes	GeLU	4	Adam	Not disclosed		Cosine to 0	Not disclosed	Not disclosed	0.1	Not disclosed		February 14, 2019
5	GPT-3	Unreleased	GPT-2	Unreleased	Learned	LayerNorm	n/a	No	Yes	GeLU	4	Adam	0.9, 0.95		Cosine to 10%	fp32 / fp16	1.0	0.0	0.10	Alternating sparse and dense layers	May 28, 2020	Paper
6	GPT-Neo	the Pile	GPT-2	GPT-Neo	Learned	LayerNorm	n/a	No	Yes	GeLU	4	Adam	0.9, 0.95		Cosine to ???	fp32 / bf16	1.0	0.0	0.10	Sliding window attention	March 22, 2021	Config file
7	GPT-J	the Pile	GPT-2	mesh-transformer-jax	Rotary	LayerNorm	n/a	Yes	Yes	GeLU	4	Adam	0.9, 0.999, 1e-08		Cosine to 10%	fp32 / bf16	1.0	0.0	0.10		June 8, 2021	Config file
8	FairSeq Dense	Unreleased	GPT-2	FairSeq	Sinusoidal	LayerNorm	n/a	No	Yes	GeLU	4	Adam	0.9, 0.98		Linear to 0	Pure fp16	1.0	0.1	0.01		December 20, 2021	Paper
9	Gopher	Unreleased	Unreleased	Unreleased	Transformer-XL-style	RMSNorm		No	Yes	GeLU	4	Adam	Not disclosed		Cosine to 10%	Pure bf16	1.0	Not disclosed	Not disclosed		December 8, 2021
10	GPT-NeoX	the Pile	GPT-NeoX-20B	GPT-NeoX	Rotary	LayerNorm	n/a	Yes	Yes	GeLU	4	Adam	0.9, 0.95		Cosine to 10%	fp32 / fp16	1.0	0.0	0.01		February 2, 2022	Paper	Default config behavior
11	PaLM	Unreleased	Unreleased	Unreleased	Rotary	LayerNorm	n/a	Yes	No	SwiGLU	4	Adafactor w/o factorization	0.9, 1 - step_num^-0.8		1/sqrt(step_num)	fp32 / bf16	1.0	0.0	lr^2.0		April 4, 2022	Paper
12	OPT	the Pile + Unreleased	GPT-2	Unreleased	Learned	LayerNorm	n/a	No	Yes	ReLU	4	AdamW	0.9, 0.95		Custom to 10%	fp32 / fp16	1.0	0.1	0.10		May 2, 2022	Paper	HF implementation
13	BLOOM	ROOTS	BLOOM	Megatron-DeepSpeed	Alibi	LayerNorm	n/a	No	Yes	GeLU	4	Adam	0.9, 0.95		Cosine to ???	fp32 / bf16	1.0	0.0	0.10		May 26, 2022	Paper	Config file
14	Pythia	the Pile	GPT-NeoX-20B	GPT-NeoX	Rotary	LayerNorm	n/a	Yes	Yes	GeLU	4	Adam	0.9, 0.95		Cosine to 10%	fp32 / fp16	1.0	0.0	0.10		December 10, 2022	Paper	Default config behavior
15	LLaMA	Unreleased	LLaMA	Unreleased	Rotary	RMSNorm		No	Yes	SwiGLU	8/3	AdamW	0.9, 0.95		Cosine to 10%	fp32 / fp16	1.0	0.0	0.10		February 24, 2023	Paper	Looking at the actual weights
16	RedPajama-INCITE	RedPajamas	GPT-NeoX-20B	GPT-NeoX	Rotary	LayerNorm	n/a	No	Yes	GeLU	4	Not disclosed		Not disclosed	Not disclosed	fp32/fp16	Not disclosed				May 5, 2023
17	MPT	C4, RP, Stack, S2ORC	GPT-NeoX-20B	Composer	Alibi	QKNorm		No	No	GeLU	4	LION	Not disclosed		Not disclosed	fp32 / bf16	Not disclosed	0.0	Not disclosed		May 5, 2023
18	CerebrasGPT	the Pile	GPT-2	Unreleased	Learned	LayerNorm	n/a	No	Yes	GeLU	4	Adam	0.9, 0.95, 1e-9	Linear over 375M tokens	Cosine to 10%	fp32 / bf16	1.0	0.0	0.10	Uses muP (but only up to 3B!)
19	LLaMA 2	Unreleased	LLaMA	Unreleased	Rotary w/ ctx extension	RMSNorm		No	No	SwiGLU	8/3	AdamW	0.9, 0.95		Cosine to 10%	fp32 / bf16	1.0	0.0	0.10		July 18, 2023	Paper	Looking at the actual weights
20	WeLab	Unreleased	WeLab	GPT-NeoX	Rotary	LayerNorm	n/a	Yes	Yes	GeLU	4	Adam	Not disclosed	Not disclosed	Not disclosed	Not disclosed	Not disclosed
21	Stable LM v2	the Pile + Unreleased	GPT-NeoX-20B	GPT-NeoX	Rotary w/ ctx extension	LayerNorm	n/a	Yes	Norms only	SwiGLU	8/3	Adam	0.9, 0.95		Cosine to 10%	fp32 / fp16	1.0	0.0	0.0001		August 5, 2023	Config file
22	Falcon-180B	Unreleased	Falcon	Unreleased	Rotary w/ ctx extension	LayerNorm	n/a	Yes	Not disclosed	GeLU	4	AdamW	Not disclosed		Cosine to 10%	fp32 / bf16	1.0	0.0	0.10		September 6, 2023	Model card
23	Mistral	Unreleased	Mistral	Unreleased	Rotary w/ ctx extension	RMSNorm		No	No	SwiGLU	8/3	Not disclosed			Not disclosed	fp32 / bf16	Not disclosed			Sliding window attention	September 27, 2023
24	Qwen	Unreleased	Qwen	Unreleased	Rotary w/ ctx extension	RMSNorm		No	QKV-only	SwiGLU	8/3	AdamW	0.9, 0.95, 1e-8	Not disclosed	Cosine to 10%	fp32 / bf16	1.0	0.1	0.10		September 24, 2023
25	Yi	Unreleased	LLaMA	Unreleased	Rotary	RMSNorm		No	No	SwiGLU	8/3	Not disclosed			Not disclosed	fp32 / bf16	Not disclosed
26	Amber	RP + RW + StarCoder	Amber	Amber-Train	Rotary	RMSNorm	1e-6	No	Yes	SwiGLU	8/3	AdamW	0.9, 0.95	Linear over 9.1B tokens	Cosine to 10%	fp32 / bf16	1.0	0.0	0.10			Paper
27	TeleChat	Unreleased	TeleChat	Megatron-DeepSpeed	Rotary w/ ctx extension	RMSNorm	1e-5	No	Yes	SwiGLU	8/3 < x < 4	Adam	0.9, 0.95, 1e-5	Linear over 1B tokens	Cosine to 10%	fp32 / bf16	1.0	0.1	0.0001	Batch size ramp-up	1/8/2024	Paper
28	InternLM	Unreleased	InternLM	InternLM	Rotary w/ ctx extension	RMSNorm	1e-6	Not disclosed	Not disclosed	SwiGLU	Not disclosed	Not disclosed	Not disclosed	Not disclosed	Not disclosed	Not disclosed	Not disclosed	Not disclosed	Not disclosed
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100