Llama vs DeepSeek

	A	B	D	E	G	H	J	K
1	Llama				DeepSeek
2
3	Batchsize	2048			Batchsize	15360
4	Seqlen	8192	Flops/Token	2.536564064	Seqlen	4096	Flops/Token	0.2662347571
5	Numlayers	126			Numlayers	61
6	Hiddensize	16384			Hiddensize	7168	w/ All Experts Active	4.32920671
7	Querygroups	8			Kvlora	512
8	Numheads	128			Qlora	1536	BF16 only	0.04789685453
9	FFN	53248			Numheads	128
10	Vocabsize	128256	Parameters (B)	405.8492436	Headdim	128	Parameters (B)	715.4077404
11	TP	8			Ropedim	64
12	PP	16			Moedim	2048
13	FSDP	128			Nexperts	256
14			BF16 One-way TP / Batch (TB)	484.8846278	Nrouted	8	FP8 One-way ZeRO to EP to 7 devices with Uniform Dist / Batch (TB)	195.7216597
15					Nshared	1
16			BF16 One-way PP / Batch (TB)	8246.337208	Vocabsize	129280	FP8 One-way PP / Batch (TB)	6.764573491
17					Nstages	16
18			FP32 Gradient Reduce Scatter	1.610714186	ZeRO	256	FP32 Gradient Inter-node Reduce Scatter / Batch (TB)	2.160574808
19					EP	64
20					Est. expert flops	1.059E+15
21					Intra-node interconnect	1.60E+11
22					Inter-node interconnect	5.00E+10	Local Expert One-way Comp Time / Batch	2.555079695
23					No grad acc local batch size	60	FP8 Same-node Expert Comm Time / Batch	9.63
24							FP8 Two diff nodes Expert Comm Time / Batch	17.62
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100