Training Comparison

	A	B	C	D	E	F	G	H	I	J	K	L	M	N	O
1	Run	Trainer	GPU	Train (h)	Max Mem (GiB)	Power (W)	Energy (kWh)	Tok/s	Steps	Optimizer	Max Seq	Batch Size	Grad Accum	Global	Notes

2	4090-unsloth-lora	unsloth	RTX 4090	0.93	19.28	393	0.36	2617.0	33	Adam8Bit	2048	2	64	128	24% faster than tortchtune compiled version
3	4090-torchtune-lora-compile	torchtune	RTX 4090	1.22	23.42	364	0.44	2280.2	74	PagedAdamW8bit	2048	1	64	64	15% faster tok/s; 11% faster wall clock
4	4090-torchtune-lora	torchtune	RTX 4090	1.38	23.22	357	0.49	1977.4	74	PagedAdamW8bit	2048	1	64	64
5	4090-axolotl-lora	axolotl	RTX 4090	1.42	23.11	338	0.48		73	adamw_bnb_8bit	2048	1	64	64
6	3090-unsloth-lora	unsloth	RTX 3090	2.15	19.11	270	0.58	1124.0	33	Adam8Bit	2048	2	64	128	28% faster than torchtune compiled version
7	7900xtx-torchtune-lora-compile	torchtune	7900 XTX	2.56	21.10	303	0.78	1073.6	74	PagedAdamW8bit	2048	1	64	64	14% faster than 3090/W7900
8	3090-torchtune-lora-compile	torchtune	RTX 3090	2.97	23.00	290	0.86	923.1	74	PagedAdamW8bit	2048	1	64	64	13% faster tok/s; 11% faster wall clock
9	W7900-torchtune-lora-compile	torchtune	W7900	2.98	21.12	240	0.71	920.2	74	PagedAdamW8bit	2048	1	64	64	got bnb sorted for less memory usage
10	3090-torchtune-lora	torchtune	RTX 3090	3.33	23.05	295	0.98	818.4	74	PagedAdamW8bit	2048	1	64	64
11	W7900-torchtune-lora-b2	torchtune	W7900	3.35	26.73	240	0.80	814.2	37	AdamW	2048	2	64	128	bitsandbytes issue w/ Adam8Bit...
12	W7900-torchtune-lora-b4	torchtune	W7900	3.50		240	0.84	777.3	18	AdamW	2048	4	64	256	b4 slower than b2
13	W7900-torchtune-lora	torchtune	W7900	3.54	21.04	240	0.85	769.1	74	PagedAdamW8bit	2048	1	64	64
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100