BDT model

	A	B	C	D	E	F
1	Background Data
2	FLOP to train xTrimo	6.00E+23			Epoch
3	Number of A100s for xtrimo in 2023	768			Chen et al
4
5	Cost to train xTrimo Epoch method 2				Method is from Epoch
6	Cost of A100 at release ($)	$15,000			For each hardware model, I used whatever unit price I could find that was reported closest to the release date of the hardware, as long as it was reported by a seemingly credible source.
7	Hardware replacement time (years)	2			Epoch
8	Cost per year to run	$7,500			Compute price trends.ipynb
9	Cost per second to run	$0.0002			Calc
10	Peak FP performance (FLOP/s)	3.1E+14			See A100 datasheet: https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-nvidia-us-2188504-web.pdf
11	Hardware price performance (FLOP/s per $)	1.3E+18			Calc
12	Utilization rate	35%			Epoch
13	Realised training compute per dollar (FLOP per $)	4.6E+17			Calc
14	FLOP to train xTrimo-100b	6.0E+23			Epoch
15	Hardware cost ($)	$1,306,722			Calc
16
17	Increase in data OOMs	7	Epoch database	4,883.72
18	Increase in compute OOMs	5
19	How much data increase for a given increase in compute	1.4
20
21	Cost to train xTrimo level performance over time from Moore's law
22	FLOP/s per $ doubles every x years	2.46			Epoch
23	OOMs per year	0.122			Epoch
24	Increase in hardware price performance per year (FLOP/s per $)	1.324341535
25	2023	$1,306,722
26	2024	$986,696
27	2025	$745,046
28	2026	$562,579
29	2027	$424,799
30	2028	$320,762
31	2029	$242,205
32	2030	$182,887
33
34	Projections of compute to train xTrimo-type model
35	Compute increase factor per year	9	0.3154648768	3.785578521
36	As a proportion	0.9542425094
37	FLOP to train BDTs	Year	realised training compute per dollar	Hardware cost
38	2023	6.0E+23	4.6E+17	$1,306,722	Compute in year n=2023
39	2024	5.4E+24	6.1E+17	$8,880,262	1
40	2025	4.9E+25	8.1E+17	$60,348,751	2
41	2026	4.4E+26	1.1E+18	$410,119,857	3	Seems implausible
42	2027	3.9E+27	1.4E+18	$2,787,104,845	4
43	2028	3.5E+28	1.9E+18	$18,940,690,854	5
44	2029	3.2E+29	2.5E+18	$128,717,716,053	6
45	2030	2.9E+30	3.3E+18	$874,743,722,593	7
46	2031	2.6E+31	4.3E+18	$5,944,609,675,159	8
47	2032	2.3E+32	5.8E+18	$40,398,557,059,954	9
48	2033	2.1E+33	7.6E+18	$274,541,728,003,826	10
49
50	Cost to train nucleotide transformer
51	FLOP to train Nucleotide transformer	1.2E+22
52	Hardware cost ($)	$26,134
53
54	Projections of compute to train Nucleotide transformer type model	Year	realised training compute per dollar	Hardware cost
55	2023	1.2E+22	4.6E+17	$26,134
56	2024	1.1E+23	6.1E+17	$177,605
57	2025	9.7E+23	8.1E+17	$1,206,975
58	2026	8.7E+24	1.1E+18	$8,202,397
59	2027	7.9E+25	1.4E+18	$55,742,097
60	2028	7.1E+26	1.9E+18	$378,813,817
61	2029	6.4E+27	2.5E+18	$2,574,354,321
62	2030	5.7E+28	3.3E+18	$17,494,874,452
63	2031	5.2E+29	4.3E+18	$118,892,193,503
64	2032	4.6E+30	5.8E+18	$807,971,141,199
65	2033	4.2E+31	7.6E+18	$5,490,834,560,077
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100