ABCDEFGHIJKLMNOPQRSTUVWXYZAAABACADAEAFAGAHAI
1
GPU type# of GPUs
teraflops per GPU
Total flops
Days to train GPT-4 with this much compute (2.1e25 based on https://epochai.org/data/epochdb/table)
Multiple of GPT-4 compute
Source on # of GPUs
For source on tflops, see "GPU flops"
Flops used to train
Calculation and source
2
GPT-3A10010243123.19E+17760.770.01
See page 8 https://arxiv.org/pdf/2104.04473.pdf
See page 8 https://arxiv.org/pdf/2104.04473.pdf
3
Llama-2 70BA10020483126.39E+17380.380.02
1.7 million GPU hours
https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md

Assume they trained on 2048 a100s, which they did for llama-1 (this would mean 30 days training)
8.10E+23
1.7 million GPU hours
https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md

Assume they trained on 2048 a100s, which they did for llama-1 (this would mean 30 days training)

Assume $10k for an a100, 19.5 TFLOPs for a100
4
GPT-4 (estimated)
H100100003,9583.96E+196.141.00
10k h100s https://techmonitor.ai/technology/cloud/google-ai-supercomputer-nvidia-h100
2.10E+25
$40K per GPU

10,000, H100 each one has 4000 TFLOP/s
5
Gemini ultraTPU770003933.03E+198.030.76
77,000 TPUs https://www.theinformation.com/articles/how-google-got-back-on-its-feet-in-ai-race?rc=c86l7t
43 days to train8.00E+25
Assume $14k for a TPU (you can’t actually buy them)

Based on pricing of nvidia v100 which is offer at similar hourly cost on google cloud
6
Eth miners
Nvidia GeForce RTX 3060 ti
163934431302.13E+210.1153.84
1k terahashes per second ethereum pow 2022. 3060 ti can do 61 megahashes per second (https://www.kryptex.com/en/hardware/nvidia-rtx-3060-ti) -> assume all GPUs are 3060 tis -> 7.5 million GPUs
Ethereum power consumption 78 TWh
https://ethereum.org/developers/docs/consensus-mechanisms/pos/pos-vs-pow#sustainability
44.5 million GPUs

1k hash rate https://www.theblock.co/data/on-chain-metrics/ethereum/ethereums-hash-rate-daily
33 million GPUs

RTX 3090 ti at $800
7
If all bittorrent users had an m1 macbook
M1 chip1700000002.64.42E+200.5511.17
170 million bittorrent users https://earthweb.com/bittorrent-statistics/
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100