2024 LifeArchitect.ai data (shared) - OLD (corrupted)

	A	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P
1	MOVED:	https://docs.google.com/spreadsheets/d/1kc262HZSMAWI6FVsh0zJwbB-ooYvzhCHaHcNUiA0_hY/edit?gid=1158069878#gid=1158069878
2	Google corrupted this one...
3	Model	Lab	Playground	Parameters (B)	Tokens trained (B)	Ratio Tokens:Params (Chinchilla scaling≥20:1)	ALScore "ALScore" is a quick and dirty rating of the model's power. The formula is: Sqr Root of (Parameters x Tokens) ÷ 300. Any ALScore ≥ 1.0 is a powerful model in mid-2023.	MMLU	MMLU -Pro	GPQA	Training dataset	Announced ▼	Public?	Paper / Repo	Arch	Notes

4	Olympus	Amazon	https://lifearchitect.ai/olympus/	2000	40000							TBA				New related Titan details: '$65m training run. 200B dense model on 4T tokens of data across 13,760 NVIDIA A100 chips. 48 days to train. Training runs soon to cross $1B' https://importai.substack.com/p/import-ai-365-wmd-benchmark-amazon
5	GPT-5	OpenAI	https://lifearchitect.ai/gpt-5/	52500								TBA				Due 2024.
6	GPT-6	OpenAI	https://lifearchitect.ai/gpt-6/									TBA				Due 2025.
7	AuroraGPT (ScienceGPT)	Argonne National Laboratory	https://www.hpcwire.com/2023/11/13/training-of-1-trillion-parameter-scientific-ai-begins/	1000								TBA	🔴			https://tpc.dev/2023/11/10/tpc-announced-with-founding-partners/ powered by Intel Ponte Vecchio GPUs.
8	Grok-2	xAI	https://twitter.com/elonmusk/status/1773655245769330757									TBA				Due 2025.
9	MAI-1	Microsoft	https://arstechnica.com/information-technology/2024/05/microsoft-developing-mai-1-language-model-that-may-compete-with-openai-report/	500	10000	20:1	7.5					TBA		https://www.reuters.com/technology/microsoft-readies-new-ai-model-compete-with-google-openai-information-reports-2024-05-06/	Dense	Due 2024. MAI=Microsoft artificial intelligence. MSFT CTO statement: https://archive.md/XRSgS
10	GPT-4o mini	OpenAI	https://chatgpt.com/	8	6000	750:1	0.7	82		40.2	🆆 📚⬆ 🕸 🌋	Jul/2024	🟢	https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/	MoE	Omnimodel. "OpenAI would not disclose exactly how large GPT-4o mini is, but said it’s roughly in the same tier as other small AI models, such as Llama 3 8b, Claude Haiku and Gemini 1.5 Flash." https://techcrunch.com/2024/07/18/openai-unveils-gpt-4o-mini-a-small-ai-model-powering-chatgpt/ "tested GPT-4o to identify potential risks, which we have addressed and plan to share the details of in the forthcoming GPT-4o system card and Preparedness scorecard." And related paper about instruction hierarchy: https://arxiv.org/abs/2404.13208
11	NeMo	Mistral	https://huggingface.co/mistralai/Mistral-Nemo-Base-2407	12	2000	167:1	0.5	68			🆆 📚⬆ 🕸 🌋	Jul/2024	🟢	https://mistral.ai/news/mistral-nemo/	Dense	With NVIDIA. "Drop-in replacement of Mistral 7B". "trained using Megatron-LM, part of NVIDIA NeMo, with 3,072 H100 80GB Tensor Core GPUs" https://blogs.nvidia.com/blog/mistral-nvidia-ai-model/
12	Codestral Mamba	Mistral	https://huggingface.co/mistralai/mamba-codestral-7B-v0.1	7	2000	286:1	0.4				🆆 📚⬆ 🕸 🌋	Jul/2024	🟢	https://mistral.ai/news/codestral-mamba/	Dense	"Unlike Transformer models, Mamba models offer the advantage of linear time inference and the theoretical ability to model sequences of infinite length."
13	Mathstral	Mistral	https://huggingface.co/mistralai/mathstral-7B-v0.1	7	2000	286:1	0.4	63.47			🆆 📚⬆ 🕸 🌋	Jul/2024	🟢	https://mistral.ai/news/mathstral/	Dense	"We’re contributing Mathstral to the science community to bolster efforts in advanced mathematical problems requiring complex, multi-step logical reasoning."
14	SpreadsheetLLM	Microsoft		1760	13000	8:1	15.9				🆆 📚⬆ 🕸 🌋	Jul/2024	🔴	https://arxiv.org/abs/2407.09025v1	Dense	Notable finetune of GPT4-0125-preview "outperforming the vanilla approach by 25.6% in GPT4’s in-context learning setting"
15	next-gen	DeepL	https://www.deepl.com/en/translator								🌋	Jul/2024	🟢	https://www.deepl.com/en/blog/next-gen-language-model	Dense	"Built using our own groundbreaking, specialized LLM technology and proprietary training data, designed specifically for translation"
16	SmolLM	Hugging Face	https://huggingface.co/collections/HuggingFaceTB/smollm-6695016cad7167254ce15966	1.7	1000	589:1	0.1	39.97			🆆 📚⬆ 🕸 🌋 ⚛️	Jul/2024	🟢	https://huggingface.co/blog/smollm	Dense	Dataset includes new Cosmopedia v2 synthetic data. 135M and 360M models,each trained on 600B tokens from Smollm-Corpus. 1.7B model trained on 1T tokens from Smollm-Corpus.
17	Mockingbird	Vectara	https://vectara.com/platform/	9	1000	112:1	0.3				🆆 📚⬆ 🕸 🌋 ⚛️	Jul/2024	🟢	https://vectara.com/blog/mockingbird-a-rag-and-structured-output-focused-llm/	Dense	"At <10B parameters it's an LLM trained to provide optimal results for RAG and structured outputs."
18	FLAMe	Google DeepMind		24	1000	42:1	0.5				👥	Jul/2024	🔴	https://arxiv.org/abs/2407.10817v1	Dense	LLM-as-a-Judge autorater. Foundational Large Autorater Models (FLAMe). Uses an instruction-tuned PaLM-2-24B model. Unrelated to Microsoft FLAME Jan/2023.
19	H2O-Danube3-4B	H2O.ai	https://h2o.ai/platform/danube/personal-gpt/	4	6000	1,500:1	0.5	55.18			🆆 📚⬆ 🕸 🌋 ⚛️	Jul/2024	🟢	https://arxiv.org/abs/2407.09276	Dense	Runs natively and fully offline on mobile phone. "H2O-Danube3 is a family of decoder only LLM models that use the general Llama model architecture adopting core principles from Llama 2 and Mistral with custom parameters determining the shape of each layer and total parameter count. We use the Mistral tokenizer..." MMLU for chat=54.74, base=55.18 via https://huggingface.co/h2oai/h2o-danube3-4b-base
20	Causal Axioms	Microsoft		0.067	1.2	18:1	0.0				⚛️	Jul/2024	🔴	https://arxiv.org/abs/2407.07612v1	Dense	"the training dataset follows a specific structure, we develop a custom tokenizer. Alphanumeric node names are tokenized at a character level, while special terms such as ‘causes’, ‘Does’, ‘cause’, ‘Yes’, and ‘No’ are tokenized at the word level... Our training setup consists of around 175k instances of sequential chains with size of chains ranging from 3 to 6 nodes... All models are trained for 100 epochs. [LifeArchitect.ai estimate is 12 tokens per node x 6 nodes x 175,000 instances x 100 epochs = 1.26B tokens]" Based on GPT-2 arch.
21	SenseNova 5.5	SenseTime	https://platform.sensenova.cn/home#/home	600	10000	17:1	8.2				⚛️	Jul/2024	🟢	https://www.sensetime.com/en/news-detail/51168278?categoryId=1072	MoE	"The model training was based on over 10TB tokens [sic, taken as 10T tokens instead of 10TB=2T tokens] of high-quality training data, including a large amount of synthetically-generated reasoning chain data, which help to enhance its reasoning capabilities." & "The updates include SenseNova 5o, the first real-time multimodal model in China, which provides a new AI interaction model on par with GPT-4o’s streaming interaction capabilities"
22	Helium 7B	Kyutai	https://moshi.chat/	7	1000	143:1	0.3				⚛️	Jul/2024	🟢	https://youtu.be/hm2IJSKcYvo	Dense	"1. The model is fine-tuned on 100K transcripts generated by Helium itself. 2. These transcripts are highly detailed, heavily annotated with emotion and style, and conversational. 3. Text to Speech Engine is further fine-tuned on 20 hours of audio recorded by Alice and licensed."
23	InternLM2.5	Shanghai AI Laboratory/SenseTime	https://huggingface.co/collections/internlm/internlm25-66853f32717072d17581bc13	7	2600	372:1	0.4	72.8		38.4	🆆 📚⬆ 🕸 🌋	Jul/2024	🟢	https://github.com/InternLM/InternLM/blob/main/model_cards/internlm2.5_7b.md	Dense	"The release of InternLM2.5 series contains 7B model size for now and we are going to release the 1.8B and 20B versions soon"
24	Llama 3 405B	Meta AI	https://wabetainfo.com/whatsapp-beta-for-android-2-24-14-7-whats-new/	405	15000	38:1	8.2	84.8		48	🆆 📚⬆ 🕸 🌋	Jun/2024	🟡		Dense	Waiting on release outside of WhatsApp Android as of 1/Jul/2024.
25	ERNIE 4.0 Turbo	Baidu	https://yiyan.baidu.com/								🆆 📚⬆ 🕸 🌋	Jun/2024	🟢	https://www.reuters.com/technology/artificial-intelligence/baidu-launches-upgraded-ai-model-says-user-base-hits-300-mln-2024-06-28/	Dense	"Ernie Bot has reached 300 million users since its launch [on 16/Mar/2023, public Aug/2023]" Jun/2024
26	Gemma 2	Google DeepMind	https://huggingface.co/google/gemma-2-27b-it	27	13000	482:1	2.0	75.2			🆆 📚⬆ 🕸 🌋	Jun/2024	🟢	https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf	Dense	Announce: https://blog.google/technology/developers/google-gemma-2/
27	CriticGPT	OpenAI									👥	Jun/2024	🔴	https://cdn.openai.com/llm-critics-help-catch-llm-bugs-paper.pdf	Dense	"LLM Critics Help Catch LLM Bugs" Announce: https://openai.com/index/finding-gpt4s-mistakes-with-gpt-4/
28	4M-21	Apple	https://github.com/apple/ml-4m/	3							🌋	Jun/2024	🟢	https://arxiv.org/abs/2406.09406	Dense	Vision model based on T5-XXL. Modalities: RGB, Caption, Bounding boxes, Semantic segmentation, Depth, Human poses, Surface normals, CLIP, DINOv2, ImageBind, Metadata, Canny edges, SAM edges, SAM instances, Color palette. Project page: https://4m.epfl.ch/
29	ESM3	EvolutionaryScale	https://github.com/evolutionaryscale/esm	98	771	8:1	0.9				🌋	Jun/2024	🟡	https://www.evolutionaryscale.ai/blog/esm3-release	Dense	Biology large language model: "sequence, structure, and function are all masked and predicted during training, ESM3 can generate in all three modalities." 1.4B only released.
30	PanGu 5.0 Super	Huawei	https://www.huaweicloud.com/intl/en-us/product/modelarts.html	1000	20000	20:1	14.9				🌋	Jun/2024	🟡	https://www.huaweicentral.com/huawei-cloud-unveils-pangu-large-model-5-0/	MoE	https://x.com/faridofanani96/status/1804079517193113850/photo/1
31	Claude 3.5 Sonnet	Anthropic	https://poe.com/Claude-3.5-Sonnet					90.4	72.83	67.2	🆆 📚⬆ 🕸 🌋	Jun/2024	🟢	https://www.anthropic.com/news/claude-3-5-sonnet	Dense	Model card: https://www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf
32	DeepSeek-Coder-V2	DeepSeek-AI	https://chat.deepseek.com/coder	236	10200	35:1	4.6	79.2	63.63		🆆 📚⬆ 🕸 🌋	Jun/2024	🟢	https://github.com/deepseek-ai/DeepSeek-Coder-V2/blob/main/paper.pdf	MoE	DeepSeek-V2 with additional 6 trillion tokens.
33	DCLM-Baseline 7B 2.6T	International	https://huggingface.co/apple/DCLM-Baseline-7B	7	2600	372:1	0.4	63.7			🕸 🌋	Jun/2024	🟡	https://arxiv.org/abs/2406.11794	Dense	New dataset: 240T tokens: 8× larger than previous SOTA dataset. DCLM-Pool is 240T, DCLM-Baseline is 3.8T: "we combine our 3.8T DCLM-BASELINE with the StarCoder and ProofPile2 data to arrive at a 4.1T token dataset. We train a 7B model for 2.5T tokens" and "We release the DCLM benchmark, framework, models, and datasets at https://datacomp.ai/dclm."
34	Nemotron-4-340B	NVIDIA	https://build.nvidia.com/nvidia/nemotron-4-340b-instruct	340	9000	27:1	5.8	81.1			🆆 📚⬆ 🕸 🌋	Jun/2024	🟢	https://d1qx31qr3h6wln.cloudfront.net/publications/Nemotron_4_340B_8T.pdf	Dense	Open-source equiv of Mar/2023 GPT-4 (1760MoE≈340B, 13T), same param count but 2x the tokens of May/2023 PaLM 2 (340B, 3.6T), competitor to Nov/2023 Grok-1 (314B, 6T). Trained on 6,144 H100s. ~1.3TB for inference. 50+ natural and 40+ coding languages. Trained between December 2023 and May 2024. MMLU 0-shot for instruct=78.7, 5-shot for base=81.1. Permalink for paper: https://research.nvidia.com/publication/2024-06_nemotron-4-340b
35	Apple On-Device model Jun/2024	Apple	https://github.com/apple/corenet/tree/main/projects/openelm	3.04	1500	494:1	0.2	26.76			🆆 📚⬆ 🕸 🌋	Jun/2024	🟢	https://arxiv.org/abs/2404.14619	Dense	https://lifearchitect.ai/apple/ Likely to be the Apple OpenELM model (Apr/2024). "two of these models — a ~3 billion parameter on-device language model, and a larger server-based language model available with Private Cloud Compute". https://machinelearning.apple.com/research/introducing-apple-foundation-models The server-based model is possibly Ferret, although it is more properly called a multimodal model (not just language). It could also be Apple GPT based on their Ajax framework: https://archive.md/f3C0r
36	MatMul-Free LM	UCSC	https://github.com/ridgerchu/matmulfreellm	2.7	100	38:1	0.1				🆆 📚⬆ 🕸 🌋	Jun/2024	🟢	https://arxiv.org/abs/2406.02528	Dense	"we explore alternative methods for mixing tokens without relying on matrix multiplications." Compared with Transformer++ based on Llama-2, not to be confused with the pre-GPT-3 American Express Transformer++ paper from 2/Mar/2020. Instead, Transformer++ is defined in the Mamba paper: 'Transformer++: A Transformer with an improved architecture, namely rotary positional encodings (Su et al. 2021) and SwiGLU MLP (Shazeer 2020)'
37	Luna	Galileo	https://www.rungalileo.io/blog/introducing-galileo-luna-a-family-of-evaluation-foundation-models	0.44	162	369:1	0.0				🆆 📚⬆ 🕸 🌋	Jun/2024	🟢	https://arxiv.org/abs/2406.00975	Dense	Based on DeBERTA-large (440M). RoBERTa=162B token dataset.
38	Qwen2	Alibaba	https://huggingface.co/spaces/Qwen/Qwen2-72B-Instruct	72	7000	98:1	2.4	84.2	55.6	37.9	🆆 📚⬆ 🕸 🌋	Jun/2024	🟢	https://arxiv.org/abs/2407.10671	Dense	Instruct MMLU=82. Instruct GPQA=41.9. https://qwenlm.github.io/blog/qwen2/
39	Qwen2-57B-A14B	Alibaba	https://github.com/QwenLM/Qwen2?tab=readme-ov-file	57	4500	79:1	1.7	76.5	43	34.3	🆆 📚⬆ 🕸 🌋	Jun/2024	🟢	https://arxiv.org/abs/2407.10671	MoE	https://qwenlm.github.io/blog/qwen2/
40	Skywork MoE 16x13B	Kunlun Tech	https://huggingface.co/Skywork/Skywork-MoE-Base	146				77.4			🆆 📚⬆ 🕸 🌋	Jun/2024	🟢	https://github.com/SkyworkAI/Skywork-MoE/blob/main/skywork-moe-tech-report.pdf	MoE	CN + EN. "(MoE) model with 146 billion parameters, 16 experts, and 22 billion activated parameters. This model is initialized from the pre-existing dense checkpoints of our Skywork-13B model."
41	Mamba-2	CMU	https://github.com/state-spaces/mamba	2.7	300	112:1	0.1				🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://arxiv.org/abs/2405.21060	Dense	Analysis: https://tridao.me/blog/2024/mamba2-part1-model/
42	MAP-Neo	International	https://map-neo.github.io/	7	4500	643:1	0.6	58.14			🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://arxiv.org/abs/2405.19327	Dense	"first fully open-sourced bilingual LLM with comparable performance to existing state-of-the-art LLMs... we open-source all details to reproduce our MAP-Neo, where the cleaned pre-training corpus, data cleaning pipeline, checkpoints, and well-optimized training/evaluation framework are provided."
43	K2	LLM360	https://huggingface.co/LLM360/K2	65	1400	22:1	1.0	64.8			🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://www.llm360.ai/blog/several-new-releases-to-further-our-mission.html	Dense	"K2-65B is a fully reproducible LLM outperforming Llama 2 70B using 35% less compute."
44	Codestral	Mistral	https://huggingface.co/mistralai/Codestral-22B-v0.1	22	2000	91:1	0.7				🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://mistral.ai/news/codestral/	Dense	Fluent in 80+ programming languages
45	Aya-23-35B	Cohere	https://huggingface.co/spaces/CohereForAI/aya-23	35	4800	138:1	1.4				🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://drive.google.com/file/d/1YKBPo61pnl97C1c_1C2ZVOnPhqf7MLSc/view	Dense
46	Yi-XLarge	01-ai	https://platform.01.ai/	2000	20000	10:1	21.1	85.1		48.2	🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://www.aixinzhijie.com/article/6845768	MoE	Still training as of May/2024: https://appserversrc.8btc.cn/FnDYlEC4STBhphu6M3NL4CKH43FW dead link, use: https://finance.china.com.cn/roll/20240513/6116857.shtml
47	Yi-Large	01-ai	https://platform.01.ai/	1000	15000	15:1	12.9	83.8	58.1	43.5	🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://www.aixinzhijie.com/article/6845768	Dense
48	Chameleon	Meta AI	https://ai.meta.com/resources/models-and-libraries/chameleon-downloads/?gk_enable=chameleon_web_flow_is_live	34	9200	271:1	1.9	65.8			🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://arxiv.org/abs/2405.09818	Dense	Multimodal
49	Sparse Llama 7B	Cerebras	https://huggingface.co/spaces/neuralmagic/llama-2-sparse-transfer-chat-deepsparse	7	145	21:1	0.1				🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://arxiv.org/abs/2405.03594	Hybrid	https://www.cerebras.net/blog/introducing-sparse-llama-70-smaller-3x-faster-full-accuracy "For the 50% sparse model, we utilized 45 billion tokens of pretraining data, while an additional 100 billion tokens were used for the 70% model. This represents approximately 2% to 8% of the original 2 trillion tokens used to train the base Llama-2 model."
50	Gemini 1.5 Flash	Google DeepMind	https://aistudio.google.com/app/prompts/new_chat					78.9	59.1	39.5	🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://goo.gle/GeminiV1-5	MoE	1M context length.
51	GPT-4o	OpenAI	https://chatgpt.com/					88.7	72.6	53.6	🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://openai.com/index/hello-gpt-4o/	MoE	Omnimodel. ‘[GPT-4o is] likely an early checkpoint of GPT-5’. https://twitter.com/drjimfan/status/1790089671365767313 ELO: https://twitter.com/LiamFedus/status/1790064963966370209 Demo: https://youtu.be/DQacCB9tDaw
52	Falcon 2 11B	TII	https://huggingface.co/tiiuae/falcon-11B	11	5500	500:1	0.8	58.37			🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://www.tii.ae/news/falcon-2-uaes-technology-innovation-institute-releases-new-ai-model-series-outperforming-metas	Dense	Announce: https://www.tii.ae/news/falcon-2-uaes-technology-innovation-institute-releases-new-ai-model-series-outperforming-metas
53	Fugaku-LLM	Fujitsu	https://huggingface.co/Fugaku-LLM/Fugaku-LLM-13B-instruct	13	380	30:1	0.2				🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://www.fujitsu.com/global/about/resources/news/press-releases/2024/0510-01.html	Dense	Japanese. CPU trained: 158,976+ A64FX CPUs (7M+ cores), zero GPUs. https://en.wikipedia.org/wiki/Fugaku_(supercomputer)
54	Yi 1.5 34B	01-ai	https://huggingface.co/01-ai/Yi-1.5-34B-Chat	34.4	3600	105:1	1.2	76.8	52.3		🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://github.com/01-ai/Yi-1.5	Dense	Uses 600B more training tokens than Yi 1.0 (Nov/2023).
55	YOCO	Microsoft	https://github.com/microsoft/unilm/tree/master/YOCO	3	1600	534:1	0.2				🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://arxiv.org/abs/2405.05254	Dense	With Tsingua. You Only Cache Once (YOCO). Long context "1M context length with near-perfect needle retrieval accuracy"
56	DeepSeek-V2	DeepSeek-AI	https://chat.deepseek.com/	236	8100	35:1	4.6	78.5	54.8		🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://arxiv.org/abs/2405.04434	MoE	Huge dataset, 12% Chinese "Therefore, we acknowledge that DeepSeek-V2 still has a slight gap in basic English capabilities with LLaMA3 70B".
57	ChuXin	Independent	https://huggingface.co/chuxin-llm/Chuxin-1.6B-Base	1.6	2300	1,438:1	0.2	41.07			🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://arxiv.org/abs/2405.04828	Dense	"results on the ”Needle In A Haystack”(NIAH) tests indicate that ChuXin-1M performs well across all context window lengths up to 1M."
58	RWKV-v6 Finch	RWKV	https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-2	7.63	2500	328:1	0.5				🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://huggingface.co/BlinkDL/rwkv-6-world	Dense	https://twitter.com/BlinkDL_AI/status/1787834625211158562
59	xLSTM	ELLIS		2.7	15	6:1	0.0				🆆 📚⬆ 🕸 🌋	May/2024	🔴	https://arxiv.org/abs/2405.04517	Dense	New method LSTM to xLSTM, see also RNNs. Code/weights doesn't seem to be released. https://github.com/AI-Guru/xlstm-resources
60	Granite Code	IBM	https://github.com/ibm-granite/granite-code-models	34	3500	103:1	1.1				🌋	May/2024	🟢	https://github.com/ibm-granite/granite-code-models/blob/main/paper.pdf	Dense	Dataset: publicly available datasets (e.g., GitHub Code Clean, Starcoder data), public code repositories, and issues from GitHub.
61	Qwen-Max	Alibaba	https://chat.lmsys.org/	300	6000	20:1	4.5				🆆 📚⬆ 🕸 🌋	May/2024	🟢	https://help.aliyun.com/zh/dashscope/developer-reference/model-introduction	Dense	https://twitter.com/JustinLin610/status/1787584325367529509
62	Med-Gemini-L 1.0	Google DeepMind	https://twitter.com/alan_karthi/status/1785117450528264216	1500	30000	20:1	22.4				🆆 📚⬆ 🕸 🌋	May/2024	🔴	https://arxiv.org/abs/2404.18416	Dense	Med-Gemini-M 1.0 and Med-Gemini-L 1.0 (Pro and Ultra finetunes) "For language tasks that require less complex reasoning, such as summarizing medical notes and creating referral letters, we introduce Med-Gemini-M 1.0 by fine-tuning the Gemini 1.0 Pro model. For other tasks that require more advanced reasoning, we introduce Med-Gemini-L 1.0 by fine-tuning the Gemini 1.0 Ultra model using a self-training method to enable the models to efficiently use web search."
63	Tele-FLM	BAAI	https://huggingface.co/CofeAI/Tele-FLM	52	2000	39:1	1.1	64			🆆 📚⬆ 🕸 🌋	Apr/2024	🟢	https://arxiv.org/abs/2404.16645	Dense	Also known as FLM-2. "We will open-source a 1T model checkpoint, namely Tele-FLM-1T, to advance further training and research." Discussion paper Jul/2024: https://arxiv.org/abs/2407.02783
64	Qwen-1.5 110B	Alibaba	https://huggingface.co/spaces/Qwen/Qwen1.5-110B-Chat-demo	111	3000	28:1	1.9	80.4	49.9	35.9	🆆 📚⬆ 🕸 🌋	Apr/2024	🟢	https://qwenlm.github.io/blog/qwen1.5-110b/	Dense	Worse performance on GPQA (72B=36.3, 110B=35.9).
65	Arctic	Snowflake AI Research	https://arctic.streamlit.app/	480	3500	8:1	4.3	67.3			🆆 📚⬆ 🕸 🌋	Apr/2024	🟢	https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/	Hybrid	"Arctic uses a unique Dense-MoE Hybrid transformer architecture. It combines a 10B dense transformer model with a residual 128×3.66B MoE MLP resulting in 480B total and 17B active parameters chosen using a top-2 gating."
66	SenseNova 5.0	SenseTime		600	10000	17:1	8.2	84.78		42.93	🆆 📚⬆ 🕸 🌋	Apr/2024	🟢	https://news.futunn.com/en/post/41290101/a-large-shangtang-multi-modal-model-with-600-billion-parameters	MoE	GPT-4 scale; low media coverage; no demo in Western world. https://www.techinasia.com/sensetime-pauses-trading-stock-rises-30-model-launch
67	OpenELM	Apple	https://huggingface.co/apple/OpenELM-3B-Instruct	3.04	1500	494:1	0.2	26.76			🆆 📚⬆ 🕸 🌋	Apr/2024	🟢	https://arxiv.org/abs/2404.14619	Dense	On-device model (laptop, phone). Open-source Efficient Language Models (OpenELM). https://venturebeat.com/ai/apple-releases-openelm-small-open-source-ai-models-designed-to-run-on-device/
68	phi-3-medium	Microsoft	https://huggingface.co/microsoft/Phi-3-medium-128k-instruct	14	4800	343:1	0.9	78.2	55.7		⚛️	Apr/2024	🟢	https://arxiv.org/abs/2404.14219	Dense	Preview only, benchmarks being investigated as of May/2024.
69	phi-3-mini	Microsoft	https://huggingface.co/microsoft/Phi-3-mini-128k-instruct	3.8	3300	869:1	0.4	68.8	45.7		⚛️	Apr/2024	🟢	https://arxiv.org/abs/2404.14219	Dense	"phi3-mini can be quantized to 4-bits so that it only occupies ≈ 1.8GB of memory. We tested the quantized model by deploying phi-3-mini on iPhone 14 with A16 Bionic chip running natively on-device and fully offline achieving more than 12 tokens per second."
70	Llama 3 70B	Meta AI	https://meta.ai/	70	15000	215:1	3.4	82	52.8		🆆 📚⬆ 🕸 🌋	Apr/2024	🟢	https://ai.meta.com/blog/meta-llama-3/	Dense	Instruct MMLU-Pro=56.2
71	HLAT	Amazon		7	1800	258:1	0.4	41.318			🆆 📚⬆ 🕸 🌋	Apr/2024	🔴	https://arxiv.org/abs/2404.10630	Dense	HLAT=High-quality LLM pre-trained on AWS Trainium. Same arch as Llama 7B. The pre-training is performed up to 64 Amazon EC2 trn1.32xlarge instances with totalling up to 1024 AWS Trainium accelerators. Read more about Trainium: https://www.aboutamazon.com/news/aws/what-you-need-to-know-about-the-aws-ai-chips-powering-amazons-partnership-with-anthropic
72	Idefics2	Hugging Face	https://huggingface.co/HuggingFaceM4/idefics2-8b	8.4							🆆 🕸	Apr/2024	🟢	https://huggingface.co/blog/idefics2	Dense	Clone of Flamingo now using Mistral 7B. Named after Asterix and Obelix's dog Idefix (Image-aware Decoder Enhanced à la Flamingo with Interleaved Cross-attentionS)
73	Reka Core	Reka AI	https://poe.com/RekaCore	300	10000	34:1	5.8	83.2		38.2	🆆 📚⬆ 🕸 🌋	Apr/2024	🟢	https://publications.reka.ai/reka-core-tech-report.pdf	Dense	https://www.reka.ai/news/reka-core-our-frontier-class-multimodal-language-model
74	WizardLM-2-8x22B	Microsoft	https://huggingface.co/MaziyarPanahi/WizardLM-2-8x22B-GGUF	141	2000	15:1	1.8				🆆 📚⬆ 🕸 🌋	Apr/2024	🟢	https://wizardlm.github.io/WizardLM2/	MoE	Base model = mistral-8x22b.
75	Pile-T5	EleutherAI	https://huggingface.co/EleutherAI/pile-t5-xxl	11	2000	182:1	0.5	53.84			🆆 📚⬆ 🕸 🌋	Apr/2024	🟢	https://blog.eleuther.ai/pile-t5/	Dense
76	Zephyr 141B-A35B	Hugging Face H4	https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1	35	2000	58:1	0.9				🆆 📚⬆ 🕸 🌋	Apr/2024	🟢	https://arxiv.org/abs/2403.07691	MoE	mixtral-8x22b finetune using Odds Ratio Preference Optimization (ORPO).
77	Rerank 3	Cohere	https://docs.cohere.com/reference/rerank-1	104	4000	39:1	2.1				📚 🕸	Apr/2024	🟢	https://txt.cohere.com/rerank-3/	Dense	RAG + semantic search, possibly backed by Command-R+.
78	gpt-4-turbo-2024-04-09	OpenAI	https://chat.openai.com/					86.5	63.7	49.1	🆆 📚⬆ 🕸 🌋	Apr/2024	🟢	https://cdn.openai.com/papers/gpt-4.pdf	MoE	This is such a significantly better model that I've added it here. This GPQA=46.5%, old GPT-4 GPQA=36%. https://twitter.com/EpochAIResearch/status/1778463039932584205 MMLU scores are unclear, but may have improved by 1%: https://twitter.com/OpenAI/status/1778602770784002136. Final benchmarks are here: https://archive.md/6Cc0Z
79	MiniCPM-2.4B	Tsinghua	https://github.com/OpenBMB/MiniCPM/	2.4	1100	459:1	0.2				🆆 📚⬆ 🕸 🌋	Apr/2024	🟢	https://arxiv.org/abs/2404.06395	Dense	MoE option=https://huggingface.co/openbmb/MiniCPM-MoE-8x2B
80	Ferret-UI	Apple	https://github.com/apple/ml-ferret	13	2000	154:1	0.5				🆆 📚⬆ 🕸 👥	Apr/2024	🟢	https://arxiv.org/abs/2404.05719	Dense	Vicuna base, multimodal. Extension of Ferret from Oct/2023.
81	mixtral-8x22b	Mistral	https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1	141	2000	15:1	1.8	77.75			🆆 📚⬆ 🕸 🌋	Apr/2024	🟢	https://mistral.ai/news/mixtral-8x22b/	MoE	MoE=22Bx8, seq=65536.
82	Sailor	Sail	https://huggingface.co/sail	7	200	29:1	0.1				🆆 📚⬆ 🕸 🌋	Apr/2024	🟢	https://arxiv.org/abs/2404.03608v1	Dense	SEA languages. Based on Qwen-1.5. https://github.com/sail-sg/sailor-llm "Generally Sailor models consume around 200B tokens, completing a full pass through the SailCraft corpus once. However, the Sailor-0.5B model undergoes training with 400B tokens, equivalent to 2 epochs."
83	JetMoE-8B	MIT	https://www.lepton.ai/playground/chat?model=jetmoe-8b-chat	8	1250	157:1	0.3	49.2			🆆 📚⬆ 🕸 🌋	Apr/2024	🟢	https://huggingface.co/jetmoe/jetmoe-8b	MoE
84	Eurus	Tsinghua	https://huggingface.co/collections/openbmb/eurus-660bc40bec5376b3adc9d1c5	70	2000	29:1	1.2				🆆 📚⬆ 🕸 🌋	Apr/2024	🟢	https://huggingface.co/collections/openbmb/eurus-660bc40bec5376b3adc9d1c5	Dense	Fine-tune of Mistral-7B and CodeLlama-70B.
85	Command-R+	Cohere	https://huggingface.co/spaces/CohereForAI/c4ai-command-r-plus	104	4000	39:1	2.1	75.7			📚 🕸	Apr/2024	🟢	https://huggingface.co/CohereForAI/c4ai-command-r-plus	Dense	purpose-built to excel at real-world enterprise use cases. Announce with no arch details: https://txt.cohere.com/command-r-plus-microsoft-azure/
86	Viking	Silo AI		33	2000	61:1	0.9				🌋	Apr/2024	🟢	https://www.silo.ai/blog/viking-7b-13b-33b-sailing-the-nordic-seas-of-multilinguality	Dense	Viking uses an architecture similar to Llama 2, with flash attention, rotary embeddings, grouped query attention and supports a 4k sequence length'
87	OLMo-Bitnet-1B	Nous Research	https://huggingface.co/NousResearch/OLMo-Bitnet-1B	1	60	60:1	0.0				🌋	Apr/2024	🟢	https://arxiv.org/abs/2402.17764	Dense	1.58-bit quantized (ternary weights) means we can run a 70B model in ~14GB VRAM. See also BitNet b1.58
88	Aurora-M	International	https://huggingface.co/collections/aurora-m/aurora-m-models-65fdfdff62471e09812f5407	15.5	2035	132:1	0.6				🌋	Mar/2024	🟢	https://arxiv.org/abs/2404.00399	Dense
89	ReALM-3B	Apple		3	134	45:1	0.1				🌋	Mar/2024	🔴	https://arxiv.org/abs/2403.20329	Dense	FLAN-T5 (Oct/2022) finetune.
90	Qwen1.5-MoE-A2.7B	Alibaba	https://qwenlm.github.io/blog/qwen-moe/	14.3	1500	105:1	0.5	62.5			🆆 📚⬆ 🕸 🌋	Mar/2024	🟢	https://qwenlm.github.io/blog/qwen-moe/	MoE	MoE. "Of particular significance is the fact that, through upcycling, the necessity for training an equivalent volume of tokens as in the original model has been eliminated." I assumed half of the original 3T tokens
91	Grok-1.5	xAI	https://grok.x.ai/	314	6000	20:1	4.6	81.3			🆆 📚⬆ 🕸 🌋	Mar/2024	🟢	https://x.ai/blog/grok-1.5	Dense	Context=128k.
92	Jamba	AI21	https://huggingface.co/ai21labs/Jamba-v0.1	52	5000	97:1	1.7	67.4			🆆 📚⬆ 🕸 🌋	Mar/2024	🟢	https://arxiv.org/abs/2403.19887	MoE	MoE. Open weights, licensed under Apache 2.0. Announce: https://arxiv.org/abs/2403.19887
93	DBRX	MosaicML	https://huggingface.co/spaces/databricks/dbrx-instruct	132	12000	91:1	4.2	73.7			🆆 📚⬆ 🕸 🌋	Mar/2024	🟢	https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm	MoE	MoE. Trained for $10M on 3,072 NVIDIA H100s connected by 3.2Tbps Infiniband.
94	Stable Code Instruct 3B	Stability AI	https://huggingface.co/stabilityai/stable-code-instruct-3b	2.7	560	208:1	0.1				🌋	Mar/2024	🟢	https://stability.ai/news/introducing-stable-code-instruct-3b	Dense	Context window=16,384. Trained on The Stack dataset.
95	EvoLLM-JP	Sakana AI	https://huggingface.co/SakanaAI/EvoLLM-JP-v1-10B	10	800	80:1	0.3				🆆 📚⬆ 🕸 🌋	Mar/2024	🟢	https://arxiv.org/abs/2403.13187	Dense	Japanese. Model merge 'our EvoLLM-JP-A is a merge of shisa-gamma-7b-v1, Arithmo2-Mistral-7B, and Abel7B-002' https://sakana.ai/evolutionary-model-merge/
96	RakutenAI-7B	Rakuten Group	https://huggingface.co/Rakuten/RakutenAI-7B	7	3000	429:1	0.5	61.31			🆆 📚⬆ 🕸 🌋	Mar/2024	🟢	https://arxiv.org/abs/2403.15484	Dense	Japanese. Mistral 7B derivative.
97	Parakeet	Independent	https://colab.research.google.com/drive/1gI8CM9Bz9ov0-E6aL2jF808rE56UtZyF?usp=sharing	0.378	3	8:1	0.0				🆆 📚⬆ 🕸 🌋	Mar/2024	🟢	https://news.ycombinator.com/item?id=39745700#39745702	Dense	Tiny model (378M) for testing
98	RWKV-v5 EagleX	RWKV	https://huggingface.co/recursal/EagleX_1-7T	7.52	1700	227:1	0.4	40.14			🆆 📚⬆ 🕸 🌋	Mar/2024	🟢	https://substack.recursal.ai/p/eaglex-17t-soaring-past-llama-7b	Dense	Built on the RWKV-v5 architecture (a linear transformer with 10-100x+ lower inference cost)
99	MM1	Apple		30	2010	67:1	0.8				🌋	Mar/2024	🔴	https://arxiv.org/abs/2403.09611	Dense	VLM, outperforms Flamingo 80B (Apr/2022) across benchmarks. 2T text tokens + ~10B+ other text (estimate). Unreleased.
100	RFM-1	Covariant	https://vimeo.com/921866765	8	160	20:1	0.1				🆆 📚⬆ 🕸 🌋	Mar/2024	🟡	https://covariant.ai/insights/introducing-rfm-1-giving-robots-human-like-reasoning-capabilities/	Dense	Commercial, multimodal for robotics
101	Command-R	Cohere	Cohere	35	700	20:1	0.5		37.9		📚 🕸	Mar/2024	🟢	https://txt.cohere.com/command-r/	Dense	RAG and tool use
102	DeepSeek-VL	DeepSeek-AI	https://github.com/deepseek-ai/DeepSeek-VL?tab=readme-ov-file	7	2000	286:1	0.4				🆆 📚⬆ 🕸 🌋	Mar/2024	🟢	https://arxiv.org/abs/2403.05525	Dense	Vision, based on DeepSeek-LLM-7B
103	AnyGPT	Fudan University	https://junzhan2000.github.io/AnyGPT.github.io/	7	2000	286:1	0.4				🆆 📚⬆ 🕸 🌋	Mar/2024	🟢	https://arxiv.org/abs/2402.12226	Dense	Llama 2 7B backbone with new matrices ('reshaping the embedding matrix and prediction layer')
104	Stable Beluga 2.5	Stability AI		70	2000	29:1	1.2				🆆 📚⬆ 🕸 🌋	Mar/2024	🟢	https://stability.ai/news/putting-the-ai-supercomputer-to-work	Dense	Mentioned in Stability release about Intel chips 11/Mar/2024, availablity unknown
105	Inflection-2.5	Inflection AI	https://inflection.ai/inflection-2	1200	20000	17:1	16.3	85.5		38.4	🆆 📚 ⬆ 🕸	Mar/2024	🟢	https://inflection.ai/inflection-2-5	Dense
106	Apollo	SRIBD/CUHK	https://apollo.llmzoo.com/	7	2500	358:1	0.4				🆆 📚🕸 🌋	Mar/2024	🟢	https://arxiv.org/abs/2403.03640	Dense	Qwen 1.8B as base. Medical focus.
107	Claude 3 Opus	Anthropic	https://claude.ai/	2000	40000	20:1	29.8	88.2	68.5	59.5	🆆 📚⬆ 🕸 🌋	Mar/2024	🟢	https://www.anthropic.com/claude-3-model-card	Dense	Original MMLU=86.8 (GPT-4=86.4). Original GPQA=50.4. 200k context, 1M for researchers.
108	Nemotron-4 15B	NVIDIA		15	8000	534:1	1.2	64.2			🆆 📚⬆ 🕸 🌋	Feb/2024	🟢	https://arxiv.org/abs/2402.16819	Dense
109	TowerLLM	Unbabel	https://unbabel.com/meet-towerllm/	7	1020	29:1	1.2				🆆 📚⬆ 🕸 🌋	Feb/2024	🟢	https://arxiv.org/abs/2402.17733	Dense	Commercial product, Llama-2 as base.
110	Hawk	Google DeepMind		7	300	43:1	0.2	35			🆆 📚🕸 🌋	Feb/2024	🟢	https://arxiv.org/abs/2402.19427	Dense	MMLU=35. RNN.
111	Griffin	Google DeepMind		14	300	22:1	0.2	49.5			🆆 📚🕸 🌋	Feb/2024	🟢	https://arxiv.org/abs/2402.19427	Dense	MMLU=49.5. RNN.
112	BitNet b1.58	Microsoft	https://huggingface.co/1bitLLM/bitnet_b1_58-xl	70	2000	29:1	1.2				🆆 📚🕸 🌋	Feb/2024	🟢	https://arxiv.org/abs/2402.17764	Dense
113	Samba-1	SambaNova	https://trysambanova.ai/	1400	20000	15:1	17.6				🌋	Feb/2024	🟡	https://sambanova.ai/press/secure-one-trillion-parameter-generative-ai-model-for-the-enterprise	CoE	CoE: Collection of experts: Llama2 7B / 13B / 70B Mistral 7B DeepSeek Coder 1.3B / 6.7B / 33B Falcon 40B DePlot CLIP Llava
114	Aya-101	Cohere	https://huggingface.co/CohereForAI/aya-101	13	1000	77:1	0.4				📚 🕸	Feb/2024	🟢	https://cohere.com/research/aya/aya-model-paper.pdf	Dense	mT5 base.
115	Cosmo-1B	HF	https://huggingface.co/HuggingFaceTB/cosmo-1b	1.8	180	100:1	0.1				⚛️	Feb/2024	🟢	https://huggingface.co/blog/cosmopedia	Dense	Synthetic data (25B tokens of synthetic data for 6 epochs + code). MMLU=32.4
116	Poro	Silo AI	https://huggingface.co/LumiOpen/Poro-34B	34.2	1000	30:1	0.6				🌋	Feb/2024	🟢	https://www.silo.ai/blog/viking-7b-13b-33b-sailing-the-nordic-seas-of-multilinguality	Dense	Uses a BLOOM architecture with ALiBi embeddings to allow for context window extrapolation. While model architecture for the initial model has been kept simple, future models under progress will support additional capabilities, such as flash attention, rotary embeddings and grouped query attention.'
117	StarCoder 2	HF/ServiceNow		15	4300	287:1	0.8				🌋	Feb/2024	🟢	https://arxiv.org/abs/2402.19173	Dense	The Stack v2=900B tokens, 5 epochs to 4.3T tokens
118	530B	ByteDance		530	300	1:1	1.3				🆆 📚⬆ 🕸 🌋	Feb/2024	🔴	https://arxiv.org/abs/2402.15627	Dense	Trained using 12,288 A100 GPUs, replicating MT-NLG size
119	175B	ByteDance		175	300	2:1	0.8				🆆 📚⬆ 🕸 🌋	Feb/2024	🔴	https://arxiv.org/abs/2402.15627	Dense	Trained using 12,288 A100 GPUs, replicating GPT-3 size
120	Mistral Small	Mistral	https://chat.mistral.ai/chat	7	3000	429:1	0.5	72.2			🆆 📚⬆ 🕸 🌋	Feb/2024	🟢	https://mistral.ai/news/mistral-large/	Dense	Optimised for latency and cost.
121	Mistral Large	Mistral	https://poe.com/Mistral-Large	300	8000	27:1	5.2	81.2			🆆 📚⬆ 🕸 🌋	Feb/2024	🟢	https://mistral.ai/news/mistral-large/	Dense	MMLU=81.2 (same as Flan-PaLM 2 340B, higher than PaLM 2 340B MMLU=78.3), 32k context window. API only (not open source).
122	Hanooman	Reliance		40							🌋	Feb/2024	🟢	https://www.hanooman.ai/	Dense	11 Indian languages like Hindi, Tamil, and Marathi
123	Ask	Apple		20							🌋	Feb/2024	🔴	https://www.macrumors.com/2024/02/22/applecare-advisors-testing-new-ask-tool/	Dense	Internal employee model only
124	Reka Edge	Reka AI	https://chat.reka.ai/	7	4500	643:1	0.6	63.1			🆆 📚⬆ 🕸 🌋	Feb/2024	🟢	https://publications.reka.ai/reka-core-tech-report.pdf	Dense
125	Reka Flash	Reka AI	https://poe.com/RekaFlash	21	5000	239:1	1.1	73.5		34	🆆 📚⬆ 🕸 🌋	Feb/2024	🟢	https://publications.reka.ai/reka-core-tech-report.pdf	Dense	My testing shows very poor performance equiv with tiny model
126	Gemma	Google DeepMind	https://labs.pplx.ai/	7	6000	858:1	0.7	64.3	33.7		🆆 📚⬆ 🕸 🌋	Feb/2024	🟢	https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf	Dense	MMLU=64.3 (Llama 2 70B=68.9, ChatGPT 20B=70). Text only. Probably dense. Largest trained dataset (6T) besides frontier models.
127	Gemini 1.5 Pro	Google DeepMind	https://aistudio.google.com/app/prompts/new_chat	1500	30000	20:1	22.4	85.9	69	46.2	🆆 📚⬆ 🕸 🌋	Feb/2024	🟢	https://goo.gle/GeminiV1-5	MoE	Sparse MoE. Context window=1M and 10M for research
128	Qwen-1.5 72B	Alibaba	https://huggingface.co/spaces/Qwen/Qwen1.5-72B-Chat	72	3000	42:1	1.5	77.5	52.6	36.3	🆆 📚⬆ 🕸 🌋	Feb/2024	🟢	https://qwenlm.github.io/blog/qwen1.5/	Dense
129	GOODY-2	BRAIN	https://www.goody2.ai/chat								🌋	Feb/2024	🟢	https://www.goody2.ai/goody2-modelcard.pdf	Dense	Satire (and hilarious). Probably Llama 2 with aggressive prompt. Wired interview: https://archive.md/toxHq
130	Natural-SQL-7B	ChatDB		7	2000	286:1	0.4				🌋	Feb/2024	🟢	https://huggingface.co/chatdb/natural-sql-7b	Dense	Based on DeepSeek-Coder 6.7B.
131	Sea-Lion	AI Singapore	https://aisingapore.org/aiproducts/sea-lion/	7.5	980	131:1	0.3				🆆 📚⬆ 🕸 🌋	Feb/2024	🟢	https://huggingface.co/aisingapore/sealion7b	Dense	MPT base. MMLU=26.87. Southeast Asian languages like Thai, Vietnamese and Bahasa Indonesia. https://www.computerweekly.com/feature/Sea-Lion-explained-Southeast-Asias-first-large-language-model
132	TimesFM	Google		0.2	100	500:1	0.0				🆆 🌋	Feb/2024	🔴	https://blog.research.google/2024/02/a-decoder-only-foundation-model-for.html	Dense	Time-series forecasting only. 'a large pretraining corpus of 100B real world time-points' may be more than 100B tokens.
133	OLMo	Allen AI	https://huggingface.co/allenai/OLMo-7B	7	2500	358:1	0.4				🆆 📚⬆ 🕸 🌋	Feb/2024	🟢	https://allenai.org/olmo/olmo-paper.pdf	Dense	Open Language Model (OLMo)
134	FLOR-6.3B	Cerebras	https://huggingface.co/projecte-aina/FLOR-6.3B	6.3	481	77:1	0.2				🆆 📚⬆ 🕸 🌋	Jan/2024	🟢	https://www.cerebras.net/press-release/cerebras-systems-and-barcelona-supercomputing-center-train-industry-leading-multilingual-spanish-catalan-english-llm	Dense	Spanish, Catalan. Bloom-7.1B (341B tok) + continued pre-training on 140B tok. Trained on Cerebras hardware.
135	Weaver	AIWaves.cn	https://www.wawawriter.com/	34	2018	60:1	0.9				📚	Jan/2024	🟢	https://arxiv.org/abs/2401.17268	Dense	Llama? 'All Weaver models are initialized from powerful open-source LLMs.' English waitlist: https://www.wawawriter.com/en/
136	miqu 70b	Mistral	https://huggingface.co/miqudev/miqu-1-70b	70	3000	43:1	1.5				🆆 📚⬆ 🕸 🌋	Jan/2024	🟢	https://huggingface.co/miqudev/miqu-1-70b	Dense	Leaked, proper version soon: https://venturebeat.com/ai/mistral-ceo-confirms-leak-of-new-open-source-ai-model-nearing-gpt-4-performance/
137	iFlytekSpark-13B	iFlyTek	https://gitee.com/iflytekopensource/iFlytekSpark-13B	13	3000	231:1	0.7	63.02			🆆 📚⬆ 🕸 🌋	Jan/2024	🟢	https://www.ithome.com/0/748/030.htm	Dense	pre-trained on a massive high-quality data set with a total of more than 3 trillion tokens, and then fine-tuned on fine-tuned diversified alignment data.'
138	Xinghuo 3.5 (Spark)	iFlyTek		200	4000	20:1	3.0				🆆 📚⬆ 🕸 🌋	Jan/2024	🟢	https://www.laitimes.com/en/article/6f50u_6vhbm.html	Dense	GPT-4 competitor. https://www.shine.cn/biz/tech/2401304331/
139	MGIE	Apple	https://github.com/tsujuifu/pytorch_mgie	7	2000	286:1	0.4				🆆 📚⬆ 🕸 🌋	Jan/2024	🟢	https://openreview.net/forum?id=S1RKWSyZ2Y	Dense	MLLM and diffusion model initialized from LLaVA-7B (Llama 2 + Vicuna) + StableDiffusion-v1.5.
140	CodeLlama-70B	Meta AI	https://huggingface.co/codellama/CodeLlama-70b-hf	70	2000	29:1	1.2				🆆 📚⬆ 🕸 🌋	Jan/2024	🟢	https://ai.meta.com/research/publications/code-llama-open-foundation-models-for-code/	Dense	Paper link is to 34B from Aug/2023. This 70B model finished training Jan/2024.
141	RWKV-v5 Eagle 7B	RWKV	https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-2	7.52	1100	147:1	0.3	33.21			🆆 📚⬆ 🕸 🌋	Jan/2024	🟢	https://blog.rwkv.com/p/eagle-7b-soaring-past-transformers	Dense	Built on the RWKV-v5 architecture (a linear transformer with 10-100x+ lower inference cost), Trained on 1.1 Trillion Tokens across 100+ languages. Original paper: https://arxiv.org/abs/2305.13048
142	MaLA-500	LMU	https://huggingface.co/MaLA-LM/mala-500	10	2000	200:1	0.5				🆆 📚⬆ 🕸 🌋	Jan/2024	🟢	https://arxiv.org/abs/2401.13303	Dense	Extends Llama 2 7B to 10B using 534 languages.
143	MambaByte	Cornell	https://github.com/kyegomez/MambaByte	0.972	37.5	39:1	0.0				📚🌋	Jan/2024	🔴	https://arxiv.org/abs/2401.13660	Dense	Used bytes instead of tokens. 4 bytes≈1 token, so 150B bytes≈37.5B tokens
144	DeepSeek-Coder	DeepSeek-AI	https://coder.deepseek.com/	33	2000	61:1	0.9				🆆 📚⬆ 🕸 🌋	Jan/2024	🟢	https://arxiv.org/abs/2401.14196	Dense	surpasses existing closed-source models like Codex and GPT-3.5... permissive license that allows for both research and unrestricted commercial use.'
145	FuseLLM	Tencent	https://github.com/fanqiwan/FuseLLM	7	2000	286:1	0.4				🆆 📚⬆ 🕸 🌋	Jan/2024	🟢	https://arxiv.org/abs/2401.10491	Dense	Fusion of Llama-2-7B (2T tok), OpenLLaMA-7B (2T tok), and MPT-7B (1T tok).
146	Fuyu-Heavy	Adept		120	5000	42:1	2.6				🆆 📚⬆ 🕸 🌋	Jan/2024	🟡	https://www.adept.ai/blog/adept-fuyu-heavy	Dense	Fuyu-Heavy is the world’s third-most-capable multimodal model, behind only GPT4-V and Gemini Ultra, which are 10-20 times bigger.' Token estimate is based on Adept Persimmon-8B using many more tokens.
147	Orion-14B	OrionStar	https://github.com/OrionStarAI/Orion	14	2500	179:1	0.6	69.6			🆆 📚⬆ 🕸 🌋	Jan/2024	🟢	https://arxiv.org/abs/2401.12246	Dense	English, Chinese, Japanese, Korean, and other languages.
148	InternLM2	Shanghai AI Laboratory/SenseTime	https://github.com/InternLM/InternLM	20	2600	130:1	0.8	67.7			🆆 📚⬆ 🕸 🌋	Jan/2024	🟢	https://arxiv.org/abs/2403.17297	Dense
149	GLM-4	Zhipu AI (Tsinghua)	https://open.bigmodel.cn/	200	4000	20:1	3.0				🆆 📚⬆ 🕸 🌋	Jan/2024	🟢	https://pandaily.com/zhipu-ai-unveils-glm-4-model-with-advanced-performance-paralleling-gpt-4/	Dense	Best Chinese model to date based on analysis. Follows OpenAI roadmap. MMLU=81.5. 'hundreds of billions of parameters' https://www.chatglm.cn/
150	DeepSeekMoE	DeepSeek-AI		16	2000	125:1	0.6				🆆 📚⬆ 🕸 🌋	Jan/2024	🔴	https://arxiv.org/abs/2401.06066	MoE	MoE activated parameters is 10-15% of dense, so I need to rethink ALScore for MoE. 'preliminary efforts to scale up DeepSeekMoE to 145B'
151	DeepSeek	DeepSeek-AI	https://chat.deepseek.com/	67	2000	30:1	1.2				🆆 📚⬆ 🕸 🌋	Jan/2024	🟢	https://arxiv.org/abs/2401.02954	Dense	Chinese/English. Outperforms Llama 2. MMLU=71.3 outperforms GPT-3.5.
152	LLaMA Pro	Tencent	https://huggingface.co/TencentARC/LLaMA-Pro-8B	8.3	2080	251:1	0.4				🆆 📚⬆ 🕸 🌋	Jan/2024	🟢	https://arxiv.org/abs/2401.02415	Dense	We pre-train LLAMA PRO’s expanded blocks on 80B tokens using open-source code and math data for 2830 GPU Hours (16 NVIDIA H800 GPUs for about 7 days).
153	TinyLlama	SUTD/Independent	https://github.com/jzhang38/TinyLlama	1.1	3000	2,728:1	0.2				🆆 📚⬆ 🕸 🌋	Jan/2024	🟢	https://arxiv.org/abs/2401.02385	Dense	Overtrained' using 2,727 tokens per parameter. Dataset was 1T: 3 epochs to 3T seen. Singapore
154	DocLLM	JPMorgan		7	2000	286:1	0.4				🆆 📚⬆ 🕸 🌋	Jan/2024	🔴	https://arxiv.org/abs/2401.00908	Dense	Document spatial layout structure.
155	Unified-IO 2	Allen AI	https://unified-io-2.allenai.org/	7	1000	143:1	0.3				🆆 📚⬆ 🕸 🌋	Dec/2023	🟢	https://arxiv.org/abs/2312.17172	Dense	600TB dataset (plus 120+ fine-tuning datasets) includes '1B imagetext pairs, 1T text tokens, 180M video clips, 130M interleaved image & text, 3M 3D assets, and 1M agent trajectories.'
156	WaveCoder-DS-6.7B	Microsoft		6.7							🆆 📚⬆ 🕸 🌋	Dec/2023	🔴	https://arxiv.org/abs/2312.14187	Dense	To obtain WaveCoder models, We choose StarCoder-15B, CodeLLaMa (7B and 13B), DeepseekCoder-6.7B as the base model and fine-tune all the base model for 3 epochs
157	YunShan	Huawei		7	1748	250:1	0.4				🆆 📚⬆ 🕸 🌋	Dec/2023	🔴	https://arxiv.org/abs/2312.17276	Dense	Finance + law fine-tune of PanGu-π
158	PanGu-Pi	Huawei		7	1600	229:1	0.4				🆆 📚⬆ 🕸 🌋	Dec/2023	🔴	https://arxiv.org/abs/2312.17276	Dense	Dense, named PanGu-π
159	YAYI 2	Wenge	https://huggingface.co/wenge-research/yayi2-30b	30	2650	89:1	0.9	80.5			🆆 📚⬆ 🕸 🌋	Dec/2023	🟢	https://arxiv.org/abs/2312.14862	Dense	Dataset=240TB filtered to 10.6TB for 2.65T tokens
160	Emu2	BAAI	https://baaivision.github.io/emu2/	37	4	1:1	0.0				🆆 📚⬆ 🕸 🌋	Dec/2023	🟢	https://arxiv.org/abs/2312.13286	Dense	VLM. Gemini clone. Outperforms Flamingo 80B. The Pile for text, but only sampled 3.6B tokens (1.4% of the dataset).
161	MedLM	Google DeepMind	https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/medlm								🆆 📚⬆ 🕸 🌋	Dec/2023	🟡	https://cloud.google.com/static/vertex-ai/docs/generative-ai/medlm/MedLM-model-card.pdf	Dense	Available to 'white-listed' orgs only.
162	SOLAR-10.7B	Upstage AI	https://huggingface.co/upstage/SOLAR-10.7B-v1.0	10.7							🆆 📚⬆ 🕸 🌋	Dec/2023	🟢	https://arxiv.org/abs/2312.15166	Dense	South Korean. Llama-2 arch. SOTA for its size (Dec/2023).
163	DeciLM-7B	Deci	https://console.deci.ai/infery-llm-demo	7.04	200	29:1	0.1				🆆 📚⬆ 🕸 🌋	Dec/2023	🟢	https://deci.ai/blog/introducing-DeciLM-7b-the-fastest-and-most-accurate-7b-large-language-model-to-date	Dense	4.4x times faster than Mistral. English only.
164	Mistral-medium	Mistral	https://poe.com/	180	3500	20:1	2.6	75.3			🆆 📚⬆ 🕸 🌋	Dec/2023	🟢	https://mistral.ai/news/la-plateforme/	Dense	MMLU=75.3% (GPT-3.5-turbo 20B=70%, Llama 2 70B=68.9%)
165	mixtral-8x7b-32kseqlen	Mistral	https://www.together.ai/blog/mixtral	46.7	8000	172:1	2.0	70.6	43.3		🆆 📚⬆ 🕸 🌋	Dec/2023	🟢	https://arxiv.org/abs/2401.04088	MoE	MoE=7Bx8, aka mistral-small. 'Concretely, Mixtral has 45B total parameters but only uses 12B parameters per token. It, therefore, processes input and generates output at the same speed and for the same cost as a 12B model.'
166	StripedHyena 7B	Together	https://api.together.xyz/playground/language/togethercomputer/StripedHyena-Hessian-7B	7.65							🌋	Dec/2023	🟢	https://www.together.ai/blog/stripedhyena-7b	Dense	RedPajama (C4), new arch beyond just Transformers
167	NexusRaven-V2 13B	Nexusflow.ai	https://huggingface.co/spaces/Nexusflow/NexusRaven-V2-Demo								🌋	Dec/2023	🟢	https://github.com/nexusflowai/NexusRaven-V2/tree/master	Dense	Based on CodeLlama. 'surpasses GPT-4 by up to 7% in function calling success rates in human-generated use cases involving nested and composite functions.'
168	Gemini Ultra 1.0	Google DeepMind	https://deepmind.google/technologies/gemini/	1500	30000	20:1	22.4	90.04		35.7	🆆 📚⬆ 🕸 🌋	Dec/2023	🟢	https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf	Dense	Original MMLU=83.7. Chinchilla (20:1), dense, maybe 600B-2000T.
169	Mamba	CMU	https://huggingface.co/havenhq/mamba-chat	2.8	300	108:1	0.1	26.2			🆆 📚⬆ 🕸 🌋	Dec/2023	🟢	https://arxiv.org/abs/2312.00752	Dense	The Pile, new arch beyond just Transformers. 2.7B MMLU=26.2. 7B MMLU=33.3.
170	LVM-3B	Berkeley/JHU		3	420	140:1	0.1				🖼	Dec/2023	🔴	https://arxiv.org/abs/2312.00785	Dense	Paper is 25MB. First Large Vision Model (LVM); no text. Based on Llama and LAION 5B (1.49B).
171	SeaLLM-13b	Alibaba	https://github.com/damo-nlp-sg/seallms	13	2000	154:1	0.5				🆆 📚⬆ 🕸 🌋	Dec/2023	🟢	https://arxiv.org/abs/2312.00738	Dense	Llama 2 for Southeast Asian (SEA) languages: Vietnamese 🇻🇳, Indonesian 🇮🇩, Thai 🇹🇭, Malay 🇲🇾, Khmer🇰🇭, Lao🇱🇦, Tagalog🇵🇭 and Burmese🇲🇲
172	pplx-70b-online	Perplexity	https://labs.perplexity.ai/	70	2000	29:1	1.2				🆆 📚⬆ 🕸 🌋	Nov/2023	🟢	https://blog.perplexity.ai/blog/introducing-pplx-online-llms	Dense	Web access. Higher 'freshness' and 'truth' scores.
173	SeamlessM4T-Large v2	Meta AI	https://seamless.metademolab.com/expressive/	2.3							🌋	Nov/2023	🟢	https://ai.meta.com/research/publications/seamless-multilingual-expressive-and-streaming-speech-translation/	Dense	Based on NLLB and older models. https://github.com/facebookresearch/seamless_communication
174	Q-Transformer	Google DeepMind	https://qtransformer.github.io/								🌋	Nov/2023	🔴	https://qtransformer.github.io/assets/qtransformer.pdf	Dense	Robotics, builds on RT-1
175	Yuan 2.0	IEIT	https://github.com/IEIT-Yuan/Yuan-2.0/blob/main/README-EN.md	102.6	288	3:1	0.6				🆆 📚⬆ 🕸 🌋	Nov/2023	🟢	https://arxiv.org/abs/2311.15786	Dense	Chinese + EN dataset include The Pile: DM, arxiv, wikipedia, book3, stack exchange, Freelaw and medical
176	MEDITRON	EPFL	https://huggingface.co/epfl-llm/meditron-70b	70	2000	29:1	1.2				🆆 📚⬆ 🕸 🌋	Nov/2023	🟢	https://arxiv.org/abs/2311.16079	Dense	Llama 2 trained on med data using NVIDIA Megatron-LM. "outperforms Llama-2-70B, GPT-3.5 (text-davinci-003, 8-shot), and Flan-PaLM on multiple medical reasoning tasks."
177	Transformers-Arithmetic	Microsoft		0.1	0.3	3:1	0.0				⬆	Nov/2023	🔴	https://arxiv.org/abs/2311.14737	Dense	Proving maths is not memorized. Uses GPT-2-style model. Sébastien Bubeck
178	Starling-7B	Berkeley	https://huggingface.co/berkeley-nest/Starling-LM-7B-alpha	7	2000	286:1	0.4		37.9		🆆 📚⬆ 🕸 🌋	Nov/2023	🟢	https://starling.cs.berkeley.edu/	Dense	Llama 2 7B -> OpenChat 7B -> Starling-7B (RLAIF)
179	Inflection-2	Inflection AI	https://inflection.ai/inflection-2	1200	20000	17:1	16.3				🆆 📚 ⬆ 🕸	Nov/2023	🟢	https://inflection.ai/inflection-2	Dense	“now the 2nd best LLM in the world”. Finished training 19/Nov/2023, waiting for fine-tuning and release.
180	Claude 2.1	Anthropic	https://claude.ai/	130	2500	20:1	1.9	78.5			🆆 📚⬆ 🕸 🌋	Nov/2023	🟢	https://www.anthropic.com/index/claude-2-1	Dense	Less hallucinations, 200k context length, tool use
181	TÜLU 2	Allen AI	https://huggingface.co/allenai/tulu-2-dpo-70b	70	2000	29:1	1.2				🆆 📚⬆ 🕸 🌋	Nov/2023	🟢	https://arxiv.org/abs/2311.10702	Dense	Llama 2 finetune with RLHF direct preference optimization (DPO).
182	Nemotron-3 22B	NVIDIA	https://huggingface.co/nvidia/nemotron-3-8b-base-4k	22	3800	173:1	1.0	54.4			🆆 📚⬆ 🕸 🌋	Nov/2023	🟢	https://developer.nvidia.com/blog/nvidia-ai-foundation-models-build-custom-enterprise-chatbots-and-co-pilots-with-production-ready-llms/	Dense	8B released, 22B internal.
183	Nemotron-2 43B	NVIDIA		43	3800	89:1	1.3				🆆 📚⬆ 🕸 🌋	Nov/2023	🔴	https://arxiv.org/abs/2311.09528	Dense	Used to train HelpSteer (16/Nov/2023): https://arxiv.org/abs/2311.09528
184	Orca 2	Microsoft		13	2001	154:1	0.5				🆆 📚⬆ 🕸 🌋	Nov/2023	🟡	https://arxiv.org/abs/2311.11045	Dense	Llama 2 13B (2T) -> Orca 2 (GPT-4 finetune). Still an imitation model, overhyped: The False Promise of Imitating Proprietary LLMs https://arxiv.org/abs/2305.15717
185	Phi-2	Microsoft	https://replicate.com/lucataco/phi-2	2.7	1400	519:1	0.2				⚛️	Nov/2023	🟢	https://huggingface.co/microsoft/phi-2	Dense	https://twitter.com/SebastienBubeck/status/1724854157004190095
186	Florence-2	Microsoft	https://huggingface.co/microsoft/Florence-2-large	0.771							🌋	Nov/2023	🟢	https://arxiv.org/abs/2311.06242	Dense	VLM, Flamingo alt
187	Mirasol3B	Google DeepMind		3							🌋	Nov/2023	🔴	https://arxiv.org/abs/2311.05698	Dense	Combiner + autoregressive transformer for video/audio/text
188	OtterHD-8B	NTU	https://github.com/Luodian/Otter	8	737	93:1	0.3				🆆 📚⬆ 🕸 🌋	Nov/2023	🟢	https://arxiv.org/abs/2311.04219	Dense	Evolution of Persimmon-9.3B and Fuyu 8B
189	Gauss	Samsung	https://koreajoongangdaily.joins.com/news/2023-11-08/business/tech/Samsung-unveils-generative-AI-model-Gauss/1908889	7								Nov/2023	🟡	https://koreajoongangdaily.joins.com/news/2023-11-08/business/tech/Samsung-unveils-generative-AI-model-Gauss/1908889	Dense	Gauss Language specializing in generating texts, Gauss Code on software and code description and Gauss Image for image creation.
190	Grok-1	xAI	https://grok.x.ai/	314	6000	20:1	4.6				🆆 📚⬆ 🕸 🌋	Nov/2023	🟢	https://github.com/xai-org/grok-1	Dense	Context window=8192. UI: https://twitter.com/TobyPhln/status/1721053802235621734
191	Grok-0	xAI	https://grok.x.ai/	33	2000	61:1	0.9				🆆 📚⬆ 🕸 🌋	Nov/2023	🔴	https://web.archive.org/web/20231105051542/https://x.ai/	Dense	Announced Nov/2023, trained Jul/2023
192	Yi-34B	01-ai	https://huggingface.co/01-ai/Yi-34B	34.4	3000	88:1	1.1	76.3	43		🆆 📚⬆ 🕸 🌋	Nov/2023	🟢	https://github.com/01-ai/Yi	Dense	Controversy about Llama 2 base. https://twitter.com/kaifulee/status/1724673131875377465 MMLU=76.3 (PaLM 2=78.3) Outperforms Llama 2. Chinese and English. https://www.bloomberg.com/news/articles/2023-11-05/kai-fu-lee-s-open-source-01-ai-bests-llama-2-according-to-hugging-face
193	GPT-4 Turbo	OpenAI	https://chat.openai.com/					86.4		46.5	🆆 📚⬆ 🕸 🌋	Nov/2023	🟢	https://cdn.openai.com/papers/gpt-4.pdf	MoE	https://openai.com/blog/new-models-and-developer-products-announced-at-devday
194	Skywork-13B	Kunlun Tech		13	3200	247:1	0.7	62.7			🆆 📚⬆ 🕸 🌋	Oct/2023	🟢	https://arxiv.org/abs/2310.19341	Dense	CN + EN.
195	Kimi Chat	Moonshot AI	https://kimi.moonshot.cn/	100	2000	20:1	1.5				🆆 📚⬆ 🕸 🌋	Oct/2023	🟢	https://www.chinadaily.com.cn/a/202403/22/WS65fce476a31082fc043be1b1.html	Dense	Chinese. Long context. No paper.
196	jina-embeddings-v2	Jina AI	https://huggingface.co/jinaai/jina-embeddings-v2-base-en	0.435							🕸	Oct/2023	🟢	https://jina.ai/news/jina-ai-launches-worlds-first-open-source-8k-text-embedding-rivaling-openai/	Dense	Alternative to text-embedding-ada-002. Related v1 paper: https://arxiv.org/abs/2307.11224
197	Fuyu	Adept	https://huggingface.co/adept/fuyu-8b	8							🆆 📚⬆ 🕸 🌋	Oct/2023	🟢	https://www.adept.ai/blog/fuyu-8b	Dense	VLM. 8B available under open licence, Medium size is closed
198	ERNIE 4.0	Baidu	https://yiyan.baidu.com/	1000	20000	20:1	14.9				🆆 📚⬆ 🕸 🌋	Oct/2023	🟢	https://reuters.com/technology/chinas-baidu-unveils-latest-version-its-ernie-ai-model-2023-10-17/	Dense	Dense (confirmed). English-dubbed launch video (2h52m): https://twitter.com/i/broadcasts/1yNGaZaeallJj & https://youtu.be/wYozcsavRuM
199	Zephyr	Hugging Face H4	https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha	7.3	800	110:1	0.3		33		🆆 📚⬆ 🕸 🌋	Oct/2023	🟢	https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha	Dense	Mistral with 'aligned' data removed from dataset
200	PaLI-3	Google DeepMind		5							🌋	Oct/2023	🔴	https://arxiv.org/abs/2310.09199	Dense	VLM. Next iteration of PaLI via Pathways. https://lifearchitect.ai/pathways/
201	Retro 48B	NVIDIA		48	1200	25:1	0.8				🆆 📚⬆ 🕸 🌋	Oct/2023	🟢	https://arxiv.org/abs/2310.07713	Dense	the largest LLM pretrained with retrieval before instruction tuning.'
202	Ferret	Apple	https://github.com/apple/ml-ferret	13	2000	154:1	0.5				🆆 📚⬆ 🕸 👥	Oct/2023	🟢	https://arxiv.org/abs/2310.07704	Dense	Vicuna base, multimodal
203	Lemur	XLANG Lab	https://github.com/OpenLemur/Lemur	70	2090	30:1	1.3				🆆 📚⬆ 🕸 🌋	Oct/2023	🟢	https://arxiv.org/abs/2310.06830	Dense	https://arxiv.org/abs/2310.06830
204	AceGPT	KAUST/Shenzhen	https://huggingface.co/FreedomIntelligence/AceGPT-13B	13	2010	155:1	0.5				🆆 📚⬆ 🕸 🌋	Oct/2023	🟢	https://github.com/FreedomIntelligence/AceGPT/tree/main	Dense	Arabic. Llama 2 + RLAIF
205	Yasa-1	Reka AI	https://reka.ai/announcing-our-multimodal-ai-assistant/								🌋	Oct/2023	🟡	https://reka.ai/product/	Dense	Multi-modal. No public arch info. Researchers from DeepMind, Google, Baidu and Meta building enterprise models
206	RT-X	Google DeepMind	https://robotics-transformer-x.github.io/	55							🌋	Oct/2023	🟢	https://robotics-transformer-x.github.io/paper.pdf	Dense	Robotics using UL2. 'RT-1 model trained using the robotic data mixture as RT-1-X, and the RT-2 model trained using the robotic data mixture as RT-2-X.'
207	MotionLM	Waymo		0.09							🌋	Sep/2023	🔴	https://arxiv.org/abs/2309.16534	Dense	LLM for autonomous vehicle forecasting. https://youtu.be/jrMMNmN21I8?t=1560
208	GAIA-1	Wayve	https://wayve.ai/thinking/scaling-gaia-1/	9							🌋	Sep/2023	🔴	https://arxiv.org/abs/2309.17080	Dense	World model, generates video. Uses T5-large 770M for language + all vision parameters
209	Qwen	Alibaba	https://huggingface.co/Qwen	72	3000	42:1	1.5				🆆 📚⬆ 🕸 🌋	Sep/2023	🟢	https://arxiv.org/abs/2309.16609	Dense	Chinese. Full name is 'Tongyi Qianwen' 通义千问. 'Lags behind both GPT-3.5 and GPT-4'. Originally 7B/14B params Apr/2023
210	Llama 2 Long	Meta AI		70	2400	35:1	1.4				🆆 📚⬆ 🕸 🌋	Sep/2023	🔴	https://arxiv.org/abs/2309.16039	Dense	Unreleased to date. Context window=32,768 tokens (compare to Llama 2=4096 tokens)
211	LeoLM	Hessian AI/LAION	https://huggingface.co/LeoLM/leo-hessianai-13b	13	2065	159:1	0.5				🆆 📚⬆ 🕸 🌋	Sep/2023	🟢	https://laion.ai/blog/leo-lm/	Dense	Llama 2 'extended' and pretrained on 2000B Llama 2 tokens + 65B tokens of German
212	Mistral 7B	Mistral	https://huggingface.co/mistralai	7.3	800	110:1	0.3		30.9		🆆 📚⬆ 🕸 🌋	Sep/2023	🟢	https://mistral.ai/news/announcing-mistral-7b/	Dense	Apache 2.0, Sliding Window Attention (SWA) to handle longer sequences at smaller cost
213	Kosmos-2.5	Microsoft		1.3							🆆 📚⬆ 🕸 🌋	Sep/2023	🔴	https://arxiv.org/abs/2309.11419	Dense
214	Baichuan 2	Baichuan	https://github.com/baichuan-inc/Baichuan2/blob/main/README_EN.md	13	2600	200:1	0.6				🆆 📚⬆ 🕸 🌋	Sep/2023	🟢	https://cdn.baichuan-ai.com/paper/Baichuan2-technical-report.pdf	Dense	Great paper. Chinese-English bilingual dataset
215	BOLT2.5B	ThirdAI	https://huggingface.co/spaces/thirdai/BOLT2.5B	2.5	40	16:1	0.0				🕸	Sep/2023	🟢	https://medium.com/thirdai-blog/introducing-the-worlds-first-generative-llm-pre-trained-only-on-cpus-meet-thirdai-s-bolt2-5b-10c0600e1af4	Dense	CPU trained
216	DeciLM	Deci	https://huggingface.co/Deci/DeciLM-6b	5.7	200	36:1	0.1				🆆 📚⬆ 🕸 🌋	Sep/2023	🟢	https://deci.ai/blog/decilm-15-times-faster-than-llama2-nas-generated-llm-with-variable-gqa/	Dense	Faster inference (4.8× throughput of Llama 2)
217	MoLM	IBM	https://github.com/ibm/moduleformer	8	300	38:1	0.2				🆆 📚⬆ 🕸 🌋	Sep/2023	🟢	https://arxiv.org/abs/2306.04640	MoE	ModuleFormer is based on the Sparse Mixture of Experts (MoE).
218	NExT-GPT	Singapore	https://next-gpt.github.io/	7	1000	143:1	0.3				🆆 📚⬆ 🕸 🌋	Sep/2023	🟢	https://arxiv.org/abs/2309.05519	Dense	Multimodal. Vicuna 7B + other modalities
219	Phi-1.5	Microsoft	https://huggingface.co/microsoft/phi-1_5	1.3	150	116:1	0.0				📚⚛️	Sep/2023	🟢	https://arxiv.org/abs/2309.05463	Dense	Textbooks only. 30B-token dataset
220	UniLM	Apple	https://jackcook.com/2023/09/08/predictive-text.html	0.034	1	30:1	0.0				🌋	Sep/2023	🟢	https://github.com/jackcook/predictive-spy	Dense	Apple's Transformer model for iOS 17 + macOS Sonoma. Announce is actually Jun/2023. GPT-2 base? 128 token context window
221	Persimmon-8B	Adept	https://www.adept.ai/blog/persimmon-8b	8	737	93:1	0.3				🆆 📚⬆ 🕸 🌋	Sep/2023	🟢	https://github.com/persimmon-ai-labs/adept-inference	Dense	Open Apache license and publicly accessible weights.
222	FLM-101B	BAAI	https://huggingface.co/CofeAI/FLM-101B	101	245	3:1	0.5				🆆 📚⬆ 🕸 🌋	Sep/2023	🟢	https://arxiv.org/abs/2309.03852	Dense	Train for $100k compute budget (on a cluster of 24 DGX-A800 GPU 8×80G servers for 21 days)
223	Falcon 180B	TII	https://huggingface.co/spaces/tiiuae/falcon-180b-demo	180	3500	20:1	2.6	70.6			🆆 📚⬆ 🕸 🌋	Sep/2023	🟢	https://arxiv.org/abs/2311.16867	Dense	Major milestone for open source models (largest open dense model to date).
224	Hunyuan	Tencent	https://www.tencent.com/en-us/articles/2201685.html	100	2000	20:1	1.5				🆆 📚⬆ 🕸 🌋	Sep/2023	🟢	https://arxiv.org/abs/2402.01723v1	Dense
225	phi-CTNL	Independent		0.1	0.01	1:1	0.0				🆆 📚⬆ 🕸 🌋	Sep/2023	🟢	https://arxiv.org/abs/2309.08632	Dense	Satire. MMLU=100. 'phi-CTNL (pronounced “fictional”) that achieves perfect results across diverse academic benchmarks'
226	Jais	Inception	https://huggingface.co/inception-mbzuai	13	395	31:1	0.2				🆆 📚⬆ 🕸 🌋	Aug/2023	🟢	https://arxiv.org/abs/2308.16149	Dense	Arabic, trained in Abu Dhabi, UAE using Cerebras.
227	Code Llama 34B	Meta AI	https://github.com/facebookresearch/codellama	34	2600	77:1	1.0				🆆 🕸	Aug/2023	🟢	https://ai.meta.com/research/publications/code-llama-open-foundation-models-for-code/	Dense	Outperforms GPT-3.5. Initial Llama 2 (2T tokens) trained on 500B tokens of code, 100B tokens of python
228	IDEFICS	Hugging Face	https://huggingface.co/spaces/HuggingFaceM4/idefics_playground	80							🆆 🕸	Aug/2023	🟢	https://huggingface.co/blog/idefics	Dense	Clone of Flamingo using Llama-1 65B. Named after Asterix and Obelix's dog Idefix (Image-aware Decoder Enhanced à la Flamingo with Interleaved Cross-attentionS)
229	Raven	UI/NVIDIA		11	40	4:1	0.1				🆆 🕸	Aug/2023	🔴	https://arxiv.org/abs/2308.07922	Dense	RAG Atlas
230	DukunLM	AzaleAI	https://huggingface.co/azale-ai/DukunLM-13B-V1.0-Uncensored	13	1500	116:1	0.5				🆆 📚⬆ 🕸 👥	Aug/2023	🟢	https://huggingface.co/azale-ai/DukunLM-13B-V1.0-Uncensored	Dense	Indonesian fine-tune of WizardLM (which is a Llama fine-tune).
231	WizardLM	Microsoft	https://huggingface.co/WizardLM/WizardLM-70B-V1.0	70	2000	29:1	1.2				🆆 📚⬆ 🕸 👥	Aug/2023	🟢	https://github.com/nlpxucan/WizardLM	Dense	Assume Llama-2 fine-tune. Outperforms text-davinci-003. May merge this entry with the Apr/2023 7B release
232	Platypus	Boston University	https://platypus-llm.github.io/	70	2000	29:1	1.2				🆆 📚⬆ 🕸 🌋	Aug/2023	🟢	https://platypus-llm.github.io/Platypus.pdf	Dense	Fine-tune of Llama 2, family includes merges with Beluga, Dolphin, and Camel fine-tunes.
233	Japanese StableLM Alpha 7B	Stability AI	https://huggingface.co/stabilityai/japanese-stablelm-base-alpha-7b	7	750	108:1	0.2				🆆 📚⬆ 🕸 🌋	Aug/2023	🟢	https://stability.ai/blog/stability-ai-new-jplm-japanese-language-model-stablelm	Dense	Best-performing openly available language model for Japanese speakers.
234	Stable Code 3B	Stability AI	https://huggingface.co/stabilityai/stablecode-completion-alpha-3b-4k	2.7	560	208:1	0.1				🌋	Aug/2023	🟢	https://stability.ai/blog/stablecode-llm-generative-ai-coding	Dense	Context window=16,384. Trained on The Stack dataset.
235	Med-Flamingo	Stanford	https://github.com/snap-stanford/med-flamingo	8.3	1000	121:1	0.3				🕸 🌋	Jul/2023	🟢	https://arxiv.org/abs/2307.15189	Dense	Uses LAION OpenFlamingo 9B, based on LLaMA-7B text + 1.3B vision
236	Alfred-40B-0723	LightOn	https://huggingface.co/lightonai/alfred-40b-0723	40	1000	25:1	0.7				🆆 📚⬆ 🕸 🌋	Jul/2023	🟢	https://www.lighton.ai/blog/lighton-s-blog-4/introducing-alfred-40b-0723-38	Dense	First finetuned version of Falcon with RLHF. Enterprise: https://www.lighton.ai/paradigm
237	LLaMA-2-7B-32K	Together	https://huggingface.co/togethercomputer/LLaMA-2-7B-32K	7	2000	286:1	0.4				🆆 📚⬆ 🕸 🌋	Jul/2023	🟢	https://together.ai/blog/llama-2-7b-32k	Dense	32k context window instead of 4k (Llama 2)
238	Med-PaLM M	Google DeepMind		540	780	2:1	2.2				🆆 📚⬆ 🕸 👥	Jul/2023	🔴	https://arxiv.org/abs/2307.14334	Dense	Uses PaLM 1. Already outperformed by Med-PaLM 2. Med-PaLM Multimodal (Med-PaLM M).
239	BTLM-3B-8K	Cerebras	https://huggingface.co/cerebras/btlm-3b-8k-base	3	627	209:1	0.1				🆆 📚⬆ 🕸 🌋	Jul/2023	🟢	https://www.cerebras.net/blog/btlm-3b-8k-7b-performance-in-a-3-billion-parameter-model/	Dense	Runs on devices with as little as 3GB of memory [iPhone, Macbook] when quantized to 4-bit
240	Stable Beluga 2	Stability AI	https://huggingface.co/stabilityai/FreeWilly2	70	2000	29:1	1.2				🆆 📚⬆ 🕸 🌋	Jul/2023	🟢	https://stability.ai/blog/stable-beluga-large-instruction-fine-tuned-models	Dense	Fine-tuned Llama 2. Non-commercial use license. Codename was FreeWilly2
241	Stable Beluga 1	Stability AI	https://huggingface.co/stabilityai/FreeWilly1-Delta-SafeTensor	65	1400	22:1	1.0				🆆 📚⬆ 🕸 🌋	Jul/2023	🟢	https://stability.ai/blog/stable-beluga-large-instruction-fine-tuned-models	Dense	Fine-tuned LLaMA-1. Non-commercial use license. Codename was FreeWilly1
242	Meta-Transformer	Shanghai AI Laboratory/CUHK	https://github.com/invictus717/MetaTransformer	2							🆆 📚⬆ 🕸 🌋	Jul/2023	🟢	https://arxiv.org/abs/2307.10802	Dense	Proto-AGI. 12 modalities (text, image, point cloud, audio, video, infrared, hyperspectral, X-ray, time-series, tabular, Inertial Measurement Unit (IMU), and graph data).
243	Llama 2	Meta AI	https://www.llama2.ai/	70	2000	29:1	1.2	68.9	37.5	26.26	🆆 📚⬆ 🕸 🌋	Jul/2023	🟢	https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/	Dense	Context window=4096. MMLU=68.9 (GPT-3.5=70.0, GPT-4=86.4)
244	WormGPT	(Undisclosed)		6	402	67:1	0.2				🆆 📚⬆ 🕸 🌋	Jul/2023	🟡	https://slashnext.com/blog/wormgpt-the-generative-ai-tool-cybercriminals-are-using-to-launch-business-email-compromise-attacks/	Dense	GPT-J (2021) finetune/module.
245	Claude 2	Anthropic	https://claude.ai/	130	2500	20:1	1.9	78.5			🆆 📚⬆ 🕸 🌋	Jul/2023	🟢	https://www-files.anthropic.com/production/images/Model-Card-Claude-2.pdf	Dense	More HHH, 200k context length
246	LongLLaMA	IDEAS/DeepMind	https://github.com/CStanKonrad/long_llama	7	1000	143:1	0.3				🆆 📚⬆ 🕸 🌋	Jul/2023	🟢	https://arxiv.org/abs/2307.03170	Dense	256k context length
247	xTrimoPGLM	Tsinghua		100	1000	10:1	1.1				🌋	Jul/2023	🔴	https://www.biorxiv.org/content/10.1101/2023.07.05.547496v1	Dense	Protein language model
248	XGen	Salesforce	https://github.com/salesforce/xgen	7	1500	215:1	0.3				🆆 📚⬆ 🕸 🌋	Jul/2023	🟢	https://blog.salesforceairesearch.com/xgen/	Dense	8K sequence length. Released under Apache-2.0.
249	Zhinao (Intellectual Brain)	360 cn	https://ai.360.com/	100	2000	20:1	1.5				🆆 📚⬆ 🕸 🌋	Jul/2023	🟢	https://arxiv.org/abs/2402.01723v1	Dense
250	Yasa	Reka AI	https://reka.ai/product/									Jun/2023	🟡	https://reka.ai/product/	Dense	No public arch info. Researchers from DeepMind, Google, Baidu and Meta building enterprise models
251	Kosmos-2	Microsoft	https://44e505515af066f4.gradio.app/	1.6	360	225:1	0.1				🆆 📚⬆ 🕸 🌋	Jun/2023	🟢	https://arxiv.org/abs/2306.14824	Dense	Proto-AGI. Multimodal large language model (MLLM). a multimodal large language model with grounding capability built upon KOSMOS-1
252	AudioPaLM	Google	https://google-research.github.io/seanet/audiopalm/examples/	340	3600	11:1	3.7				🆆 📚⬆ 🕸 👥	Jun/2023	🔴	https://arxiv.org/abs/2306.12925	Dense	a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation
253	Inflection-1	Inflection AI	https://docs.google.com/forms/d/e/1FAIpQLScM9Iz1KzaRlfgDrYrldoPDnXbhO5LW3-hqmQCd56YpheEN7g/viewform	120	2000	17:1	1.6				🆆 📚 ⬆ 🕸	Jun/2023	🟢	https://inflection.ai/assets/Inflection-1_0622.pdf	Dense	Comparable with benchmarking results from InternLM 104B, 1-2% better. ‘Inflection-1 was trained using thousands of NVIDIA H100 GPUs on a very large dataset.’
254	Phi-1	Microsoft		1.3	51	40:1	0.0				📚⚛️	Jun/2023	🔴	https://arxiv.org/abs/2306.11644	Dense	Code model. ‘breaking existing scaling laws by training a 1.3B-parameter model, which we call phi-1, for roughly 8 passes over 7B tokens (slightly over 50B total tokens seen) followed by finetuning on less than 200M tokens.’
255	InternLM	Shanghai AI Laboratory/SenseTime	https://internlm-org.translate.goog/?_x_tr_sl=zh&_x_tr_tl=en	104	1600	16:1	1.4				🆆 📚⬆ 🕸 🌋	Jun/2023	🔴	https://github.com/InternLM/InternLM-techreport	Dense	Outperforms ChatGPT, LLaMA on RACE-h, Chinese + English
256	BlenderBot 3x	Meta AI	https://parl.ai/projects/bb3x/	175	300	2:1	0.8				🆆 📚 ⬆ 🕸	Jun/2023	🟢	https://arxiv.org/abs/2306.04707	Dense	OPT-175B with new dialogue data
257	Orca	Microsoft	https://aka.ms/orca-lm	13	1000	77:1	0.4				🆆 📚⬆ 🕸 🌋	Jun/2023	🟡	https://arxiv.org/abs/2306.02707	Dense	LLaMA -> Vicuna -> Orca (GPT-4 finetune). Still an imitation model, overhyped: The False Promise of Imitating Proprietary LLMs https://arxiv.org/abs/2305.15717
258	PassGPT	ETH Zürich									🌋	Jun/2023	🔴	https://arxiv.org/abs/2306.01545	Dense	GPT-2 trained on leaked passwords
259	DIDACT	Google DeepMind			37900						🌋	Jun/2023	🔴	https://ai.googleblog.com/2023/05/large-sequence-models-for-software.html	Dense	Iterative coding model trained on Google's monorepo. Jacob: https://twitter.com/jacobaustin132/status/1663972128176128002
260	LTM-1	Magic	https://magic.dev/blog/ltm-1								🌋	Jun/2023	🔴	https://magic.dev/blog/ltm-1	Dense	Context window=5M
261	GPT-4 MathMix	OpenAI		1760	13000	8:1	15.9				🆆 📚⬆ 🕸 🌋	May/2023	🔴	https://arxiv.org/abs/2305.20050	Dense	Unreleased, includes step by step research
262	PandaGPT	Cambridge/Tencent	https://panda-gpt.github.io/	13	1000	77:1	0.4				🆆 📚⬆ 🕸 🌋	May/2023	🟢	https://github.com/yxuansu/PandaGPT/blob/main/PandaGPT.pdf	Dense	Proto-AGI. 6 modalities (text, image/video, audio, depth, thermal, and IMU/accelerometer/gyroscope/compass). Based on Vicuna.
263	Falcon	TII	TS	40	1000	25:1	0.7				🆆 📚⬆ 🕸 🌋	May/2023	🟢	https://www.tii.ae/news/uaes-technology-innovation-institute-launches-open-source-falcon-40b-large-language-model	Dense	Abu Dhabi
264	202305-refact2b-mqa-lion	Refact	https://refact.ai/blog/2023/applying-recent-innovations-to-train-model/	1.6							🆆 📚⬆ 🕸 🌋	May/2023	🟡	https://refact.ai/blog/2023/applying-recent-innovations-to-train-model/	Dense	LiON vs Adam, code, RedPajama+The Stack
265	Guanaco	UW	https://huggingface.co/spaces/uwnlp/guanaco-playground-tgi	65	1400	22:1	1.0				🆆 📚⬆ 🕸 🌋	May/2023	🟢	https://arxiv.org/abs/2305.14314	Dense	LLaMA-65B via QLoRA
266	LIMA	Meta AI		65							🆆 📚⬆ 🕸 🌋	May/2023	🔴	https://arxiv.org/abs/2305.11206	Dense	LLaMA-65B with nearly no fine-tuning, no RLHF
267	Formosa (FFM)	Asus/TWS		176	366	3:1	0.8				🆆 📚⬆ 🕸 🌋	May/2023	🟡	https://www.asus.com/news/xxifirl2s2tzesl0/	Dense	BLOOMZ finetune? Chinese, Taiwan's first LLM. Subscription hardware: https://archive.md/cVdJt
268	CodeT5+	Salesforce	https://huggingface.co/Salesforce/codet5p-16b	16							🕸 🌋	May/2023	🟢	https://arxiv.org/abs/2305.07922	Dense	InstructCodeT5+ 16B sets new SoTA results of 35.0% pass@1 and 54.5% pass@10 against other open code LLMs, even surpassing the closed-source OpenAI code-cushman-001'
269	PaLM 2	Google	https://console.cloud.google.com/vertex-ai/generative/language/create/chat	340	3600	11:1	3.7				🆆 📚⬆ 🕸 👥	May/2023	🟢	https://ai.google/static/documents/palm2techreport.pdf	Dense	“What we found in our work is that it’s not really the sort of size of model — that the larger is not always better,” Deepmind VP Zoubin Ghahramani said in a press briefing ahead of today’s announcement. “That’s why we’ve provided a family of models of different sizes. We think that actually parameter count is not really a useful way of thinking about the capabilities of models and capabilities are really to be judged by people using the models and finding out whether they’re useful in the tests that they try to achieve with these models.”
270	StarCoder	HF/ServiceNow	https://huggingface.co/bigcode/starcoderbase	15.5	1000	65:1	0.4				🌋	May/2023	🟢	https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0kl4BYs/view	Dense
271	MPT	MosaicML	https://huggingface.co/mosaicml/mpt-7b	7	1000	143:1	0.3				🆆 📚⬆ 🕸 🌋	May/2023	🟢	https://twitter.com/NaveenGRao/status/1654496162492084227	Dense	Llongboi' -Apache 2.0 license suitable for commercial use. -Base 7B LLM trained on 1T tokens outperforms LLaMA and GPT3. -64K+ context length. -$200k to train from scratch.
272	Pi	Inflection AI	https://pi.ai/talk	60							🆆 📚⬆ 🕸 👥	May/2023	🟢	https://www-cnbc-com.cdn.ampproject.org/c/s/www.cnbc.com/amp/2022/03/08/reid-hoffman-has-set-up-a-new-ai-company-with-deepminds-co-founder.html	Dense	No indication of params/tokens. Devs from DeepMind.
273	GPT-2B-001	NVIDIA	https://huggingface.co/nvidia/GPT-2B-001	2	1100	550:1	0.2				🆆 📚⬆ 🕸 🌋	May/2023	🟢	https://huggingface.co/nvidia/GPT-2B-001	Dense	No paper yet
274	Titan	Amazon	https://aws.amazon.com/bedrock/titan/	200	4000	20:1	3.0	70.4			🆆 📚⬆ 🕸 👥	Apr/2023	🟢	https://www.techrepublic.com/article/amazon-bedrock-titan-cloud-artificial-intelligence/	Dense	No official information at all. 2nd hand via Jack Clark: https://importai.substack.com/p/import-ai-365-wmd-benchmark-amazon '$65m training run. Specifically, they trained a 200B dense model on 4T tokens of data across 13,760 NVIDIA A100 chips (using 1,720 P4d nodes). It took 48 days to train.'
275	WizardLM	Microsoft	https://6f8173a3550ed441ab.gradio.live/	7	1000	143:1	0.3				🆆 📚⬆ 🕸 👥	Apr/2023	🟢	https://arxiv.org/abs/2304.12244	Dense	LLaMA 7B self-instructed fine-tune.
276	MPT	MosaicML	https://huggingface.co/mosaicml/mpt-1b-redpajama-200b-dolly	1.3	200	154:1	0.1				🆆 📚⬆ 🕸 🌋	Apr/2023	🟢	https://twitter.com/jefrankle/status/1649060478910357504	Dense	More 1B models coming with different datasets. Many more.
277	StableLM	Stability AI	https://huggingface.co/spaces/stabilityai/stablelm-tuned-alpha-chat	65	1500	24:1	1.0				🆆 📚⬆ 🕸 🌋	Apr/2023	🟢	https://github.com/stability-AI/stableLM/	Dense	contains 1.5 trillion tokens, roughly 3x the size of The Pile. These models will be trained on up to 1.5 trillion tokens. The context length for these models is 4096 tokens.
278	Dolly 2.0	Databricks	https://huggingface.co/databricks/dolly-v2-12b	12	300	25:1	0.2				🆆 📚⬆ 🕸 🌋	Apr/2023	🟢	https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm	Dense	Fine-tuned Pythia 12B
279	Pythia	EleutherAI	https://huggingface.co/EleutherAI/pythia-12b	12	300	25:1	0.2				🆆 📚⬆ 🕸 🌋	Apr/2023	🟢	https://arxiv.org/abs/2304.01373	Dense
280	Koala-13B	Berkeley	https://chat.lmsys.org/?model=koala-13b	13							🆆 📚⬆ 🕸 👥	Apr/2023	🟢	https://bair.berkeley.edu/blog/2023/04/03/koala/	Dense	LLaMA base. Academic licence only.
281	C1.2	Character.ai	https://blog.character.ai/character-ai/	33	1000	31:1	0.6				🆆 📚⬆ 🕸 🌋	Mar/2023	🟢	https://blog.character.ai/character-ai/	Dense	No details released.
282	BloombergGPT	Bloomberg		50	569	12:1	0.6	39.2			🆆 📚⬆ 🕸 🌋	Mar/2023	🔴	https://arxiv.org/abs/2303.17564	Dense	Video: https://youtu.be/m2Scj2SO85Y Underperforms GPT-3, based on BLOOM. Tokens: 'We select a model size motivated by Hoffmann et al. (2022) and train a 50 billion parameter model on 569 billion tokens from our corpus of over 700 billion tokens to produce a model that is competitive with larger models.'
283	OpenFlamingo-9B	LAION	https://huggingface.co/openflamingo/OpenFlamingo-9B	8.3	1000	121:1	0.3				🕸 🌋	Mar/2023	🟢	https://laion.ai/blog/open-flamingo/	Dense	Uses LLaMA-7B. Demo: https://7164d2142d11.ngrok.app/
284	GPT4All-LoRa	Nomic	https://github.com/nomic-ai/gpt4all	7	1000	143:1	0.3				🆆 📚⬆ 🕸 🌋	Mar/2023	🟢	https://s3.amazonaws.com/static.nomic.ai/gpt4all/2023_GPT4All_Technical_Report.pdf	Dense	chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa
285	Cerebras-GPT	Cerebras	https://huggingface.co/cerebras	13	260	20:1	0.2				🆆 📚⬆ 🕸 🌋	Mar/2023	🟢	https://www.cerebras.net/blog/cerebras-gpt-a-family-of-open-compute-efficient-large-language-models/	Dense	20:1 tokens to parameters as per https://lifearchitect.ai/chinchilla/
286	PanGu-Sigma	Huawei		1085							🌋	Mar/2023	🔴	https://arxiv.org/abs/2303.10845	MoE	Sparse. 1.085T parameters named PanGu-Σ.
287	CoLT5	Google		5.2							🆀🅰 🕸	Mar/2023	🔴	https://arxiv.org/abs/2303.09752	Dense	up to 64k context window [48k words or about 96 pages -Alan]
288	Med-PaLM 2	Google DeepMind									🆆 📚⬆ 🕸 👥	Mar/2023	🔴	https://blog.google/technology/health/ai-llm-medpalm-research-thecheckup/	Dense	Recently, our next iteration, Med-PaLM 2, consistently performed at an “expert” doctor level on medical exam questions, scoring 85%. This is an 18% improvement from Med-PaLM’s previous performance and far surpasses similar AI models.
289	GPT-4 Classic (gpt-4-0314 & gpt-4-0613, non-Turbo)	OpenAI	https://chat.openai.com/	1760	13000	8:1	15.9	90.1		35.7	🆆 📚⬆ 🕸 🌋	Mar/2023	🟢	https://cdn.openai.com/papers/gpt-4.pdf	MoE	Original MMLU=86.4. Proto-AGI. 1.76T parameters MoE.
290	Alpaca	Stanford	https://crfm.stanford.edu/alpaca/	7	1000	143:1	0.3				🆆 📚⬆ 🕸 👥	Mar/2023	🟢	https://github.com/tatsu-lab/stanford_alpaca	Dense	Stanford Alpaca: An Instruction-following LLaMA model'
291	Jurassic-2	AI21	Studio	178							🆆 📚 ⬆ 🕸	Mar/2023	🟢	https://www.ai21.com/blog/introducing-j2	Dense
292	GPT-NeoX-Chat-Base-20B	Together	https://huggingface.co/spaces/togethercomputer/OpenChatKit	20				33.6			🆆 📚 ⬆ 🕸 🌋	Mar/2023	🟢	https://github.com/togethercomputer/OpenChatKit	Dense	instruction-tuned 20 billion parameter language model, a 6 billion parameter moderation model, and an extensible retrieval system for including up-to-date responses from custom repositories. It was trained on the OIG-43M training dataset, which was a collaboration between Together, LAION, and Ontocord.ai. '
293	Kosmos-1	Microsoft		1.6	360	225:1	0.1				🆆 📚⬆ 🕸 🌋	Feb/2023	🔴	https://arxiv.org/abs/2302.14045	Dense	Proto-AGI. Multimodal large language model (MLLM). Raven’s Progressive Matrices as real images, not digits as in testing of text-davinci-003 at https://lifearchitect.ai/ravens/
294	LLaMA-65B	Meta AI	Weights leaked: https://github.com/facebookresearch/llama/pull/73/files	65	1400	22:1	1.0	68.9			🆆 📚⬆ 🕸 👥	Feb/2023	🟢	https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/	Dense	Researchers only, noncommercial only. 'LLaMA-65B is competitive with the best models, Chinchilla70B and PaLM-540B.'
295	MOSS	Fudan University	https://moss.fastnlp.top/	16	430	27:1	0.3				🕸 🌋	Feb/2023	🟢	https://txsun1997.github.io/blogs/moss.html	Dense	Major bandwidth issues: https://www.reuters.com/technology/china-fudan-university-team-apologises-after-chatgpt-style-platform-crashes-2023-02-21/
296	Palmyra	Writer	https://huggingface.co/models?search=palmyra	20	300	15:1	0.3				🌋	Feb/2023	🟢	https://writer.com/blog/palmyra/	Dense	Only up to 5B available open-source 'trained on over 300 billion tokens of text data, and the size of the resulting model is over 20 billion parameters. ' https://writer.com/product/cowrite/
297	Luminous Supreme Control	Aleph Alpha	https://app.aleph-alpha.com/playground/completion	70	588	9:1	0.7				🆆 📚⬆ 🕸 👥	Feb/2023	🟢	https://www.aleph-alpha.com/pdf/2023_02_AA_Benchmarks_doc.pdf	Dense	‘Control’ means instruction tuned
298	Toolformer+Atlas 11B+NLLB 54B	Meta AI	Replicated: https://github.com/conceptofmind/toolformer	6.7	402	60:1	0.2				🆆 📚⬆ 🕸 🌋	Feb/2023	🔴	https://arxiv.org/abs/2302.04761	Dense	Based on GPT-J 6.7B + access to other models via API
299	Multimodal-CoT	Amazon	https://github.com/amazon-science/mm-cot	0.738							🌋	Feb/2023	🟢	https://arxiv.org/abs/2302.00923	Dense	Models <1B with vision CoT
300	FLAME	Microsoft		0.06	9	150:1	0.0				🌋	Jan/2023	🔴	https://arxiv.org/abs/2301.13779	Dense	T5 for Excel formulas, very small 60M params, "We start from a dataset of 927M formulas" estimate 10x multiplier for 9B tokens
301	Med-PaLM 1	Google DeepMind		540	780	2:1	2.2				🆆 📚⬆ 🕸 👥	Dec/2022	🔴	https://arxiv.org/abs/2212.13138	Dense	Collab between Google & DeepMind. Makes 1% less errors than humans
302	OPT-IML	Meta AI	https://github.com/facebookresearch/metaseq/tree/main/projects/OPT-IML	175	300	2:1	0.8				🆆 📚⬆ 🕸	Dec/2022	🟢	https://arxiv.org/abs/2212.12017	Dense	Instruct
303	RL-CAI	Anthropic		52	400	8:1	0.5				🆆 📚⬆ 🕸 👥	Dec/2022	🔴	https://arxiv.org/abs/2212.08073	Dense	RLAIF=reinforcement learning with AI feedback
304	ERNIE-Code	Baidu		0.56							🕸	Dec/2022	🟢	https://arxiv.org/abs/2212.06742#baidu	Dense
305	RT-1	Google		0.035							🌋	Dec/2022	🔴	https://robotics-transformer.github.io/assets/rt1.pdf	Dense
306	ChatGPT (gpt-3.5-turbo)	OpenAI	https://chat.openai.com/	20				70		28.1	🆆 📚 ⬆ 🕸	Nov/2022	🟢	https://openai.com/blog/chatgpt	Dense	Instruct with strict policies ("extremely limited")
307	text-davinci-003	OpenAI	https://chat.openai.com/								🆆 📚 ⬆ 🕸	Nov/2022	🟢	https://openai.com/blog/chatgpt	Dense
308	GPT-JT	Together	https://huggingface.co/spaces/togethercomputer/GPT-JT	6							🆆 📚 ⬆ 🕸 🌋	Nov/2022	🟢	https://www.together.xyz/blog/releasing-v1-of-gpt-jt-powered-by-open-source-ai	Dense
309	RWKV-4	RWKV	https://huggingface.co/BlinkDL	14	332	24:1	0.2				🆆 📚 ⬆ 🕸 🌋	Nov/2022	🟢	https://arxiv.org/abs/2305.13048	Dense	RNN, not transformer: https://www.reddit.com/r/MachineLearning/comments/yxt8sa/r_rwkv4_7b_release_an_attentionfree_rnn_language/
310	Galactica	Meta AI	https://galactica.org/	120	450	4:1	0.8	52.6			📚	Nov/2022	🟢	https://galactica.org/static/paper.pdf	Dense	scientific only
311	SED	DeepMind									🕸	Nov/2022	🔴	https://arxiv.org/abs/2211.04236	Dense	SED 420M (diffusion text model)
312	mT0	BigScience	https://github.com/bigscience-workshop/xmtf	13	1000	77:1	0.4				🆀🅰 🕸	Nov/2022	🟢	https://arxiv.org/abs/2211.01786	Dense	fine-tuned
313	BLOOMZ	BigScience	https://github.com/bigscience-workshop/xmtf	176	366	3:1	0.8				⬆ 🕸	Nov/2022	🟢	https://arxiv.org/abs/2211.01786	Dense	fine-tuned
314	PACT	Microsoft	https://github.com/microsoft/PACT		0.03		0.0				🌋	Oct/2022	🟢	https://arxiv.org/abs/2209.11133	Dense	Trained on ~5TB data, 2GB model download. 'In general we see an improvement in model performance as we increase the number of training tokens. Interestingly, larger models did not necessarily result in better performance for robot navigation. Even though larger models consistently presented better loss values for action prediction on a static dataset, (Fig. 7 b), when it comes to real-time deployment the larger network capacity introduces inference delays that become a disadvantage and lead to earlier crashes. For example, while LiDAR perception measurements arrive to the vehicle every 0.077s (13Hz), the largest model of 24 layers takes on average 0.023s for inference with a RTX3090 GPU, roughly 40% longer the 3 layer model (0.016s). These time differences can amount to even larger performance gaps in small embedded systems, and further emphasize the importance of multiple downstream task architectures sharing a common representation branch for real-time robotics applications.'
315	Flan-T5	Google	TS	11	1100	100:1	0.4				🆀🅰 🕸	Oct/2022	🟢	https://arxiv.org/abs/2210.11416	Dense	T5=1T tokens + LM-adapted T5 as 100B tokens
316	Flan-PaLM	Google		540	780	2:1	2.2	73.5			🆆 📚⬆ 🕸 👥	Oct/2022	🔴	https://arxiv.org/abs/2210.11416	Dense
317	U-PaLM	Google		540	780	2:1	2.2	74.1			🆆 📚⬆ 🕸 👥	Oct/2022	🔴	https://arxiv.org/abs/2210.11399	Dense
318	VIMA	NVIDIA	Open: https://vimalabs.github.io/	0.2							🌋	Oct/2022	🟢	https://arxiv.org/abs/2210.03094	Dense
319	OpenChat	Tsinghua	https://huggingface.co/openchat/openchat_3.5	13	2000	154:1	0.5				🆆 📚⬆ 🕸 🌋	Sep/2022	🟢	https://arxiv.org/abs/2309.11235	Dense	Llama 2 13B -> OpenChat 13B
320	WeLM	Wechat	https://welm.weixin.qq.com/docs/playground/	10	300	30:1	0.2				🆆 📚⬆ 🕸 🌋	Sep/2022	🟢	https://arxiv.org/abs/2209.10372	Dense	13% English tokens and 87% Chinese
321	CodeGeeX	Tsinghua		13	850	66:1	0.4				🌋	Sep/2022	🟢	https://github.com/THUDM/CodeGeeX	Dense
322	Sparrow	DeepMind		70	1400	20:1	1.0				🆆 📚⬆ 🕸 🌋	Sep/2022	🔴	https://storage.googleapis.com/deepmind-media/DeepMind.com/Authors-Notes/sparrow/sparrow-final.pdf	Dense	Chatbot as a fine-tuned version of Chinchilla 70B
323	PaLI	Google		17							🌋	Sep/2022	🔴	https://arxiv.org/abs/2209.06794	Dense	PaLM Vision model, new datasets of 10B multilingual text-image pairs
324	NeMo Megatron-GPT 20B	NVIDIA	https://huggingface.co/nvidia/nemo-megatron-gpt-20B	20							🆆 📚⬆ 🕸 🌋	Sep/2022	🟢	https://huggingface.co/nvidia/nemo-megatron-gpt-20B	Dense
325	Z-Code++	Microsoft		0.71	500	705:1	0.1				🆆 🕸	Aug/2022	🔴	https://arxiv.org/abs/2208.09770v1	Dense	abstractive text summarization, 710M, outperforms PaLM 540B. "Due to the limited computational resource, Z-Code++LARGE is trained with only 500B tokens instead of 1T tokens as that for mT5 training."
326	Atlas	Meta AI		11	40	4:1	0.1	47.9			🆆 🕸	Aug/2022	🟢	https://arxiv.org/abs/2208.03299	Dense
327	BlenderBot 3	Meta AI	blenderbot.ai (US only)	175	300	2:1	0.8				🆆 📚 ⬆ 🕸	Aug/2022	🟢	https://github.com/facebookresearch/ParlAI/blob/main/projects/bb3/BB3_main_tech_report.pdf	Dense
328	GLM-130B	Tsinghua	https://huggingface.co/spaces/THUDM/GLM-130B	130	400	4:1	0.8				🆆 📚 ⬆ 🕸	Aug/2022	🟢	https://arxiv.org/abs/2210.02414	Dense	50% English (200B tokens), so included here
329	AlexaTM 20B	Amazon	Github (train/deploy)	20	1300	65:1	0.5				🆆 🕸	Aug/2022	🟢	https://assets.amazon.science/ee/20/3abcf2304d9b8d68da2006ff7107/alexatm-20b-few-shot-learning-using-a-large-scale-multilingual-seq2seq-model.pdf	Dense	Wikipedia and mC4 only. seq2seq
330	6.9B FIM	OpenAI		6.9	100	15:1	0.1				🆆 📚 ⬆ 🕸	Jul/2022	🔴	https://arxiv.org/pdf/2207.14255.pdf	Dense	Several models: 8 sizes, NLP, Code, FIM/non-FIM. 100B tokens for 6.9B params... beyond chinchilla
331	‘monorepo-Transformer’	Google		0.5							🕸	Jul/2022	🔴	https://ai.googleblog.com/2022/07/ml-enhanced-code-completion-improves.html	Dense	Unnamed. Writes >3% of internal google code.
332	PanGu-Coder	Huawei		2.6							🌋	Jul/2022	🔴	https://arxiv.org/abs/2207.11280	Dense	Python via GH
333	NLLB	Meta AI	Github (train/deploy)	54.5							🌋	Jul/2022	🟢	https://research.facebook.com/publications/no-language-left-behind/	MoE	54.5B MOE, 3.3B dense. 200+ languages
334	J-1 RBG	AI21	ask-rbg.ai	178	300	2:1	0.8				🆆 📚 ⬆ 🕸	Jul/2022	🟢	https://www.ai21.com/blog/announcing-ai21-studio-and-jurassic-1	Dense	J-1 fine-tuned with RBG law corpus
335	BLOOM (tr11-176B-ml)	BigScience	https://huggingface.co/spaces/huggingface/bloom_demo	176	366	3:1	0.8	39.1			⬆ 🕸	Jul/2022	🟢	https://github.com/bigscience-workshop/bigscience/tree/master/train/tr11-176B-ml	Dense
336	Minerva	Google		540	818.5	2:1	2.2				🆆 📚⬆ 🕸 🌋	Jun/2022	🔴	https://ai.googleblog.com/2022/06/minerva-solving-quantitative-reasoning.html	Dense	PaLM finetuned on LaTeX/arXiv maths
337	GODEL-XL	Microsoft		2.7							🆆 📚⬆ 🕸 👥	Jun/2022	🟢	https://arxiv.org/abs/2206.11309#microsoft	Dense	XL: GPT-3 175B in paper, GPT-J 2.7B released
338	YaLM 100B	Yandex	Github (train/deploy)	100	300	3:1	0.6				🆆 📚⬆ 🕸	Jun/2022	🟢	https://github.com/yandex/YaLM-100B	Dense	Megatron-LM clone, Russian/English: https://medium.com/yandex/yandex-publishes-yalm-100b-its-the-largest-gpt-like-neural-network-in-open-source-d1df53d0e9a6
339	Unified-IO	Allen AI	Limited demo	2.8							🆆 📚⬆ 🕸 🌋	Jun/2022	🔴	https://github.com/jiasenlu/unified-io/blob/main/UnifiedIOv1.pdf	Dense	Based on T5. Demo only
340	Perceiver AR	DeepMind		1							🆆 📚⬆ 🕸 🌋	Jun/2022	🔴	https://arxiv.org/abs/2202.07765	Dense	Context window=100,000. Params=364m wiki, 975M pg-19, 826M books, music=?, imagenet=770M,
341	LIMoE	Google		5.6							🆆 📚⬆ 🕸 👥	Jun/2022	🔴	https://ai.googleblog.com/2022/06/limoe-learning-multiple-modalities-with.html	MoE
342	GPT-4chan	Independent	https://huggingface.co/ykilcher/gpt-4chan/discussions/4	6							🆆 📚⬆ 🕸 🌋	Jun/2022	🟢	https://arxiv.org/abs/2001.07487	Dense	Warning for inappropriate content. GPT-J.
343	Diffusion-LM	Stanford	Github (train/deploy)	0.3							🌋 👥	May/2022	🟢	https://arxiv.org/abs/2205.14217	Dense	GPT-J with synthetic data
344	UL2 20B	Google		20	1000	50:1	0.5	39.2			🕸	May/2022	🔴	https://arxiv.org/abs/2205.05131	Dense	Unifying Language model. C4 only.
345	Gato (Cat)	DeepMind		1							🆆 📚⬆ 🕸 🌋	May/2022	🔴	https://storage.googleapis.com/deepmind-media/A%20Generalist%20Agent/Generalist%20Agent.pdf	Dense	Proto-AGI. Generalist agent (LLM, VLM, robot)
346	LaMDA 2	Google	YouTube (video only)	137							⬆ 🕸 👥	May/2022	🟡	https://arxiv.org/abs/2201.08239	Dense	Chatbot with tiny walled garden demo TBA
347	OPT-175B	Meta AI	HF (train/deploy)	175	300	2:1	0.8				🆆 📚 ⬆ 🕸	May/2022	🟢	https://arxiv.org/abs/2205.01068	Dense	Only 30B available (Jun/2022)
348	Tk-Instruct	Hugging Face	https://instructions.apps.allenai.org/demo	11							🆀🅰 🕸	Apr/2022	🟢	https://arxiv.org/abs/2204.07705	Dense	Based on T5.
349	InCoder	Meta AI	https://huggingface.co/spaces/facebook/incoder-demo	6.7							GH, StackOverflow	Apr/2022	🟢	https://arxiv.org/abs/2204.05999	Dense	Python and JavaScript
350	NOOR	TII		10							🆆 📚 🕸 🇦🇪	Apr/2022	🔴	https://www.tii.ae/news/technology-innovation-institute-announces-launch-noor-worlds-largest-arabic-nlp-model	Dense	Arabic. "World’s largest high-quality cross-domain Arabic dataset, combining web data with books, poetry, news articles, and technical information"
351	mGPT	Sber	Hugging Face	13							🆆 🕸	Apr/2022	🟡	https://arxiv.org/abs/2204.07580	Dense	60 languages. Only 1.3B model available
352	PaLM-Coder	Google		540	780	2:1	2.2				🕸	Apr/2022	🔴	https://storage.googleapis.com/pathways-language-model/PaLM-paper.pdf	Dense
353	PaLM	Google		540	780	2:1	2.2				🆆 📚⬆ 🕸 👥	Apr/2022	🔴	https://storage.googleapis.com/pathways-language-model/PaLM-paper.pdff	Dense
354	SeeKeR	Meta AI		2.7							🆆 📚 ⬆ 🕸	Mar/2022	🟢	https://arxiv.org/abs/2203.13224	Dense	BART and compared to GPT-2
355	CodeGen	Salesforce	TS, Goose	16							🕸 BigQuery, BigPython	Mar/2022	🟢	https://arxiv.org/abs/2203.13474	Dense	Code
356	VLM-4	LightOn	Muse	10							🕸	Mar/2022	🟢	https://www-cnbc-com.cdn.ampproject.org/c/s/www.cnbc.com/amp/2022/03/08/reid-hoffman-has-set-up-a-new-ai-company-with-deepminds-co-founder.html	Dense	Params corrected 25/Apr/2022
357	CM3	Meta AI		13							🆆 🕸	Jan/2022	🟢	https://arxiv.org/abs/2201.0752	Dense	LLM with multimodal capabilities
358	Luminous	Aleph Alpha	AA playground	200							🕸	Nov/2021	🟢	https://www.aleph-alpha.de/pricing	Dense	Devs from EleutherAI
359	Chinchilla	DeepMind		70	1400	20:1	1.0	67.5			🆆 📚⬆ 🕸 🌋	Mar/2022	🔴	https://arxiv.org/abs/2203.15556	Dense	First to double tokens per size increase
360	GPT-NeoX-20B	EleutherAI	TS, Goose	20							🆆 📚⬆ 🕸 🌋	Feb/2022	🟢	https://github.com/EleutherAI/gpt-neox	Dense	Latest model to Feb/2022
361	ERNIE 3.0 Titan	Baidu		260							🆆 📚⬆ 🕸 🌋	Dec/2021	🟢	https://arxiv.org/abs/2112.12731	Dense
362	XGLM	Meta AI		7.5							🕸	Dec/2021	🟢	https://arxiv.org/abs/2112.10668	Dense	Multilingual: 30 languages, 16 families.
363	Fairseq	Meta AI	TS, Goose	1100							🆆 📚⬆ 🕸 🕸 🕸	Dec/2021	🟢	https://arxiv.org/abs/2112.10684	Dense	13B & 1100B param models.
364	Gopher	DeepMind		280	300	2:1	1.0	60			🆆 📚⬆ 🕸 🌋	Dec/2021	🔴	https://arxiv.org/abs/2112.11446	Dense	Dataset: https://lifearchitect.ai/whats-in-my-ai/
365	GLaM	Google		1200							🆆 📚⬆ 🕸 👥	Dec/2021	🔴	https://arxiv.org/abs/2112.06905	MoE
366	Anthropic-LM 52B	Anthropic		52	400	8:1	0.5				🆆 📚⬆ 🕸 👥	Dec/2021	🔴	https://arxiv.org/abs/2112.00861	Dense	Internal research only
367	RETRO	DeepMind		7.5							🆆 📚⬆ 🕸 🌋	Dec/2021	🔴	https://arxiv.org/abs/2112.04426	Dense	with retrieval
368	DeBERTaV3	Microsoft		1.5	162	108:1	0.1				🆆 📚 ⬆ 🕸	Nov/2021	🟢	https://arxiv.org/abs/2111.09543	Dense	RoBERTa=162B token dataset.
369	BERT-480	Google		480							🆆 📚 🕸	Nov/2021	🔴	https://cloud.google.com/blog/topics/tpus/google-showcases-cloud-tpu-v4-pods-for-large-model-training	Dense	Submission to benchmarks. Original dataset was BookCorpus + Wikipedia: https://arxiv.org/pdf/1810.04805.pdf
370	BERT-200	Google		200							🆆 📚 🕸	Nov/2021	🔴	https://cloud.google.com/blog/topics/tpus/google-showcases-cloud-tpu-v4-pods-for-large-model-training (same as above)	Dense	Submission to benchmarks. Original dataset was BookCorpus + Wikipedia: https://arxiv.org/pdf/1810.04805.pdf
371	Cedille FR-Boris	Coteries	Cedille, TS	6							🆆 📚 🕸 🇫🇷	Nov/2021	🟢	https://github.com/coteries/cedille-ai	Dense	French only. GPT-J.
372	MT-NLG	Microsoft/NVIDIA		530	270	1:1	1.3				🆆 📚⬆ 🌋 🕸 🕸	Oct/2021	🔴	https://arxiv.org/abs/2201.11990	Dense
373	FLAN	Google		137							⬆ 🕸 👥	Sep/2021	🔴	https://arxiv.org/abs/2109.01652	Dense	Fine-tuned LaMDA
374	Command xlarge	Cohere	Cohere	52.4							📚 🕸	Sep/2021	🟢	https://arxiv.org/abs/2108.07790	Dense	Stealth 'ebooks and webpages'. 52B: https://crfm.stanford.edu/helm/v1.0/?models=1
375	PLATO-XL	Baidu	Baidu	11							⬆ 👥	Sep/2021	🟢	https://arxiv.org/abs/2109.09519	Dense	Chatbot. Reddit comments + CN social
376	Macaw	Allen AI	Allen (static demo only)	11							🆀🅰	Sep/2021	🟡	https://arxiv.org/abs/2109.02593	Dense	Chatbot
377	CodeT5	Salesforce		0.7							🕸 BigQuery, BigPython	Mar/2022	🟢	https://arxiv.org/abs/2109.00859	Dense	Code. Large introduced in https://arxiv.org/pdf/2207.01780.pdf
378	Codex	OpenAI	Playground	12							🕸	Aug/2021	🟢	https://arxiv.org/abs/2107.03374	Dense	Code
379	Jurassic-1	AI21	Studio	178	300	2:1	0.8				🆆 📚 ⬆ 🕸	Aug/2021	🟢	https://www.ai21.com/blog/announcing-ai21-studio-and-jurassic-1	Dense	Emulated GPT-3 dataset
380	BlenderBot 2.0	Meta AI		9.4							👥 🌋	Jul/2021	🟢	https://parl.ai/projects/blenderbot2/	Dense	Chatbot
381	GPT-J	EleutherAI	TS, Goose	6	402	67:1	0.2				🆆 📚⬆ 🕸 🌋	Jun/2021	🟢	https://github.com/kingoflolz/mesh-transformer-jax	Dense	Popular
382	LaMDA	Google	YouTube (video only)	137							⬆ 🕸 👥	Jun/2021	🔴	https://arxiv.org/abs/2201.08239	Dense	Chatbot
383	ruGPT-3	Huawei/Sberbank	Sber Cloud	1.3							🕸 "170GB data"	Feb/2021	🟢	https://github.com/sberbank-ai/ru-gpts	Dense	Russian GPT-3 with input from Huawei
384	Switch	Google		1600	576	1:1	3.2				🆀🅰 🕸	Jan/2021	🟢	https://arxiv.org/abs/2101.03961	MoE
385	GPT-3	OpenAI	Sunset/deprecated :-(	175	300	2:1	0.8	43.9			🆆 📚 ⬆ 🕸	May/2020	🟢	https://arxiv.org/abs/2005.14165	Dense	No RLHF (base only). Popular: 3.1M wpm. Dataset: https://lifearchitect.ai/whats-in-my-ai/
386	Megatron-11B	Meta AI	InferKit	11	2200	200:1	0.5				🆆 📚 ⬆ 🕸	Apr/2020	🟢	https://github.com/pytorch/fairseq/tree/main/examples/megatron_11b	Dense	My favourite model until GPT-3 and GPT-4 came along: https://github.com/facebookresearch/fairseq/blob/main/examples/megatron_11b/README.md
387	Transformer++	American Express		0.212	0.1	1:1	0.0				📚	Mar/2020	🔴	https://arxiv.org/abs/2003.04974	Dense	Not to be confused with the more common usage of Transformer++, the ~2023 Transformer++ based on Llama. See Mamba paper.
388	Meena	Google		2.6	10000	3,847:1	0.5				👥 🌋	Jan/2020	🔴	https://arxiv.org/abs/2001.09977	Dense	Dialogue model. Trained 61B tokens for 164x epochs to 10T tokens!
389	T5	Google		11	1000	91:1	0.3				🆀🅰 🕸	Oct/2019	🟢	https://arxiv.org/abs/1910.10683	Dense	C4 + NLP language problems. "compared the following three configurations: First, the standard baseline model, which was pre-trained on 235 ≈ 34B tokens; second, the baseline trained instead for about 1 trillion tokens (i.e. the same amount of pre-training used for T5), which we refer to as “baseline-1T”; and third, T5-Base."
390	Megatron-LM	NVIDIA		8.3	800	97:1	0.3				🆆 📚 ⬆ 🕸	Sep/2019	🟢	https://arxiv.org/abs/1909.08053	Dense
391	RoBERTa	Meta AI	Hugging Face	0.355	2200	6,198:1	0.1	27.9			🆆 📚 ⬆ 🕸	Jul/2019	🟢	https://arxiv.org/abs/1907.11692	Dense	calcs: "In total, this batch size and number of steps corresponds to pre-training on 235 ≈ 34B tokens. This is considerably less than BERT (Devlin et al., 2018), which used roughly 137B tokens, or RoBERTa (Liu et al., 2019c), which used roughly 2.2T tokens. Using only 2 35 tokens results in a reasonable computational budget while still providing a sufficient amount of pre-training for acceptable performance. We consider the effect of pre-training for more steps in Sections 3.6 and 3.7. Note that 2 35 tokens only covers a fraction of the entire C4 data set, so we never repeat any data during pre-training." https://arxiv.org/pdf/1910.10683.pdf MMLU shows RoBERTa-base 125M only=27.9 (not 355M)
392	GPT-2	OpenAI	Hugging Face	1.5	10	7:1	0.0	32.4			⬆	Feb/2019	🟢	https://openai.com/blog/better-language-models/	Dense	Reddit outbound only
393	BERT	Google	Hugging Face	0.34	137	403:1	0.0				🆆 📚	Oct/2018	🟢	https://arxiv.org/abs/1810.04805	Dense
394	GPT-1	OpenAI	Hugging Face	0.117	0.003	1:1	0.0				📚	Jun/2018	🟢	https://openai.com/blog/language-unsupervised/	Dense	Books only. "We train for 100 epochs on minibatches of 64 randomly sampled, contiguous sequences of 512 tokens." =3,276,800
395	ULMFiT	Fast.ai	https://docs.fast.ai/tutorial.text.html	0.034	0.1	3:1	0.0				🆆	Jan/2018	🟢	https://arxiv.org/abs/1801.06146	Dense	Aussie Prof Jeremy Howard: https://www.abc.net.au/news/science/2023-11-15/jeremy-howard-taught-ai-to-the-world-and-helped-invent-chatgpt/103092474
396	Transformer (big)	Google	https://github.com/tensorflow/tensor2tensor?tab=readme-ov-file#walkthrough	0.213	0.1	1:1	0.0				📚	Jun/2017	🟢	https://arxiv.org/abs/1706.03762	Dense	"We trained on the standard WMT 2014 English-German dataset consisting of about 4.5 million sentence pairs... For English-French, we used the significantly larger WMT 2014 English-French dataset consisting of 36M sentences and split tokens into a 32000 word-piece vocabulary. Sentence pairs were batched together by approximate sequence length. Each training batch contained a set of sentence pairs containing approximately 25000 source tokens and 25000 target tokens."
397	Transformer (base)	Google	https://github.com/tensorflow/tensor2tensor?tab=readme-ov-file#walkthrough	0.065	0.1	2:1	0.0				📚	Jun/2017	🟢	https://arxiv.org/abs/1706.03762	Dense	"We trained on the standard WMT 2014 English-German dataset consisting of about 4.5 million sentence pairs... For English-French, we used the significantly larger WMT 2014 English-French dataset consisting of 36M sentences and split tokens into a 32000 word-piece vocabulary. Sentence pairs were batched together by approximate sequence length. Each training batch contained a set of sentence pairs containing approximately 25000 source tokens and 25000 target tokens."
398	About this sheet															About this sheet

	A	B	C	F	G	H
1	About the Models Table and this spreadsheet			Column headers	Format	Description
2	Model count	388		Model	Text	Name of the large language model. Sometimes uses filename syntax.
3	Cite	Thompson, A. D. (2024). Models Table. LifeArchitect.ai. https://lifearchitect.ai/models-table/		Lab	Text	Name of the organization or group responsible for training or publishing the model. Sometimes lists a consortium such as "International". Color highlights popular lab names.
4	Author	Dr Alan D. Thompson		Playground	URI	URI pointing to a playground of the model, or HuggingFace repository for hosting weights.
5	Created date	2021-present		Parameters (B)	Float	Total number of parameters (weights) in the model. Using total weights for Dense, and total weights (not just active weights) for MoE.
6	Permalink	https://lifearchitect.ai/models-table/		Tokens trained (B)	Integer	Total number of tokens (sub-words) used to train the model end-to-end, taking into account reported dataset, epochs, pretraining, and fine-tuning tokens.
7	Homepage	https://lifearchitect.ai/		Ratio Tokens:Params	Ratio	Number of tokens trained per parameter. Chinchilla scaling ≥ 20:1. Color highlights RED=0–7, ORANGE=8–16, GREEN=17–499, DARK GREEN=500–9999.
8	The Memo	https://lifearchitect.ai/memo/		ALScore	Float	Quick and dirty rating of the model's power. The formula is: Sqr Root of (Parameters x Tokens) ÷ 300. Any ALScore ≥ 1.0 is a powerful model in mid-2023. Color highlights centerpoint 15.
9	Timeline view	https://lifearchitect.ai/timeline/		MMLU	Float	Benchmark score 0–100 on Massive Multitask Language Understanding, released Sep/2020: https://arxiv.org/abs/2009.03300 Color highlights centerpoint 80.
10	Script join A	https://onlinetexttools.com/join-text		MMLU-Pro	Float	Benchmark score 0–100 on Massive Multitask Language Understanding Pro, released Jun/2024: https://arxiv.org/abs/2406.01574 Color highlights centerpoint 55.
11	Script join A-D	https://poe.com/TheMemoModelsBot		GPQA	Float	Benchmark score 0–100 on Google-Proof Q&A, released Nov/2023: https://arxiv.org/abs/2311.12022 Color highlights centerpoint 40.
12	Corrections	Corrections are welcomed (with citations).		Training dataset	Symbol	Rough guide of major datasets used to train the model. See key. Note increasing use of synthetic data from 2023.
13		When logged in via Google, select cell, click Insert > Comment		Announced	Date	Date as month/year. All data sorted by this column descending.
14				Public?	Symbol	Ternary: GREEN=publicly accessible (weights, API, playground...), YELLOW=video or scripted demo only, RED=held in lab and never released.
15	Key	Italics: estimates and hypothesis only, based on current information.		Paper / Repo	URI	URI pointing to official paper, technical note, or model card. Sometimes shows link to GitHub repository.
16		🆆 Wikipedia	👥 Dialogue	Arch	Text	Architecture: Dense versus Mixture of Experts (MoE).
17		📚 Books	🆀🅰 Questions and answers	Notes	Text	Any further comments or useful highlights.
18		⬆ Reddit outbound	🌋 Special
19		🕸 Common Crawl	⚛️ Synthetic data
20
21
22	Alternative sheets showing large language models
23	Alternative viz of some frontier models in this sheet (Apr/2024)		https://posts.voronoiapp.com/Technology/The-size-of-LLMs-1020
24	Alternative viz using some of this sheet data (Jul/2023)		https://informationisbeautiful.net/visualizations/the-rise-of-generative-ai-large-language-models-llms-like-chatgpt/
25	Data for training cost trends in machine learning (Jul/2022)		https://docs.google.com/spreadsheets/d/1beiholXkLpiYoeyDL64bOHLI-mXZ-_8ZnY1Ed6Sfjh0/edit#gid=0
26	100+ imitation models with eval/rankings (Jun/2023)		https://docs.google.com/spreadsheets/d/1ikqqIaptv2P4_15Ytzro46YysCldKY7Ub2wcX5H1jCQ/edit#gid=0
27	LLM chat leaderboard, only ~12 imitation models (May/2023)		https://chat.lmsys.org/?leaderboard
28	Limited paper; some inaccuracies (Jun/2023)		https://arxiv.org/abs/2303.18223
29	Anonymous (ends around Mar/2023)		https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYpb63e1ZR3aePczz3zlbJW-Y4/edit#gid=0
30	100+ Chinese LLMs (Jul/2023)		https://twitter.com/AdeenaY8/status/1679435164747960320
31	Stella @ EleutherAI (seems to end Dec/2021)		https://docs.google.com/spreadsheets/d/1gc6yse74XCwBx028HV_cvdxwXkmXejVjkO-Mz2uwE0k/edit#gid=0
32	Jaime @ Epoch AI (seems to end Jun/2021)		https://docs.google.com/spreadsheets/d/1AAIebjNsnJj_uKALHbXNfn3_YsT6sHXtCU0q7OIPuc4/edit#gid=0
33

	A	B	C	D	F	H	J	K	L	M	N	O	P
1	MOVED:	https://docs.google.com/spreadsheets/d/1kc262HZSMAWI6FVsh0zJwbB-ooYvzhCHaHcNUiA0_hY/edit?gid=1158069878#gid=1158069878
2	Google corrupted this one...
3	(39) Permalink:	https://lifearchitect.ai/datasets-table/			Paper: What's in my AI?	https://lifearchitect.ai/whats-in-my-ai/			The Memo:	https://lifearchitect.ai/memo
4	Dataset	Lab	Total tokens (T) ▼	Total size (GB, uncompressed)	Web (CC/C4) data (GB, uncompressed)	Other data (GB, uncompressed)	ALQual (rates quality of data)	Announced	Public?	Model example	Paper	Project page	Notes

5	DCLM-Pool	International	240	1000000	1000000		★☆☆☆☆	Jun/2024	🟡	DCLM-Baseline 7B	https://arxiv.org/abs/2406.11794	https://www.datacomp.ai/dclm/	All CC from 2008-2022, new extraction using resiliparse framework. https://x.com/Vaishaal/status/1803198069888229817/photo/1 "DCLM-POOL contains 200B documents (370TB after gzip compression)"
6	RedPajama-Data-v2	Together AI	30	125000	125000		★★☆☆☆	Oct/2023	🟢			https://together.ai/blog/redpajama-data-v2
7	Piper monorepo	Google	37.9	86000		86000	★★★☆☆	Jun/2023	🔴	DIDACT			Piper PDF from 2016: https://dl.acm.org/doi/pdf/10.1145/2854146
8	The Stack v2	BigCode	20.25	67500		67500	★★☆☆☆	Feb/2024	🟢	StarCoder 2	https://arxiv.org/abs/2402.19173	https://huggingface.co/datasets/bigcode/the-stack-v2	"The Stack v2 is ten times larger than its predecessor, yielding a raw dataset of 67.5 TB. Through extensive cleaning, filtering, and subsampling of the source code, along with the incorporation of other high-quality code-related datasets, we created a training set of approximately 3TB (900B+ tokens)."
9	MNBVC (Massive Never-ending BT Vast Chinese corpus	MNBVC.253874	30	40000	40000		★☆☆☆☆	Oct/2023	🟢		https://github.com/esbatmop/MNBVC	https://mnbvc.253874.net/	Chinese only.
10	Claude-3.5 dataset	Anthropic	20	40000	5000	35000	★★★★★	Jun/2024	🔴	Claude 3.5 Sonnet		https://www.anthropic.com/news/claude-3-5-sonnet	Michael Gerstenhaber, head of product at Anthropic, says the company’s new Claude 3.5 Sonnet model is larger than its predecessor but draws much of its new competence from innovations in training. For example, the model was given feedback designed to improve its logical reasoning skills. https://archive.md/iH4vg & Michael Gerstenhaber, product lead at Anthropic, says that the improvements are the result of architectural tweaks and new training data, including AI-generated data. Which data specifically? Gerstenhaber wouldn’t disclose, but he implied that Claude 3.5 Sonnet draws much of its strength from these training sets. https://techcrunch.com/2024/06/20/anthropic-claims-its-latest-model-is-best-in-class/
11	FineWeb	HF	15	44000	44000		★★☆☆☆	Apr/2024	🟢			https://huggingface.co/datasets/HuggingFaceFW/fineweb	"FineWeb was originally meant to be a fully open replication of 🦅 RefinedWeb"
12	GPT-4 dataset	OpenAI	13	40000			★★★★☆	Mar/2023	🔴	GPT-4			My estimate: https://lifearchitect.ai/gpt-4/#dataset
13	FineWeb-Edu-score-2	HF	5.4	31500	31500		★★★☆☆	May/2024	🟢			https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu-score-2
14	CulturaX	UOregon	6.3	27000	27000		★★★☆☆	Sep/2023	🟢		https://arxiv.org/abs/2309.09400	https://huggingface.co/datasets/uonlp/CulturaX
15	HPLT (High Performance Language Technologies)	Helsinki	5.6	50100	50100		★★☆☆☆	Mar/2024	🟢		https://arxiv.org/abs/2403.14009		"In total, after de-duplication, we release a collection of 5.25 billion documents (approximately corresponding to web pages), totaling 50.1 TB of uncompressed texts and approximately 5.6 trillion whitespace-separated word tokens"
16	RefinedWeb	TII	5	23240	23240		★★☆☆☆	Jun/2023	🔴	Falcon	https://arxiv.org/pdf/2306.01116.pdf
17	MassiveText ML	DeepMind	5	20000	4544	15655	★★★★☆	Dec/2021	🔴	Retro
18	Matrix	International	4.69	21600	2465	2112	★★★☆☆	May/2024	🟢	MAP-Neo	https://arxiv.org/pdf/2405.19327	https://cdn-uploads.huggingface.co/production/uploads/654907a4a1faff97850c4eff/1FWMF_t_Mhy0UQmu65Bb1.png	Combines RedPajama, Dolma, Culturax, Amber, SlimPajama, Falcon, CulturaY
19	Cultura-Y	UOregon	4	16000			★★★☆☆	Mar/2024	🟢	Vistral-7B-Chat		https://www.ontocord.ai/blog/cultura-y
20	DCLM-Baseline	International	4	13000	13000		★★★☆☆	Jun/2024	🟢	DCLM-Baseline 7B	https://arxiv.org/abs/2406.11794	https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0	"All CC from 2008-2022, new extraction using resiliparse framework. https://x.com/Vaishaal/status/1803198069888229817/photo/1 ""DCLM-POOL contains 200B documents (370TB after gzip compression)"""
21	PaLM 2 dataset	Google	3.6	13000			★★★★☆	May/2023	🔴	PaLM 2			My estimate: https://lifearchitect.ai/bard/#dataset
22	Dolma	AI2	3	11519	9832.4	1686.6	★★★☆☆	Aug/2023	🟢	OLMo	https://arxiv.org/abs/2402.00159
23	Infiniset	Google	2.8	12616	1569	11047	★★★★☆	May/2021	🔴	LaMDA			My calcs: https://lifearchitect.ai/bard/#dataset & It's not clear why Google chose to use 1.5TB of the ~14TB history within Wikipedia
24	MADLAD-400	Google	3	12000	12000		★★☆☆☆	Sep/2023	🟢	MADLAD400-8B	https://arxiv.org/abs/2309.04662	https://huggingface.co/datasets/allenai/MADLAD-400
25	MassiveText EN	DeepMind	2.35	10550	5173	5376.5	★★★★☆	Dec/2021	🔴	Chinchilla, Gopher
26	The Stack v1	BigCode	2	6400		6400	★★☆☆☆	Nov/2022	🟢	Megatron-LM fork	https://arxiv.org/abs/2211.15533	https://huggingface.co/datasets/bigcode/the-stack-dedup	Various dedupes down to 2.7TB and 1.5TB.
27	InternLM	Shanghai AI	1.6	5100	3616	1199	★★☆☆☆	Jun/2023	🟡	InternLM			Chinese/English. My rough estimates only by multiplying tokens (billions) by 3 to get GB
28	Stability New Pile	Stability AI	1.5	5000			★★★☆☆	Apr/2023	🔴	StableLM			Announced but not detailed
29	FineWeb-Edu 1.3T	HF	1.3	8840	8840		★★★★☆	May/2024	🟢			https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu
30	LLaMA	Meta AI	1.2	4083	4083	666	★★☆☆☆	Feb/2023	🟡	LLaMA, Alpaca
31	RedPajama	Together AI	1.2	4033	3510	524	★★★☆☆	Apr/2023	🟢	MPT			Open
32	SlimPajama	Cerebras	0.627	2685	706	145	★★★☆☆	Jun/2023	🟢		https://huggingface.co/datasets/cerebras/SlimPajama-627B	https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama	The dataset consists of 59166 jsonl files and is ~895GB compressed. It is a cleaned and deduplicated version of Together's RedPajama.
33	Common Corpus	PleIAS	0.65	2000	2000		★★☆☆☆	Mar/2024	🟢			https://huggingface.co/collections/PleIAs/common-corpus-65d46e3ea3980fdcd66a5613	500tok=650w.
34	ROOTS	BigScience	0.341	1600			★★☆☆☆	Mar/2024	🟢	BLOOM	https://arxiv.org/abs/2303.03915
35	The Pile v1	EleutherAI	0.247	825	227	629.71	★★★★☆	Dec/2020	🟢	GPT-Neo, GPT-J			Some dupes in my older calcs. Deliberately excludes US Congressional minutes (slavery) and literotica (sex).
36	StarCoder dataset (The Stack 1.2 subset)	BigCode	0.25	783		783	★★★☆☆	May/2023	🟢	https://huggingface.co/datasets/bigcode/starcoderdata	https://arxiv.org/abs/2305.06161		It contains 783GB of code in 86 programming languages, and includes 54GB GitHub Issues + 13GB Jupyter notebooks in scripts and text-code pairs, and 32GB of GitHub commits, which is approximately 250 Billion tokens.
37	GPT-3 dataset	OpenAI	0.499	753	620	133.4	★★★☆☆	May/2020	🔴	GPT-3
38	RoBERTa dataset	Meta AI		161	145	16	★★★☆☆	Jul/2019	🟡	RoBERTa, Megatron-11B
39	YouTube-Commons	PleIAS	0.03	110		110	★★★★☆	Apr/2024	🟢		https://huggingface.co/datasets/HuggingFaceTB/cosmopedia	https://huggingface.co/datasets/PleIAs/YouTube-Commons	286x parquet files x 385MB each
40	Cosmopedia v2	HF	0.028	103		103	★★★★★	Jul/2024	🔴	SmolLM		https://huggingface.co/blog/smollm	"Cosmopedia v2 is an enhanced version of Cosmopedia, the largest synthetic dataset for pre-training, consisting of over 30 million textbooks, blog posts, and stories generated by Mixtral-8x7B-Instruct-v0.1. Most of the samples are generated by prompting the model to generate content on specific topics using a web page referred to as a "seed sample", as shown in Figure 1. We use web samples to increase diversity and expand the range of prompts."
41	Cosmopedia v0.1	HF	0.025	92		92	★★★★★	Mar/2024	🟢			https://huggingface.co/blog/cosmopedia	Replication of phi-1.5, very high quality synthetic data
42	GPT-2 dataset	OpenAI		40	40		★★☆☆☆	Feb/2019	🟡	GPT-2		Popular web
43	GPT-1 dataset	OpenAI		4.6		4.6	★☆☆☆☆	Jun/2018	🟡	GPT-1		Books
44	About this sheet												About this sheet

	A	B	C	D	E	F	G	H	I	J	K	L
1	Table by Alan D. Thompson via GPT-4V + GPT-4 translation using data from: https://twitter.com/AdeenaY8/status/1679435164747960320						The Memo:	https://lifearchitect.ai/memo
2	#	Lab	Model	Region	General?	Date	Description	公司 (Company)	大模型名称 (Model Name)	所属领域 (Region)	通用 (General)	说明 (Description)

3	1	Baidu	Wenxin Yiyen	Beijing	✓	Jul/2023	Trial, contact required	百度	文心一言	北京	✓	试用,需要联系
4	2	iFLYTEK	Sibichi	Anhui	✓	Jul/2023	Trial, contact required	科大讯飞	思必驰	安徽	✓	试用,需要联系
5	3	DataGrand	MOOC	Shanghai	✓	Jul/2023	Trial, contact required	达观数据	慕课	上海	✓	试用,需要联系
6	4	Huawei Cloud	Daoyi Tianwen	Xinjiang	✓	Jul/2023	Trial, contact required	华为云	道义天问	新疆	✓	试用,需要联系
7	5	Chongqing Univ.	MOSS	Shanghai	✓	Jul/2023	Open source 6B, Smart AI	重庆大学	MOSS	上海	✓	开源6B, 智慧AI
8	6	Zhixin Technology	ChatGLM	Beijing	✓	Jul/2023	Open source 6B	智芯科技	ChatGLM	北京	✓	开源6B
9	7	Qingmang	Qingmang, Qing+Guang, Qing-Wang	Undisclosed	✓	Jul/2023	Qingmang's victory	轻芒	轻芒,轻芒+光锥,轻芒-王	不详	✓	轻芒+胜场
10	8	Intengine	Daoyi Tianwen	Beijing	✓	Jul/2023	Recommend 3.0, AQUILA intelligence, Aquila-7B, AquilaChat-7B, AquilaCode-7B-NV, AquilaCode-7B-TS	Intengine	道义天问	北京	✓	推荐3.0, 版本, AQUILA天智感, Aquila-7B, AquilaChat-7B, AquilaCode-7B-NV, AquilaCode-7B-TS
11	9	Shanxi Univ.	BenSe	Jiangsu	✓	Jul/2023	Medicine; Recommends LLaMa; Also has a Med-ChatGLM based on ChatGLM	问答赛道山大学	本色	江苏	✓	医学; 推荐LLaMa; 另有基于 ChatGLM 的 Med-ChatGLM
12	10	Shell	BELLE	Beijing	✓	Jul/2023	Recommends multiple types of BLOOM interactive LaMa	贝壳	BELLE	北京	✓	推荐BLOOM交互LaMa的多个类型
13	11	Baichuan Intelligence	Baichuan	Beijing	✓	Jul/2023	Recommends multiple types of BLOOM interactive LaMa	百川智能	baichuan	北京	✓	推荐BLOOM交互LaMa的多个类型
14	12	OpenBMB	CPM	Beijing	✓	Jul/2023	Recommends BLOOM	OpenBMB	CPM	北京	✓	推荐BLOOM
15	13	Intengine	Yingjie: Qingyuan, OpenMEDLab	Shanghai	✓	Jul/2023	Technical report, Shanghai AI Summit recommended Chinese large models	Intengine	英杰:清源, OpenMEDLab	上海	✓	技术报告,上海AI峰会相关推荐中文大量
16	14	Yunhe Technology	Shanhai	Beijing	✓	Jul/2023	Recommend communication	云和智	山海	北京	✓	推荐交流
17	15	NEU	TechGPT	Jiangsu	✓	Jul/2023	Recommends BELLE->LLaMa, Image intelligence and context understanding Q&A	NEU	TechGPT	江苏	✓	推荐BELLE->LLaMa, 图像智能和语境理解问答
18	16	Zhongwen Shenzhen	Ji Wei, Lv Ying	Shenzhen	✓	Jul/2023	University Chinese (Shenzhen) and Shenzhen Technical Alliance Research Institute, Medicine, Demo, Huazhi is based on BLOOMZ	智中文深圳	纪伟, 吕颖	深圳	✓	高校中文大学（深圳）和深圳市技术联盟研究院, 医学,Demo,华和知源是基于 BLOOMZ
19	17	Chinese Academy of Sciences	Enhanced Liu	Beijing	✓	Jul/2023	Enhanced Liu 2.0 claims 100B parameters, worth attention	中科院	增强大刘	北京	✓	增强大刘2.0号称100B参数, 值得关
20	18	Ideal Science & Tech.	TigerBot	Shanghai	✓	Jul/2023	Recommends BLOOM	理想科技	TigerBot	上海	✓	推荐BLOOM
21	19	IDEA Institute	Little Philosopher MindBot	Undisclosed	✓	Jul/2023	Recommend the Tian series models	IDEA研究院	小哲科技MindBot	不详	✓	推荐天系列模型
22	20	Shanghai Jiao Tong Univ.	K2, Magnolia	Shanghai	✓	Jul/2023	Demo, GeoLLaMa, Recommends LLaMa, HuggingFace	上海交通大学	K2,白玉兰	上海	✓	Demo, GeoLLaMa, 推荐LLaMa, HuggingFace
23	21	360	Zhinai, Yijian	Beijing	✓	Jul/2023	Recommends BLOOM	360	智脑,一见	北京	✓	推荐BLOOM
24	22	Du Xiaoman Financial	Qianyan	Beijing	✓	Jul/2023	Recommends BLOOM	度小满	千言	北京	✓	推荐BLOOM
25	23	Dr. Science & Tech. Institute	ProactiveHealthGPT, Heihei, SoulChat	Guangdong	✓	Jul/2023	Recommends Guanaoo->English LLaMa, Use LoRA	科学博士工程技术研究院	ProactiveHealthGPT,嘿嘿,忽如SoulChat	广东	✓	推荐Guanaoo->英LLaMa, 使用LoRA
26	24	Wenzi Technology	Anima	Zhejiang	✓	Jul/2023	Recommends Guanaoo->English LLaMa, Use LoRA	文字科技	Anima	浙江	✓	推荐Guanaoo->英LLaMa, 使用LoRA
27	25	Peking University Law AI Institute	ChatLaw	Beijing	✓	Jul/2023	ChatLaw-13B based on Ziya-LLaMa-13B-v1->LLaMa, ChatLaw-33B based on Anima33B->Guanaco->LLaMa	北京大学法律人工智能研究院	ChatLaw	北京	✓	ChatLaw-13B基于Ziya-LLaMa-13B-v1->LLaMa,ChatLaw-33B基于Anima33B->Guanaco->LLaMa
28	26	Xiangde Technology Co., Ltd.	Mu Yuan	Beijing	✓	Jul/2023		想得科技有限公司	慕元	北京	✓
29	27	Horgos	MiniMax	Shanghai	✓	Jul/2023	GLOW social rendering	霍尔果斯	MiniMax	上海	✓	GLOW渲染社交
30	28	Tencent Cloud	Tencent	Xinjiang	✓	Jul/2023		腾讯云	腾讯	新疆	✓
31	29	Competitive Tech + Chongqing Review Network	Competitive XPT	Sichuan	✗	Jul/2023		比赛科技+重庆复盘网	比赛型XPT	四川	✗
32	30	Institute of Computing Technology, CAS	White Horse	Beijing	✓	Jul/2023	Based on LLaMa, aims for Diff download of 7B and 13B, demo	中国科学院计算技术研究所	白马	北京	✓	基于LLaMa, 目标Diff下载7B和13B,demo
33	31	Beijing Language and Culture University	Bang Bang	Beijing	✓	Jul/2023	Based on LLaMa, academic collaboration with Tsinghua, Peking University	北京语言大学	棒棒	北京	✓	基于LLaMa,北语清华学术共同, 北京大学
34	32	SenseTime	Ri Ri Xin	Shanghai	✓	Jul/2023		商汤科技	日日新	上海	✓
35	33	National Supercomputing Center in Tianjin	Tianjin Tianyuan	Tianjin	✗	Jul/2023		国家超级计算天津中心	天津天元	天津	✗
36	34	Guoke Technology	Weightless, Saisen	Shanghai	✗	Jul/2023	Weightless - Finance; Saisen - Big data analysis	国科科技	无重量, 赛森	上海	✗	无重量—金融; 赛森—大数据分析
37	35	Competitive Tech + Tianjin University	Haihe Mint	Tianjin	✗	Jul/2023		比赛科技+天津大学	海河·造币厂	天津	✗
38	36	Bian Sheng Electronics	LightGPT	Xinjiang	✗	Jul/2023		便生电子	LightGPT	新疆	✗
39	37	Telecom Smart Science	Star Print	Beijing	✗	Jul/2023	General vision, China Telecom	电信智科	星印	北京	✗	通用视觉, 中国电信
40	38	Xiamen Yunji Technology	Xiamen GPT	Beijing	✓	Jul/2023	Medical, Nursing Key	厦门云集	厦云GPT	北京	✓	医疗, 护理Key
41	39	Wisdom Eye	Jing Shi	Hunan	✗	Jul/2023	Medically oriented	智慧眼	镜石	湖南	✗	医疗导向
42	40	TAL Education Group	MathGPT	Beijing	✗	Jul/2023	Xueersi (a brand of TAL)	好未来	MathGPT	北京	✗	学而思
43	41	Shugan Space-Time	Great Wall	Beijing	✗	Jul/2023	Natural resources, remote sensing	数感时空	长城	北京	✗	自然资源, 遥感
44	42	Ideal Science & Technology	Da Dao	Beijing	✗	Jul/2023	Legal representative type	理想科技	大道Dao	北京	✗	法律代表型
45	43	Huisheng Intelligence	Zhi Xin	Liaoning	✗	Jul/2023		慧生智能	智芯	辽宁	✗
46	44	China Internet Network	Zhi Gong	Beijing	✗	Jul/2023	Cooperative with the National NLP Laboratory, Industrial intelligence	中国互联网	智工	北京	✗	与全国NLP实验室联合, 工业智能
47	45	Entrepreneurial Dark Horse	Tian Qi	Beijing	✗	Jul/2023	Entrepreneurial Dark Horse in cooperation with 360, and venture capital services industry	创业黑马	天启	北京	✗	创业黑马与360合作,和创投服务行业
48	46	Together Technology	Bowen	Dongguan	✗	Jul/2023		一起科技	博文Bowen	东莞	✗
49	47	NetEase Youdao	Yu Chuan	Beijing	✗	Jul/2023		网易有道	于川	北京	✗
50	48	NetEase Youdao	Wang Yan	Guangdong	✗	Jul/2023		网易有道	王言	广东
51	49	Undecided	Tian Ji	Beijing	✓	Jul/2023	Five-dimensional interactive experience	未定	天玑	北京	✓	5维智能互动体验
52	50	Zhihu	Zhihu Think Tank	Beijing	✗	Jul/2023	Collaboration between Zhihu and Think Tank Technology	知乎	知乎智库	北京	✗	知乎和智库科技合作
53	51	EasyGo Network Science	Uni-talk	Shanghai	✗	Jul/2023	Supported by Shanghai R&D Center for Intelligent Vehicles, SJTU	易行网科	Uni-talk	上海	✗	上海交通大学山西智能车上海研发中心支持
54	52	Luwen Education	Luwen	Xinjiang	✗	Jul/2023	Capable of voice and multimodal tasks simultaneously	路文教育	路文	新疆	✗	接活语音和多模态两不误
55	53	Chinese Academy of Sciences	Rubik's Cube	Beijing	✗	Jul/2023		中科创达	魔方Rubik	北京	✗
56	54	Tencent	Bubble	Guangdong	✗	Jul/2023		腾讯	泡泡	广东	✗
57	55	TikTok Vision	DouTian	Beijing	✗	Jul/2023		抖音愿景	抖天	北京	✗
58	56	Leyan Technology	Leyan	Shanghai	✗	Jul/2023	TRSGPT	乐言科技	乐言	上海	✗	TRSGPT
59	57	DiDi Intelligence	Xiang	Beijing	✗	Jul/2023	Based on Qiantang experience data	滴滴智能	先向	北京	✗	基于钱塘体验数据
60	58	ZhiZi Engine	Metaverse	Jiangsu	✓	Jul/2023		智子引擎	元宇宙	江苏	✓
61	59	TikTok Technology	TikTok	Jiangxi	✗	Jul/2023		抖音科技	抖音	江西	✗
62	60	Micro Environment Intelligence	RongGu	Beijing	✓	Jul/2023	Micro Environment Intelligence, iFLYTEK, Huawei	微环智能	融古	北京	✓	微环智能,科大讯飞,华为
63	61	Evernote	Elephant GPT	Beijing	✗	Jul/2023		印象笔记	大象GPT	北京	✗
64	62	Hummingbird Unity	Hummingbird	Beijing	✓	Jul/2023		蜂鸟团结	蜂鸟	北京	✓
65	63	Cosmos Leap	Grace	Beijing	✗	Jul/2023	Internal code name	宇宙跃动	Grace	北京	✗	内部代号
66	64	Aomen Nuomen	Kang Jia Nuo	Beijing	✓	Jul/2023		凹门诺门	康嘉诺	北京	✓
67	65	Digital Tribe Technology	SocialGPT	Guangdong	✗	Jul/2023		数族技术	SocialGPT	广东	✗
68	66	CloudWalk Technology	Cong Rong	Guangdong	✓	Jul/2023		云从科技	从容	广东	✓
69	67	EKT University	Xiao Ke	Beijing	✗	Jul/2023	Officially used by our institute for the industry	电科大校	小可	北京	✗	我院正式业界用
70	68	Agricultural Bank of China	Honeybee ChatABC	Beijing	✗	Jul/2023		中国农业银行	小蜜ChatABC	北京	✗
71	69	Tencent Fusion	Tianlai AllMe	Beijing	✓	Jul/2023		腾讯融合	天籁AllMe	北京	✓
72	70	Taiwan Liquor Cloud	EnSpace FFM	Taiwan	✓	Jul/2023	Cross-border cloud company	台酒云	恩斯佩斯FFM	台湾	✓	跨境云公司
73	71	MedTech	medGPT	Sichuan	✗	Jul/2023		医医科技	medGPT	四川	✗
74	72	Super Thinking Science	MindGPT	Beijing	✗	Jul/2023		超思科学	MindGPT	北京	✗
75	73	Spirit Realm Multi-AI	Dongni	Beijing	✓	Jul/2023		灵境多人工智能	Dongni	北京	✓
76	74	Changhong IT	Changhong Totem	Henan	✗	Jul/2023		长红IT	长红图腾	河南	✗
77	75	KidsKing	KidsGPT	Jiangsu	✗	Jul/2023		孩子王	KidsGPT	江苏	✗
78	76	CAS Wenhao	Dao Yi	Beijing	✗	Jul/2023		中科闻道	道意	北京	✗
79	77	DiDi Technology	Lan Zi	Beijing	✓	Jul/2023		滴滴科技	蓝子	北京	✓
80	78	JD.com	Ji Xing, ChatJD	Beijing	✗	Jul/2023		京东	吉星,ChatJD	北京	✗
81	79	Zhishuan Intelligence	Hua Jun	Shanghai	✗	Jul/2023	Novel-writing robot	智算智能	华峻	上海	✗	小说机器人
82	80	H3C New Hua III	White Cloud House	Xinjiang	✗	Jul/2023		新华三H3C	白山云屋	新疆	✗
83	81	Tencent Blue Whale	Tencent Brain·Brainsea	Guangdong	✗	Jul/2023	Peng Cheng Mind	腾讯蓝鲸	腾讯脑·脑海	广东	✗	Peng Cheng Mind
84	82	Uniview Technologies	Wise Eye	Xinjiang	✗	Jul/2023	AIoT industry	宇视科技	慧目	新疆	✗	AIoT行业
85	83	China Unicom	Yu Xiang	Beijing	✗	Jul/2023		中国联通	宇翔	北京	✗
86	84	Meituan Technology	Bumblebee	Shenzhen	✗	Jul/2023	Public safety	美团技术	大黄蜂	深圳	✗	公共安全
87	85	Purple Power Technology	Darwin	Guangdong	✗	Jul/2023	Meituan Power, Triangle Research Institute with Tsinghua University, Tsinghua Bio-motion, Hexagonal District Innovation Service Center	紫天动力科技	达尔文	广东	✗	美团动力,清华大校三角研究院,清华生物动,六角区科技创新服务中心
88	86	Real Smart	Zhao Bin	Xinjiang	✗	Jul/2023	TARS	实在智能	赵斌	新疆	✗	TARS
89	87	Jiadu Technology	Jiadu Zhiyin	Guangdong	✗	Jul/2023	Transportation domain	佳都科技	佳都知音	广东	✗	交通领域
90	88	Wisdom Environment Research Institute	Wisdom	Guangdong	✓	Jul/2023	Based on LLaMa, comprehensive insurance package	智慧环境研究院	智慧	广东	✓	基于LLaMa, 团团包赔体
91	89	Chip Cloud Research Institute	EmoGPT, EduChat	Shanghai	✗	Jul/2023	EmoGPT is supported by Shanghai's mental health and intelligence project, in cooperation with Smart Tech company, EduChat is based on BELLE (BELLE is based on LLaMa)	芯云研究院科学	EmoGPT,EduChat	上海	✗	EmoGPT是上海市心理健康与智能机行项目点支撑与与智能科技公司合作完成, 教学教育大规模EduChat基于BELLE (BELLE基于LLaMa)
92	90	Yandao Intelligent	ArynGPT	Jiangsu	✗	Jul/2023		演道智能	ArynGPT	江苏	✗
93	91	Tencent	WAI	Shanghai	✓	Jul/2023		腾讯	WAI	上海	✓
94	92	Northwestern Polytechnical University	Purple Light·Observation	Shaanxi	✗	Jul/2023	Fluid dynamics large model, aerospace	西北工业大学华为技术	紫光·观测	陕西	✗	流体力学大模型,航空航天
95	93	Singularity Intelligence	Singularity OpenAPI	Beijing	✓	Jul/2023	Intelligence and balance scales	奇点智能	Singularity OpenAPI	北京	✓	智机和天秤
96	94	Lenovo Technology	Lenovo	Xinjiang	✓	Jul/2023	OmModel Om Manufacturing Workshop (First Love Archive) large model	联想科技	联想	新疆	✓	OmModel欧姆智造坊（初恋档案）大模型
97	95	ShanghaiTech University	DoctorGLM	Shanghai	✓	Jul/2023	Medical large model, iterative	上海科技大学	DoctorGLM	上海	✓	医学大模型, 迭次
98	96	Xuanen High	Zhiwei Couple System	Jiangsu	✗	Jul/2023		宣恩高	知味夫妇系统	江苏	✗
99	97	Hong Kong University of Science and Technology	Robin Robin	Hong Kong	✓	Jul/2023	Based on LLaMa, open-source LMFlow by HKUST	香港科技大学	罗宾Robin	香港	✓	基于LLaMa,港科大开源LMFlow
100	98	Shengong Communication	Source	Shandong	✗	Jul/2023	Source	深港通信	源	山东	✗	源
101	99	China Mobile	Da Tian	Beijing	✗	Jul/2023		中国移动	大天	北京	✗
102	100	China Telecom	TeleChat	Beijing	✗	Jul/2023		中国电信	TeleChat	北京	✗
103	101	RongCloud	Fan Ke	Beijing	✗	Jul/2023	Customer service, marketing	融云云	范克	北京	✗	客服, 营销
104	102	Cloud Flying Technology	Heavenly Book	Guangdong	✗	Jul/2023		云天励飞	天书	广东	✗
105	103	Smart Technology	CityGPT	Shanghai	✗	Jul/2023	City large model	智能科技	CityGPT	上海	✗	城市大模型
106
107
108
109
110
111
112
113
114
115

	A	B	C	D	E	F	G	H	I	J	L
1	Model	Training end	Chip type	TFLOP/s (max)	Chip count	Wall clock time (days)	Total time (hours)	Total time (years)	Retail cost ($US)	MMLU ▼	Check (calculated column)
2	GPT-3	Apr/2020	V100	130	10,000	15 days	3,552,000	405 years	$9M	43.9	$8,808,960
3	Llama 1	Jan/2023	A100	312	2,048	21 days	1,032,192	118 years	$4M	63.4	$4,056,515
4	Llama 2	Jun/2023	A100	312	2,048	35 days	1,720,320	196 years	$7M	68.0	$6,760,858
5	Titan	Apr/2023	A100	312	13,760	48 days	11,558,400	1,319 years	$45M	70.4	$45,424,512
6	GPT-4	Aug/2022	A100	312	25,000	95 days	57,000,000	6,503 years	$224M	86.4	$224,010,000
7	Gemini	Nov/2023	TPUv4	275	57,000	100 days	136,800,000	15,606 years	$440M	90.0	$440,496,000
8	Llama 3 70B	Apr/2024	H100	989	24,576	11 days	6,300,000	719 years	$7M	82.0	$7,560,000
9	Llama 3 405B	Apr/2024	H100	989	24,576	50 days	29,491,200	3,364 years	$125M	85.0	$125,337,600
10	GPT-5	Mar/2024	H100	989	50,000	120 days	144,000,000	16,428 years	$612M		$612,000,000
11	Olympus	Aug/2024	H100	989							$0
12	Grok 2	Jun/2024	H100	989	20,000	50 days	57,600,000	6,571 years	$245M		$244,800,000
13	Gemini 2	Nov/2024	TPUv6	1847							$0
14	Grok 3	Dec/2024	H100	989	100,000	50 days	288,000,000	32,855 years	$1.2B		$1,224,000,000
15											$0
16
17	Model	Source type (primary, analysis, informed estimate)	Link	Quote		Chip type	Pricing date	$ per chip-hour	Source	1M hours
18	GPT-3	Primary	https://arxiv.org/abs/2005.14165	“All models were trained on V100 GPU’s on part of a high-bandwidth cluster provided by Microsoft.”		V100	2020	$0.66	https://web.archive.org/web/20200611063415/https://lambdalabs.com/service/gpu-cloud	$660,000	Big pricing disparity between lambda and GCP
19		Analysis	https://arxiv.org/pdf/2204.05149.pdf	Note: this is a nearly-primary source by authors including Google's Dr Jeff Dean. “GPT-3 was trained on 10,000 V100 GPUs... GPT-3 took 405 V100 years to train in 2020.”		V100	2020	$2.48	https://www.top500.org/news/google-expands-its-gpu-cloud-options/	$2,480,000	Using this number for GPT-3 in final table
20	GPT-4	Analysis	https://www.semianalysis.com/p/google-gemini-eats-the-world-gemini cited in https://www.lesswrong.com/posts/tJAD2LG9uweeEfjwq/estimating-efficiency-improvements-in-llm-pre-training	“According to SemiAnalysis, GPT-4 was trained on 25,000 A100’s for roughly 95 days”		A100	2023	$3.93	https://gpus.llm-utils.org/a100-gpu-cloud-availability-and-pricing/	$3,930,000
21	Llama 1	Primary	https://arxiv.org/abs/2302.13971	“When training a 65B-parameter model, our code processes around 380 tokens/sec/GPU on 2048 A100 GPU with 80GB of RAM. This means that training over our dataset containing 1.4T tokens takes approximately 21 days.”		H100	2023	$4.25	https://web.archive.org/web/20240108002155/https://coreweave.com/gpu-cloud-pricing	$4,250,000
22	Llama 2	Primary	https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md	“Llama 2 70B: Time (GPU Hours) = 1720320”		TPUv4	2023	$3.22	https://web.archive.org/web/20240105115832/https://cloud.google.com/tpu/pricing	$3,220,000
23		Estimate only		2,048 chips is an assumption based on Llama 1. Could/should be 2x-20x.		TPUv5e	2024	$1.20	https://web.archive.org/web/20240105115832/https://cloud.google.com/tpu/pricing	$1,200,000
24	Titan	Analysis	https://importai.substack.com/p/import-ai-365-wmd-benchmark-amazon	"200B dense model on 4T tokens of data across 13,760 NVIDIA A100 chips (using 1,720 P4d nodes). It took 48 days to train. "		TPUv5p	2024	$4.20	https://web.archive.org/web/20240105115832/https://cloud.google.com/tpu/pricing	$4,200,000
25		Primary	https://aws.amazon.com/blogs/aws/build-rag-and-agent-based-generative-ai-applications-with-new-amazon-titan-text-premier-model-available-in-amazon-bedrock/	"MMLU=70.4"		TPUv6 (Trillium)	2024			$0
26	Gemini	Primary	https://arxiv.org/abs/2312.11805	“Training Gemini Ultra used a large fleet of TPUv4 accelerators across multiple datacenters.”
27		Analysis	https://www.semianalysis.com/p/google-gemini-eats-the-world-gemini cited in https://www.lesswrong.com/posts/tJAD2LG9uweeEfjwq/estimating-efficiency-improvements-in-llm-pre-training	“The TPUv4 has a maximum performance of 275 TFLOP/s in bf16.”
28		Analysis	https://www.semianalysis.com/p/google-gemini-eats-the-world-gemini cited in https://www.lesswrong.com/posts/tJAD2LG9uweeEfjwq/estimating-efficiency-improvements-in-llm-pre-training	“According to SemiAnalysis... Gemini Ultra was trained on roughly 57.000 TPUv4’s for 100 days.”		This sheet is owned and maintained by Dr Alan D. Thompson at LifeArchitect.ai.
29	GPT-5	Estimate only	https://lifearchitect.ai/gpt-5/	Alan's estimate based on comparison and extrapolating from Morgan Stanley research note and other sources. May be 2x-5x.
30	Llama 3	Primary	https://blogs.nvidia.com/blog/meta-llama3-inference-acceleration/#:~:text=Meta%20engineers%20trained%20Llama%203,NVIDIA%20Quantum%2D2%20InfiniBand%20networks.	"Meta engineers trained Llama 3 on computer clusters packing 24,576 NVIDIA H100 Tensor Core GPUs"
31		Primary	https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md	"Llama 3 70B 6.4M [GPU hours]" extrap to 405B params
32	Olympus
33	Grok 2	Primary	https://www.datacenterdynamics.com/en/news/elon-musk-xais-grok-2-requires-20000-nvidia-h100-gpus-grok-3-may-need-100000/	"In an interview with Norway wealth fund CEO Nicolai Tangen on Twitter/X spaces... Musk said training the Grok 2 model takes about 20,000 Nvidia H100 GPUs... training the Grok 3 model and beyond will require 100,000 Nvidia H100s."
34	Gemini 2
35	Grok 3	Primary	https://www.datacenterdynamics.com/en/news/elon-musk-xais-grok-2-requires-20000-nvidia-h100-gpus-grok-3-may-need-100000/	"In an interview with Norway wealth fund CEO Nicolai Tangen on Twitter/X spaces... Musk said training the Grok 2 model takes about 20,000 Nvidia H100 GPUs... training the Grok 3 model and beyond will require 100,000 Nvidia H100s."
36	Other	1. Training end: Most training end dates assume 1 month before release for easy figures.
37		2. Total time (hours): column hidden in final report.
38		3. US$ estimate: Cost estimates using GCP in 2023: A100 @$3.93/hr, TPUv4 @$3.22/hr, H100 @$4.25/hr.
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010

	A	B	C	D	E	F	G	H	I
1	Field	Achievement	Result	Outperforms human avg?	Engine	Testing date	Peer- reviewed?	Paper/ link	Extract

2	Humor	ChatGPT is funnier than humans.	-	Yes	ChatGPT	Jul/2024	Yes	🔗	"ChatGPT outperformed the majority of our human humor producers on each task. ChatGPT 3.5 performed above 73% of human producers on the acronym task, 63% of human producers on the fill-in-the-blank task, and 87% of human producers on the roast joke task."
3	Legal	Claude 3 Opus works at least 5,000 times faster than humans do, while producing work of similar or better quality…	-	Yes	Claude 3 Opus	Jun/2024	No	🔗	"Of the 37 merits cases decided so far this Term, Claude decided 27 in the same way the Supreme Court did. In the other 10 (such as Campos-Chaves), I frequently was more persuaded by Claude’s analysis than the Supreme Court’s…" Adam Unikowsky is a biglaw partner, a former law clerk to Justice Antonin Scalia, and has won eight Supreme Court cases as lead counsel. Using Claude 3 Opus, he explores the potential of AI in adjudicating Supreme Court cases.
4	Investment	GPT-4’s stock selection accuracy is as high as 60% vs humans at 52%.	-	Yes	GPT-4	May/2024	No	🔗	"With the step-by-step prompts, GPT-4 achieved a prediction accuracy of 60.35 per cent, significantly higher than the 52.71 per cent accuracy of human analysts. Moreover, GPT-4’s F1-score, which balances the accuracy and relevance of predictions, also outperformed that of the human analysts."
5	Investment	GPT-4 returned 15% on the stock market (limited evidence, 11 test runs over 12 months)	-	Yes	GPT-4	May/2024	No	🔗	"The average return for the two LLMs used by the GPT Investor is as follows: GPT-4: 15.54% (with a corresponding average $SPY return of 9.74%)"
6	Information security	GPT-4 can exploit zero-day security vulnerabilities all by itself	87	Yes	GPT-4	Apr/2024	Yes	🔗	"When given the CVE description, GPT-4 is capable of exploiting 87% of these vulnerabilities compared to 0% for every other model we test"
7	Psychology	GPT-4 outperforms 100% of psychologists in Social Intelligence	92.18	Yes	GPT-4	Apr/2024	Yes	🔗	"In ChatGPT-4, the score on the SI scale was 59, exceeding 100% of specialists, whether at the doctoral or the bachelor’s levels.... the average scores were 39.19 of bachelor’s students and 46.73 of PhD holders. While the raw scores of the AI models were treated as representing independent individual samples (one total score for each model); the scores of SI were 59 of GPT4" https://www.psypost.org/chatgpt-4-outperforms-human-psychologists-in-test-of-social-intelligence-study-finds/
8	Psychiatry	GPT-4 outperforms human psychiatrists	75	Yes	GPT-4	Apr/2024	Yes	🔗	"GPT-4 performance was highest in psychiatry, with a median 75th percentile among physicians (95% CI, 66.3 to 81.0)... Compared with the performance of 849 physicians who took the board medical examination in 2022, GPT-4 performed above the median physician in internal medicine and psychiatry and ranked above a considerable fraction of physicians in other disciplines."
9	Art (via prompting Midjourney)	GPT-4 outperforms humans in image creation	-	Yes	GPT-4	Mar/2024	Yes	🔗	"We conduct an extensive human evaluation experiment, and find that AI excels human experts, and Midjourney is better than the other text-to-image generators... On average, prompts improve 20.39% for short text descriptions compared to human-generated social media creatives"
10	Persuasion/argument/debate	Persuasion: GPT-4 better than human debater	-	Yes	GPT-4	Mar/2024	Yes	🔗	"participants who debated GPT-4 with access to their personal information had 81.7% (p < 0.01; N=820 unique participants) higher odds of increased agreement with their opponents compared to participants who debated humans. Without personalization, GPT-4 still outperforms humans, but the effect is lower and statistically non-significant (p=0.31)."
11	Aerospace	GPT-4 and GPT-4V can help fly a plane	-	Yes	GPT-4	Mar/2024	Yes	🔗	‘[GPT-4V and GPT-4 was used] to interpret and generate human-like text from cockpit images and pilot inputs, thereby offering real-time support during flight operations. To the best of our knowledge, this is the first work to study the virtual co-pilot with pretrained LLMs for aviation... The case study revealed that GPT-4, when provided with instrument images and pilot instructions, can effectively retrieve quick-access references for flight operations. The findings affirmed that the V-CoP can harness the capabilities of LLM to comprehend dynamic aviation scenarios and pilot instructions.’
12	Relationships	ChatGPT converses with 5,239 girls for Russian programmer.	-	Yes	GPT-4	Feb/2024	No	🔗	‘In total, the bot met 5,239 girls, out of which Alexander selected four most suitable ones. Ultimately, he chose one of them named Karina…’
13	Academia	GPT-4 writes better essays than humans	-	Yes	GPT-4	30/Oct/2023	Yes	🔗	"ChatGPT generates essays that are rated higher regarding quality than human-written essays."
14	Psychotherapy	Seligman: “This is a rare moment in the history of scientific psychology: [GPT-4] now promises much more effective psychotherapy and coaching.”	-	Yes	GPT-4	29/Sep/2023	Yes	🔗	https://www.eurekalert.org/news-releases/1003232
15	Legal	ChatGPT officiates wedding	-	Yes	ChatGPT	5/Jul/2023	No	🔗	‘ChatGPT planned the welcome, the speech, the closing remarks — everything except the vows — making ChatGPT, in essence, the wedding officiant.’
16	Chemistry	GPT-4 helps with ‘instructions, to robot actions, to synthesized molecule.’	-	Yes	GPT-4	19/Jun/2023	Yes	🔗	‘We report a model that can go from natural language instructions, to robot actions, to synthesized molecule with an LLM. We synthesized catalysts, a novel dye, and insect repellent from 1-2 sentence instructions. This has been a seemingly unreachable goal for years!’
17	Chip design	ChatGPT helps design an accumulator, part of a CPU	-	Yes	ChatGPT	22/May/2023	Yes	🔗	‘two hardware engineers “talked” in standard English with ChatGPT-4 – a Large Language Model (LLM) built to understand and generate human-like text type – to design a new type of microprocessor architecture. The researchers then sent the designs to manufacture.’ Paper: https://arxiv.org/abs/2305.13243
18	Medical	ChatGPT 'higher quality' and 'more empathetic' than human doctors	-	Yes	ChatGPT	28/Apr/2023	Yes	🔗	Chatbot responses were rated of significantly higher quality than physician responses… 9.8 times higher prevalence of empathetic or very empathetic responses for the chatbot.
19	Lottery	Human wins lottery with numbers provided by ChatGPT (this is tongue-in-cheek, but it did happen!)	-	Yes	ChatGPT	17/Apr/2023	No	🔗	Patthawikorn Boonrin revealed that he put in a few hypothetical questions... and received the numbers 57, 27, 29, and 99 from the chatbot... the numbers ended up winning him a [lottery] prize...
20	Quantum computing	GPT-4 achieves a 'B' grade (73/100) on exam.	73	No	GPT-4	11/Apr/2023	No	🔗	The result: GPT-4 scored 73 / 100. (Because of extra credits, the max score on the exam was 120, though the highest score that any student actually achieved was 108.) For comparison, the average among the students was 74.4 (though with a strong selection effect—many students who were struggling had dropped the course by then!). While there’s no formal mapping from final exam scores to letter grades (the latter depending on other stuff as well), GPT-4’s performance would correspond to a solid B.
21	Jurisprudence/ legal rulings	ChatGPT helps a judge with a verdict (India).	-	-	ChatGPT	29/Mar/2023	No	🔗	‘Armed with [ChatGPT's] legal expertise, Chitkara ultimately rejected the defendant’s bail bid on the grounds that they did act cruelly before the victim died.’
22	Japan: National Medical Licensure Examination	Bing Chat would achieve 78% [above cut-off grade of 70%], ChatGPT would achieve 38%	78	Yes	Bing Chat	9/Mar/2023	Yes	🔗	‘The accuracy of ChatGPT was lower than prior studies using the United States Medical Licensing Examination. The limited amount of Japanese language data may have affected the ability of ChatGPT to correctly answer medical questions in Japanese... Bing has an accuracy level to pass the national medical licensing exam in Japan.’
23	Spanish medical examination (MIR)	Bing Chat would achieve 93%, ChatGPT would achieve 70%, both above cut-off grade	93	Yes	Bing Chat	2/Mar/2023	No	🔗	‘I asked 185 questions, excluding the 25 that required images, which I removed. To balance the exam, I added the 10 reserve questions for the challenges. Out of the 185 questions, Bing Chat answered 172 correctly and failed on 13, resulting in a success rate of 93%’
24	Cover of TIME magazine	ChatGPT made the 27/Feb/2023 cover of TIME magazine.	-	Yes	ChatGPT	27/Feb/2023	No	🔗	Alan: this one is not really an ability, but definitely an achievement!
25	CEO	ChatGPT appointed to CEO of CS India.	-	-	ChatGPT	9/Feb/2023	No	🔗	‘As CEO, ChatGPT will be responsible for overseeing the day-to-day operations of CS India and driving the organization's growth and expansion. ChatGPT will use its advanced language processing skills to analyze market trends, identify new impact opportunities, and develop strategies...’
26	Software dev job	ChatGPT would be hired as L3 Software Developer at Google: the role pays $183,000/year.	-	Yes	ChatGPT	31/Jan/2023	No	🔗	https://www.pcmag.com/news/chatgpt-passes-google-coding-interview-for-level-3-engineer-with-183k-salary https://www.cnbc.com/2023/01/31/google-testing-chatgpt-like-chatbot-apprentice-bard-with-employees.html "ChatGPT gets hired at L3 when interviewed for a coding position"
27	Jurisprudence/ legal rulings	ChatGPT helps a judge with a verdict (Colombia).	-	-	ChatGPT	31/Jan/2023	No	🔗	English: https://interestingengineering.com/innovation/chatgpt-makes-humane-decision-columbia Spanish: https://www.bluradio.com/judicial/sentencia-la-tome-yo-chatgpt-respaldo-argumentacion-juez-de-cartagena-uso-inteligencia-artificial-pr30 "On January 31, the first labor court of Cartagena resolved a guardianship action with the help of the famous artificial intelligence known as ChatGPT, arguing that it applied Law 2213 of 2022, which says that in certain cases these virtual tools can be used."
28	Politics	ChatGPT writes several Bills (USA).	-	-	ChatGPT	26/Jan/2023	Yes	🔗	Regulate ChatGPT: https://malegislature.gov/Bills/193/SD1827 Mental health & ChatGPT: https://malegislature.gov/Bills/193/HD676
29	MBA	ChatGPT would pass an MBA degree exam at Wharton (UPenn).	B/B-	Yes	ChatGPT	22/Jan/2023	Yes	🔗	"Considering this performance, ChatGPT would have received a B to B- grade on the exam."
30	Accounting	GPT-3.5 would pass the US CPA exam.	57.6%	Yes	text-davinci-003	11/Jan/2023	Yes	🔗	"the model answers 57.6% of questions correctly"
31	Legal	GPT-3.5 would pass the bar in the US.	50.3%	Yes	text-davinci-003	29/Dec/2022	Yes	🔗	"GPT-3.5 achieves a headline correct rate of 50.3% on a complete NCBE MBE practice exam"
32	Medical	ChatGPT would pass the United States Medical Licensing Exam (USMLE).	>60%	Yes	ChatGPT	20/Dec/2022	Yes	🔗	"ChatGPT performed at >50% accuracy across all examinations, exceeding 60% in most analyses. The USMLE pass threshold, while varying by year, is approximately 60%. Therefore, ChatGPT is now comfortably within the passing range."
33	IQ (fluid/aptitude)	ChatGPT outperforms college students on the Raven's Progressive Matrices aptitude test.	>98%	Yes	text-davinci-003	19/Dec/2022	Yes	🔗	More info at: https://lifearchitect.ai/ravens/
34	AWS certificate	ChatGPT would pass the AWS Certified Cloud Practitioner exam.	80%	Yes	ChatGPT	8/Dec/2022	No	🔗	"Final score: 800/1000; a pass is 720"
35	IQ (verbal only)	ChatGPT scores IQ=147, 99.9th %ile.	>99.9%	Yes	ChatGPT	6/Dec/2022	No	🔗	"Psychology Today Verbal-Linguistic Intelligence IQ Test, it gets a score of 147!"
36	SAT exam	ChatGPT scores 1020/1600 on SAT exam.	52%	Yes	ChatGPT	2/Dec/2022	No	🔗	"According to collegeboard, a 1020/1600 score is ~52nd percentile."
37	General knowledge	GPT-3 would beat IBM Watson on Jeopardy! questions.	100%	Yes	davinci	20/Sep/2021	No	🔗	Watson scored 88%, GPT-3 scored 100%.
38	IQ (Binet-Simon Scale, verbal only)	GPT-3 scores in 99.9th %ile (estimate only).	99.9%	Yes	davinci	11/May/2021	No	🔗	"As of 2021, I expect that it would not be simple to assess the intelligence of an AI using current IQ instrument design... some subtests where the AI would be easily in the top 0.01% of the world population (processing speed, memory), while other subtests may be far lower."
39	General knowledge	GPT-3 outperforms average humans on trivia.	73%	Yes	davinci	12/Mar/2021	No	🔗	"GPT-3 got 73% of 156 trivia questions correct. This compares favorably to the 52% user average."
40	Reasoning	GPT-3 would pass the SAT Analogies subsection.	65.2%	Yes	davinci	28/May/2020	Yes	🔗	"GPT-3 achieves 65.2% in the few-shot setting... average score among college applicants was 57% (random guessing yields 20%)."
41
42
43		See more benchmarks for large language models 2020-2023:			https://lifearchitect.ai/iq-testing-ai/
44		This sheet is owned and maintained by Dr Alan D. Thompson at LifeArchitect.ai. It should be up-to-date, but please see tab date and sheet header.
45
46

	A	B	C	D	E	F	G	H
1	Lab name	Abbrev	Location	Founded	Headcount estimated (2022)	Focus	Website	Charter

2	Baidu Research	Baidu	Beijing, China; USA	2014	13,000	Models	research.baidu.com	Baidu Research brings together top talents from around the world to focus on future-looking fundamental research in artificial intelligence.
3	OpenAI	OAI	San Francisco, USA	2015	1200	Models	openai.com	OpenAI’s mission is to ensure that artificial general intelligence (AGI) benefits all of humanity.
4	DeepMind	DM	London, UK	2010	1200	Models	deepmind.com	Our teams research and build safe AI systems. We're committed to solving intelligence, to advance science and benefit humanity.
5	Meta AI	Meta	Menlo Park, USA	2013	1000	Models	ai.facebook.com	Bring the world closer together by advancing AI
6	Microsoft Research	MSR	Redmond, USA	1991	500	General	microsoft.com/en-us/research/lab/microsoft-research-redmond/	Microsoft Research is doing foundational research, pursuing ambitious disruptions, and performing high-impact tech transfer. We want to create a better future for Microsoft and society through research.
7	Google AI	GAI	Mountain View, USA	2017	500	General	ai.google	Advancing AI for everyone
8	Tencent AI Lab	Tencent	Shenzhen, China	2016	400	General	ai.tencent.com/ailab/en/index	Make AI everywhere
9	Allen AI	AI2	Seattle, USA	2015	150	General	allenai.org	AI2 is a non-profit research institute founded in 2014 with the mission of conducting high-impact AI research and engineering in service of the common good.
10	BAAI	BAAI	Beijing, China	2018	150	Models	www-pre.baai.ac.cn/en	BAAI encourages scientists to explore “no man’s land”, to take on the most foundational and challenging problems, to promote revolutionary or disruptive breakthroughs in AI theories, methodologies, tools, systems and applications.
11	Eleuther AI	EAI	Remote	2020	100 (vol)	Models	www.eleuther.ai	EleutherAI is a decentralized collective of volunteer researchers, engineers, and developers focused on AI alignment, scaling, and open source AI research.
12	AI21 Labs	AI21	Tel Aviv, Israel	2017	90	Models	ai21.com	AI21 Labs is reimagining the way humans read and write, for the better. They say two heads are better than one. We say two heads - one human and one intelligent machine - are the best.
13	Aleph Alpha	Aleph Alpha	Heidelberg, Germany	2019	30	Models	aleph-alpha.com	To be the leading European company researching and creating next-generation strong artificial intelligence.
14	Anthropic AI	Anthropic	San Francisco, USA	2021	15	Alignment	anthropic.com	Anthropic is an AI safety and research company that’s working to build reliable, interpretable, and steerable AI systems.
15	Inflection AI	Inflection	Remote; SF; NYC; London	2022	10	Integration	inflection.ai	Inflection is an AI-first company, redefining human-computer interaction.
16	Adept AI Labs	Adept	San Francisco, USA	2022	10	Integration	adept.ai	Adept is an ML research and product lab building general intelligence by enabling humans and computers to work together creatively.
17
18
19		This sheet is owned and maintained by Dr Alan D. Thompson at LifeArchitect.ai. It should be up-to-date, but please see tab date and sheet header.
20		LifeArchitect.ai
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

	A	B	C	D	E	F	G	H	I	J	K	L	M	N	O
1	Lab	Robot	Country	Release	Height (cm)	Weight (kg)	Walk (km/h)	Run (km/h)	Payload/carry (kg)	Power (hours)	Link	Video	Clothed?	US$ ('000)	Notes

2	1X	NEO	Norway	2023	160	30	4	12	20	2-4	https://www.1x.tech/neo		Yes		Due Dec/2023
3	Figure	01	USA	2023	167	60	4.3	4.3	20	5	https://www.figure.ai/	https://youtu.be/9sHUggKsWTk	Yes
4	Fourier	GR-1	Singapore	2023	165	55	5	5	50		https://robots.fourierintelligence.com/	https://youtu.be/SHPxcRBlXN0	No		100x units due 2023
5	Tesla	Optimus Gen 1	USA	2023	173	57	8	8	20.4		https://archive.md/QNvx9	https://youtu.be/XiQkeWOFwmk	No		Lift 68kg
6	Tesla	Optimus Gen 2	USA	2023	173	47	8	8	20.4		https://archive.md/mEWsJ	https://www.youtube.com/watch?v=fPeGPf7jvEQ	Yes	20
7	Boston Dynamics	Atlas	USA	2022	150	89	9	9	11		https://www.bostondynamics.com/atlas	https://youtu.be/-e1_QhJ1EhQ	No
8	Sanctuary AI	Phoenix	Canada	2023	170	70	5	5	25		https://sanctuary.ai/resources/news/sanctuary-ai-unveils-phoenix-a-humanoid-general-purpose-robot-designed-for-work/	https://youtu.be/k2GhgO7SnZQ	No
9	Agility	Digit	USA	2023	175	64	5.4	5.4	16	3	https://agilityrobotics.com/robots	https://youtu.be/rnFZAB9ogEE	No	250
10	Unitree	H1	China	2023	180	47	5.4	18			https://www.unitree.com/en/h1/	https://youtu.be/GtPs_ygfaEA	Yes	90
11	Apptronik	Apollo	USA	2023	172	72			25	4	https://apptronik.com/product-page		No		Released
12	Kepler	Forerunner	China	2024	178	85					https://www.gotokepler.com/home	https://youtu.be/A5vshTgDbKE	No	30
13	LimX	CL-1	China	2024							https://www.limxdynamics.com/en	https://youtu.be/sihIDeJ4Hmk	No
14	Mentee Robotics	MenteeBot	Israel	2024	175	70	5.4		25		https://www.menteebot.com/bot/	https://youtu.be/Z3T9S1Arbdk	No
15	Humanoid Robots (Shanghai) Limited	Qinglong	China	2024	185	80					https://technode.com/2024/07/05/qinglong-chinas-first-full-sized-general-purpose-humanoid-robot-unveiled-at-world-artificial-intelligence-conference-2024/	https://youtu.be/ilwG5zkNOz8	No
16
17			This sheet is owned and maintained by (but not a focus of) Dr Alan D. Thompson at LifeArchitect.ai. Alternative sheets are probably more up-to-date:
18			Alternative data:		Cern Basher & Scott Walter (no metrics, Apr/2024):			https://twitter.com/CernBasher/status/1780700653335335137
19					Robots Guide (old, but bigger list, still missing bots):			https://robotsguide.com/robots?category=humanoids
20					NVIDIA (limited view, Mar/2024):			https://twitter.com/leeron/status/1770047411827933455
21
22
23
24
25
26
27	https://lifearchitect.ai/humanoids/		Alternative viz by Harrison Schell from madevisual.co

	A	B
1	Auxiliary sheets	Link
2	Video playlist	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=1023028020
3	Contents of GPT-3/Pile/Megatron/CC	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=0
4	Contents of Chinese models	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=995838717
5	Leta template	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=272212998
6	IQ testing AI Q32022	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=741722385
7	IQ testing AI Q12022	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=57569659
8	GPT2/3/J questions	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=1115077966
9	BMI	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=958745455
10	Watson vs GPT-3	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=1989481885
11	Leta-prompt-explain	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=1289129393
12	WebGPT sample questions	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=467273092
13	BMI vs BCI	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=1549223773
14	PaLM Explaining jokes	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=923122602
15	LLM languages	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=138672226
16	Model sizes	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=373202700
17	ChatGPT v Bard NPR	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=1298952257
18	Context windows 2023	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=8769
19	GPT-4 Enterprise Customers	https://docs.google.com/spreadsheets/d/18WTt3bmLAad5rJZi6eRtDhj4vehK7-u4xTpkho4X-4U/edit#gid=1455583713
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999