https://lifearchitect.ai/models-table/

Timeline view:

https://lifearchitect.ai/timeline

https://lifearchitect.ai/memo

Parameters
(B)

Tokens
trained (B)

Ratio Tokens:Params
(Chinchilla scaling≥20:1)

Training dataset

Announced
▼

https://lifearchitect.ai/olympus/

New related Titan details: '$65m training run. 200B dense model on 4T tokens of data across 13,760 NVIDIA A100 chips. 48 days to train. Training runs soon to cross $1B' https://importai.substack.com/p/import-ai-365-wmd-benchmark-amazon

https://lifearchitect.ai/gpt-5/

AuroraGPT (ScienceGPT)

Argonne National Laboratory

https://arstechnica.com/information-technology/2024/05/microsoft-developing-mai-1-language-model-that-may-compete-with-openai-report/

Due 2024. MAI=Microsoft artificial intelligence. MSFT CTO statement: https://archive.md/XRSgS

Dense

"the training dataset follows a specific structure, we develop a custom tokenizer. Alphanumeric node names are tokenized at a character level, while special terms such as ‘causes’, ‘Does’, ‘cause’, ‘Yes’, and ‘No’ are tokenized at the word level... Our training setup consists of around 175k instances of sequential chains with size of chains ranging from 3 to 6 nodes... All models are trained for 100 epochs. [LifeArchitect.ai estimate is 12 tokens per node x 6 nodes x 175,000 instances x 100 epochs = 1.26B tokens]" Based on GPT-2 arch.

https://platform.sensenova.cn/home#/home

"The model training was based on over 10TB tokens [sic, taken as 10T tokens instead of 10TB=2T tokens] of high-quality training data, including a large amount of synthetically-generated reasoning chain data, which help to enhance its reasoning capabilities." & "The updates include SenseNova 5o, the first real-time multimodal model in China, which provides a new AI interaction model on par with GPT-4o’s streaming interaction capabilities"

Shanghai AI Laboratory/SenseTime

https://huggingface.co/collections/internlm/internlm25-66853f32717072d17581bc13

https://github.com/InternLM/InternLM/blob/main/model_cards/internlm2.5_7b.md

Dense

"The release of InternLM2.5 series contains 7B model size for now and we are going to release the 1.8B and 20B versions soon"

https://wabetainfo.com/whatsapp-beta-for-android-2-24-14-7-whats-new/

Dense

https://yiyan.baidu.com/

Dense

"Ernie Bot has reached 300 million users since its launch [on 16/Mar/2023, public Aug/2023]" Jun/2024

Google DeepMind

https://huggingface.co/google/gemma-2-27b-it

https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf

Dense

Dense

"LLM Critics Help Catch LLM Bugs" Announce: https://openai.com/index/finding-gpt4s-mistakes-with-gpt-4/

https://github.com/apple/ml-4m/

Dense

Vision model based on T5-XXL. Modalities: RGB, Caption, Bounding boxes, Semantic segmentation, Depth, Human poses, Surface normals, CLIP, DINOv2, ImageBind, Metadata, Canny edges, SAM edges, SAM instances, Color palette. Project page: https://4m.epfl.ch/

EvolutionaryScale

https://github.com/evolutionaryscale/esm

Dense

Biology large language model: "sequence, structure, and function are all masked and predicted during training, ESM3 can generate in all three modalities." 1.4B only released.

https://www.huaweicloud.com/intl/en-us/product/modelarts.html

Claude 3.5 Sonnet

https://poe.com/Claude-3.5-Sonnet

Dense

Model card: https://www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf

DeepSeek-Coder-V2

https://chat.deepseek.com/coder

DCLM-Baseline 7B 2.6T

https://www.datacomp.ai/dclm/

Dense

New dataset: 240T tokens: 8× larger than previous SOTA dataset. DCLM-Pool is 240T, DCLM-Baseline is 3.8T: "we combine our 3.8T DCLM-BASELINE with the StarCoder and ProofPile2 data to arrive at a 4.1T token dataset. We train a 7B model for 2.5T tokens" and "We release the DCLM benchmark, framework, models, and datasets at https://datacomp.ai/dclm." Model not yet available.

Nemotron-4-340B

https://build.nvidia.com/nvidia/nemotron-4-340b-instruct

Dense

Open-source equiv of Mar/2023 GPT-4 (1760MoE≈340B, 13T), same param count but 2x the tokens of May/2023 PaLM 2 (340B, 3.6T), competitor to Nov/2023 Grok-1 (314B, 6T). Trained on 6,144 H100s. ~1.3TB for inference. 50+ natural and 40+ coding languages. Trained between December 2023 and May 2024. MMLU 0-shot for instruct=78.7, 5-shot for base=81.1. Permalink for paper: https://research.nvidia.com/publication/2024-06_nemotron-4-340b

Apple On-Device model Jun/2024

https://github.com/apple/corenet/tree/main/projects/openelm

Dense

https://lifearchitect.ai/apple/ Likely to be the Apple OpenELM model (Apr/2024). "two of these models — a ~3 billion parameter on-device language model, and a larger server-based language model available with Private Cloud Compute". https://machinelearning.apple.com/research/introducing-apple-foundation-models The server-based model is possibly Ferret, although it is more properly called a multimodal model (not just language). It could also be Apple GPT based on their Ajax framework: https://archive.md/f3C0r

https://github.com/ridgerchu/matmulfreellm

Dense

"we explore alternative methods for mixing tokens without relying on matrix multiplications." Compared with Transformer++ based on Llama-2, not to be confused with the pre-GPT-3 American Express Transformer++ paper from 2/Mar/2020. Instead, Transformer++ is defined in the Mamba paper: 'Transformer++: A Transformer with an improved architecture, namely rotary positional encodings (Su et al. 2021) and SwiGLU MLP (Shazeer 2020)'

https://www.rungalileo.io/blog/introducing-galileo-luna-a-family-of-evaluation-foundation-models

Dense

https://huggingface.co/spaces/Qwen/Qwen2-72B-Instruct

Dense

Instruct MMLU=82. Instruct GPQA=41.9. Qwen1.5 trained on ~3T tokens, using this number for Qwen2 while waiting for official number.

https://github.com/QwenLM/Qwen2?tab=readme-ov-file

Skywork MoE 16x13B

https://huggingface.co/Skywork/Skywork-MoE-Base

CN + EN. "(MoE) model with 146 billion parameters, 16 experts, and 22 billion activated parameters. This model is initialized from the pre-existing dense checkpoints of our Skywork-13B model."

https://github.com/state-spaces/mamba

https://arxiv.org/abs/2405.21060

Dense

https://map-neo.github.io/

https://arxiv.org/abs/2405.19327

Dense

"first fully open-sourced bilingual LLM with comparable performance to existing state-of-the-art LLMs... we open-source all details to reproduce our MAP-Neo, where the cleaned pre-training corpus, data cleaning pipeline, checkpoints, and well-optimized training/evaluation framework are provided."

https://huggingface.co/LLM360/K2

https://www.llm360.ai/blog/several-new-releases-to-further-our-mission.html

Dense

"K2-65B is a fully reproducible LLM outperforming Llama 2 70B using 35% less compute."

https://huggingface.co/mistralai/Codestral-22B-v0.1

https://mistral.ai/news/codestral/

Dense

https://huggingface.co/spaces/CohereForAI/aya-23

Dense

https://platform.01.ai/

Still training as of May/2024: https://appserversrc.8btc.cn/FnDYlEC4STBhphu6M3NL4CKH43FW dead link, use: https://finance.china.com.cn/roll/20240513/6116857.shtml

https://platform.01.ai/

Dense

https://ai.meta.com/resources/models-and-libraries/chameleon-downloads/?gk_enable=chameleon_web_flow_is_live

https://arxiv.org/abs/2405.09818

Dense

https://huggingface.co/spaces/neuralmagic/llama-2-sparse-transfer-chat-deepsparse

https://arxiv.org/abs/2405.03594

Hybrid

Google DeepMind

https://aistudio.google.com/app/prompts/new_chat

https://goo.gle/GeminiV1-5

Omnimodel. ‘[GPT-4o is] likely an early checkpoint of GPT-5’. https://twitter.com/drjimfan/status/1790089671365767313 ELO: https://twitter.com/LiamFedus/status/1790064963966370209 Demo: https://youtu.be/DQacCB9tDaw

https://huggingface.co/tiiuae/falcon-11B

https://www.tii.ae/news/falcon-2-uaes-technology-innovation-institute-releases-new-ai-model-series-outperforming-metas

Dense

Announce: https://www.tii.ae/news/falcon-2-uaes-technology-innovation-institute-releases-new-ai-model-series-outperforming-metas

https://huggingface.co/Fugaku-LLM/Fugaku-LLM-13B-instruct

Dense

Japanese. CPU trained: 158,976+ A64FX CPUs (7M+ cores), zero GPUs. https://en.wikipedia.org/wiki/Fugaku_(supercomputer)

https://huggingface.co/01-ai/Yi-1.5-34B-Chat

Dense

https://github.com/microsoft/unilm/tree/master/YOCO

Dense

With Tsingua. You Only Cache Once (YOCO). Long context "1M context length with near-perfect needle retrieval accuracy"

https://chat.deepseek.com/

https://arxiv.org/abs/2405.04434

https://huggingface.co/chuxin-llm/Chuxin-1.6B-Base

https://arxiv.org/abs/2405.04828

Dense

https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-2

https://huggingface.co/BlinkDL/rwkv-6-world

Dense

https://arxiv.org/abs/2405.04517

Dense

New method LSTM to xLSTM, see also RNNs. Code/weights doesn't seem to be released. https://github.com/AI-Guru/xlstm-resources

https://github.com/ibm-granite/granite-code-models

https://github.com/ibm-granite/granite-code-models/blob/main/paper.pdf

Dense

Dataset: publicly available datasets (e.g., GitHub Code Clean, Starcoder data), public code repositories, and issues from GitHub.

https://chat.lmsys.org/

https://help.aliyun.com/zh/dashscope/developer-reference/model-introduction

Dense

Med-Gemini-L 1.0

Google DeepMind

https://twitter.com/alan_karthi/status/1785117450528264216

https://arxiv.org/abs/2404.18416

Dense

Med-Gemini-M 1.0 and Med-Gemini-L 1.0 (Pro and Ultra finetunes) "For language tasks that require less complex reasoning, such as summarizing medical notes and creating referral letters, we introduce Med-Gemini-M 1.0 by fine-tuning the Gemini 1.0 Pro model. For other tasks that require more advanced reasoning, we introduce Med-Gemini-L 1.0 by fine-tuning the Gemini 1.0 Ultra model using a self-training method to enable the models to efficiently use web search."

https://huggingface.co/CofeAI/Tele-FLM

https://arxiv.org/abs/2404.16645

Dense

Also known as FLM-2. "We will open-source a 1T model checkpoint, namely Tele-FLM-1T, to advance further training and research." Discussion paper Jul/2024: https://arxiv.org/abs/2407.02783

https://huggingface.co/spaces/Qwen/Qwen1.5-110B-Chat-demo

https://qwenlm.github.io/blog/qwen1.5-110b/

Dense

Snowflake AI Research

https://arctic.streamlit.app/

Hybrid

"Arctic uses a unique Dense-MoE Hybrid transformer architecture. It combines a 10B dense transformer model with a residual 128×3.66B MoE MLP resulting in 480B total and 17B active parameters chosen using a top-2 gating."

GPT-4 scale; low media coverage; no demo in Western world. https://www.techinasia.com/sensetime-pauses-trading-stock-rises-30-model-launch

https://huggingface.co/apple/OpenELM-3B-Instruct

Dense

On-device model (laptop, phone). Open-source Efficient Language Models (OpenELM). https://venturebeat.com/ai/apple-releases-openelm-small-open-source-ai-models-designed-to-run-on-device/

https://huggingface.co/microsoft/Phi-3-medium-128k-instruct

Dense

https://huggingface.co/microsoft/Phi-3-mini-128k-instruct

Dense

"phi3-mini can be quantized to 4-bits so that it only occupies ≈ 1.8GB of memory. We tested the quantized model by deploying phi-3-mini on iPhone 14 with A16 Bionic chip running natively on-device and fully offline achieving more than 12 tokens per second."

Dense

Dense

HLAT=High-quality LLM pre-trained on AWS Trainium. Same arch as Llama 7B. The pre-training is performed up to 64 Amazon EC2 trn1.32xlarge instances with totalling up to 1024 AWS Trainium accelerators. Read more about Trainium: https://www.aboutamazon.com/news/aws/what-you-need-to-know-about-the-aws-ai-chips-powering-amazons-partnership-with-anthropic

https://huggingface.co/HuggingFaceM4/idefics2-8b

Dense

Clone of Flamingo now using Mistral 7B. Named after Asterix and Obelix's dog Idefix (Image-aware Decoder Enhanced à la Flamingo with Interleaved Cross-attentionS)

https://poe.com/RekaCore

https://publications.reka.ai/reka-core-tech-report.pdf

Dense

https://www.reka.ai/news/reka-core-our-frontier-class-multimodal-language-model

WizardLM-2-8x22B

https://huggingface.co/MaziyarPanahi/WizardLM-2-8x22B-GGUF

https://wizardlm.github.io/WizardLM2/

https://huggingface.co/EleutherAI/pile-t5-xxl

Dense

Zephyr 141B-A35B

Hugging Face H4

https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1

https://arxiv.org/abs/2403.07691

https://docs.cohere.com/reference/rerank-1

Dense

gpt-4-turbo-2024-04-09

https://chat.openai.com/

This is such a significantly better model that I've added it here. This GPQA=46.5%, old GPT-4 GPQA=36%. https://twitter.com/EpochAIResearch/status/1778463039932584205 MMLU scores are unclear, but may have improved by 1%: https://twitter.com/OpenAI/status/1778602770784002136. Final benchmarks are here: https://archive.md/6Cc0Z

https://github.com/OpenBMB/MiniCPM/

https://arxiv.org/abs/2404.06395

Dense

https://github.com/apple/ml-ferret

Dense

https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1

https://mistral.ai/news/mixtral-8x22b/

https://huggingface.co/sail

https://arxiv.org/abs/2404.03608v1

Dense

SEA languages. Based on Qwen-1.5. https://github.com/sail-sg/sailor-llm "Generally Sailor models consume around 200B tokens, completing a full pass through the SailCraft corpus once. However, the Sailor-0.5B model undergoes training with 400B tokens, equivalent to 2 epochs."

https://www.lepton.ai/playground/chat?model=jetmoe-8b-chat

Dense

https://huggingface.co/spaces/CohereForAI/c4ai-command-r-plus

Dense

purpose-built to excel at real-world enterprise use cases. Announce with no arch details: https://txt.cohere.com/command-r-plus-microsoft-azure/

https://www.silo.ai/blog/viking-7b-13b-33b-sailing-the-nordic-seas-of-multilinguality

Dense

Viking uses an architecture similar to Llama 2, with flash attention, rotary embeddings, grouped query attention and supports a 4k sequence length'

https://huggingface.co/NousResearch/OLMo-Bitnet-1B

https://arxiv.org/abs/2402.17764

Dense

1.58-bit quantized (ternary weights) means we can run a 70B model in ~14GB VRAM. See also BitNet b1.58

https://huggingface.co/collections/aurora-m/aurora-m-models-65fdfdff62471e09812f5407

https://arxiv.org/abs/2404.00399

Dense

https://arxiv.org/abs/2403.20329

Dense

Qwen1.5-MoE-A2.7B

https://qwenlm.github.io/blog/qwen-moe/

https://qwenlm.github.io/blog/qwen-moe/

MoE. "Of particular significance is the fact that, through upcycling, the necessity for training an equivalent volume of tokens as in the original model has been eliminated." I assumed half of the original 3T tokens

https://x.ai/blog/grok-1.5

https://huggingface.co/ai21labs/Jamba-v0.1

https://arxiv.org/abs/2403.19887

https://huggingface.co/spaces/databricks/dbrx-instruct

https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm

Stable Code Instruct 3B

https://huggingface.co/stabilityai/stable-code-instruct-3b

Dense

https://huggingface.co/SakanaAI/EvoLLM-JP-v1-10B

https://arxiv.org/abs/2403.13187

Dense

https://huggingface.co/Rakuten/RakutenAI-7B

https://arxiv.org/abs/2403.15484

Dense

https://colab.research.google.com/drive/1gI8CM9Bz9ov0-E6aL2jF808rE56UtZyF?usp=sharing

https://news.ycombinator.com/item?id=39745700#39745702

Dense

https://huggingface.co/recursal/EagleX_1-7T

https://substack.recursal.ai/p/eaglex-17t-soaring-past-llama-7b

Dense

https://arxiv.org/abs/2403.09611

Dense

VLM, outperforms Flamingo 80B (Apr/2022) across benchmarks. 2T text tokens + ~10B+ other text (estimate). Unreleased.

https://vimeo.com/921866765

https://covariant.ai/insights/introducing-rfm-1-giving-robots-human-like-reasoning-capabilities/

Dense

Dense

https://github.com/deepseek-ai/DeepSeek-VL?tab=readme-ov-file

https://arxiv.org/abs/2403.05525

Dense

https://junzhan2000.github.io/AnyGPT.github.io/

https://arxiv.org/abs/2402.12226

Dense

Llama 2 7B backbone with new matrices ('reshaping the embedding matrix and prediction layer')

https://stability.ai/news/putting-the-ai-supercomputer-to-work

Dense

Mentioned in Stability release about Intel chips 11/Mar/2024, availablity unknown

https://inflection.ai/inflection-2

https://inflection.ai/inflection-2-5

Dense

https://apollo.llmzoo.com/

https://arxiv.org/abs/2403.03640

Dense

https://www.anthropic.com/claude-3-model-card

Dense

Original MMLU=86.8 (GPT-4=86.4). Original GPQA=50.4. 200k context, 1M for researchers.

Dense

https://unbabel.com/meet-towerllm/

https://arxiv.org/abs/2402.17733

Dense

Google DeepMind

https://arxiv.org/abs/2402.19427

Dense

Google DeepMind

https://arxiv.org/abs/2402.19427

Dense

https://huggingface.co/1bitLLM/bitnet_b1_58-xl

https://arxiv.org/abs/2402.17764

Dense