https://lifearchitect.ai/models-table

Timeline view:

https://lifearchitect.ai/timeline

The Memo:

https://lifearchitect.ai/memo

Playground

Parameters
(B)

Tokens
trained (B)

Ratio Tokens:Params
(Chinchilla scaling≥20:1)

Training dataset

Announced
▼

AuroraGPT (ScienceGPT)

Argonne National Laboratory

Three models targeted in Jul/2024: AuroraGPT-7B-P (Ponte Vecchio GPU testing) AuroraGPT-7B-A (Aurora) AuroraGPT-7B-A-S (Aurora + Science).

Due April 2025. Hybrid MoE, 1.2TA78B. 5.2PB corpus=1.3Qa tokens. 1.3 quadrillion tokens = 1,300T tokens = 1,300,000B tokens "Constructed a 5.2 PB high-quality corpus covering vertical domains such as finance, law, and patents." http://jiuyangongshe.com/a/1h4gq724su0 and translated at: https://docs.google.com/document/d/e/2PACX-1vTmx-A5sBe_3RsURGM7VvLWsAgUXbcIb2pFaW7f1FTPgK7mGvYENXGQPoF2u4onFndJ_5tzZ02su-vg/pub

Google DeepMind

https://huggingface.co/zai-org/GLM-4.6

https://z.ai/blog/glm-4.6

https://huggingface.co/inclusionAI/Ring-1T-preview

https://huggingface.co/inclusionAI/Ring-1T-preview

https://claude.ai/

https://www.anthropic.com/news/claude-sonnet-4-5

Gemini Robotics 1.5

Google DeepMind

https://storage.googleapis.com/deepmind-media/gemini-robotics/Gemini-Robotics-1-5-Tech-Report.pdf

2. "vision-language-action (VLA) model turns visual information and instructions into motor commands for a robot to perform a task." Available to select partners.

Gemini Robotics-ER 1.5

Google DeepMind

https://storage.googleapis.com/deepmind-media/gemini-robotics/Gemini-Robotics-1-5-Tech-Report.pdf

1. "vision-language model (VLM) reasons about the physical world, natively calls digital tools and creates detailed, multi-step plans to complete a mission." Available to all devs.

https://research.google/blog/time-series-foundation-models-can-be-few-shot-learners/

Dense

TimesFM-ICF is 6.8% more accurate than TimesFM (Base). Time-series forecasting only. 'a large pretraining corpus of 100B real world time-points' may be more than 100B tokens.

https://chat.qwen.ai/

"Qwen3-Max-Thinking — still under active training — is already demonstrating remarkable potential. When augmented with tool usage and scaled test-time compute, the Thinking variant has achieved 100% on challenging reasoning benchmarks such as AIME 25 and HMMT. "

https://github.com/QwenLM/Qwen3-Omni?tab=readme-ov-file

"Qwen3-Omni is a unified end-to-end model capable of processing multiple modalities, such as text, audio, image and video, and generating real-time text or speech response."... "pretraining utilizes a large-scale dataset containing approximately 2 trillion tokens, with the following distribution across modalities: text (0.57 trillion), audio (0.77 trillion), image (0.82 trillion), video (0.05 trillion), and video-audio (0.05 trillion)."

DeepSeek-V3.1-Terminus

Dense

"perceptive-language model...delivering capabilities that meet or exceed those of models over 50 times its size. Founded by the team behind Meta's Chameleon multimodal models, Perceptron is tackling a fundamental challenge: bringing the power of physical AI to the dynamic, multimodal, and real-time environments we live and work in."

https://grok.com/

"2M token context window, and a unified architecture that blends reasoning and non-reasoning modes in one model."

Google DeepMind

https://huggingface.co/google/vaultgemma-1b

Dense

Qwen3-Next-80B-A3B

https://huggingface.co/collections/Qwen/qwen3-next-68c25fd6838e585db8eeea9d

"Qwen3-Next introduces several key improvements: a hybrid attention mechanism, a highly sparse Mixture-of-Experts (MoE) structure, training-stability-friendly optimizations, and a multi-token prediction mechanism for faster inference."

Dense

"Built on the Qwen2.5 base model, our system shows that smaller models can compete at the highest levels by combining advanced post-training and test-time computation techniques. The approach is based on six key technical pillars: Long Chain-of-thought Supervised Finetuning, Reinforcement Learning with Verifiable Rewards (RLVR), Agentic planning prior to reasoning, Test-time Scaling, Speculative Decoding, and Inference-optimized Hardware, all using publicly available open-source datasets."

Dense

"a modern multilingual encoder trained on 3T tokens and 1833 languages. We introduce several novel elements in training: an inverse masking schedule and a cascading annealed language learning schedule for multilingual data" Announce: https://huggingface.co/blog/mmbert

ERNIE-4.5-21B-A3B-Thinking

https://huggingface.co/baidu/ERNIE-4.5-21B-A3B-Thinking

https://huggingface.co/Kwai-Klear/Klear-46B-A2.5B-Instruct

https://huggingface.co/TildeAI/TildeOpen-30b

Dense

Qwen3-Max-Preview

https://chat.qwen.ai/

GPQA score is SuperGPQA. "our biggest model yet, with over 1 trillion parameters"

Kimi K2-Instruct-0905

https://huggingface.co/moonshotai/Kimi-K2-Instruct

https://huggingface.co/swiss-ai/Apertus-70B-Instruct-2509

Dense

"Apertus – Latin for “open”" 1,811 languages. Announce: https://ethz.ch/en/news-and-events/eth-news/news/2025/09/press-release-apertus-a-fully-open-transparent-multilingual-language-model.html

https://microsoft.ai/news/two-new-in-house-models/

MAI=Microsoft artificial intelligence. "MAI’s first foundation model trained end-to-end... MAI-1-preview is an in-house mixture-of-experts model, pre-trained and post-trained on ~15,000 NVIDIA H100 GPUs. This model is designed to provide powerful capabilities to consumers seeking to benefit from models that specialize in following instructions and providing helpful responses to everyday queries. We will be rolling MAI-1-preview out for certain text use cases within Copilot"

https://github.com/features/copilot

https://data.x.ai/2025-08-26-grok-code-fast-1-model-card.pdf

"We built grok-code-fast-1 from scratch, starting with a brand-new model architecture. To lay a robust foundation, we carefully assembled a pre-training corpus rich with programming-related content. For post-training, we curated high-quality datasets that reflect real-world pull requests and coding tasks." Announce: https://x.ai/news/grok-code-fast-1

https://huggingface.co/NousResearch/Hermes-4-405B-FP8

https://arxiv.org/abs/2508.18255

Dense

https://github.com/NVlabs/Jet-Nemotron

Dense

DeepSeek-V3.1-Base

https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Base

https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-Base

Dense

Google DeepMind

https://huggingface.co/google/gemma-3-270m-it

Dense

https://poe.com/GPT-5

https://openai.com/index/gpt-5-system-card/

Announce: https://openai.com/index/introducing-gpt-5/. MMLU is based on ES and PT translated from EN.

https://huggingface.co/openai/gpt-oss-120b

https://cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7637/oai_gpt-oss_model_card.pdf

https://huggingface.co/openai/gpt-oss-20b

https://cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7637/oai_gpt-oss_model_card.pdf

https://claude.ai/

https://www.anthropic.com/news/claude-opus-4-1

https://huggingface.co/zai-org/GLM-4.5

https://z.ai/blog/glm-4.5

China Telecom Artificial Intelligence Research Institute

https://github.com/Tele-AI/T1

Dense

Shanghai AI Laboratory/SenseTime

https://huggingface.co/internlm/Intern-S1

https://huggingface.co/internlm/Intern-S1

41T tokens assumes base model of Qwen3. "Built upon a 235B MoE language model and a 6B Vision encoder, Intern-S1 has been further pretrained on 5 trillion tokens of multimodal data"

https://www.stepfun.com/

https://github.com/stepfun-ai/Step3/blob/main/Step3-Sys-Tech-Report.pdf

Qwen3-235B-A22B-Thinking-2507

https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507

https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507

235B-A22B. "Qwen3 is pre-trained on 36 trillion tokens across 119 languages" MMLU score is MMLU-Redux.

200BA40B. In training as of Jul/2025. "to address the overthinking problem in reasoning-intensive tasks"

https://huggingface.co/Kwaipilot/KAT-V1-40B

Dense

Qwen3-Coder-480B-A35B-Instruct

https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct

Qwen3-235B-A22B-Instruct-2507

https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507

235B-A22B. "Qwen3 is pre-trained on 36 trillion tokens across 119 languages" MMLU score is MMLU-Redux.

https://huggingface.co/allenai/FlexOlmo-7x7B-1T

https://arxiv.org/abs/2507.07024v1

37B-A20B. "We adopt the OLMo-2 7B setup, starting from a a checkpoint pre-trained on 4T tokens and annealed for 50B tokens to produce a public expert. We then train two additional experts on math and code, each for 50B tokens, and combine them with the public expert to form a three-expert version of FLEXOLMO."

https://huggingface.co/LGAI-EXAONE/EXAONE-4.0-32B

https://www.lgresearch.ai/data/cdn/upload/EXAONE_4_0.pdf

Dense

“EXAONE”=“EXpert AI for EveryONE”. Training tokens/ratio: EXAONE-3 7.8B=8T tokens (Aug/2024) -> EXAONE-3.5 7.8B=9T -> EXAONE-3.5 32B=6.5T tokens -> EXAONE 4.0 32B=14T tokens. MMLU score is MMLU-Redux. Interesting: "To focus [RL] training on more informative data samples, we perform accuracy-based filtering by generating eight responses from the SFT model and excluding samples where all eight responses are correct, a pre-filtering step that removes problems that are easy for the model to avoid inefficient training."

https://huggingface.co/moonshotai/Kimi-K2-Instruct

https://huggingface.co/RekaAI/reka-flash-3.1

https://www.reka.ai/news/introducing-reka-flash

Dense

https://chat.mistral.ai/chat

https://mistral.ai/news/devstral-2507

Dense

https://grok.com/

"The smartest AI in the world, 100% on SAT, etc, questions that it's never seen before."

Phi-4-mini-flash-reasoning

https://huggingface.co/microsoft/Phi-4-mini-flash-reasoning

Dense

"Pre-training: 5T tokens; Reasoning training: 150B tokens" "At the core of Phi-4-mini-flash-reasoning is the newly introduced decoder-hybrid-decoder architecture, SambaY, whose central innovation is the Gated Memory Unit (GMU), a simple yet effective mechanism for sharing representations between layers. The architecture includes a self-decoder that combines Mamba (a State Space Model) and Sliding Window Attention (SWA), along with a single layer of full attention. The architecture also involves a cross-decoder that interleaves expensive cross-attention layers with the new, efficient GMUs. This new architecture with GMU modules drastically improves decoding efficiency, boosts long-context retrieval performance and enables the architecture to deliver exceptional performance across a wide range of tasks. "

Google DeepMind

https://huggingface.co/google/t5gemma-9b-9b-ul2-it

Dense

Google DeepMind

https://huggingface.co/google/medgemma-27b-it

https://arxiv.org/abs/2507.05201

Dense

https://huggingface.co/tngtech/DeepSeek-TNG-R1T2-Chimera

Assembly of Experts-method of V3-0324, R1, R1-0528. Announce: https://x.com/tngtech/status/1940531045432283412?s=46

Dense

"Spectra-1.1, an open suite of TriLMs trained on up to 1.2 trillion tokens, demonstrating sustained performance gains at scale. Furthermore, to improve inference efficiency, we propose novel 2-bit and 1.6-bit packing schemes for ternary weights"

https://github.com/apple/ml-diffucoder

Dense

"We adapt our model from Qwen2.5-Coder (Hui et al., 2024) as the base model to perform continual pre-training using the adaptation approach from Gong et al. (2025). During this pre-training, we use a 400B-token code pre-training corpus from RefineCode (Huang et al., 2024) and Stackv2 (Lozhkov et al., 2024)."

https://huggingface.co/tencent/Hunyuan-A13B-Instruct

https://huggingface.co/tencent/Hunyuan-A13B-Instruct

80B-A13B. 'We have open-sourced Hunyuan-A13B-Pretrain , Hunyuan-A13B-Instruct , Hunyuan-A13B-Instruct-FP8 , Hunyuan-A13B-Instruct-GPTQ-Int4 on Hugging Face.'

https://chat.inceptionlabs.ai/

https://blogs.windows.com/windows-insider/2025/06/13/announcing-windows-11-insider-preview-build-26200-5651-dev-channel/

Dense

"distillation from Microsoft’s Phi models...Mu is an efficient 330M encoder–decoder language model optimized for small-scale deployment, particularly on the NPUs on Copilot+ PCs. It follows a transformer encoder–decoder architecture"

Gemini Robotics On-Device

Google DeepMind

https://docs.google.com/forms/u/0/d/1sM5GqcVMWv-KmKY3TOMpVtQ-lDFeAftQ-d9xQn92jCE/viewform?ts=67cef986&edit_requested=true

See Mar/2025 Gemini Robotics-ER model for comparison. Announce: https://deepmind.google/discover/blog/gemini-robotics-on-device-brings-ai-to-local-robotic-devices/

https://huggingface.co/ICONNAI/ICONN-1

"ICONN-1 (this version) is optimized for natural, emotionally resonant, and conversational interactions. ICONN-e1 is a specialized variant of the model fine-tuned for advanced reasoning, critical analysis, and complex problem-solving."

https://huggingface.co/MiniMaxAI/MiniMax-M1-80k

https://chat.mistral.ai/chat

https://mistral.ai/static/research/magistral.pdf

Dense

https://huggingface.co/common-pile/comma-v0.1-2t

Dense

"Comma v0.1-2T is a decoder-only transformer that uses the same architecture as Llama 3. Training was done in two stages: first on 1.93 trillion tokens with a cosine learning rate schedule, and second a "cool-down" training phase on 75.5 billion tokens from high-quality sources. The final model is the average of 10 checkpoints during this cool-down phase. Both training phases use a batch size of 8.3 million tokens per step. Training was performed using lingua on 512 AMD MI300A GPUs."

Xiaohongshu/RedNote

https://huggingface.co/rednote-hilab/dots.llm1.base

https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf

142B-A14B. "dots.llm1, a large-scale MoE model that activates 14 billion parameters out of a total of 142 billion parameters, delivering performance on par with state-of-the-art models while reducing training and inference costs. Leveraging our meticulously crafted and efficient data processing pipeline, dots.llm1 achieves performance comparable to Qwen2.5-72B after pretraining on 11.2T high-quality tokens and post-training to fully unlock its capabilities. Notably, no synthetic data is used during pretraining. To foster further research, we open-source intermediate training checkpoints at every one trillion tokens, providing valuable insights into the learning dynamics of large language models."

Gemini 2.5 Pro 06-05

Google DeepMind

https://deepmind.google/models/gemini-diffusion/

https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf

Dense

"an upgraded preview of Gemini 2.5 Pro, our most intelligent model yet. Building on the version we released in May and showed at I/O, this model will be the generally available, stable version starting in a couple of weeks, ready for enterprise-scale applications."

https://huggingface.co/XiaomiMiMo/MiMo-7B-RL-0530

Dense

"[2025.05.30] During the RL training, by continuously expanding the training window size (from 32K to 48K), the performance of MiMo-7B-RL-0530 on AIME24 can be continuously improved and eventually surpass that of DeepSeek R1... MiMo-7B-Base is pre-trained on approximately 25 trillion tokens."

Google DeepMind

https://arxiv.org/abs/2505.23735

Dense

"Atlas, a long-term memory module with high capacity that learns to memorize the context by optimizing the memory based on the current and past tokens, overcoming the online nature of long-term memory models. Building on this insight, we present a new family of Transformer-like architectures, called DeepTransformers, that are strict generalizations of the original Transformer architecture."

Google DeepMind

https://arxiv.org/abs/2505.23735

Dense

"Atlas, a long-term memory module with high capacity that learns to memorize the context by optimizing the memory based on the current and past tokens, overcoming the online nature of long-term memory models. Building on this insight, we present a new family of Transformer-like architectures, called DeepTransformers, that are strict generalizations of the original Transformer architecture."

DeepSeek-R1-0528

https://chat.deepseek.com/

https://huggingface.co/deepseek-ai/DeepSeek-R1-0528

Censorship increased significantly. "overall performance is now approaching that of leading models, such as o3 and Gemini 2.5 Pro." MMLU shows MMLU-Redux score with lower error rate.

https://huggingface.co/FractalAIResearch/Fathom-R1-14B

Dense

https://huggingface.co/Tongyi-Zhiwen/QwenLong-L1-32B

Dense

"the first long-context LRM trained with reinforcement learniing for long-context reasoning."

https://claude.ai/

Dense

"Claude Opus 4 is our most intelligent model to date, pushing the frontier in coding, agentic search, and creative writing. With advanced reasoning and powerful collaboration capabilities…Both models can also alternate between reasoning and tool use—like web search—to improve responses…Claude Opus 4 can work continuously for hours on complex, long-running tasks"

https://huggingface.co/tiiuae/Falcon-H1-34B-Instruct-GGUF

https://huggingface.co/papers/2507.22448

Dense

Google DeepMind

https://deepmind.google/models/gemini-diffusion/

https://deepmind.google/models/gemini-diffusion/

Dense

"Gemini Diffusion’s external benchmark performance is comparable to much larger models [like Gemini-2.0-Flash-Lite], whilst also being faster."

Google DeepMind

https://ai.google.dev/gemma/docs/gemma-3n

https://developers.googleblog.com/en/introducing-gemma-3n/

MatFormer

Matryoshka Transformer or MatFormer model architecture. 850M (696M / 620M / 582M).

https://huggingface.co/ParScale/ParScale-4.7B-P8-Python

Dense

"We introduce the third scaling paradigm for scaling LLMs: leverages parallel computation during both training and inference time (Parallel Scaling, or ParScale)... ParScale can use up to 22× less memory increase and 6× less latency increase compared to parameter scaling that achieves the same performance improvement. It can also recycle an off-the-shelf pre-trained model into a parallelly scaled one by post-training on a small amount of tokens, further reducing the training budget." MMLU shows for 1.8B models, not the 4.7B models.

o3 base. "codex-1, a version of OpenAI o3 optimized for software engineering. It was trained using reinforcement learning on real-world coding tasks in a variety of environments to generate code that closely mirrors human style and PR preferences, adheres precisely to instructions, and can iteratively run tests until it receives a passing result."

https://huggingface.co/tiiuae/Falcon-E-3B-Instruct

https://huggingface.co/blog/tiiuae/falcon-edge

Dense

https://windsurf.com/blog/windsurf-wave-9-swe-1

https://windsurf.com/blog/windsurf-wave-9-swe-1

https://chat.primeintellect.ai/

https://storage.googleapis.com/public-technical-paper/INTELLECT_2_Technical_Report.pdf

Dense

https://github.com/pangu-tech/pangu-ultra

https://arxiv.org/abs/2505.04519

718B-A39B. Trained on 6,000 Ascend NPUs (Kunpeng 920 processors in Huawei Atlas 800T A2 servers).

https://chat.mistral.ai/chat

https://mistral.ai/news/mistral-medium-3

Dense

Granite-4.0-Tiny-Preview

https://huggingface.co/ibm-granite/granite-4.0-tiny-preview

https://www.ibm.com/new/announcements/ibm-granite-4-0-tiny-preview-sneak-peek

"the model is only partially trained—it has only seen 2.5T of a planned 15T or more training tokens...Granite 4.0 Tiny-Preview, specifically, is a fine-grained hybrid mixture of experts (MoE) model, with 7B total parameters and only 1B active parameters at inference time... Like its predecessors in Granite 3.2 and Granite 3.3, Granite 4.0 Tiny Preview offers toggleable thinking on and thinking off functionality (though its reasoning-focused post-training is very much incomplete)."

Phi-4-reasoning-plus

https://huggingface.co/microsoft/Phi-4-reasoning-plus

Dense

"Phi-4-reasoning-plus is a state-of-the-art open-weight reasoning model finetuned from Phi-4 using supervised fine-tuning on a dataset of chain-of-thought traces and reinforcement learning."

https://huggingface.co/ibm-ai-platform/Bamba-9B-v2

https://huggingface.co/blog/ibm-ai-platform/bamba-9b-v2

Dense

"During Christmas of 2024, IBM, Princeton, CMU, and UIUC released, Bamba v1, a performant Mamba2 based pretrained model with full data lineage trained to 2T tokens. Since then, we have been busy cooking an update with new datasets. Today, we are excited to release Bamba v2, trained for an additional 1T tokens that significantly improves on Bamba v1. The L1 and L2 leaderboard scores outperform Llama 3.1 8B, which was trained with nearly 5x the amount of data. All of this with the inference speedup that we get from Mamba2 based architecture, which with the latest vLLM is 2-2.5x faster than similar sized transformer models."

https://huggingface.co/Qwen/Qwen3-235B-A22B

Qwen3-235B-A22B. Qwen3-30B-A3B. "Qwen3 is pre-trained on 36 trillion tokens across 119 languages"

https://huggingface.co/microsoft/MAI-DS-R1

https://techcommunity.microsoft.com/blog/machinelearningblog/introducing-mai-ds-r1/4405076

DeepSeek-R1 base. "MAI-DS-R1, a new open weights DeepSeek R1 model variant... post-trained by the Microsoft AI team to improve its responsiveness on blocked topics and its risk profile, while maintaining its reasoning capabilities and competitive performance."

Gemini 2.5 Flash Preview

Google DeepMind

https://aistudio.google.com/prompts/new_chat?model=gemini-2.5-flash-preview-04-17

https://deepmind.google/technologies/gemini/flash/

Context in=1M, out=64k. Knowledge cutoff Jan/2025. Codename 'nebula'. Note: Gemini outputs are watermarked. I do not use GDM models. https://lifearchitect.ai/watermarking/