https://lifearchitect.ai/models-table

Timeline view:

https://lifearchitect.ai/timeline

The Memo:

https://lifearchitect.ai/memo

Playground

Parameters
(B)

Tokens
trained (B)

Ratio Tokens:Params
(Chinchilla scaling≥20:1)

Training dataset

Announced
▼

AuroraGPT (ScienceGPT)

Argonne National Laboratory

Three models targeted in Jul/2024: AuroraGPT-7B-P (Ponte Vecchio GPU testing) AuroraGPT-7B-A (Aurora) AuroraGPT-7B-A-S (Aurora + Science).

Due April 2025. Hybrid MoE, 1.2TA78B. 5.2PB corpus=1.3Qa tokens. 1.3 quadrillion tokens = 1,300T tokens = 1,300,000B tokens "Constructed a 5.2 PB high-quality corpus covering vertical domains such as finance, law, and patents." http://jiuyangongshe.com/a/1h4gq724su0 and translated at: https://docs.google.com/document/d/e/2PACX-1vTmx-A5sBe_3RsURGM7VvLWsAgUXbcIb2pFaW7f1FTPgK7mGvYENXGQPoF2u4onFndJ_5tzZ02su-vg/pub

Google DeepMind

Announced, coming soon. Llama5 cancelled: https://x.com/stablequan/status/1977235217879552103

Google DeepMind

https://abehrouz.github.io/files/NL.pdf

Dense

"Combining our self-modifying sequence model with the continuum memory system, we present a learning module, called HOPE, showing promising results in language modeling, continual learning, and long-context reasoning tasks." Announce: https://research.google/blog/introducing-nested-learning-a-new-ml-paradigm-for-continual-learning/ May be released after paper is public.

https://kimi.com/

1TA32B. 1T parameters and 384 experts. Open source SOTA. HLE=51.0 on text-only subset, compare to Grok-4 HLE=50.7 also on text-only, but Grok-4 HLE=44.4 on HLE full, ∴ Kimi K2 Thinking HLE≈44 full (estimated).

https://generalistai.com/blog/nov-04-2025-GEN-0

Dense

"GEN-0, a new class of embodied foundation models built for multimodal training directly on high-fidelity raw physical interaction. Its architecture builds on the strengths of vision and language models while also going beyond them—natively designed to capture human-level reflexes and physical commonsense. One core feature is Harmonic Reasoning, in which the models are trained to simultaneously think and act seamlessly... GEN-0 is pretrained on our in-house robotics dataset, which includes over 270,000 hours of real-world diverse manipulation data, growing at a rate of 10,000 hours a week and accelerating."

https://github.com/shaochenze/calm

Dense

"Continuous Autoregressive Language Models (CALM), a paradigm shift from discrete next-token prediction to continuous next-vector prediction. CALM uses a high-fidelity autoencoder to compress a chunk of K tokens into a single continuous vector, from which the original tokens can be reconstructed with over 99.9% accuracy... We train our models on the Pile uncopyrighted dataset (Gao et al., 2020). The raw text is processed with the Llama 3 tokenizer (Grattafiori et al., 2024), resulting in a training set of ∼230B tokens."

https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct

48B-A3B. "Kimi Linear is a hybrid linear attention architecture that outperforms traditional full attention methods across various contexts, including short, long, and reinforcement learning (RL) scaling regimes. At its core is Kimi Delta Attention (KDA)—a refined version of Gated DeltaNet that introduces a more efficient gating mechanism to optimize the use of finite-state RNN memory."

https://huggingface.co/MiniMaxAI/MiniMax-M2

https://huggingface.co/microsoft/UserLM-8b

Dense

"we trained UserLM-8b to simulate the “user” role in conversation (by training it to predict user turns in a large corpus of conversations called WildChat)."

https://huggingface.co/Salesforce/CoDA-v0-Instruct

Dense

Dense

"Tiny Recursive Model (TRM), a much simpler recursive reasoning approach that achieves significantly higher generalization than HRM, while using a single tiny network with only 2 layers"

https://huggingface.co/ibm-granite/granite-4.0-h-small

https://www.ibm.com/granite/docs/models/granite

32B-A9B. Announce: https://www.ibm.com/new/announcements/ibm-granite-4-0-hyper-efficient-high-performance-hybrid-models

https://huggingface.co/zai-org/GLM-4.6

https://z.ai/blog/glm-4.6

https://huggingface.co/inclusionAI/Ring-1T-preview

https://huggingface.co/inclusionAI/Ring-1T-preview

https://claude.ai/

https://assets.anthropic.com/m/12f214efcc2f457a/original/Claude-Sonnet-4-5-System-Card.pdf

Gemini Robotics 1.5

Google DeepMind

https://storage.googleapis.com/deepmind-media/gemini-robotics/Gemini-Robotics-1-5-Tech-Report.pdf

2. "vision-language-action (VLA) model turns visual information and instructions into motor commands for a robot to perform a task." Available to select partners.

Gemini Robotics-ER 1.5

Google DeepMind

https://aistudio.google.com/?model=gemini-robotics-er-1.5-preview

https://storage.googleapis.com/deepmind-media/gemini-robotics/Gemini-Robotics-1-5-Tech-Report.pdf

1. "vision-language model (VLM) reasons about the physical world, natively calls digital tools and creates detailed, multi-step plans to complete a mission." Available to all devs.

https://huggingface.co/collections/google/timesfm-release-66e4be5fdb56e960c1e482a6

https://research.google/blog/time-series-foundation-models-can-be-few-shot-learners/

Dense

TimesFM-ICF is 6.8% more accurate than TimesFM (Base). Time-series forecasting only. 'a large pretraining corpus of 100B real world time-points' may be more than 100B tokens.

https://chat.qwen.ai/

"Qwen3-Max-Thinking — still under active training — is already demonstrating remarkable potential. When augmented with tool usage and scaled test-time compute, the Thinking variant has achieved 100% on challenging reasoning benchmarks such as AIME 25 and HMMT. "

https://github.com/QwenLM/Qwen3-Omni?tab=readme-ov-file

"Qwen3-Omni is a unified end-to-end model capable of processing multiple modalities, such as text, audio, image and video, and generating real-time text or speech response."... "pretraining utilizes a large-scale dataset containing approximately 2 trillion tokens, with the following distribution across modalities: text (0.57 trillion), audio (0.77 trillion), image (0.82 trillion), video (0.05 trillion), and video-audio (0.05 trillion)."

DeepSeek-V3.1-Terminus

Dense

"perceptive-language model...delivering capabilities that meet or exceed those of models over 50 times its size. Founded by the team behind Meta's Chameleon multimodal models, Perceptron is tackling a fundamental challenge: bringing the power of physical AI to the dynamic, multimodal, and real-time environments we live and work in."

https://grok.com/

"2M token context window, and a unified architecture that blends reasoning and non-reasoning modes in one model."

Google DeepMind

https://huggingface.co/google/vaultgemma-1b

Dense

Qwen3-Next-80B-A3B

https://huggingface.co/collections/Qwen/qwen3-next-68c25fd6838e585db8eeea9d

"Qwen3-Next introduces several key improvements: a hybrid attention mechanism, a highly sparse Mixture-of-Experts (MoE) structure, training-stability-friendly optimizations, and a multi-token prediction mechanism for faster inference."

Dense

"Built on the Qwen2.5 base model, our system shows that smaller models can compete at the highest levels by combining advanced post-training and test-time computation techniques. The approach is based on six key technical pillars: Long Chain-of-thought Supervised Finetuning, Reinforcement Learning with Verifiable Rewards (RLVR), Agentic planning prior to reasoning, Test-time Scaling, Speculative Decoding, and Inference-optimized Hardware, all using publicly available open-source datasets."

Dense

"a modern multilingual encoder trained on 3T tokens and 1833 languages. We introduce several novel elements in training: an inverse masking schedule and a cascading annealed language learning schedule for multilingual data" Announce: https://huggingface.co/blog/mmbert

ERNIE-4.5-21B-A3B-Thinking

https://huggingface.co/baidu/ERNIE-4.5-21B-A3B-Thinking

https://huggingface.co/Kwai-Klear/Klear-46B-A2.5B-Instruct

https://huggingface.co/TildeAI/TildeOpen-30b

Dense

Qwen3-Max-Preview

https://chat.qwen.ai/

GPQA score is SuperGPQA. "our biggest model yet, with over 1 trillion parameters"

Kimi K2-Instruct-0905

https://huggingface.co/moonshotai/Kimi-K2-Instruct

https://huggingface.co/swiss-ai/Apertus-70B-Instruct-2509

Dense

"Apertus – Latin for “open”" 1,811 languages. Announce: https://ethz.ch/en/news-and-events/eth-news/news/2025/09/press-release-apertus-a-fully-open-transparent-multilingual-language-model.html

https://longcat.ai/

560B-A18.6B–31.3B (27B on average). Announce: https://lmsys.org/blog/2025-09-01-sglang-longcat-flash/

https://microsoft.ai/news/two-new-in-house-models/

MAI=Microsoft artificial intelligence. "MAI’s first foundation model trained end-to-end... MAI-1-preview is an in-house mixture-of-experts model, pre-trained and post-trained on ~15,000 NVIDIA H100 GPUs. This model is designed to provide powerful capabilities to consumers seeking to benefit from models that specialize in following instructions and providing helpful responses to everyday queries. We will be rolling MAI-1-preview out for certain text use cases within Copilot"

https://github.com/features/copilot

https://data.x.ai/2025-08-26-grok-code-fast-1-model-card.pdf

"We built grok-code-fast-1 from scratch, starting with a brand-new model architecture. To lay a robust foundation, we carefully assembled a pre-training corpus rich with programming-related content. For post-training, we curated high-quality datasets that reflect real-world pull requests and coding tasks." Announce: https://x.ai/news/grok-code-fast-1

https://huggingface.co/NousResearch/Hermes-4-405B-FP8

https://arxiv.org/abs/2508.18255

Dense

https://github.com/NVlabs/Jet-Nemotron

Dense

DeepSeek-V3.1-Base

https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Base

https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-Base

Dense

Google DeepMind

https://huggingface.co/google/gemma-3-270m-it

Dense

https://poe.com/GPT-5

https://openai.com/index/gpt-5-system-card/

Announce: https://openai.com/index/introducing-gpt-5/. MMLU is based on ES and PT translated from EN.

https://huggingface.co/openai/gpt-oss-120b

https://cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7637/oai_gpt-oss_model_card.pdf

https://huggingface.co/openai/gpt-oss-20b

https://cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7637/oai_gpt-oss_model_card.pdf

https://claude.ai/

https://www.anthropic.com/news/claude-opus-4-1

https://huggingface.co/zai-org/GLM-4.5

https://z.ai/blog/glm-4.5

China Telecom Artificial Intelligence Research Institute

https://github.com/Tele-AI/T1

Dense

Shanghai AI Laboratory/SenseTime

https://huggingface.co/internlm/Intern-S1

https://huggingface.co/internlm/Intern-S1

41T tokens assumes base model of Qwen3. "Built upon a 235B MoE language model and a 6B Vision encoder, Intern-S1 has been further pretrained on 5 trillion tokens of multimodal data"

https://www.stepfun.com/

https://github.com/stepfun-ai/Step3/blob/main/Step3-Sys-Tech-Report.pdf

Qwen3-235B-A22B-Thinking-2507

https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507

https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507

235B-A22B. "Qwen3 is pre-trained on 36 trillion tokens across 119 languages" MMLU score is MMLU-Redux.

200BA40B. In training as of Jul/2025. "to address the overthinking problem in reasoning-intensive tasks"

https://huggingface.co/Kwaipilot/KAT-V1-40B

Dense

Qwen3-Coder-480B-A35B-Instruct

https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct

Qwen3-235B-A22B-Instruct-2507

https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507

235B-A22B. "Qwen3 is pre-trained on 36 trillion tokens across 119 languages" MMLU score is MMLU-Redux.

https://huggingface.co/allenai/FlexOlmo-7x7B-1T

https://arxiv.org/abs/2507.07024v1

37B-A20B. "We adopt the OLMo-2 7B setup, starting from a a checkpoint pre-trained on 4T tokens and annealed for 50B tokens to produce a public expert. We then train two additional experts on math and code, each for 50B tokens, and combine them with the public expert to form a three-expert version of FLEXOLMO."

https://huggingface.co/LGAI-EXAONE/EXAONE-4.0-32B

https://www.lgresearch.ai/data/cdn/upload/EXAONE_4_0.pdf

Dense

“EXAONE”=“EXpert AI for EveryONE”. Training tokens/ratio: EXAONE-3 7.8B=8T tokens (Aug/2024) -> EXAONE-3.5 7.8B=9T -> EXAONE-3.5 32B=6.5T tokens -> EXAONE 4.0 32B=14T tokens. MMLU score is MMLU-Redux. Interesting: "To focus [RL] training on more informative data samples, we perform accuracy-based filtering by generating eight responses from the SFT model and excluding samples where all eight responses are correct, a pre-filtering step that removes problems that are easy for the model to avoid inefficient training."

https://huggingface.co/moonshotai/Kimi-K2-Instruct

https://huggingface.co/RekaAI/reka-flash-3.1

https://www.reka.ai/news/introducing-reka-flash

Dense

https://chat.mistral.ai/chat

https://mistral.ai/news/devstral-2507

Dense

https://grok.com/

"The smartest AI in the world, 100% on SAT, etc, questions that it's never seen before."

Phi-4-mini-flash-reasoning

https://huggingface.co/microsoft/Phi-4-mini-flash-reasoning

Dense

"Pre-training: 5T tokens; Reasoning training: 150B tokens" "At the core of Phi-4-mini-flash-reasoning is the newly introduced decoder-hybrid-decoder architecture, SambaY, whose central innovation is the Gated Memory Unit (GMU), a simple yet effective mechanism for sharing representations between layers. The architecture includes a self-decoder that combines Mamba (a State Space Model) and Sliding Window Attention (SWA), along with a single layer of full attention. The architecture also involves a cross-decoder that interleaves expensive cross-attention layers with the new, efficient GMUs. This new architecture with GMU modules drastically improves decoding efficiency, boosts long-context retrieval performance and enables the architecture to deliver exceptional performance across a wide range of tasks. "

Google DeepMind

https://huggingface.co/google/t5gemma-9b-9b-ul2-it

Dense

Google DeepMind

https://huggingface.co/google/medgemma-27b-it

https://arxiv.org/abs/2507.05201

Dense

https://huggingface.co/tngtech/DeepSeek-TNG-R1T2-Chimera

Assembly of Experts-method of V3-0324, R1, R1-0528. Announce: https://x.com/tngtech/status/1940531045432283412?s=46

Dense

"Spectra-1.1, an open suite of TriLMs trained on up to 1.2 trillion tokens, demonstrating sustained performance gains at scale. Furthermore, to improve inference efficiency, we propose novel 2-bit and 1.6-bit packing schemes for ternary weights"

https://github.com/apple/ml-diffucoder

Dense

"We adapt our model from Qwen2.5-Coder (Hui et al., 2024) as the base model to perform continual pre-training using the adaptation approach from Gong et al. (2025). During this pre-training, we use a 400B-token code pre-training corpus from RefineCode (Huang et al., 2024) and Stackv2 (Lozhkov et al., 2024)."

https://huggingface.co/tencent/Hunyuan-A13B-Instruct

https://huggingface.co/tencent/Hunyuan-A13B-Instruct

80B-A13B. 'We have open-sourced Hunyuan-A13B-Pretrain , Hunyuan-A13B-Instruct , Hunyuan-A13B-Instruct-FP8 , Hunyuan-A13B-Instruct-GPTQ-Int4 on Hugging Face.'

https://chat.inceptionlabs.ai/

https://blogs.windows.com/windows-insider/2025/06/13/announcing-windows-11-insider-preview-build-26200-5651-dev-channel/

Dense

"distillation from Microsoft’s Phi models...Mu is an efficient 330M encoder–decoder language model optimized for small-scale deployment, particularly on the NPUs on Copilot+ PCs. It follows a transformer encoder–decoder architecture"

Gemini Robotics On-Device

Google DeepMind

https://docs.google.com/forms/u/0/d/1sM5GqcVMWv-KmKY3TOMpVtQ-lDFeAftQ-d9xQn92jCE/viewform?ts=67cef986&edit_requested=true

See Mar/2025 Gemini Robotics-ER model for comparison. Announce: https://deepmind.google/discover/blog/gemini-robotics-on-device-brings-ai-to-local-robotic-devices/

https://huggingface.co/ICONNAI/ICONN-1

"ICONN-1 (this version) is optimized for natural, emotionally resonant, and conversational interactions. ICONN-e1 is a specialized variant of the model fine-tuned for advanced reasoning, critical analysis, and complex problem-solving."

https://huggingface.co/MiniMaxAI/MiniMax-M1-80k

https://chat.mistral.ai/chat

https://mistral.ai/static/research/magistral.pdf

Dense

https://huggingface.co/common-pile/comma-v0.1-2t

Dense

"Comma v0.1-2T is a decoder-only transformer that uses the same architecture as Llama 3. Training was done in two stages: first on 1.93 trillion tokens with a cosine learning rate schedule, and second a "cool-down" training phase on 75.5 billion tokens from high-quality sources. The final model is the average of 10 checkpoints during this cool-down phase. Both training phases use a batch size of 8.3 million tokens per step. Training was performed using lingua on 512 AMD MI300A GPUs."

Xiaohongshu/RedNote

https://huggingface.co/rednote-hilab/dots.llm1.base

https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf

142B-A14B. "dots.llm1, a large-scale MoE model that activates 14 billion parameters out of a total of 142 billion parameters, delivering performance on par with state-of-the-art models while reducing training and inference costs. Leveraging our meticulously crafted and efficient data processing pipeline, dots.llm1 achieves performance comparable to Qwen2.5-72B after pretraining on 11.2T high-quality tokens and post-training to fully unlock its capabilities. Notably, no synthetic data is used during pretraining. To foster further research, we open-source intermediate training checkpoints at every one trillion tokens, providing valuable insights into the learning dynamics of large language models."

Gemini 2.5 Pro 06-05

Google DeepMind

https://deepmind.google/models/gemini-diffusion/

https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf

Dense

"an upgraded preview of Gemini 2.5 Pro, our most intelligent model yet. Building on the version we released in May and showed at I/O, this model will be the generally available, stable version starting in a couple of weeks, ready for enterprise-scale applications."

https://huggingface.co/XiaomiMiMo/MiMo-7B-RL-0530

Dense

"[2025.05.30] During the RL training, by continuously expanding the training window size (from 32K to 48K), the performance of MiMo-7B-RL-0530 on AIME24 can be continuously improved and eventually surpass that of DeepSeek R1... MiMo-7B-Base is pre-trained on approximately 25 trillion tokens."

Google DeepMind

https://arxiv.org/abs/2505.23735

Dense

"Atlas, a long-term memory module with high capacity that learns to memorize the context by optimizing the memory based on the current and past tokens, overcoming the online nature of long-term memory models. Building on this insight, we present a new family of Transformer-like architectures, called DeepTransformers, that are strict generalizations of the original Transformer architecture."

Google DeepMind

https://arxiv.org/abs/2505.23735

Dense

"Atlas, a long-term memory module with high capacity that learns to memorize the context by optimizing the memory based on the current and past tokens, overcoming the online nature of long-term memory models. Building on this insight, we present a new family of Transformer-like architectures, called DeepTransformers, that are strict generalizations of the original Transformer architecture."

DeepSeek-R1-0528

https://chat.deepseek.com/

https://huggingface.co/deepseek-ai/DeepSeek-R1-0528

Censorship increased significantly. "overall performance is now approaching that of leading models, such as o3 and Gemini 2.5 Pro." MMLU shows MMLU-Redux score with lower error rate.

https://huggingface.co/FractalAIResearch/Fathom-R1-14B

Dense

https://huggingface.co/Tongyi-Zhiwen/QwenLong-L1-32B

Dense

"the first long-context LRM trained with reinforcement learniing for long-context reasoning."

https://claude.ai/

Dense

"Claude Opus 4 is our most intelligent model to date, pushing the frontier in coding, agentic search, and creative writing. With advanced reasoning and powerful collaboration capabilities…Both models can also alternate between reasoning and tool use—like web search—to improve responses…Claude Opus 4 can work continuously for hours on complex, long-running tasks"

https://huggingface.co/tiiuae/Falcon-H1-34B-Instruct-GGUF

https://huggingface.co/papers/2507.22448

Dense

Google DeepMind

https://deepmind.google/models/gemini-diffusion/

https://deepmind.google/models/gemini-diffusion/

Dense

"Gemini Diffusion’s external benchmark performance is comparable to much larger models [like Gemini-2.0-Flash-Lite], whilst also being faster."

Google DeepMind

https://ai.google.dev/gemma/docs/gemma-3n

https://developers.googleblog.com/en/introducing-gemma-3n/

MatFormer

Matryoshka Transformer or MatFormer model architecture. 850M (696M / 620M / 582M).

https://huggingface.co/ParScale/ParScale-4.7B-P8-Python

Dense

"We introduce the third scaling paradigm for scaling LLMs: leverages parallel computation during both training and inference time (Parallel Scaling, or ParScale)... ParScale can use up to 22× less memory increase and 6× less latency increase compared to parameter scaling that achieves the same performance improvement. It can also recycle an off-the-shelf pre-trained model into a parallelly scaled one by post-training on a small amount of tokens, further reducing the training budget." MMLU shows for 1.8B models, not the 4.7B models.

o3 base. "codex-1, a version of OpenAI o3 optimized for software engineering. It was trained using reinforcement learning on real-world coding tasks in a variety of environments to generate code that closely mirrors human style and PR preferences, adheres precisely to instructions, and can iteratively run tests until it receives a passing result."

https://huggingface.co/tiiuae/Falcon-E-3B-Instruct

https://huggingface.co/blog/tiiuae/falcon-edge

Dense

https://windsurf.com/blog/windsurf-wave-9-swe-1

https://windsurf.com/blog/windsurf-wave-9-swe-1

https://chat.primeintellect.ai/

https://storage.googleapis.com/public-technical-paper/INTELLECT_2_Technical_Report.pdf

Dense