Fantastic Pretraining Optimizers
and Where to Find Them
A Fair Comparison of Modern Optimizers + Hyperball
Wen, Hall, Ma, Liang (2025) | Wen, Dang, Lyu, Ma, Liang (2026)
This Talk
Finding 1
With fair tuning across multiple scales, matrix-based optimizers including Muon & Soap yield clear speedup over AdamW.
However, all modern optimizers reach only 10% speedup over AdamW at 1.2B scale / ~100B tokens mysteriously.
Finding 2
The disappearing speedup is (partly) due to a constant weight decay (~0.1) shaping the effective learning rate.
Hyperball (weight-normalized training) eliminates this effect — persistent 20-30% speedup across scales.
Fantastic Optimizers and Where to Find Them (2025)�Fantastic Optimizers and Where to Find Them 2.1 (2026)
Pretraining as an Optimization Task
Pretraining is the mostly costly component of LLM training
Many recent optimizers claim 1.4-2x speedup over AdamW: Sophia [Liu et al.’24], SOAP [Vyas et al.’25], Muon [Liu et al.’25], MARS [Yuan et al.’25], C-AdamW / C-Lion (Cautious Optimizers) [Liang et al.’25], FOCUS [Liu et al.’25], SCION [Pethick et al.’25], SWAN [Ma et al.’25]
Puzzling phenomenon:
Why is AdamW still popular despite these claims?
A 1.5x speedup could save ~$20M on LLaMA 3.1 405B training
The Optimizer Landscape
Baseline: Adam(W)
Scalar Variants
AdamW
NAdam
The Optimizer Landscape
Matrix-Based
Shampoo
Scalar-Based
Change Subject
Shampoo
Muon
Confusing Results in the Literature
Different evaluation protocols lead to vastly different conclusions about optimizer effectiveness.
Muon can't be both worse than AdamW AND 2x better!
Figure 1 of MARS
Figure 1 of Moonlight
Key Reason: Optimizers are not evaluated under respectively best practices!
Methodology: Three-Phase Fair Tuning
Fairness Principles
Phase I
Fine-grained sweeps for every hyperparameter per-optimizer
Phase II
Sweeping�scale-sensitive hypers�for larger models or more data
Phase III
Hyper scaling laws�extrapolating to 1.2B�models
Phase I: Coordinate Descent Algorithm
Algorithm:
Coordinate descent on a discrete hyperparameter grid: tune one hyperparameter at a time, keep updates only if validation loss improves and stop when no single change helps.
Settings:
0.1B model 1-8x Chinchilla
0.3B, 0.5B 1x Chinchilla
1 Chinchilla = 20x token per parameter
Results: One table on the sensitivity of all hyperparameters for each settings
Phase I: Coordinate Descent Results
Observations:
Takeaway:
Narrow later sweeps to only scaling-sensitive hyperparameters like learning rate.
Phase II: Sweeps on 0.1B - 0.5B & 1-8x Chinchilla
Speedup:
Additional percentage of tokens AdamW needs to match the performance of optimizer X
Observations:
Phase III: Extrapolation to 1.2B
Fit hyperparameter scaling laws on 12 tested settings per optimizer and extrapolate to 1.2B models
Observations:
Qualitative Observations
Early Training is Misleading
Validation-loss curves during initial LR decay tend to exaggerate performance gaps and can even reverse the eventual ranking.
Shared Norm Dynamics
Weight norm rises/falls with LR schedule. Gradient norm increases during LR decay. Consistent across optimizers. [Defazio et.al.’25]
Key Takeaways: Fair Optimizer Comparison
With fair tuning, many claimed 1.4-2x speedups shrink to 1.1-1.3x.
Key parameters for tuning: learning rate
Matrix-preconditioned optimizers are best-in-class, but gains decay with scale
Scalar variants provides small but consistent ~1.1x speedup
Always compare at target budget with LR decay to near zero, across scales and data ratios
Extension: Batch size matters empirically
Based on discussions over X. We have some more reflections on our study.
Our objective:
Find the best hyperparameters that reaches lowest loss in one pass of a fixed amount of data (this includes tuning the batch size).
In practice:
Wall time matters more, and it crucially depends on the MFUs (hardware utilization rate)
Especially, batch size positively correlates with MFUs.
Muon has much better performance compared to AdamW at large batch size.
Extension: Preserving the speedup
The relative slowdown of matrix optimizer at larger compute budget is very puzzling.
[Qiu et.al. 2025] points out that properly scaling the weight decay with 1/width can preserve the speedup of matrix-based optimizer at 1x Chinchilla regime
This indicates that:
Controlling the weight norm dynamics of optimizers is crucial for preserving the speedup of optimizers.
[Qiu et.al. 2025], Fig 1
Hyperball: Weight-Normalized Training
Two simple modifications:
1. Remove weight decay
2. Normalize weight and update norms
Compatible with any optimizer:
Adam-Hyperball (AdamH)
Muon-Hyperball (MuonH)
Why doesn't this hurt representation power?
Modern Transformers use RMSNorm with rescaling parameters. These parameters steer the output norm when weight norm is fixed.
Hyperball: Persistent Speedup
AdamH and MuonH show persistent 20-30% speedup over weight decay counterparts with increasing scale with AdamH even outperforms MuonW at 1.2B scale!
We also observe MuonH to provide constant loss gap with respect to MuonW on 130M model with extreme overtraining.
Hyperball: Hyperparameter transfer out-of-box
Thanks to constant ratio between update and weight norm, we observe a consistent learning rate scaling for Hyperball.
AdamH enables highly predictable scaling in Marin (0.005 difference on 8B model & 32B model on the way).
Applied in Current 535B Marin Open Research Training
https://wandb.ai/marin-community/marin_moe/reports/535B-A23B-18T-Token-Hero-Run-Scaling-Ladder--VmlldzoxNzc2MDM5Ng