1 of 54

Understanding the Bitter Lesson in�Time Series Foundation Models�

Danielle Maddix Robinson

CME 500 SEMINAR

06.02.2026

Senior Applied Scientist, AWS AI

1

2 of 54

Collaborators

2

Christos

faloutso@

Annan

annanyu@

3 of 54

Time series data

Time series are measurements made at regular intervals

3

Energy

Finance

Retail

Weather

Healthcare

Traffic

Time

Value

2024-01-01

2024-01-02

2024-01-03

4 of 54

Time series forecasting

What will happen in the future given the past?

4

5 of 54

Probabilistic forecasting

Probabilistic forecast captures uncertainty in predictions

5

6 of 54

Local statistical models

Fit a separate model for each individual time series
Examples: ARIMA, ETS, Theta

Strong baseline (esp. limited data)
Often interpretable
Low flexibility
Slow inference

6

7 of 54

Global deep learning models

Fit a single model for each task
Examples: DeepAR, TFT, PatchTST

High flexibility
Fast inference
Slow training
Data hungry

7

8 of 54

ChatGPT Moment for Forecasting?

Can we develop a single model that both

requires no dataset-specific training and
performs well on new time series tasks?

8

9 of 54

Language modeling and forecasting

9

10 of 54

Tokenization

10

Text language models have a discrete vocabulary

Time series are real-valued signals

11 of 54

Time series tokenization

11

12 of 54

Regression via classification

12

Probability p given by softmax

Cross-entropy loss seems to be important in Chronos’ good performance on chaotic systems data compared to TSFMs based on regression losses
Zhang, Y., Gilpin, W., Zero-shot forecasting of chaotic systems, ICLR 2025.
Zhang Y., Gilpin, W., Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learning, https://arxiv.org/abs/2505.11349, 2025.
Lai, J., Bao, A., Gilpin, W., Panda: A pretrained forecast model for universal representation of chaotic dynamics, https://arxiv.org/abs/2505.13755, 2025.�

Has been suggested in the literature:
Torgo, Luis, and Joao Gama. "Regression using classification algorithms." Intelligent Data Analysis 1.4 (1997): 275-292.
Stewart, Lawrence, et al. "Regression as classification: Influence of task formulation on neural network features." AISTATS (2023).

13 of 54

Chronos

Requires no changes to the language model architecture & training procedure
Probabilistic by design

13

✔ probabilistic by design

✔ requires no changes to the language model architecture or training procedure

Ansari, A.F., et al., “Chronos: Learning the Language of Time Series”, TMLR, 2024.

14 of 54

Training corpus

14

Language: 15T tokens

Time Series: 84B tokens

15 of 54

Training corpus

84B tokens

15

16 of 54

Training datasets

28 datasets from various domains and frequencies
890K time series with 84B observations

16

17 of 54

Augmenting the training set

TSMix augmentations

Improve pattern diversity by mixing time series from different datasets

Synthetic data

Generate synthetic time series from Gaussian processes

17

18 of 54

Baselines

18

19 of 54

Evaluation Metrics

19

20 of 54

Benchmarks

20

21 of 54

Chronos: In-domain Results

21

22 of 54

Chronos: Zero-shot Results

22

23 of 54

Chronos-Bolt⚡

More accurate and 250x faster than the original Chronos models

23

	Chronos-Bolt	Chronos
Input (tokens)	Patches	Individual observations
Output (forecast)	Multi-step quantile forecast	Autoregressive sampling
Loss function	Quantile loss	Cross-entropy loss
Context length	2048	512
Inference device	CPU or GPU	GPU

Ansari, A.F., et al., “Fast and accurate zero-shot forecasting with Chronos-Bolt and AutoGluon”, AWS Technical Report, 2025.

24 of 54

Chronos-Bolt⚡: 250x faster than Chronos

24

25 of 54

Chronos-Bolt: zero-shot results

25

26 of 54

Long-term Behavior on Chaotic Systems

26

Zhang, Y. et al. ”Zero-shot Forecasting of Chaotic Systems," ICLR, 2025.

Zhang, Y. et al., “Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learning”, arXiv preprint arXiv:2505.11349, 2025.

Interestingly, Chronos beats Chronos-Bolt and other regression-loss trained time series foundation models on chaotic systems data! Potentially showing the importance of the use of the cross-entropy loss.

Chaotic systems are inherently unpredictable due to their sensitivity to initial conditions, even though they are governed by deterministic laws. This means that tiny changes in the starting point of a chaotic system can lead to vastly different outcomes over time, making long-term prediction extremely difficult.

Caption (top): “Zero-shot forecast models effectively capture attractor geometry. (A) Example forecasts produced by the zero-shot and trained models, for 20 initial conditions from the Lorenz chaotic attractor. (B) The correlation between the fractal dimension of the predicted attractor and the true attractor (Spearman’s rank-order coefficient, N = 2420 points, p < 10−3 for all cases), versus the VPT of the corresponding model. The red markers represent variants of Chronos with different model sizes: tiny (8M parameters), mini (20M), small (46M), base (200M), and large (710M). The blue markers represent the baseline models. Models closer to the top capture the attractor geometry better and models closer to the right make accurate point forecasts for longer. Error bars are standard errors over 135 dynamical systems, each with 20 different initial conditions.”

Caption (bottom): “Context parroting outperforms foundation models in zero-shot forecasting. Left: Forecast error of each model as a function of the forecast horizon. The context length is set to 512 for all models. Right: Valid prediction time of each model as a function of the context length. Dashed lines indicate context lengths exceeding the maximum context window of the model. All results are averaged over the 135 chaotic systems in the dysts database, with 20 trajectories from random initial conditions for each system.�

27 of 54

Design Choices of TSFMs

27

Yu, A. et al., “Understanding the Implicit Biases of Design Choices for Time Series Foundation Models”, ICLR, 2026.

28 of 54

Inductive Biases Overview

28

29 of 54

How Do TSFMs Learn Time?

29

30 of 54

Temporal Frequency Bias

30

31 of 54

Frequency Bias: Good or Bad?

31

32 of 54

Temporal Periodicity Bias�

32

Periodicity: loosely-recurring “periods” in a time series, not necessarily occurring precisely at any fixed frequency
Maintaining strict periodic fidelity facilitates precisely-recurring pattern forecasting, but allowing more flexibility can be useful when time series are nonstationary or patterns drift.

(a): Autocorrelation in output of Transformer given periodic input context, which measure periodicity preservation of a model.
(b): Leading principle components of the encoded context of Chronos –bolt with and without reg token with reg periodic
(c) best matching score on 10,000 randomly sampled ts in Chronos training set with and without patching. Best matching score is computed by comparing last motif of length 16 to all previous motifs.

There are several knobs control this:
1. Patching

If sequence’s recurring motif doesn’t align with the patch size, then the patching process can introduce aliasing effects
These effects distorts the original freq content and this effectively changes the periodicity of the embedded sequence ( c)
2. Architecture:

Encoder-only, encoder-decoder (Bolt), decoder-only
Encoders are more effective at preserving the periodicity since they process entire input bidirectionally to capture long-range periodic patterns (a)
Moirai: encoder-only
Chronos/Bolt: encoder-decoder
Times-FM: decoder-only

3. Reg token

Used in encoder-decoder to aggregate global information
Without masking, it can contaminate the preceding context –disrupts many periodic patterns that otherwise would be preserved

33 of 54

Periodicity Bias: Good or Bad?

33

Top
Pros and cons of preserving periodicity bias: consider signal formed from a chaotic system’s data
X(t) = Ind(0, Delta)(t)F(t) + Ind(T-Delta, T)(t)F(t-T + Delta), F is motif from Lorentz oscillator
F is repeated twice in the context
Task can be solved simply by matching the repeating motifs.
Periodicity bias is good: Chronos bolt repeat period (encoder-decoder architecture vs. decoder only TimesFM)
(1) model preserves periodicity better (Chronos-Bolt) performs better than one that destroys it more TimesFM (decoder –only)

Bottom
Bad in second example when period slightly changes
(2) But if one focuses too much on the periodic patterns, then one may overlook the non-periodicities
X(t) = Ind(R1)tsin(t) + Ind R2 sin(t), R1 is disjoint union of all even ones from an evenly divided set and R2 is complement.
Bolt focuses heavily on the periodic patterns of R2 bad forecast
TimesFM gives a more accurate partition of R1 and R2 in its forecast

34 of 54

How Do TSFMs Learn Geometry?

34

35 of 54

Design Choice: Embedding Type

Quantization

Continuous Embedding

35

.

36 of 54

Geometric Angular Bias�

36

�- (a) Angles between embedded vectors Theta(x,y) = arccos(|phi_Q(x) dot phi_Q(y)| / ||phi_Q(x)|| ||phi(y)||

- theta_Q(x,y) is significantly larger than theta_C as a function of |x-y|, which makes input appear more different from its close neighbors (increases with number of bins used in quantization)

(b) Histogram of all self-attention scores when inferring two models on a corpus of contexts

- Self-attention scores of Chronos are bimodal - tokens place high attention weights on their immediate neighbors and low weights on distant tokens

- Attention scores of Chronos-Bolt are more evenly distributed, indicating broader mixing of contextual info

Similar for cross-attention
- Chronos focuses almost entirely on most relevant part of the context
-Chronos-Bolt attends more broadly across the context
-Makes Chronos good at parroting and better on chaotic systems
- But comes at cost of less mixing across context – hurt performance on tasks with more complex reasoning

(c) forecasts from both models on context formed by repeating a motif of chaotic system. Chronos lower MASE. Cross-attention scores are aggregated across all layers when generating the first prediction token.

- Learn local info vs global info. Bias affects self-attention and cross-attention

37 of 54

Angular Bias: Good or Bad?��

37

38 of 54

Geometric Distance Bias

38

Just as learned quantization embedding phi_Q bends the input space R into R^d with angle measurement.
It also stretches or squeezes R
Continuous embedding is naturally expected to map nearby numbers to nearby vectors in hidden space, learned phi_Q is less scale-preserving (a)

(a) Distance between embedded vectors for Chronos embedding and Bolt

Quantization-based embedding magnifies smaller scales, which makes nearby number appear more distinct in the hidden space.

(b) Chronos’ and Chronos-Bolts’ forecasts of multi-scale time series

Chronos is more sensitive and better at learning the fine-scale patterns than Bolt
While Chronos and Bolt are good at learning the large-scale pattern, only Chronos nicely learns the small-scale patterns

© Forecasting quality by computing relative MSE as we increase ratio between two scales

as ratio of the two scales increases, all continuous embedding methods are less capable of learning fine-scale pattern

39 of 54

Distance Bias: Good or Bad?

39

40 of 54

Geometric Norm Bias

40

�- Norm bias: map large numbers to large vectors

- Mapping of single element rather than pair of elements as above.

Continuous embedding many TSFMs use ReLU-activated MLP
ReLU activation is positive homogeneous, phi_C maps small input values to small vector in R^d and large values to large vectors

Quant embedding behaves differently

Mapping of each bin, phi_q(D_i) is initialized same distribution, it has no inherent bias toward preserving norm (a)
(a) norm of embedded vectors

(b): scale of each period is same; only difference is their offsets (forecasts of multi-offset time series)

When period is far from zero, its embedded vectors occupy a larger portion of the embedded context, which makes it easier for the model to learn
Conversely, a period near zero is mapped to small vectors that are more easily overwhelmed by rest of the context

© as offset increases (as period’s oscillation amplitude decreases while offset stays fixed), Bolt struggles to forecast near-zero period.

For regular signals with multiple offsets, undesirable imbalance
For sparse signals, it can be advantageous by allowing the model to focus on nonzero entries and ignoring zeros

41 of 54

Norm Bias: Good or Bad?

41

42 of 54

How Do TSFMs Regress to the Mean?

42

43 of 54

Regression-to-the-Mean Bias

43

�-L2: favors mean by averaging all possible outcomes

- L1 favors median – central split of probability mass in half

CE loss: models full probability distribution and settles on a/the mode. Avoid compromise between possible outcomes and allows model to represent distributions with large variances and stronger hetereogenity.
Unlike mean or median base forecasts regressing to the mode lets prediction reflect a sharp, high-prob outcomes instead of smoothed central tendency

(a) Periodic walk between two branches 0 and 1 – fully deterministic all similar
(b) random walk ½ on each branch – uncertainty in context drives TimesFM toward the mean, Bolt toward any value p in [0,1] and Chronos toward the mode (0 or 1)
- Bridge gap deterministic and random, for each 0 <= p <= 0.5, periodic walk and diffuse it with probability p, as p increases more uncertain, models with continuous loss regress more strongly than one trained with CE

For chaotic systems, fractal dimension is measurement of long-term geometry of the trajectories and regressing to median or mean can damage it

44 of 54

Regression-to-the-Mean Bias: Good or bad?

44

45 of 54

Understanding Transformers for Time Series

45

Yu, A., Maddix, D.C., et al., ”Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility”, ICLR, 2026.

46 of 54

Understanding Transformers for Time Series

46

47 of 54

Conclusions

�

Identify design choices in TSFMs that cause 3 inductive biases:

Temporal, geometric and regression-to-mean

Careful numerical analysis and design of TSFMs is required
Temporal data is a different data-modality, e.g., more compressible, has frequency parameters, continuity in time
Bitter Lesson

Adding traditional forecasting inductive biases can help improve performance on classical benchmarks
But it can hurt generalization on unseen domains and tasks,

e.g., chaotic systems

47

48 of 54

Chronos-2: From Univariate to Multivariate

48

Ansari, A.F., et al., ”Chronos-2: From Univariate to Universal Forecasting”, arXiv preprint arXiv:2510.15821, 2025.

49 of 54

fev-bench: Realistic benchmark for time series forecasting

Large-scale evaluation on real-world forecasting tasks

100 univariate & multivariate tasks (incl. 46 with covariates)

Statistically sound aggregation methods

Reliable model comparisons using bootstrap confidence intervals

Extensible infrastructure for reproducible evaluation

Lightweight Python wrapper on top of 🤗 datasets library

Paper: arxiv.org/abs/2509.26468
Code: github.com/autogluon/fev
Leaderboard: huggingface.co/spaces/autogluon/fev-bench

49

Shchur, O., et al., ”fev-bench: A Realistic Benchmark for Time Series Forecasting”, arXiv preprint arXiv:2509.26468, 2025.

50 of 54

Chronos in the Open Source

Inference code available on GitHub

Model weights available on Hugging Face 🤗

Deploy Chronos-2 on AWS using SageMaker JumpStart

Run Chronos with 1 line of code using AutoGluon

50

(Chronos-2 coming soon!)

51 of 54

ProbHardE2E

51

Utkarsh, U., Maddix, D.C., Ma, R., Mahoney, M., Wang, Y., "End-to-End Probabilistic Framework for Learning with Hard Constraints”, ICLR 2026.

52 of 54

ProbHardE2E

52

Utkarsh, U., Maddix, D.C., Ma, R., Mahoney, M., Wang, Y., "End-to-End Probabilistic Framework for Learning with Hard Constraints”, ICLR 2026.

53 of 54

Mitra: Tabular Foundation Model

53

Blog

Hugging Face

AutoGluon 1.4

Zhang, X., Maddix, D.C., et al., ”Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models”, NeurIPS, 2025.

54 of 54

Danielle Maddix Robinson

dcmaddix@gmail.com

https://dcmaddix.github.io

54

Thank you!