1 of 37

Understanding the Bitter Lesson in�Time Series Foundation Models�

Danielle Maddix Robinson

BERT²S NEURIPS 2025 WORKSHOP

12.07.2025

Senior Applied Scientist, AWS AI

1

2 of 37

Collaborators

2

Christos

faloutso@

Annan

annanyu@

3 of 37

Chronos

Requires no changes to the language model architecture & training procedure
Probabilistic by design

3

✔ probabilistic by design

✔ requires no changes to the language model architecture or training procedure

Ansari, A.F., et al., “Chronos: Learning the Language of Time Series”, TMLR, 2024.

4 of 37

Baselines

4

5 of 37

Benchmarks

5

6 of 37

Chronos: In-domain Results

6

7 of 37

Chronos: Zero-shot Results

7

8 of 37

Chronos-Bolt⚡

More accurate and 250x faster than the original Chronos models

8

	Chronos-Bolt	Chronos
Input (tokens)	Patches	Individual observations
Output (forecast)	Multi-step quantile forecast	Autoregressive sampling
Loss function	Quantile loss	Cross-entropy loss
Context length	2048	512
Inference device	CPU or GPU	GPU

Ansari, A.F., et al., “Fast and accurate zero-shot forecasting with Chronos-Bolt and AutoGluon”, AWS Technical Report, 2025.

9 of 37

Chronos-Bolt⚡: 250x faster than Chronos

9

10 of 37

Chronos: zero-shot results

10

11 of 37

Chronos-Bolt: zero-shot results

11

12 of 37

Long-term Behavior on Chaotic Systems

12

Zhang, Y. et al. ”Zero-shot Forecasting of Chaotic Systems," ICLR, 2025.

Zhang, Y. et al., “Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learning”, arXiv preprint arXiv:2505.11349, 2025.

Interestingly, Chronos beats Chronos-Bolt and other regression-loss trained time series foundation models on chaotic systems data! Potentially showing the importance of the use of the cross-entropy loss.

Chaotic systems are inherently unpredictable due to their sensitivity to initial conditions, even though they are governed by deterministic laws. This means that tiny changes in the starting point of a chaotic system can lead to vastly different outcomes over time, making long-term prediction extremely difficult.

Caption (top): “Zero-shot forecast models effectively capture attractor geometry. (A) Example forecasts produced by the zero-shot and trained models, for 20 initial conditions from the Lorenz chaotic attractor. (B) The correlation between the fractal dimension of the predicted attractor and the true attractor (Spearman’s rank-order coefficient, N = 2420 points, p < 10−3 for all cases), versus the VPT of the corresponding model. The red markers represent variants of Chronos with different model sizes: tiny (8M parameters), mini (20M), small (46M), base (200M), and large (710M). The blue markers represent the baseline models. Models closer to the top capture the attractor geometry better and models closer to the right make accurate point forecasts for longer. Error bars are standard errors over 135 dynamical systems, each with 20 different initial conditions.”

Caption (bottom): “Context parroting outperforms foundation models in zero-shot forecasting. Left: Forecast error of each model as a function of the forecast horizon. The context length is set to 512 for all models. Right: Valid prediction time of each model as a function of the context length. Dashed lines indicate context lengths exceeding the maximum context window of the model. All results are averaged over the 135 chaotic systems in the dysts database, with 20 trajectories from random initial conditions for each system.�

13 of 37

Design Choices of TSFMs

13

Yu, A. et al., “Understanding the Implicit Biases of Design Choices for Time Series Foundation Models”, ICLR, 2026..

14 of 37

Inductive Biases Overview

14

15 of 37

How Do TSFMs Learn Time?

15

16 of 37

Temporal Frequency Bias

16

17 of 37

Frequency Bias: Good or Bad?

17

18 of 37

Temporal Periodicity Bias�

18

Periodicity: loosely-recurring “periods” in a time series, not necessarily occurring precisely at any fixed frequency
Maintaining strict periodic fidelity facilitates precisely-recurring pattern forecasting, but allowing more flexibility can be useful when time series are nonstationary or patterns drift.

There are several knobs control this:
1. Patching

If sequence’s recurring motif doesn’t align with the patch size, then the patching process can introduce aliasing effects
These effects distorts the original freq content and this effectively changes the periodicity of the embedded sequence ( c)
2. Architecture:

Encoder-only, encoder-decoder (Bolt), decoder-only
Encoders are more effective at preserving the periodicity since they process entire input bidirectionally to capture long-range periodic patterns (a)
Moirai: encoder-only
Chronos/Bolt: encoder-decoder
Times-FM: decoder-only

3. Reg token

Used in encoder-decoder to aggregate global information
Without masking, it can contaminate the preceding context –disrupts many periodic patterns that otherwise would be preserved

(a): Autocorrelation in output of Transformer given periodic input context, which measure periodicity preservation of a model.
(b): Leading principle components of the encoded context of Chronos –bolt with and without reg token with reg periodic
© best matching score on 10,000 randomly sampled ts in Chronos training set with and without patching.Best matching score is computed by comparing last motif of length 16 to all previous motifs.

19 of 37

Periodicity Bias: Good or Bad?

19

Top
Pros and cons of preserving periodicity bias: consider signal formed from a chaotic system’s data
X(t) = Ind(0, Delta)(t)F(t) + Ind(T-Delta, T)(t)F(t-T + Delta), F is motif from Lorentz oscillator
F is repeated twice in the context
Task can be solved simply by matching the repeating motifs.
Periodicity bias is good: Chronos bolt repeat period (encoder-decoder architecture vs. decoder only TimesFM)
(1) model preserves periodicity better (Chronos-Bolt) performs better than one that destroys it more TimesFM (decoder –only)

Bottom
Bad in second example when period slightly changes
(2) But if one focuses too much on the periodic patterns, then one may overlook the non-periodicities
X(t) = Ind(R1)tsin(t) + Ind R2 sin(t), R1 is disjoint union of all even ones from an evenly divided set and R2 is complement.
Bolt focuses heavily on the periodic patterns of R2 bad forecast
TimesFM gives a more accurate partition of R1 and R2 in its forecast

20 of 37

How Do TSFMs Learn Geometry?

20

21 of 37

Design Choice: Embedding Type

Quantization

Continuous Embedding

21

.

22 of 37

Geometric Angular Bias�

22

�- (a) Angles between embedded vectors Theta(x,y) = arccos(|phi_Q(x) dot phi_Q(y)| / ||phi_Q(x)|| ||phi(y)||

- theta_Q(x,y) is significantly larger than theta_C as a function of |x-y|, which makes input appear more different from its close neighbors (increases with number of bins used in quantization)

(b) Histogram of all self-attention scores when inferring two models on a corpus of contexts

- Self-attention scores of Chronos are bimodal - tokens place high attention weights on their immediate neighbors and low weights on distant tokens

- Attention scores of Chronos-Bolt are more evenly distributed, indicating broader mixing of contextual info

Similar for cross-attention
- Chronos focuses almost entirely on most relevant part of the context
-Chronos-Bolt attends more broadly across the context
-Makes Chronos good at parroting and better on chaotic systems
- But comes at cost of less mixing across context – hurt performance on tasks with more complex reasoning

(c) forecasts from both models on context formed by repeating a motif of chaotic system. Chronos lower MASE. Cross-attention scores are aggregated across all layers when generating the first prediction token.

- Learn local info vs global info. Bias affects self-attention and cross-attention

23 of 37

Angular Bias: Good or Bad?��

23

24 of 37

Geometric Distance Bias

24

Just as learned quantization embedding phi_Q bends the input space R into R^d with angle measurement.
It also stretches or squeezes R
Continuous embedding is naturally expected to map nearby numbers to nearby vectors in hidden space, learned phi_Q is less scale-preserving (a)

(a) Distance between embedded vectors for Chronos embedding and Bolt

Quantization-based embedding magnifies smaller scales, which makes nearby number appear more distinct in the hidden space.

(b) Chronos’ and Chronos-Bolts’ forecasts of multi-scale time series

Chronos is more sensitive and better at learning the fine-scale patterns than Bolt
While Chronos and Bolt are good at learning the large-scale pattern, only Chronos nicely learns the small-scale patterns

© Forecasting quality by computing relative MSE as we increase ratio between two scales

as ratio of the two scales increases, all continuous embedding methods are less capable of learning fine-scale pattern

25 of 37

Distance Bias: Good or Bad?

25

26 of 37

Geometric Norm Bias

26

�- Norm bias: map large numbers to large vectors

- Mapping of single element rather than pair of elements as above.

Continuous embedding many TSFMs use ReLU-activated MLP
ReLU activation is positive homogeneous, phi_C maps small input values to small vector in R^d and large values to large vectors

Quant embedding behaves differently

Mapping of each bin, phi_q(D_i) is initialized same distribution, it has no inherent bias toward preserving norm (a)
(a) norm of embedded vectors

(b): scale of each period is same; only difference is their offsets (forecasts of multi-offset time series)

When period is far from zero, its embedded vectors occupy a larger portion of the embedded context, which makes it easier for the model to learn
Conversely, a period near zero is mapped to small vectors that are more easily overwhelmed by rest of the context

© as offset increases (as period’s oscillation amplitude decreases while offset stays fixed), Bolt struggles to forecast near-zero period.

For regular signals with multiple offsets, undesirable imbalance
For sparse signals, it can be advantageous by allowing the model to focus on nonzero entries and ignoring zeros

27 of 37

Norm Bias: Good or Bad?

27

28 of 37

How Do TSFMs Regress to the Mean?

28

29 of 37

Regression-to-the-Mean Bias

29

�-L2: favors mean by averaging all possible outcomes

- L1 favors median – central split of probability mass in half

CE loss: models full probability distribution and settles on a/the mode. Avoid compromise between possible outcomes and allows model to represent distributions with large variances and stronger hetereogenity.
Unlike mean or median base forecasts regressing to the mode lets prediction reflect a sharp, high-prob outcomes instead of smoothed central tendency

(a) Periodic walk between two branches 0 and 1 – fully deterministic all similar
(b) random walk ½ on each branch – uncertainty in context drives TimesFM toward the mean, Bolt toward any value p in [0,1] and Chronos toward the mode (0 or 1)
- Bridge gap deterministic and random, for each 0 <= p <= 0.5, periodic walk and diffuse it with probability p, as p increases more uncertain, models with continuous loss regress more strongly than one trained with CE

For chaotic systems, fractal dimension is measurement of long-term geometry of the trajectories and regressing to median or mean can damage it

30 of 37

Regression-to-the-Mean Bias: Good or Bad?

30

31 of 37

Understanding Transformers for Time Series

31

Yu, A., Maddix, D.C., et al., ”Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility”, ICLR, 2026.

.

32 of 37

Understanding Transformers for Time Series

32

33 of 37

Conclusion: Have we achieved a BERT moment?�

Identify design choices in TSFMs that cause 3 inductive biases:

Temporal, geometric and regression-to-mean

Careful numerical analysis and design of TSFMs is required
Temporal data is a different data-modality, e.g., more compressible, has frequency parameters, continuity in time
Bitter Lesson

Adding traditional forecasting inductive biases can help improve performance on classical benchmarks
But it can hurt generalization on unseen domains and tasks,

e.g., chaotic systems

33

34 of 37

Chronos-2: From Univariate to Multivariate

34

Ansari, A.F., et al., ”Chronos-2: From Univariate to Universal Forecasting”, arXiv preprint arXiv:2510.15821, Under Review, 2025.

35 of 37

fev-bench: Realistic benchmark for time series forecasting

Large-scale evaluation on real-world forecasting tasks

100 univariate & multivariate tasks (incl. 46 with covariates)

Statistically sound aggregation methods

Reliable model comparisons using bootstrap confidence intervals

Extensible infrastructure for reproducible evaluation

Lightweight Python wrapper on top of 🤗 datasets library

Paper: arxiv.org/abs/2509.26468
Code: github.com/autogluon/fev
Leaderboard: huggingface.co/spaces/autogluon/fev-bench

35

Shchur, O., et al., ”fev-bench: A Realistic Benchmark for Time Series Forecasting”, arXiv preprint arXiv:2509.26468, 2025.

36 of 37

Chronos in the Open Source

Inference code available on GitHub

Model weights available on Hugging Face 🤗

Deploy Chronos-2 on AWS using SageMaker JumpStart

Run Chronos with 1 line of code using AutoGluon

36

(Chronos-2 coming soon!)

37 of 37

Danielle Maddix Robinson

dmmaddix@amazon.com

https://dcmaddix.github.io

37

Thank you!