SAGA
Structure-Attended Generative Action Embedding Model
for Multi-Surface User Action Sequences
Tsz Fung Pang · Po Jen Chen · Nimish Ronghe · Farhad Farahani · Bo Zhang
PayPal AI · San Jose, CA
Context-Aware Recommender Systems Workshop · RecSys 2026 · Minneapolis
THE PROBLEM
One user. Six pipelines that never meet.
THE SAME PERSON
one continuous stream of actions
Checkout
transactions, merchant category
Data pipeline 1
Ranker 1
P2P
send / request
Data pipeline 2
Ranker 2
In-App
engagement
Data pipeline 3
Ranker 3
campaign sends
Data pipeline 4
Ranker 4
Push
notifications
Data pipeline 5
Ranker 5
Wallet & Account
cards, banks, login
Data pipeline 6
Ranker 6
The six streams never reconverge — there is no shared representation of this user
Twelve separately built and maintained artifacts for one user — and pipeline 1's checkout signal never reaches ranker 4.
Icons: Font Awesome 4.7 by Dave Gandy — SIL OFL 1.1
SAGA · CARS @ RecSys 2026
2
DATA COLLECTION
Seven surfaces, one sequence
Every interaction on any of these becomes one event, in a single chronological sequence per user.
SAGA · CARS @ RecSys 2026
3
Home tile
in-app
Onboarding flow
in-app
Post-transaction
in-app
End of flow
in-app
outbound
Hero placement
web
Push notification
outbound
One chronological event sequence per user · every interaction becomes one event, in the same six-field schema
FRAMING FOR CARS
Three designs. Three different ideas of what an event is.
These models do not share a field set. What separates them is how much of an event survives into the token sequence — and whether item ID is in the vocabulary at all.
PinFM
K = 1
FUSED AT THE INPUT LAYER · 3 FIELDS
item ID
action type
surface type
Σ
1 fused token
1 token / event
HSTU
K = 2
TWO TOKENS, INTERLEAVED · 2 FIELDS
item / content ID
action type
item ID
action
2 tokens / event
SAGA
K = 7
NO ITEM ID · NOTHING FUSED · 6 FIELDS
surface
parent prod
product
action intent
interaction
mcc
surface
parent prod
product
action intent
interaction
mcc
BOS
separator, prepended — not a field
7 tokens / event
If fusion happens at the input layer, before any attention, training cannot recover collapsed signals.
SAGA · CARS @ RecSys 2026
4
mcc = merchant category code
APPROACH
Structure-attended tokenization
Each event unrolls into K = 7 tokens in a fixed round-robin order, so attention can reach individual fields — within an event and across events.
Paper Figure 1
SAGA · CARS @ RecSys 2026
5
ARCHITECTURE
One backbone, two heads
An ordinary causal decoder in the middle — deliberately, so the ablations isolate tokenization and objective rather than backbone tricks.
BOS
surf
pprod
prod
act
int
mcc
event i
BOS
surf
pprod
prod
act
int
mcc
event i+1
···
Per-field embedding tables E₁ , E₂ , … , E₇ · 378 tokens total
Decoder-only transformer · 12 layers · H = 512 · causal mask
grouped-query attention (8 query heads / 4 key-value heads) · RoPE · SwiGLU · RMSNorm · ~37M parameters
hᵢ ∈ ℝ⁵¹²
Language-modeling (LM) head — every token
fires 7× per event
supervises one TOKEN · detail → next slide
Contrastive head — event's last token
fires 1× per event
supervises one EVENT · detail → slide after
ℒ = ℒ_LM + λ_c · ℒ_con λ_c = 0.5
Causal mask is a compute choice, not a quality claim — bidirectional comparison is future work
SAGA · CARS @ RecSys 2026
6
ARCHITECTURE · HEAD 1 OF 2
LM head: seven vocabularies, one per field
Position modulo seven picks the projection, so a prediction is always made inside the right field.
hᵢ → router: field(i) = i mod 7 → its own linear projection
W₁
BOS
W₂
surface
W₃
parent prod
W₄
product
W₅
action intent
W₆
interaction
W₇
mcc
Each projection scores only its own field's vocabulary — every out-of-field logit is masked to −∞, so a field-type error is impossible by construction.
TRAINING TARGETS
BOS
0
surf
1
pprod
2
prod
3
act
4
int
5
mcc
6
BOS
7
surf
8
pprod
9
prod
10
act
11
int
12
mcc
13
BOS
14
surf
15
event i
event i+1
event i+2
Next-token loss (NTL) · stride 1 — the very next token, which is the next field
+7
+14
Multi-token loss (MTL) · stride s = K = 7 — the same field, k = 1 … 5 events ahead (+7 and +14 drawn)
ℒ_LM = α₁ · NTL + α₂ · MTL α₁ = α₂ = 1 sharpened cross-entropy (logits ÷ 0.07)
SAGA · CARS @ RecSys 2026
7
ARCHITECTURE · HEAD 2 OF 2
Contrastive head: next event, not next token
Fires once per event, on the event's last token — only then has the whole event been read.
✕
✓
✕
the true next event tᵢ₊₁
identical 7-field composite — masked out
valid negative
ℒ_con = − InfoNCE( uᵢ , tᵢ₊₁ )
softmax over in-batch candidates, minus exact-duplicate targets · learnable τ_c (init 0.07) · fired only at each event's last token (mcc), once per event
SAGA · CARS @ RecSys 2026
8
NEGATIVE SAMPLING candidates for one anchor · every anchor's next-event target in the batch
PREDICTED — uᵢ
from the backbone's hidden states
TARGET — tᵢ₊₁
from the shared per-field embedding tables
h₁ , h₂ , … , h₇
the event's seven hidden states
7 × 512
Concat
along the feature axis — every field keeps its own slice
3584
φ_out
Linear · activation · Linear · post-norm (two-layer MLP)
512
L2-normalise
projects onto the unit sphere
512 · ‖uᵢ‖ = 1
E₁(tok₁) , … , E₇(tok₇)
next event's field embeddings — same tables as the model input
7 × 512
Element-wise sum over f
one vector per event — order-free
512
stop-gradient
no gradient reaches E_f along this branch
512
L2-normalise
projects onto the unit sphere
512 · ‖tᵢ₊₁‖ = 1
Intended to avoid representation collapse.
PRE-TRAINING DATA
Dataset and evaluation protocol
Four behavioral sources, mapped to one unified six-field event schema.
Sources — all mapped to the same six-field event schema
Source | Events captured | Cadence | Per-event signal |
Branded checkout | transactions · funding instrument · MCC · amount bins | low | strong |
P2P | send / request · friends-and-family vs goods-and-services · cross-border | low | strong |
In-app engagement | marketing messages: impression · click · conversion | high | weak |
Account & wallet | add card · add bank · login success / failure | medium | mixed |
Corpus and protocol
Sequences
~5.4M
History window
730 days (2 years), US only
Max events per user
146 (= 1024 tokens ÷ K = 7)
Vocabulary
378 tokens total, 9–239 per field
Foundation-model pre-training
≈ 14.5 months
Downstream ranker training
the following 2 months
Downstream evaluation
the following 1 month
Split
temporal — eval follows training
Chronological order, no reweighting. Impressions and other non-positive events stay in the context but are excluded from the loss.
SAGA · CARS @ RecSys 2026
9
THREE FINDINGS
What the ablations
actually told us
01
Multi-domain data scaling is non-monotonic
02
At a fixed budget, resolution beats horizon
03
The two objectives are complementary
SAGA · CARS @ RecSys 2026
10
01
THE ONE WE DID NOT EXPECT
Adding a domain made it worse.
Adding all of them made it better.
Data scaling has mostly meant volume. We scale diversity instead, and the gain is not monotonic.
+3.1%
1 of 4 sources
engagement only
+2.5%
2 of 4 sources
+ checkout
+6.5%
4 of 4 sources
all surfaces
the dip ↓
KEY
CK
P2P
ENG
ACC
filled = included in pre-training
CK checkout · P2P peer-to-peer · ENG in-app engagement · ACC account & wallet
Why partial mixing backfires
Transactions are sparse and strong. Engagement is dense and weak. Put only those two together and the model faces a bimodal distribution it cannot reconcile — negative transfer wins.
Add the remaining surfaces and there is enough distributional diversity that cross-domain structure becomes learnable.
Paper Table 5 · Touchpoint A click lift shown; Touchpoint B and Touchpoint C follow the same ordering
SAGA · CARS @ RecSys 2026
11
NDCG@3 lift on Touchpoint A over a production-feature baseline with no embedding. Architecture and objective fixed; only the mix changes.
02
AGAINST THE LIFELONG-HISTORY TREND
At a fixed token budget, resolution beats history length
Every arm holds the same 1024-token context. Coarser tokenization buys more events; finer buys more detail per event.
| Touchpoint A | Touchpoint B | Touchpoint C | ||
Arm | click | conversion | click | conversion | click |
SAGA K = 7, 146 evt | +6.5 | +3.3 | +1.7 | +0.6 | −1.4 |
Fused K = 1, 1024 evt | +6.4 | +3.0 | +0.9 | +0.5 | −16.0 |
Interleaved K = 2, 512 evt | +5.6 | +3.1 | −1.3 | −0.5 | −10.5 |
Relative % NDCG@3 lift over the no-embedding baseline (production features, no pre-trained embedding). Best per column in bold. Paper Table 4, Set B.
146 rich events ≥ 1024 coarse ones.
SAGA matches or beats fused with seven times fewer events in the window — recent events carry most of the signal.
Cross-touchpoint stability is the real gap.
Fused is competitive on the strongest touchpoint and collapses elsewhere: −16.0 on Touchpoint C.
Hard field selection is worst for a multi-domain setup.
K = 2 discards surface — the field that marks which domain an event came from. That costs little in a single-domain corpus, which is what HSTU was built for; across domains it loses on every touchpoint.
SAGA · CARS @ RecSys 2026
12
03
DUAL-HEAD ABLATION
The two objectives are complementary, not redundant
Neither head alone captures both granularities — field-level transitions and event-level intent.
| Touchpoint A | Touchpoint B | Touchpoint C | ||
Training objective | click | conversion | click | conversion | click |
Dual-head λ_c = 0.5 | +6.5 | +3.3 | +1.7 | +0.6 | −1.4 |
LM heads only | +3.5 | +1.6 | +0.4 | +0.3 | −4.2 |
Contrastive only | +5.8 | +3.9 | +0.5 | +0.4 | −8.6 |
Relative % NDCG@3 lift over the no-embedding baseline. Best per column in bold. Single run per arm — read the ordering, not the decimals. Paper Table 3.
Different granularities, not the same signal twice.
The LM heads learn how fields transition; the contrastive head learns to tell whole events apart — which the per-field loss never asks for.
The dual head is not best at everything.
Contrastive-only wins Touchpoint A conversion outright, and degrades worst on the sparse touchpoint.
SAGA · CARS @ RecSys 2026
13
Touchpoint C is that sparse touchpoint.
Low engagement volume and very little signal per user, so every arm degrades there. Detail in the appendix.
PRODUCTION
It transfers online — frozen, with no fine-tuning
Relative lift over the control, bars starting at zero. One result clears significance: click-through rate on Touchpoint B.
What the control was
The prior production deep-learning ranker — user sequences built from engagement events only, plus user-level aggregates from the other surfaces. No pre-trained embedding.
What to take from this
A real lift, but on one surface and one metric — not a foundation model yet.
It does agree with the offline ablations, and that is the useful part: the offline protocol predicts the online direction.
Touchpoint C is low-engagement with very little signal per user. Offline, every arm degrades there too.
Paper Table 6
SAGA · CARS @ RecSys 2026
14
Touchpoint C: click-attributed metrics were invalidated by an impression-rendering bias between arms, so only conversion is reported.
+5%
+2.5%
0
+5.01%
click-through rate
+1.53%
conversion rate
Touchpoint B
−0.06%
conversion rate
Touchpoint C
statistically significant
not statistically significant
TAKEAWAYS
What we would tell the next team
01
Multi-domain data scaling is non-monotonic
Partial mixing can be worse than no mixing.
02
At a fixed budget, spend tokens on resolution and not horizon
146 richly-tokenized events beat 1024 coarse ones in the same context window — and transfer far more evenly across touchpoints.
03
Know where the representation stops helping
Every variant degraded performance on a sparse-signal touchpoint. The embedding does not help everywhere.
Open questions
tspang@paypal.com · arXiv 2608.15429
SAGA · CARS @ RecSys 2026
15
· Contribution from fine-tuning
· Comparison with a bidirectional encoder
· Surface-conditioned generation
· Evaluation across more surfaces and use cases
APPENDIX
Kept back for
questions
A1
How the two heads interleave — the 7:1 supervision ratio
A2
Where the method breaks — the sparsest touchpoint
SAGA · CARS @ RecSys 2026
16
ARCHITECTURE · TRAINING CADENCE
The two heads fire at different rhythms
Round-robin tokenization gives the loss a fixed beat: field-level supervision at every position, event-level supervision once per event.
0
1
2
3
4
5
6
7
8
9
10
11
12
13
position
input xᵢ
BOS
surf
pprod
prod
act
int
mcc
BOS
surf
pprod
prod
act
int
mcc
ℒ_LM
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
7× / event
ℒ_con
—
—
—
—
—
—
✓
—
—
—
—
—
—
✓
1× / event
target ŷᵢ
surf
pprod
prod
act
int
mcc
BOS
surf
pprod
prod
act
int
mcc
BOS
shifted by one
event i
event i+1
Why the beat matters
The LM head gets seven gradient updates per event, one per field — it learns the micro-dynamics of how fields follow each other. The contrastive head gets one, on the event's last token, where the whole event has just been read.
The last token does the work
mcc closes every event, so by the time the model reaches it the whole event has been read. That is why it carries the contrastive loss and nothing else does.
Rebuilt from paper Figure 3
SAGA · CARS @ RecSys 2026
17
WHERE IT BREAKS
On the sparsest touchpoint, every version
of this model makes things worse.
Touchpoint C click lift for every arm we trained. This is a boundary condition on the method, not a result we are presenting as a win.
SAGA K = 7 (146 evt)
-1.4%
Fused K = 1 (146 evt)
-3.9%
LM only (K = 7, 146 evt)
-4.2%
Contrastive only (K = 7)
-8.6%
Interleaved K = 2 (512 evt)
-10.5%
Interleaved K = 2 (146 evt)
-12.6%
Fused K = 1 (1024 evt)
-16.0%
We are not claiming a win here
Touchpoint C has a very low click rate; the signal-to-noise ratio is poor before any embedding is added. A representation that encodes general interaction patterns interferes with an already-weak signal.
SAGA degrades least, but least-bad is not good. The operational answer is a threshold: below some signal density, leave the embedding out of the ranker.
Consistent with an earlier internal finding: the embedding helps click prediction and slightly hurts conversion prediction. Same pattern, different model generation.
Paper Tables 3 and 4 · Touchpoint C click lift, all arms
SAGA · CARS @ RecSys 2026
18