1 of 13

Deciding When to Rely on Visual Information: Gated Multimodal Fusion in Sequential Recommendation

Natalija Glisovic, Danica Kragic, Martin Tegner

CARS @ RecSys ‘26

2 of 13

The Problem: Modality Choice Is Also Context

CARS has long modeled context as predefined dimensions such as time, location
We argue which modality to trust is itself a context variable that is latent and inferred from interaction state, not predefined
When fusing multimodal content in recommenders it can be uniformal or using gating approach (e.g. MMSR, HM4SR)
But the gate itself is usually not examined for what it reveals
Our approach: treat visual utility as a latent, measurable variable, and use the gate as a diagnostic lens

2

How we typically think of context in CARS - traditionally meant predefined dimensions — time of day, location, device, session — things you can specify in advance and condition a model on.
We argue differently, which modality to trust, whethr to rely on items image or collaborative signal – is ittself a context variable. But unlike time or location, it's not something you can define upfront. It's latent, and it has to be inferred from the user's interaction state at the point of recommendation.
Now in multimodal rec more broadly, people fuse visual and collaborative signals in ways – uniformal (same weight all items) or gate – so gate not new
Whats missing is gate is usually treated as a method approach to improve ranking, but what does the gate reveal?
Treat visual utility – how much an items image help recommendation – as a latent variable that can be measured

3 of 13

Motivating Example

Text descriptions in e-commerce are often sparse or generic
Images can capture visual characteristics that drive purchase decisions but aren't in text
When to depend on images is related to strong collaborative signals, but also visual utility
Under sparsity/cold-start, ID embeddings alone have little signal so content can fill the gap, but this may differ depending on the item itself

3

4 of 13

Research Questions

Is visual utility item-dependent, or can uniform fusion work?

3

2

1

Does visual utility increase under sparsity, when collaborative signals are weak?

Does visual utility correlate with visual distinctiveness?

4

5 of 13

VisGate: Overview

Item-level, context-aware gate: decides how much to rely on visual vs. ID signal, per item, per point in a user's sequence
Dual purpose: competitive recommender and measurement tool for modality utility
Built on a BERT4Rec-style backbone

5

6 of 13

VisGate: Grounding and gating

Visual projection (grounding)

MLP projects CLIP features into a new space via contrastive learning over temporally co-occurring items
Kept separate from the ID embedding space to preserve complementarity instead of forcing alignment

Adaptive gating (fusion)

Gate conditioned on ID embedding + visual embedding + current sequence context (transformer hidden state)
Same item can get different gate values early vs. late in a user's history

Projection

Gating

6

So the method has two pieces working together — grounding the visual signal, and then deciding how much to trust it…

don’t just take raw clip image features and plug in – pass thrugh small learnable MLP that projects into new space and train that MLP through contrastive learning over temporally co-occuring items
Means concretely: if two products show up close together in same user interaction sequence we treat that as they are visually related in a way that matters for rec
So projector learns to pull those items closer in new visual space and unrelated apart, this is proxy label – inferring from behaviour
Projected space we keep searate frm id embedding space so not aligning images to collaborative embedding like some retrieval_based methods as we argu if we align they end up carying redundant info so gate has harder time chppsing between thm

Second gating: once we have ID embedding and grounded visual then gate decides and this is also conditioned on current sequence context so transformer hidden state at that point in user history

7 of 13

Experimental Setup

Four datasets, with varying sparsity: IKEA (proprietary), Amazon Scientific, Video Games, Health & Household
Baselines across CF, visually-aware, multimodal, and multimodal sequential families

7

8 of 13

Main Ranking Results

VisGate competitive performance against other multimodal methods, especially on the sparsest datasets
Gains shrink on denser datasets, where text-aware baselines (FREEDOM, HM4SR) stay competitive

8

9 of 13

RQ1: Is Visual Utility Is Item-Dependent?

Item-level gating beats any fixed gate value (0.3 / 0.5 / 0.7)
Complementary pattern: strong-image/weak-ID items get low gates (~0.21); strong-ID/weak-image items get high gates (~0.76)

Gate value as a function of ID and image embedding strength on IKEA dataset

9

10 of 13

RQ2: Does Visual Utility Increases Under Sparsity?

Largest performance gains are on the two sparsest datasets
At the item level: weak-ID items consistently get lower gate values (more visual reliance)
Temporally: early interactions get lower gate values than recent ones, across all datasets
The gate is tracking how much collaborative evidence exists yet, not just static item properties

Mean gate value for early (first half) vs. recent (second half) interactions by sequence position, averaged across users.

10

11 of 13

RQ3: Does Visual Utility Tracks Visual Distinctiveness?

Low-gate (vision-reliant) items are visually distinctive with patterns, clear branding
High-gate (collaborative-reliant) items are visually generic, e.g. a plain white bookshelf
Known failure mode: visually distinctive but interaction-sparse items confuse the gate - both signals unreliable due to limited co-occurrence supervision

Pictures of images with highest gate (top) and lowest (bottom) based on their average gating values

11

12 of 13

Main Take-Aways

Fusion granularity matters as much as fusion mechanism, item-level beats uniform fusion

Latent context is measurable. A learned gate is an interpretable proxy for per-item modality contribution, usable anywhere signal informativeness varies across items, users, or interaction states

Modality utility is a latent, dynamic context variable, not a fixed item propert

12

13 of 13

Thank you

Natalija Glisovic (Industrial PhD Student/Data Scientist)

natalija.glisovic@ingka.ikea.com