1 of 28

CARS @ RecSys, September 2026

Seeing the Context

Acquiring Situational Context from Review Images

Tal Cordova · Tomer Geva · Moshe Unger

Coller School of Management · Tel Aviv University

2 of 28

How Context Has Been Captured, Until Now

Context has traditionally been represented as structured features derived from time, location, user-provided details (Adomavicius et al., 2022), or text reviews (Bauman & Tuzhilin, 2022).

Images have been used to represent items (Hu et al., 2023; Liu et al., 2024), but not to acquire context.

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

The Gap

To our knowledge, no prior work has acquired context from images.

3 of 28

Context Categories

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

Physical - captures observable circumstances of the experience, including the physical setting - e.g., ambiance, time of day, outdoor / indoor.

Social - captures the presence of other people and the apparent social setting of the experience - e.g., group size, occasion.

We acquire context based on CARS taxonomy that defines three categories (Adomavicius et al., 2011):

Modal - captures the mood and tone of the experience as visible in the image - e.g., formality, relaxed vs. rushed.

4 of 28

Example Image

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

5 of 28

Example Image

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

What is in the image?

A pedestrian street storefront, an outdoor table with one seated diner, a small crowd near the entrance

6 of 28

Example Image

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

What is in the image?

A pedestrian street storefront, an outdoor table with one seated diner, a small crowd near the entrance

What is the context?

Physical - Outdoor seating on a pedestrian street, daytime, overcast weather
Social - One person dining alone at a small outdoor table. A separate group of adults gathered near the entrance, standing rather than seated.
Modal - Relaxed mood, casual setting.

7 of 28

Research Questions

RQ1 - How does context acquired from images affect recommendation performance compared to established contextual signals?

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

Seeing the Context

RQ2 - Does image-based context improve recommendation performance when combined with other contextual and content signals?

8 of 28

To address the RQs - we built a pipeline for a comparative analysis of context types and their effect on rating prediction performance.

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

9 of 28

From Photo to Context Vector

Gemma 3-12B - IT

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

Social Context

Physical Context

Modal Context

Context Categories

10 of 28

From Photo to Context Vector

Gemma 3-12B - IT (Gemma Team, 2025)

Outdoor seating on a pedestrian street, daytime, overcast lighting

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

One person dining alone

Relaxed dining

Social Context

Physical Context

Modal Context

Interpretation Layer

Context Categories

11 of 28

From Photo to Context Vector

Gemma 3-12B - IT

Outdoor seating on a pedestrian street, daytime, overcast lighting

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

One person dining alone

Relaxed dining

Social Context

Physical Context

Modal Context

0.69, 0.21, 0.63 …

0.19, 0.02, 0.86 …

0.1, 0.92, 0.55 …

Embedding Layer

E5 Small (Wang et, al. 2024)

Interpretation Layer

Context Categories

12 of 28

From Photo to Context Vector

Gemma 3-12B - IT

Outdoor seating on a pedestrian street, daytime, overcast lighting

Fused Context Representation

0.20

0.70

0.42

0.22

0.14

0.09

0.21

…

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

One person dining alone

Relaxed dining

Gating

Social Context

Physical Context

Modal Context

0.69, 0.21, 0.63 …

0.19, 0.02, 0.86 …

0.1, 0.92, 0.55 …

Interpretation layer

Context Categories

Embedding Layer

E5 Small (Wang et, al. 2024)

13 of 28

Context and Content Representation

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

CITY-DATE

Timestamp + location

One-hot Vector

IMAGE EMBEDDING

Image

Embedding

Radford et al., 2021

Embedding Model (CLIP)

TEXT-BASED

CONTEXT

Review Text

Discrete Context Features

Bauman & Tuzhilin, 2022

Context Signals

Content Signal

14 of 28

Recommendation Algorithm

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

RGCL (Shuai et al., 2022)

Used as the recommendation algorithm

Context integrated into the edge representations

Fused context shapes user and item embeddings during training

Inference requires no context

Prediction uses only the learned representations

15 of 28

Our Evaluation Pipeline

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

16 of 28

Evaluation on TripAdvisor Reviews

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

40K

Reviews

1,441 Users, 1,341 Items

400K

Reviews with images

Publicly available TripAdvisor dataset

zenodo.org/records/5644892

7-core filtering

(≥7 image reviews per user/item)

Years 2010-2019 | Split: train < 2017, validation 10% of train, test 2017-2019 | Metric: RMSE

17 of 28

Rating Prediction of Context Signals

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

CONTEXT TYPE	DESCRIPTION	RMSE
No context	CF Signal, no context	0.954
City-date	Explicit context from city and timestamp	0.903
Text-based context	Context acquired from reviews (Bauman & Tuzhilin, 2022)	0.899
Image-based context	Context acquired from images (our proposed method)	0.910

18 of 28

The Additive Value of Visual Context

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

Significance of the improvement relative to the best alternative

Test used is two-sided Wilcoxon test

* P-value < 0.05, ** P-value < 0.01, *** P-value < 0.001

Signal	DESCRIPTION	RMSE
City-date	Explicit context from city and timestamp	0.903
City-date + Image-Based Context	City-date integrated with visual context	0.902***

Text-based context	Context acquired from reviews (Bauman & Tuzhilin, 2022)	0.899
Tex- based context + Image-Based Context	Text based context integrated with visual context	0.896

Image-Embedding	Direct image embedding using CLIP (Radford et al., 2021)	0.908
Image-embedding + Image-Based Context	Image embeddings integrated with visual context	0.903***

19 of 28

On its own, image context doesn't outperform the existing signals - but combined with them, it improves rating prediction.

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

Seeing the Context

20 of 28

Context Category Ablation Analysis

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

Category Subset	Physical Weight	Social Weight	Modal Weight	RMSE
Physical, Social, Modal	0.3286	0.3034	0.368	0.9102***
Physical, Modal	0.392	-	0.608	0.9106
Physical, Social	0.3926	0.6074	-	0.9118
Social, Modal	-	0.3956	0.6043	0.9125

Significance of the improvement relative to the best alternative

Test used is two-sided Wilcoxon test

* P-value < 0.05, ** P-value < 0.01, *** P-value < 0.001

The gate assigns non-zero weight to every category - none is ignored
In pairwise combinations, modal context receives the most weight, physical the least

21 of 28

Contributions

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

Contextual Signal

We propose a new contextual signal from images for recommendation systems.

Comprehensive Evaluation

We created a comprehensive evaluation of image-based context performance, compared to and in addition to other context and content signals.

22 of 28

Future Work

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

Domain Expansion

Extend the use of image context to other datasets and domains.

Algorithm Evaluation

Evaluate other recommendation algorithms using the image-based context.

Context Acquisition

Test different approaches to improve visual context acquisition.

23 of 28

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

Prof. Tomer Geva

Dr. Moshe Unger

24 of 28

SEEING THE CONTEXT · CARS @ RecSys 2026

Thank you.

Questions welcome

talcordova@mail.tau.ac.il

Connect on LinkedIn

25 of 28

References

Adomavicius, G., Bauman, K., Tuzhilin, A., & Unger, M. (2022). Context-Aware Recommender Systems: From Foundations to Recent Developments. In Recommender Systems Handbook (3rd ed.). Springer US.

Adomavicius, G., Mobasher, B., Ricci, F., & Tuzhilin, A. (2011). Context-Aware Recommender Systems.

Bauman, K., & Tuzhilin, A. (2022). Know Thy Context: Parsing Contextual Information from User Reviews for Recommendation Purposes. Information Systems Research, 33(1), 179–202. https://doi.org/10.1287/isre.2021.1036

Hu, X., Yu, W., Wu, Y., & Chen, Y. (2023). Multi-modal recommendation algorithm fusing visual and textual features. PLOS ONE, 18(6), e0287927. https://doi.org/10.1371/journal.pone.0287927

Gemma Team. (2025). Gemma 3 Technical Report. arXiv preprint arXiv:2503.19786.

Liu, Q., Hu, J., Xiao, Y., Zhao, X., Gao, J., Wang, W., Li, Q., & Tang, J. (2024). Multimodal Recommender Systems: A Survey (No. arXiv:2302.03883). arXiv. https://doi.org/10.48550/arXiv.2302.03883

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision (arXiv:2103.00020). arXiv. https://doi.org/10.48550/arXiv.2103.00020

Shuai, J., Zhang, K., Wu, L., Sun, P., Hong, R., Wang, M., & Li, Y. (2022). A Review-aware Graph Contrastive Learning Framework for Recommendation. Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 1283–1293. https://doi.org/10.1145/3477495.3531927

Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., & Wei, F. (2024). Text Embeddings by Weakly-Supervised Contrastive Pre-training (arXiv:2212.03533). arXiv. https://doi.org/10.48550/arXiv.2212.03533

26 of 28

Appendix & backup

APPENDIX

27 of 28

Context Acquisition from Image

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION

Models Used:

For context acquisition from images - Gemma3 12B-IT (Gemma Team, 2025)

For Embedding - E5 small (Wang et al., 2024)

28 of 28

Semantic Complementarity of Image and Review Context

BACKGROUND

METHOD

EVALUATION

RESULTS

CONTRIBUTION