Linguistic Insights �Deepen our Understanding of AI Systems��The Cases of Reference Frames and Logical Reasoning
Freda Shi
University of Waterloo, Vector Institute, Canada CIFAR AI Chair
fhs@uwaterloo.ca
Interaction between Linguistics and CS
2
1950s-1960s
Russian (Romanized) | English Translation |
Mi pyeryedayem mislyi posryedstvom ryechyi. | We transmit thoughts by means of speech. |
The Georgetown--IBM experiment
1990s
Recurrent Neural Networks
2020s
Are linguistic insights still helpful?
Noam Chomsky
Jeffery Elman
Language Models for Linguistics
3
This Talk: Linguistics for Language Models
What tasks are hard for current large language and vision-language models?
The analyses also offer insights on what we should do to improve the models.
4
[Levinson. 2003. Space in language and cognition: Explorations in cognitive diversity. Vol. 5. Cambridge University Press]
[Kripke. 1959. A completeness theorem in modal logic. The journal of symbolic logic, 24(1), 1-14]
[Kripke. 1963. Semantical considerations on modal logic. Acta Philosophica Fennica, 16]
This Talk: Linguistics for Language Models
What tasks are hard for current large language and vision-language models?
The analyses also offer insights on what we should do to improve the models.
5
[Levinson. 2003. Space in language and cognition: Explorations in cognitive diversity. Vol. 5. Cambridge University Press]
[Kripke. 1959. A completeness theorem in modal logic. The journal of symbolic logic, 24(1), 1-14]
[Kripke. 1963. Semantical considerations on modal logic. Acta philosophica fennica, 16]
Common Vision-Language Models
Image-Text Mapping, e.g., CLIP (Radford et al., 2021)
The similarity metrics (e.g., probability after softmax) quantify the endorsement level.
6
[Kiros et al. 2014. Unifying visual-semantic embeddings with multimodal neural language models]
[Radford et al. 2021. Learning transferable visual models from natural language supervision; Figure credit: Radford et al.]
Common Vision-Language Models
Conditioned Generative Models, e.g., LLaVA (Liu et al., 2023)
Predict natural language response conditioned on visual and textual context.
The probability of generated text quantifies the endorsement level.
7
[Vinyals et al. 2015. Show and tell: A neural image caption generator. In CVPR;
Liu et al. 2023. Visual instruction tuning. In NeurIPS; Figure credit: Liu et al.]
Spatial Reasoning in VLMs
Liu et al. (2023): state-of-the-art VLMs fail in spatial reasoning.
Source of complexity: spatial relations are subjective and not always clearly defined.
8
The pizza is at the edge of the dining table.
The pizza is behind the dining table.
The cat is in front of the potted plant.
The cat is to the left of the potted plant.
[Liu et al. 2023. Visual Spatial Reasoning. In TACL; Figure credit: MSCOCO, Lin et al. (2014)]
Why Endorsement?
An alternative way to evaluate generative models is matching-based accuracy.
However, there are two major issues:
With ground-truth answer No, case 1 is better than case 2,� although both are considered correct.
Most generative models are built on top of probability.
It is natural to use probability-based metrics to quantify models’ competence.
9
[Dentella et al. 2023. Systematic testing of three language models reveals low language accuracy, absence of response stability, and a yes-response bias. In PNAS]
Frame of Reference (FoR): An Example
Frame of reference theories offer a taxonomy of part of the spatial relations.
For each of the following captions, do you think it’s a good description of the image?
Do VLMs also endorse these descriptions?
10
The tree is behind the car.
(reflection-relative frame)
The tree is in front of the car.
(translation-relative frame)
The tree is to the right of the car.
(ground-intrinsic frame)
FoR Knowledge in VLMs
A synthetic dataset that enables analyzing FoR knowledge in VLMs.
11
Ziqiao Ma
Zheyuan Zhang
[Zhang et al. 2025. Do vision-language models represent space and how? Evaluating spatial frame of reference under ambiguities. In ICLR]
(Camera) Q: From the camera’s viewpoint, is the basketball to the right of the car? A: Yes.
(Addressee) Q: From the woman’s viewpoint, is the basketball to the right of the car? A: Yes.
(Ground Object) Q: From the car’s viewpoint, is the basketball to the right of the car? A: Yes.
Evaluation Metric
Soft accuracy (normalized ground-truth probability)
A few other metrics (for cross-validation purposes) can be found in the paper.
12
(Camera) Q: From the camera’s viewpoint, is the basketball to the right of the car? A: Yes.
(Addressee) Q: From the woman’s viewpoint, is the basketball to the right of the car? A: Yes.
(Ground Object) Q: From the car’s viewpoint, is the basketball to the right of the car? A: Yes.
Soft Accuracy (↑): Results
13
Only some of the models give nontrivial accuracies from the camera’s view.
Random guess baseline
From the Camera’s View: A Closer Look
14
in front of
to the right of
From the Camera’s View: A Closer Look
Plot: , where serves as an intuitive reference curve.
Is the red ball [relation] the blue ball?
15
XComposer
to the left of to the right of in front of behind
Instruct-BLIP-13B
in front of
[Hayward and Tarr. 1995. Spatial language and spatial representations. In Cognition]
From the Camera’s View: A Closer Look
XComposer (behind)
16
What happened here?
Similar phenomena also observed in strong models like GPT-4o.
The tree is behind the car.
(reflection-relative frame)
The tree is in front of the car.
(translation-relative frame)
From the Camera’s View: A Closer Look
17
XComposer� (behind)
(reflection-relative frame)
(translation-relative frame)
Least liked by humans.
The drop of “in front of” is much milder than that of “behind.”
[Bender et al. 2020. Being in front is good---but where is in front? Preferences for spatial referencing affect evaluation. In CogSci]
Cross-Lingual FoR Adoption
Different language users have clearly shown different preferences in FoR adoption.
Bender et al. (2020): Chinese, German, and Japanese native speakers are more likely to adopt the translation-relative FoR than English native speakers.
Hill (1982): Hausa native speakers adopt the translation-relative �frame more than the reflection-relative one.
Clark (1973): Tamil native speakers generally adopt the rotation-�relative frame more than English native speakers.
Are we able to observe similar preference in multilingual VLMs?
18
[Bender et al. 2020. Being in front is good---but where is in front? Preferences for spatial referencing affect evaluation. In CogSci]
[Hill. 1982. Up/down, front/back, left/right. A contrastive study of Hausa and English. In Here and there: Cross-linguistic studies on deixis and demonstration]
[Clark. 1973. Space, time, semantics, and the child. In Cognitive development and the acquisition of language]
Cross-Lingual FoR Adoption in VLMs?
GPT-4o on 109 languages (darker = more favoring the reflection-relative frame).
No preference towards the intrinsic frame observed!
Conjecture: this may be because of the dominance of English in training data.
19
A Complementary View
Data scarcity: VLMs don’t see sufficient training examples for many spatial relations.
Top 17% of VSR spatial relations account for >90% MSCOCO training examples.
Proposed solution: use LLMs to synthesize spatial QA pairs from detailed captions and use the synthesized image QA data to finetune VLMs.
20
[Ogezi and Shi. 2025. SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data. Under Review]
Mike Ogezi
A front view of a small gray elephant figurine on the left; in the middle, there is an orange and black tiger; and on the right, there is a papier-mâché rhino head. They are all positioned side by side, with space between them…
What is to the right of the orange tiger? | The rhino head. |
Which animal figurine is located on the leftmost side? | The elephant. |
What animal is in the middle of the arrangement? | The tiger. |
What is in the background? | Wallpaper. |
Finetuning Results
Finetuning improves the spatial reasoning capability over the base models (Qwen2-VL; Wang et al., 2024), with comparable performance on other tasks.
There is still a large gap between human and VLM performance.
21
[Wang et al. 2024. Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution]
This Talk: Linguistics for Language Models
What tasks are hard for current large language and vision-language models?
The analyses also offer insights on what we should do to improve the models.
22
[Levinson. 2003. Space in language and cognition: Explorations in cognitive diversity]
[Kripke. 1959. A completeness theorem in modal logic. The journal of symbolic logic]
[Kripke. 1963. Semantical considerations on modal logic. Acta philosophica fennica]
LLMs perform better on high-frequency tasks
23
[Gonen et al. 2023. Demystifying Prompts in Language Models via Perplexity Estimation. In Findings of EMNLP.]
[McCoy et al. 2024. Embers of autoregression show how large language models are shaped by the problem they �are trained to solve. In PNAS]
[Source: Gonen et al.]
[Source: McCoy et al.]
A Complementary View
Besides random noises, is probability/perplexity everything about accuracy?
Quick answer: No. Logical forms complement it.
Propositional and modal logic gives us a framework to synthesize material.
24
[Wang and Shi. 2025. Logical forms complement probability in understanding language model (and human) performance. Under Review]
Yixuan Wang
Hypothetical and Disjunctive Syllogisms
We consider both logic sequents and fallacies.
25
Adding Modalities
All formulas still hold if we add the same modality quantifier (must) and (may) to each propositional variable.
26
Adding Modalities
All formulas still hold if we add the same modality quantifier (must) and (may) to each propositional variable.
27
It is certain that… (must)
It is possible that … (may)
Interpreting Logic Formulas
Real-world scenarios to create the material for main experiment.
We manually selected possible interpretations to minimize content effect.
28
If Freda is giving a talk, then Freda is reading a book.
Freda is giving a talk.
Is Freda reading a book?
Fitting a Mixed-Effects Model
We obtain the LLM soft accuracy and fit a mixed-effects model.
Fixed effects: analogous to the coefficients in linear regression.
Random effects: individual-specific effects.
29
Results: Fixed Effects of Modality and Argument Forms
LLMs are more likely to respond Yes under the possibility modality, and No under the necessity modality.
Results on argument forms resonate with Wason (1968)---modus tollens is harder for humans; related results on LLMs also reported by Huang and Wurgaft (2023).
30
[Wason. 1968. Reasoning about a rule. In Quarterly journal of experimental psychology]
[McKenzie et al. 2023. Inverse Scaling: When bigger isn’t better. In TMLR; Credit to Huang and Wurgaft: Task Modus Tollens]
Results: Mixed Effects of Perplexity
31
99.9% confidence intervals: the mixed effects of perplexity for all LLMs are negative, which is in line with the results by Gonen et al. (2023) and McCoy et al. (2024).
Results: Constant Random Effects per LLM
32
Model soft-accuracy ranking (numbers in parentheses; ↓) shows a similar trend as the constant random effects (↑).
Accuracy Distributions
33
Left and middle: similar perplexity, but clearly different performance better explained by modality and argument forms.
Right: significantly different perplexity, but similar performance.
This Talk: Linguistics for Language Models
What tasks are hard for current large language and vision-language models?
The analyses also offer insights on what we should do to improve the models.�What’s next?
34
[Levinson. 2003. Space in language and cognition: Explorations in cognitive diversity. Vol. 5. Cambridge University Press]
[Kripke. 1959. A completeness theorem in modal logic. The journal of symbolic logic, 24(1), 1-14]
[Kripke. 1963. Semantical considerations on modal logic. Acta philosophica fennica, 16]
Thoughts on Synthetic Data
All work in this talk is based on synthetic data.
Synthetic data doesn’t represent the entire world.
Synthetic data provides a clean testbed for analysis purposes.
35
[Shumailov et al. 2023. The curse of recursion: Training on generated data makes models forget]
Thoughts on Generative AI
Generative AI has exciting potentials.
But what is our goal of building generative AI?
Generative AI as (1) powerful human assistants, or (2) human cognitive models?
36
Thanks!
The Computation, Language, Intelligence, and Grounding (CompLING) �Laboratory at the University of Waterloo: compling-wat.com
37