Nons, Not Modifiers

OpenVLA Parses Objects but Fails at Spatial Composition

90%

Absolute

instructions

48–53%

Relational

instructions (chance)

15% vs 80%

Left vs Right

accuracy

The Compositional Gap

Can VLAs combine object nouns with spatial modifiers?

From naming to composing

ABSOLUTE

"pick up the coke can"

Dominant in training data. Object → action.

RELATIONAL

"pick up the can on the left"

Requires composing noun + modifier + scene.

The unresolved question

Language grounding in OpenVLA is treated as an emergent property of data scale. But training is dominated by absolute instructions — no direct supervision for relational language.

Does compositional spatial reasoning emerge from demonstration-based learning alone?

A matched-pair evaluation of OpenVLA-7B in SimplerEnv

Matched-Pair Experimental Design

Same scenes, six instruction variants — isolating the linguistic variable

20

custom scenes

5 layouts × 2 triplets × 2 L/R swaps

6

instruction types

2 absolute · 2 distance · 2 spatial

120

matched trials

With 120 trials

Six instructions, one scene

Setup

MODEL

OpenVLA-7B, pretrained

ENVIRONMENT

SimplerEnv / ManiSkill2

Google Robot · 40 steps

METRIC

EE closer to correct can (50% chance baseline)

Each scene resets identically across the 6 prompts → any behavior delta is attributable to instruction text.

Relational Instructions Collapse to Chance

Absolute accuracy is near-ceiling. Distance and spatial performance are statistically indistinguishable from a coin flip.

50% chance baseline

Absolute vs. distance

OR = 8.14

Fisher's exact, Bonferroni-corrected p = 0.001 (vs. distance); p < 0.001 (vs. spatial)

Distance vs. spatial

p = 1.0

Both relational families fail in the same way — a shared compositional bottleneck, not two separate issues.

Wilson 95% CIs. Bonferroni correction across n = 3 comparisons.

The Rightward Default

Spatial failures are not random — the model defaults to "right" almost unconditionally.

"on the LEFT"

15%

3 of 20 correct

95% CI [5.2, 36.0]

vs.

"on the RIGHT"

80%

16 of 20 correct

95% CI [58.4, 91.9]

Fisher's exact, Bonferroni-corrected p < 0.001

Failure mode decomposition

93%

of all failures → the end-effector reached for the wrong can, not the distractor.

All 21 spatial failures: reached the wrong can. The model parses "can" correctly — it just cannot apply the relational modifier.

Distance instructions fail symmetrically (60% vs. 45%, p = 1.0) → a different mechanism: undirected chance, not a positional default.

What This Means for VLA Deployment

Shallow language grounding — and a safety-relevant failure that is silent by default.

FINDING

Compositional gap is real

OpenVLA maps object names to motor actions accurately, but cannot compose those names with spatial modifiers. Grounding does not emerge from data scale alone.

RISK

Silent high-confidence failure

The robot confidently reaches for the wrong can — no uncertainty signal. A left/right positional prior inherited from the VLM persists into action output.

NEXT

Where to push

Fine-tune on relational demonstrations; test directional bias across VLA architectures; evaluate whether 3D spatial representations close the compositional bottleneck.

Recommendation: verify VLA instruction-following on relational prompts before real-world deployment.