Do We Mean It When �We Say We Want Robots to Engage?

Jesse Thomason

http://glamor.rocks/

Human–Robot Dialogue

“While most robot learning works focus on unidirectional communication from humans to robots, having robots that not only understand what human users say but also initiate and engage in conversations can help them learn and deploy more effectively in human environments.”

• https://human-robot-dialogue.github.io/

One Instruction, One Scene, Multiple Policies

SWAP: Stepwise Action Policy Routing for Vision-Language-Action Models (in submission)

One Instruction, One Scene, Multiple Policies

​

SWAP: Stepwise Action Policy Routing for Vision-Language-Action Models (in submission)

One Instruction, One Scene, Multiple Policies

​

SWAP: Stepwise Action Policy Routing for Vision-Language-Action Models (in submission)

What Does “Engagement” Look Like for People?

Better Slow than Sorry: Introducing Positive Friction for Reliable Dialogue Systems, TACL 2025

Okay, but when should we add friction?

SAFE: Multitask Failure Detection for Vision-Language-Action Models, NeurIPS'25

SAFE: Sequential Conformal Failure Prediction

SAFE: Multitask Failure Detection for Vision-Language-Action Models, NeurIPS'25

Insight from NLP: Test in a neighborhood around inputs

Contrast Sets for Evaluating Language-Guided RoboNLP

Contrast Sets for Evaluating Language-Guided Robot Policies, CoRL 2024

Go right past the couch and stop at the table.

Go left past the couch and stop at the table.

Contrast Sets for Evaluating Language-Guided RoboNLP

Contrast Sets for Evaluating Language-Guided Robot Policies, CoRL 2024

Estimate Policy Metrics with Less Experimenter Labor!

Contrast Sets for Evaluating Language-Guided Robot Policies, CoRL 2024

Transfer insight: Probe in a neighborhood around inputs

SAFECAST: Robust Failure Detection for VLA Policies with Contrast-Set Training and Calibration (under review)

SAFECAST improves risk control with cost on autonomy

SAFECAST: Robust Failure Detection for VLA Policies with Contrast-Set Training and Calibration (under review)

We’ve made progress on when to slow down; now what?

• Given suspected future failure, we could fall back to requesting human teleoperation
• Do we need a full demo?
• Is the high fidelity control of teleoperation even needed, or would a “nudge” do the trick?

Really cool ongoing work from Harshitha Rajaprakash

We’ve made progress on when to slow down; now what?

• What if we could reverse these perturbations?
• Given suspected future failure, predict what needs to change to undo that failure
• Could we ask operators to rearrange the scene? Clarify their language-based intent?

Really cool ongoing work from Harshitha Rajaprakash

Can we give commands and corrections in speech at all?

Should We Type or Talk to LLM Agents? A Comprehensive Study of Voice and Keyboard Input Perturbations (under review)

Modern LLMs are so much worse at speech vs text input

Should We Type or Talk to LLM Agents? A Comprehensive Study of Voice and Keyboard Input Perturbations (under review)

Speech Recognition with Environmental Cues

Multimodal Speech Recognition for Language-Guided Embodied Agents, INTERSPEECH 2023

Speech Recognition with Environmental Cues

Multimodal Speech Recognition for Language-Guided Embodied Agents, INTERSPEECH 2023

Speech Recognition with Environmental Cues

Multimodal Speech Recognition for Language-Guided Embodied Agents, INTERSPEECH 2023

Call to Action: Speech and Interaction-first Robotics

• Typing to robots will always be silly
• Speaking to robots is currently also silly, but it shouldn’t be
• Socially and physically embodied speech recognition is hard
• Human-human dialogue and interaction involves so much more than just the speech signal

Human-Robot Spoken Dialogue Systems

Towards an Extensible Benchmark for Spoken Dialogue with Social Robots; HRD Workshop @ IROS'26

Human-Robot Building Game

Towards an Extensible Benchmark for Spoken Dialogue with Social Robots; HRD Workshop @ IROS'26

Gaze

Gesture

Speech

Motion

Cooperative Building Games

Player A Goal

Player B Goal

Reachable Space

Gaze

Gesture

Speech

Motion

Implicit requests

Planning

Cooperative Building Games

Player A Goal

Player B Goal

Reachable Space

Competition�Ready??

Cooperative Transportation Games

Speech

Intonation

Push

Pull

Shear forces

Rotation

Planning

Stress

Facial expression

Thanks to the GLAMOR ✨ Lab PhD students and alumni

Jesse Zhang

Lee Kezar

Tejas Srinivasan

Leticia Pinto-Alva

Ting-Yun Chang

Abrar Anwar

Ishika Singh

Wang Zhu

Harshitha Rajaprakash

Zizhao Hu

Yuliang Cai

Jie Cai

Wei Yang

Dhananjay Ashok

Gabriela Pinto

GLAMOR robotics MS students applying for Fall’27 PhDs!

Mousumi Das

Aditeya Prajapati

SWAP, SAFECAST, Sign Language, Lip Reading

SWAP, RL, best paper award @ SIGDIAL’26

Do We Mean It When �We Say We Want Robots to Engage?

Jesse Thomason

http://glamor.rocks/