Human-Swarm Control Using Vision, Speech & Multi-Agent LLMs
An HRI Project Presented by Moses Ebere & Joseph Adeola
Motivation
Can Large Language Models be used
to enhance performance in today’s robotic systems? Specifically in the Human-Swarm Interaction domain.
Motivation
Literature
HRI
Pipeline
HRI
Vision System
Chatbox
AutoGen
Control Stack
Reynold’s Arrival Behaviour with Omnidirectional Points in RViz
Reynold’s Arrival Behaviour with Differential Drive Robots in Gazebo
Results
Gesture Formation
Letter Formation
Results
Text-based Control Command and Data Extraction
Results
Text-based Control Command and Data Extraction
Participants
Greta
French & Russian Lang.
Matilda
Elec. Engineering
Matko
Economics
Adrien
Computer Science
Disclaimer: Images do not depict real people.
Some Participants
Experiment
Images from User Case Studies
Our Goals vs Observation
Preference Test.
Pre-Experiment Goals
Recognition Check: Ability to identify formations.
Comfort & Expectations: User reactions and desired features.
AutoGen Interaction: Effect on understanding agent behavior.
Description
Word Cloud
WordCloud showcasing the most frequently mentioned expectations or functionalities.
Comments From Participants
HRI
Focus on just one component at a time
Matilda
I dont think the Human should be replaced.
Adrien
Focus on just one component at a time
Matko
I prefer the text control method because
it allows you to properly check the
commands you’re sending to the robots. Also, some people with speech impediments can easily make use of it.
The human in the loop should not be removed too.
Greta
I like the fact that we can get information about the system from the language agents. It makes task management easy but also introduces a loophole due to a lack of control of them.
Adrien
Overall, the gesture response is the fastest. The latency in the speech command can also be improved. Overall,
I think focusing on one control method
rather than several methods is more
intuitive.
Matilda
It's an intuitive idea. I prefer the text command mode.
Matko
Thank You