Active Acoustic Sensing for Robot Manipulation
Shihan Lu and Heather Culbertson
Department of Computer Science
University of Southern California
[1] Yoshioka, K., et al., Through the Looking Glass: Diminishing Occlusions in Robot Vision Systems with Mirror Reflections. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1578-1584, 2021.
[2] Taylor, Ian H., et al., "GelSlim 3.0: High-resolution measurement of shape, force and slip in a compact tactile-sensing finger." In IEEE International Conference on Robotics and Automation, pp. 10781-10787, 2022.
[3] Du, Maximilian, et al., "Play it by ear: Learning skills amidst occlusion through audio-visual imitation learning." RSS: Robotics Science and Systems, 2022.
State-Aware Manipulation
Visual
Tactile
Acoustic
Global - occlusion, blur, lighting
Local - limited to surface features in static contacts
Vulnerable to ambient noises
Microphone
Can we infer the global object states from a localized contact?
From a local contact between fork and cup, each tapping generates unique resonance
State-Aware Manipulation
Acoustic
Active
Sensing
State-Aware Manipulation
Acoustic
for Robot Manipulation
Active
Sensing
Sensor setup on Franka Hand and its explosive view in CAD
Sensor Design
Different excitation signals, such as impulse, linear sweep, and exponential sweep, can be applied as input.
Sensing Principle
Sensing Characteristics
Sensor Simulation
(a) Viscous contact damping for updating modal damping
(b) Two-DoF mass-spring-damper system for contact collision computation
(c) Robot simulation based on PyBullet for extracting contact dynamic datastream
Experimental Evaluation
Real
Simulation
Experimental Evaluation
Real
Sim
Real
Sim
Object Recognition
Single-object scene
Multi-object scene
Single Single
Single Multiple
Grasping Position Estimation
13 uniformly-spaced grasping positions along half of main axis
Contributions – Active Acoustic Sensing for Robot Manipulation
Sensing Principle
Sensor Design
Sensor Simulation
Sensor Evaluation