1 of 16

Cocoon: On-body microphone collaboration for spatial awareness

Bhawana Chhaglani^✳, Utku Gunay Acer^✝, Si Young Jang^✝, Fahim Kawsar^✝, Chulhong Min^✝

^✳University of Massachusetts Amherst

^✝Nokia Bell Labs

2 of 16

Growing number of wearable devices

Number of devices per person

[1] https://www.researchgate.net/figure/Estimated-Number-of-Connected-Devices-Per-Person-By-2025_fig4_322050538

3 of 16

On-body Microphone Collaboration

Microphones are Ubiquitous

Range of Applications

Speech Enhancement

Virtual Microphone array

Spatial Awareness

Body-pose estimation

Reliable Sensing

Collaboration

4 of 16

Spatial Awareness: Cocoon

Allows users to enable different applications at different locations using on-body microphone collaboration.

This is done by profiling and detecting a location (Cocoon) using acoustic signals.

5 of 16

Spatial Profiling: Primer

Room response

Impulse Response: Changes with microphone position

Test Room

ESS signal

𝑓₁,𝑓₂ are the initial and final frequency of the sine sweep signal, 𝑇 is the duration of the sweep, and 𝑅 = ln(𝑓₂/𝑓₁)

Inverse filter

Ⓧ

Inverse filter

𝑓(𝑡) = 𝑥_𝑖𝑛𝑣/𝑘

𝑘 = exp (𝑡𝑅/𝑇)

Reflections from objects

6 of 16

Spatial Profiling: Primer

Room response

Impulse Response

Test Room

ESS signal

𝑓₁,𝑓₂ are the initial and final frequency of the sine sweep signal, 𝑇 is the duration of the sweep, and 𝑅 = ln(𝑓₂/𝑓₁)

Inverse filter

Ⓧ

Inverse filter

𝑓(𝑡) = 𝑥_𝑖𝑛𝑣/𝑘

𝑘 = exp (𝑡𝑅/𝑇)

Reflections from objects

We capture spatial profiles simultaneously from on-body microphones to profile/detect a Cocoon.

7 of 16

Why on-body microphones for spatial profiling?

Enhanced Spatial Resolution

Increased Feature Dimensionality

Robustness to slight movements

How to combine on-body microphone signals to profile a Cocoon?

Now we ask the question : why on-body microphones for profiling Cocoon?��Prior work mainly used single microphone present on smartphone for spatial profiling.

We use multiple microphones as they offer several advantages:

Enhanced spatial resolution: as the microphones are significant distance apart, they capture significantly different information. As microphones are sensing from different vantage points, the dominant reflection arriving at different wearables would be different.
Increased feature dimensionality: As microphones are sensing simultaneously, we have more information about the surroundings being sensed simultaneously. Less training effort.
Robustness to slight movements: We hypothesize that multiple microphones based system will be more robust to slight movements than a single microphone based system.

Now the question is how do we combine these on-body microphones signals to detect cocoon

8 of 16

Challenges in On-body Microphone Collaboration

1.Synchronization

3. Efficient and Robust Collaboration

2. User movement

Need to synchronize audio streams from multiple wearable devices for collaboration.

Changes in relative position of wearables; Varying spatial profiles

Missing/Adding wearable should be seamless; Minimum collaboration cost

Challenges

Synchronization: However, many small-form factor wearable platforms such as commercial earbuds and smart wristbands do not support time-synchronization capability for audio streaming. Also, even with powerful devices supporting time-synchronization, such as smartphones and smartwatches, it is not easy to synchronize the audio streaming from different devices as the audio streams do not contain the timestamp information.
User movement: When the user moves, wearables move. The relative position of wearables change. This can lead change in spatial profiles from that one that was recorded during training. This might lead to retraining.
Efficient & Robust collaboration: Collaboration technique should be smart enough to take care of adding/missing wearable scenario. (Number of devices that are connected keep changing through the day). And the collaboration cost should be minimum.

9 of 16

Proposed System: Cocoon

A case-study for on-body microphone collaboration for spatial awareness using earables.

10 of 16

System Overview

Collaboration Type:

Feature Concatenation

Vs

Ensemble Learning

Feature Ranking

Peak Amplitude
Peak time differences
Peak times

✔

11 of 16

Evaluation

Experimental Setup

2 different setups

Controlled setup
Real-world setup

Recorded data in

2 conference rooms
1 bedroom

20 positions; 10 times per location.

1

5

4

3

2

12 of 16

Results: Accuracy

10.0% points and 21.5% points accuracy increase in the controlled and real-world setup over single microphone.

Methods	Controlled Setup	Real-World Setup
Cocoon	100%	98%
L (Left)	90%	80%
R (Right)	90%	73%

13 of 16

Results: Robustness

The proposed system increases robustness by 17.5% over single microphone-based system

14 of 16

Limitations & Future Work

Eliminate the requirement of speaker in the environment by utilizing ambient sound sources.

Extend the system to more wearable devices like smartwatch.

Enhancing robustness to user and environmental factors by opportunistically sensing spatial profiles and incremental learning.

15 of 16

Conclusions

Proposed Cocoon, a collaborative acoustic sensing mechanism using on-body wearables equipped with microphones.

Cocoon outperforms the existing single microphone-based methods, in terms of accuracy and robustness.

15

Contact: bchhaglani@cs.umass.edu

16 of 16

Thank You!