1 of 56

�CS60055: Ubiquitous Computing

Contactless Sensing

Part III: mmWave Sensing

Thanks to Argha Sen and Debjit Chatterjee for helping with the slides!

INDIAN INSTITUTE OF TECHNOLOGY

KHARAGPUR

Sandip Chakraborty

sandipc@cse.iitkgp.ac.in

Department of Computer Science and Engineering

2 of 56

5G New Radio (NR)

Indian Institute of Technology Kharagpur

3 of 56

Ubiquitous Computing over 5G

Communication

Sensing

Computation

Indian Institute of Technology Kharagpur

4 of 56

Background: How Sensing Works

Signals gets reflected from human subjects and other objects in the periphery: Explore the signal reflection properties

Indian Institute of Technology Kharagpur

5 of 56

Human Activity Recognition

Source

Sensor

Feature

Inference

Gesture

Vital Signs

Presence & movement

Location

Signal Processing: AoA, Point-cloud, RSSI/SINR
Deep Learning based feature extraction n/w

Speech recognition
Indoor positioning
Breathing monitoring
Facial expression recognition
Gesture recognition
Localization/tracking

Indian Institute of Technology Kharagpur

6 of 56

Various Sensing Modalities

Signal	Frequency	Key properties signifying applicability
Acoustics	20 Hz - 20 kHz(audible) >20 kHz	Affected by acoustic interference in environment
UWB	3.1 GHz to 10.6 GHz	Interference with sub 6 GHz Communicating Devices
Wifi	2.4 GHz/ 5 GHz	Low Range resolution, less accuracy with CSI
mmWave	30–300 GHz	Various commercial devices. Prominent technology of the near future
Bluetooth	2.4GHz	Low Resolution
LoRA	169 MHz, 433 MHz, 868 MHz and 915 MHz	Long range sensing but not apt for indoor

Indian Institute of Technology Kharagpur

7 of 56

Various Sensing Modalities

Signal	Frequency	Key properties signifying applicability
Acoustics	20 Hz - 20 kHz(audible) >20 kHz	Affected by acoustic interference in environment
UWB	3.1 GHz to 10.6 GHz	Interference with sub 6 GHz Communicating Devices
WiFi	2.4 GHz/ 5 GHz	Low Range resolution, less accuracy with CSI
mmWave	30–300 GHz	Various commercial devices. Prominent technology of the near future
Bluetooth	2.4GHz	Low Resolution
LoRA	169 MHz, 433 MHz, 868 MHz and 915 MHz	Long range sensing but not apt for indoor

Indian Institute of Technology Kharagpur

8 of 56

A Vast Literature on RF Sensing

Wifi

mmWave

UWB

Acoustics

Presence Detection

Heartbeat monitoring

Temperature Monitoring

Breathing monitoring

Heart rate monitoring

Eyelid movement

Hand gesture recognition

Counting crowd

Breathing monitoring

Heart rate monitoring

Hand gesture recognition

Fire detection

localization/tracking

Fall detection

Smoking detection

Mouth movement

Keystroke Recognition

Traffic Monitoring

Driver Fatigue Detection

LOS detection

Metal detection

Finger Gesture Recognition

Multi-User Gesture Recognition

Human Identification

Gesture recognition

Speech recognition

Respiration Monitoring

Eyelid movement

Gait Abnormality Detection

Speech recognition

Indoor positioning

Multi-user breathing monitoring

Facial expression recognition

Gesture recognition

Localization/tracking

Fall detection

Gait identification

Emotion detection

Heart beat monitoring

Parking space monitoring

Presence detection

Parking space monitoring

Gas monitoring

Indian Institute of Technology Kharagpur

9 of 56

mmWave Sensing – A Vast Literature

Indian Institute of Technology Kharagpur

10 of 56

mmWave Sensing

mmWave sensing follows the radar principles

Uses various modulation schemes widely used in commercial radars

Phase modulation: Use the phase difference to capture the AoA
Continuous Wave modulation: Uses a constant frequency carrier; cannot measure range as the carrier signal is unmodulated
Frequency Modulated Continuous Wave (FMCW)

Indian Institute of Technology Kharagpur

11 of 56

FMCW Radars

The basic objectives of a radar are the followings:

Indian Institute of Technology Kharagpur

12 of 56

FMCW Radars

FMCW radar transmits chirps
A chirp is characterized by a fc, B and T_c.
The Slope (S) defines the rate at which the chirp ramps up.

Indian Institute of Technology Kharagpur

13 of 56

FMCW Radars

FMCW radar transmits chirps
A chirp is characterized by a fc, B and T_c.
The Slope (S) defines the rate at which the chirp ramps up.

Indian Institute of Technology Kharagpur

14 of 56

FMCW Radar: The IF Signal

The Rx is the replica of the Tx shifted in time t_f= 2d/c
Tx and Rx has the same linear modulation, the difference is a constant tone

Equals to the round trip delay multiplied by the slope of the signal
Slope S = B/T_c, B is the bandwidth
The resultant frequency is called the intermediate frequency (IF)

Indian Institute of Technology Kharagpur

15 of 56

FMCW Radar: The IF Signal

The frequency difference of the TX-chirp and RX-chirp.
Object in front of the radar produces an IF signal with a constant frequency tone.
The frequency of this tone is f_IF = S.τ = S.2d/c
τ = 2d/c is the round-trip time

Indian Institute of Technology Kharagpur

16 of 56

FMCW Radar: Range Resolution

Range Resolution: Ability to resolve two closely spaced objects.
When the two objects are too close that they show up as a single peak in the frequency spectrum.

The two objects can be resolved by increasing the length of the IF signal.

Indian Institute of Technology Kharagpur

17 of 56

FMCW Radar: Range Resolution

Range Resolution: Ability to resolve two closely spaced objects.
When the two objects are too close that they show up as a single peak in the frequency spectrum.
This also proportionally increases the bandwidth (as we need to increase the chirp duration)!
Thus increasing the bandwidth provides better resolution

Indian Institute of Technology Kharagpur

18 of 56

FMCW Radar: Range Resolution

Range Resolution: Ability to resolve two closely spaced objects.
When the two objects are too close that they show up as a single peak in the frequency spectrum.
Increasing the length of the IF signal?
This also proportionally increases the bandwidth!
Thus intuitively: Greater the Bandwidth => better the resolution.

To resolve two different tones in the frequency spectrum, they must be spaced more than 1/T, where T is the window duration. In this case, T equals to T_c.

Indian Institute of Technology Kharagpur

19 of 56

An Important Observation from FFT

To resolve two different tones in the frequency spectrum, they must be spaced more than 1/T, where T is the window duration – Why?

We often perform the Fourier transform on finite window (duration T), as we cannot analyze an infinitely long signal

Frequency Resolution: The smallest frequency difference that can be distinguished in the Fourier-transformed signal

Indian Institute of Technology Kharagpur

20 of 56

An Important Observation from FFT

Frequency Resolution: The smallest frequency difference that can be distinguished in the Fourier-transformed signal

The sine waves or tones in the signal are transformed into peaks in the frequency domain.
Each tone, after the Fourier transform, results in a spread of energy across several frequency bins due to the finite window.
For two tones to be resolved, the distance between their frequencies must be larger than the spread of each tone, which is approximately 1/T

Indian Institute of Technology Kharagpur

21 of 56

FMCW Radar: Range Resolution

Range Resolution: Ability to resolve two closely spaced objects.
When the two objects are too close that they show up as a single peak in the frequency spectrum.
Increasing the length of the IF signal?
This also proportionally increases the bandwidth!
Thus intuitively: Greater the Bandwidth => better the resolution.

The minimum range resolution can be computed as:

To resolve two different tones in the frequency spectrum, they must be spaced more than 1/T, where T is the window duration. In this case, T equals to T_c.

Indian Institute of Technology Kharagpur

22 of 56

Equidistant Objects

Two objects are equidistant from the radar. How will the range-FFT look like?

Indian Institute of Technology Kharagpur

23 of 56

Equidistant Objects

Two objects are equidistant from the radar. How will the range-FFT look like?

Indian Institute of Technology Kharagpur

24 of 56

Equidistant Objects

Two objects are equidistant from the radar. How will the range-FFT look like?

How do we separate these two objects?

Indian Institute of Technology Kharagpur

25 of 56

Equidistant Objects

Two objects are equidistant from the radar. How will the range-FFT look like?

How do we separate these two objects?

The phase information helps!

Indian Institute of Technology Kharagpur

26 of 56

FMCW Radar: Phase of the IF Signal

Fourier Transforms: A sinusoid in the time domain produces a peak in the frequency domain.
The signal in the Frequency domain is complex (i.e., each value is a phasor with an amplitude and a phase)

Indian Institute of Technology Kharagpur

27 of 56

FMCW Radar: Phase of the IF Signal

Fourier Transforms: A sinusoid in the time domain produces a peak in the frequency domain.
The signal in the Frequency domain is complex (i.e., each value is a phasor with an amplitude and a phase)

Indian Institute of Technology Kharagpur

28 of 56

FMCW Radar: Phase of the IF Signal

Indian Institute of Technology Kharagpur

29 of 56

FMCW Radar: Phase of the IF Signal

Phase difference between (A and D) or (C and F) is

Indian Institute of Technology Kharagpur

30 of 56

FMCW Radar: Phase of the IF Signal

Phase difference between (A and D) or (C and F) is

Indian Institute of Technology Kharagpur

31 of 56

FMCW Radar: AoA Estimation

TX antenna transmits a frame of chirps
The 2D-FFT corresponding to each RX antenna will have peaks in the same location but with differing phase.
The measured phase difference (ω) can be used to estimate the angle of arrival of the object.

Indian Institute of Technology Kharagpur

32 of 56

Visualizing and Analyzing mmWave Radar Data

Rangle-Doppler heatmap:

Represents the detected objects' distances (range) and relative velocities (Doppler shift); each point corresponds to a unique combination of range and Doppler values
intensity of each point indicates the strength of the reflected signal (probability that an object is being present in that range)

Indian Institute of Technology Kharagpur

33 of 56

Visualizing and Analyzing mmWave Radar Data

Rangle-Doppler heatmap:

Represents the detected objects' distances (range) and relative velocities (Doppler shift); each point corresponds to a unique combination of range and Doppler values
intensity of each point indicates the strength of the reflected signal (probability that an object is being present in that range)

Why do we see a high intensity at zero-doppler?

Indian Institute of Technology Kharagpur

34 of 56

Visualizing and Analyzing mmWave Radar Data

Point Cloud Data (PCD):

A collection of data points in 3D space that represents the positions of objects detected by the radar (A point where a radar reflection is detected)
Generated from distance (range), relative velocity (Doppler) and AoA -- Each detected point is mapped to a specific 3D coordinate based on range, angle, and Doppler information.

Indian Institute of Technology Kharagpur

35 of 56

Analyzing mmWave Data

1D FFT (Range FFT) to extract the Range Information

Use the beat frequency (FFT over the IF signal) to compute the range bins
Each peak in the range FFT corresponds to an object at a specific distance

2D FFT across Chirps (Doppler FFT)

A second FFT is performed across the slow time axis, which corresponds to the sequence of chirps
Reveals the Doppler frequency shift caused by the relative motion of objects with respect to the radar

Angle FFT

A third FFT at the spatial dimension (across antennas) to capture the AoA

Indian Institute of Technology Kharagpur

36 of 56

MARS

Continuous Multi-user Activity Tracking via Room-Scale mmWave Sensing

ARGHA SEN, ANIRBAN DAS, SWADHIN PRADHAN, SANDIP CHAKRABORTY

IIT KHARAGPUR

ACM/IEEE IPSN 2024

Indian Institute of Technology Kharagpur

37 of 56

Your Home Knows You Well

Generated through GPT-4

Indian Institute of Technology Kharagpur

38 of 56

Requirements

Continuous subject tracking
Monitoring multiple subjects
Monitoring different activities over time

Real-time inference of activities
Multi-activity support

Both in micro-scale as well as macro-scale

IPSN 2024, May 13-16, 2024, Hong Kong, China

Let us start with the key requirements we outlined for MARS. These requirements are essential for achieving our goals and ensuring a robust system.

The Core Needs

Continuous Subject Tracking: We need to track subjects throughout the monitoring process in a continuous fashion rather than an opportunistic way.
Monitoring Multiple Subjects: Our system must be able to handle scenarios with more than one person present, following each individual's activities.
Monitoring Different Activities Over Time: The system should be able to identify a wide range of activities and track how they change over time.
Real-Time Inference of Activities: We need activity recognition with a considerable delay, allowing for a near-instantaneous understanding of what's happening.

Advanced Capabilities

Multi-Activity Support: The system should be able to recognize and handle situations where multiple individuals perform different activities.
Micro-Scale and Macro-Scale Monitoring: We need the system to function effectively on both small and large scales, identifying subtle movements and broader actions.

We can even quickly compare where MARS stands with the existing works in this direction to address the above-mentioned aspects!

39 of 56

The Broad Vision Towards a Feasible Solution

Indian Institute of Technology Kharagpur

40 of 56

Which Activities Can be Sensed?

We primarily consider

Activities of Daily Living (ADLs)
Instrumental Activities of Daily Living (IADLs)
Exercises

Need different doppler resolutions for macro (-8 to 8) and micro (-64 to 64) activities to separate
The temporal changes are also important

Running has much faster temporal changes than walking

Indian Institute of Technology Kharagpur

41 of 56

Examples: Exercise

Indian Institute of Technology Kharagpur

42 of 56

Examples: Different Jumping Activities

Indian Institute of Technology Kharagpur

43 of 56

Examples: Micro Activities

Using Laptop

Using Phone

Indian Institute of Technology Kharagpur

44 of 56

Impact of Static Clutters

Any object which are stationary but can reflect the mmWave signals

Walls, furniture, etc.

We observe multiple peaks at the range bins
Static clutters produce a higher magnitude along the zero doppler axis

Signifying near-zero or no movements

Indian Institute of Technology Kharagpur

45 of 56

Impact of Non Line of Sight (NLoS)

The subject stands near the wall and then �makes some movements

The NLoS reflections from the wall creates a �signature on the range-doppler heatmap �(we call them as zombie)

Zombie has a lower peak than the original�subject

Need a method to extract the signatures from the�zombies

Indian Institute of Technology Kharagpur

46 of 56

Radar Configuration

Different types of activities are detectable at different doppler resolutions!

Indian Institute of Technology Kharagpur

47 of 56

MARS (Multi-user Activity Tracking via Room-scale Sensing)

Mount the radar on top of a servo

A magnetometer to track the movement�of the servo

Lifecycle of the radar

Detect a user from point-cloud data
Localize and track the user
Check the macro activity
Switch the radar configuration
Check the micro activity
Switch the radar configuration

IPSN 2024, May 13-16, 2024, Hong Kong, China

48 of 56

Point and Track with Configuration Switching

IPSN 2024, May 13-16, 2024, Hong Kong, China

49 of 56

MARS System Architecture

Localization and Tracking Pipeline

Isolate subjects from static clutters (using zero doppler)
Rotating the radar changes the coordinates of the point clouds: Set up a global reference coordinate with respect to the magnetometer

IPSN 2024, May 13-16, 2024, Hong Kong, China

50 of 56

MARS System Architecture

Tracking Multiple Subjects: Use DBSCAN to cluster the pointcloud data

Compare two clusters to check whether a new subject has been introduced in the system

IPSN 2024, May 13-16, 2024, Hong Kong, China

51 of 56

Handling Blind Spots

Two user crosses each other, creating blind spots
We use a Recursive Kalman Filter (RKF) to track the subjects movement states continuously

Estimate when the actual Point Cloud Data (PCD) is unavailable due to occlusion

IPSN 2024, May 13-16, 2024, Hong Kong, China

52 of 56

The High Level Processing Pipeline

Rotate the servo when a subject moves out of the main lobe of the radar

Capture PCD and cluster the PCD to infer the subjects

Apply Kalman Filter to estimate the subject coordinates for the missing values

Random Forest Classifier on the PCD

Extract non-zero doppler range bins and stack 1sec data at different (macro and micro) configurations

Use 2D CNN to classify the activity levels (separate classifiers for macro and micro)

Static and Performing some activities

Walking/Running

IPSN 2024, May 13-16, 2024, Hong Kong, China

53 of 56

Implementation and Testing

The processing pipeline has been �implemented on a Raspberry Pi
We experimented over three �different rooms of different sizes
7 subjects (3 female, 4 male)

IPSN 2024, May 13-16, 2024, Hong Kong, China

54 of 56

Accuracy vs Response Times

Single User

Multi User (N=3)

Each user performs different activities, no activity switching over time

Perform different activities over time

IPSN 2024, May 13-16, 2024, Hong Kong, China

In comparison to the baselines, We consider different scenarios to evaluate -- the first one is single subject doing, multiple activities over time denoting Temporal activity diversity. As shown in the figure, MARS takes the least response time in inferring the activities with the highest accuracy in comparison to the baselines.

We observe that the response time for the first activity takes approx 2 sec, which involves the bootstrap time to denoise and cluster the data for localizing the subject. When a configuration switch is necessary (macro to micro or vice versa), the average response time is approx 3.14 seconds. Without a configuration switch, the average response time is approx 1 seconds. THe primary cause that challenges the baselines to achieve this performance is extended frame stacking or longer classifier inference time impacting response the time. RadHAR and Vid2Doppler show poor performance as they are built focused on macro activities only and are trained only for single-user activity classification. Pantomime, on the other hand, uses separate pointclouds for the activity classification task. These pointclouds easily overlap with each other, especially in scenarios where three subjects are performing macro activities, and thus show lower accuracy.

Here, we asked the subjects to first do some exercise through jumping (macro), then sit (micro), then take their phone and type a message (micro), and finally walk to leave the room (macro). We observe that the response time for the first activity takes approx 2 sec, which involves the bootstrap time to denoise and cluster the data for localizing the subject. When a configuration switch is necessary (macro to micro or vice versa), the average response time is approx 3.14 seconds. Without a configuration switch, the average response time is approx 1 seconds. In comparison, the baselines perform worst in the response time due to more extended frame stacking (2 sec and 3 sec, respectively, for RadHAR and Vid2Doppler) and longer classifier inference time (approx 4 sec for VGG-16 and Pantomime). Longer response times of the baselines directly impact lowering the number of hits in the activity time window w.r.t. MARS, which has a low response time due to smaller frame stacking (1 sec) and reduced inference time (approx 0.08 sec) with a light-weight model architecture. ��In the second scenario, we pick four subjects and ask three of them to choose macro activities and the remaining one to choose one of micro activities. After determining the subjects' location and states, MARS configures the low doppler resolution and classifies the macro activities simultaneously, with a response time of approx 3s at the beginning. However, using 1 sec of frame-stacked data, it can gradually infer the three macro activities simultaneously with a response time of 1.04s. For the subject performing the micro activity, it switches the configuration to high doppler resolution and classifies the same with a response time of 2.08 sec, resulting in an average response time of 1.9 sec with an accuracy of 98% in the entire activity time window of 10 sec. RadHAR and Vid2Doppler show poor performance as they are built focused on macro activities only and are trained only for single-user activity classification. Pantomime, on the other hand, uses separate pointclouds (within 1 x 1 x 2 m^3) for the activity classification task. These pointclouds easily overlap with each other, especially in scenarios where three subjects are performing macro activities, and thus show lower accuracy. Interestingly, we observe that the average number of correct inference for MARS is higher in this case (spatial diversity) compared to the previous one (temporal diversity), as the radar needs less configuration switching.

55 of 56

Accuracy vs Response Times

Multi User (N=3), activity switching over time

IPSN 2024, May 13-16, 2024, Hong Kong, China

56 of 56

Happy Learning!

Some resources related to this topic

Introduction	Related Work	Background	Observation	Methodology	Evaluation	Conclusion