CS60055: Ubiquitous Computing

ML Pipelines for Sensor Data

From features to foundation models

INDIAN INSTITUTE OF TECHNOLOGY

KHARAGPUR

Sandip Chakraborty

sandipc@cse.iitkgp.ac.in

Department of Computer Science and Engineering

Where We Are in the Course

Sensing

Transduction, IMU, physiological, acoustic, wireless (CSI)

Signals

Sampling, filtering, phase cleaning, spectra

Learning

Today: turning streams of numbers into decisions

• So far: how a sensor turns the physical world into a time series, and how we clean it
• Today: the learning side of the pipeline, for any sensor stream (IMU, PPG, CSI, radar, ambient)
• Running example: human activity recognition (HAR), the "hello world" of ubiquitous computing

Same pipeline thinking applies to health, gestures, localization and context sensing

Indian Institute of Technology Kharagpur

Today's Plan

1

Part 1: The Classical Pipeline

Acquire, preprocess, segment, extract features, classify, evaluate, deploy

2

Part 2: Sensor Foundation Models

Why pretraining at scale, definitions, a lifecycle taxonomy, three research directions (survey: Bian et al., IMWUT 2026)

3

Part 3: Case Study: SensorFM

Google Research: one trillion minutes, five million people, 35 health tasks, agents on top

Indian Institute of Technology Kharagpur

Why Learn From Sensor Data?

• Hand-written rules work for simple cases
• step = peak in |a| above a threshold
• fall = big spike followed by stillness
• Rules break when users, devices or contexts change
• phone in pocket vs hand vs bag
• an elderly person walks very differently from an athlete
• Machine learning learns the mapping from signal to label using examples
• the pipeline decides what the model sees and how we judge it

Tri-axial accelerometer (synthetic, 50 Hz): each activity has a distinct pattern in level, energy and rhythm

Goal of a pipeline: a model that works on people and devices it has never seen

Indian Institute of Technology Kharagpur

Sensor Data Is Not Like Images or Text

These differences shape every stage of the pipeline, and later, the design of sensor foundation models

Indian Institute of Technology Kharagpur

The Activity Recognition Chain (ARC)

1. Acquire

Sensors, rate, placement

2. Preprocess

Filter, resample, sync, normalize

3. Segment

Sliding windows or events

4. Features

Hand-crafted or learned

5. Classify

kNN, SVM, RF, CNN, LSTM

6. Post-process

Smoothing, fusion, decisions

• A standard way to think about sensor ML (Bulling, Blanke, Schiele, ACM Computing Surveys 2014)
• Each stage has design choices; errors made early cannot be fixed later
• Deep learning merges stages 4 and 5; foundation models also learn stage 4 from unlabeled data

Plus one more stage that decides everything: 7. Evaluate (how you split the data)

Indian Institute of Technology Kharagpur

Stage 1: Data Acquisition

• Sampling rate: Nyquist says fs > 2 × highest frequency of interest
• human motion is mostly below 20 Hz, so 50 Hz is a common choice
• higher rate costs battery and storage
• Placement: wrist, hip, ankle, pocket, chest; each sees different motion
• Synchronization: devices have separate clocks; timestamps drift
• Ground truth: video, observer notes, diary, app prompts (EMA)
• labelling is the most expensive part of the pipeline

Choose the rate from the physics of the target, not the maximum the chip allows

Indian Institute of Technology Kharagpur

Stage 2: Preprocessing, Filtering and Gravity

• Denoise: low pass or median filter removes sensor noise and spikes
• Separate gravity: accelerometer = gravity + body acceleration
• low pass (around 0.3 Hz) gives gravity, which tells posture
• subtract it to get body acceleration, which tells motion
• Band pass to keep only the band of the target (breathing below 1 Hz, gait 1 to 3 Hz)
• Resample all channels to a common rate before fusion

Pick filters from the physics of the target

Indian Institute of Technology Kharagpur

Stage 2: Normalization and Orientation

• Problem: the same walk looks different if the phone is rotated in the pocket
• Orientation-invariant features: use the magnitude |a| = √(ax² + ay² + az²)
• loses direction information but removes rotation
• Rotate to a world frame using gravity (and magnetometer) from sensor fusion
• Scale normalization: z-score per channel; per-user normalization reduces body differences
• compute statistics only on training data, otherwise you leak test information

Device frame

ax, ay, az depend on how the device is held

World frame

Rotate using gravity: vertical and horizontal parts

Normalized

Zero mean, unit variance, per channel

Indian Institute of Technology Kharagpur

Stage 2: Missing Data Is the Norm

One illustrative day from a wrist wearable: channels switch on and off independently

• Why gaps happen
• battery saving and duty cycling
• watch off wrist, charging
• sensor needs good skin contact
• packet loss over BLE

• Classic options: drop incomplete windows (wastes data) or impute (can add bias)
• Better: give the model a mask so it knows what is missing; SensorFM (Part 3) learns directly from gaps

Remember this slide: handling missingness is a key design choice in SensorFM

Indian Institute of Technology Kharagpur

Stage 3: Segmentation With Sliding Windows

• Window: fixed number of samples treated as one example
• Overlap: 50% is common; gives more examples and smoother output
• Label: majority label inside the window

• Other options: event-based windows (start at a detected peak) and activity-defined windows (from change-point detection)
• Transformers often split a window further into patches, which become tokens

Indian Institute of Technology Kharagpur

Choosing the Window Size

• Rule: a window should contain at least one or two cycles of the slowest pattern you care about
• Trade-off: longer windows give more context but slower response and fewer examples
• Activities are hierarchical: gestures inside actions inside routines

Window size is a hyperparameter. Tune it on validation data, and report it. Many papers differ only in this choice.

Indian Institute of Technology Kharagpur

Stage 4: Time-Domain Features

• Computed per window, per channel
• Cheap: can run on a microcontroller
• Interpretable: you can explain what the model uses
• A typical feature vector has 50 to 500 numbers per window

Example: mean of z-axis separates sitting vs standing; std separates walking vs running

Indian Institute of Technology Kharagpur

Stage 4: Frequency-Domain Features

• FFT of each window reveals rhythm
• dominant frequency (step rate, breathing rate)
• spectral energy in bands
• spectral entropy: periodic vs random motion
• Wavelets handle short transient events better than FFT
• Spectrograms (STFT over time) turn a stream into an image for CNNs

Walking and running can have similar variance, but their dominant frequencies differ clearly

Indian Institute of Technology Kharagpur

Stage 5: Classical Classifiers

Random forest on good features is still a strong baseline; always compare against it

Indian Institute of Technology Kharagpur

Deep Learning: Learn the Features Too

Raw window

C channels × T samples

Conv layers

Learn local motion patterns

LSTM / GRU

Learn temporal order

Classifier

Softmax over activities

• DeepConvLSTM (Ordóñez and Roggen, 2016): convolution plus LSTM became a standard HAR baseline
• 1D CNNs / TCNs: fast and simple; dilated convolutions see long context
• Transformers: attention over patches of the window; good with large data
• But: deep models need many labels, and HAR datasets are small

The bottleneck moved from feature design to label collection

Indian Institute of Technology Kharagpur

Stage 6: Post-processing

• Activities last much longer than one window
• Isolated flips are usually errors
• Majority vote or HMM / CRF smoothing fixes them

• Fusion: combine decisions from several sensors or devices (late fusion) or features (early fusion)
• Rejection: say "unknown" when confidence is low; important for open-world use

Indian Institute of Technology Kharagpur

Stage 7: Evaluation Protocols

Random split

Windows of the same person (even the same minute) in train and test

Optimistic, often 95%+

Leave-one-subject-out

Test person never seen in training

Measures generalization to new users

Cross-dataset

Train on dataset A, test on dataset B

New devices, placements, protocols

• Leakage: overlapping windows share samples, so a random split puts nearly the same data in train and test
• Report the protocol with every number; user-independent results can be 10 to 30 points lower

Accuracy without a protocol is not a result

Indian Institute of Technology Kharagpur

Metrics: Accuracy Is Not Enough

• Class imbalance: in daily life, sitting dominates; falls are rare
• a model that always says "sitting" can get 70% accuracy
• Macro F1: average F1 over classes; treats rare classes fairly
• Confusion matrix shows which activities get mixed (upstairs vs downstairs)
• Other metrics: AUROC for health risk, Pearson r or MAE for regression, recall@K for retrieval

Illustrative confusion matrix (%)

Indian Institute of Technology Kharagpur

Deployment: The Pipeline on a Device

• Edge vs cloud: where should each stage run?
• features on the watch, model on the phone, training in the cloud
• TinyML: CNNs with tens of KB run on Cortex-M class chips
• Pruning, distillation, quantization shrink models

This question returns for foundation models: a 100M parameter model does not fit on a watch

Indian Institute of Technology Kharagpur

Recap: The Pipeline So Far

Acquire

Rate, placement

Preprocess

Filter, sync, mask

Segment

Windows

Features

Time, frequency

Classify

RF, CNN

Evaluate

LOSO, macro F1

• Works well inside one dataset, with one sensor setup, and enough labels
• Fails when the user, device, placement or environment changes
• Every new task means new labels and a new model

Question for Part 2: can we learn reusable representations from unlabeled data?

Indian Institute of Technology Kharagpur

Why Sensor HAR Is Hard

• Signal: long, high-rate streams
• Data: different modalities need alignment
• User: body and placement cause domain shift
• Semantic: fuzzy activity boundaries, nested activities
• Corpus: small datasets, costly labels

Source: Bian et al., "Foundation Models Defining A New Era In Sensor-based HAR", ACM IMWUT 10(2), 2026 (CC BY 4.0)

Indian Institute of Technology Kharagpur

How the Field Evolved

• 2004: hand-crafted features + SVM, RF, HMM
• 2013: deep learning (CNN, LSTM)
• 2016: transfer and domain generalization
• 2019: self-supervised learning from unlabeled data
• 2023: foundation models with large-scale pretraining and language alignment

Source: Bian et al., "Foundation Models Defining A New Era In Sensor-based HAR", ACM IMWUT 10(2), 2026 (CC BY 4.0)

Indian Institute of Technology Kharagpur

Bridge 1: Transfer and Domain Adaptation

Source domain

Many labels (lab data, other users)

Shared model

Pretrain, then reuse layers

Target domain

Few labels (new user, new device)

• Fine-tuning: start from source weights, train a little on target data
• Domain adaptation: align feature distributions of source and target (e.g., adversarial training)
• Personalization: small per-user update on top of a population model
• Limitation: still needs a labelled source dataset of decent size

Indian Institute of Technology Kharagpur

Bridge 2: Self-Supervised Learning (SSL)

• Idea: create a training task from the data itself, no human labels
• Pretext tasks for sensors
• predict which transformation was applied (rotation, noise, scaling)
• mask part of the signal and reconstruct it
• make two views of the same window agree (contrastive)
• Unlabeled data is cheap: every watch records it all day

Pretrain

Large unlabeled data, pretext task

Adapt

Small labelled data, target task

Haresamudram et al. (IMWUT 2022): SSL clearly helps when labels are scarce

Indian Institute of Technology Kharagpur

Contrastive Learning for Sensors

• Two augmented views of the same window = positive pair
• Other windows in the batch = negatives
• Loss pulls positives together, pushes negatives apart (SimCLR, InfoNCE)
• Sensor augmentations: jitter, scaling, rotation, time warp, channel dropout

• Variants: BYOL (no negatives), TF-C (time and frequency views), RelCon (relative similarity between windows)

Augmentations encode what the model should ignore; choose them from sensor physics

Indian Institute of Technology Kharagpur

Masked Reconstruction for Sensors

• LIMU-BERT (SenSys 2021): mask IMU samples, predict them
• Model must learn motion dynamics to fill gaps
• No negatives, simple to scale
• Same idea powers LSM and SensorFM

• Masked autoencoders (MAE): large encoder sees only visible patches; small decoder rebuilds the masked ones

Pretrain-then-finetune with SSL is the direct ancestor of sensor foundation models

Indian Institute of Technology Kharagpur

Using a Pretrained Encoder: Probe or Fine-tune

Linear probing

Fine-tuning

Frozen encoder

Pretrained weights, not updated

Linear head

Only this layer is trained

Encoder

All (or some) layers updated

Task head

Trained together

• Tests the quality of the representation
• Cheap; one encoder serves many tasks

• Usually more accurate
• Costly; one copy of the model per task

SensorFM (Part 3) reports results mainly with linear probes on a frozen encoder

Indian Institute of Technology Kharagpur

Part 2

Sensor Foundation Models

Following Bian et al., "Foundation Models Defining A New Era In Sensor-based HAR: A Survey And Outlook", ACM IMWUT 2026

What is an FM?

Criteria for HAR FMs

Lifecycle taxonomy

Three directions and open problems

Indian Institute of Technology Kharagpur

What Is a Foundation Model?

"Any model trained on broad data (generally using self-supervision at scale) that can be adapted to a wide range of downstream tasks"

Stanford CRFM, 2021

Broad data

Unlabeled, diverse, huge

Self-supervised pretraining

Masking, contrast, next token

Adaptation

Probe, fine-tune, prompt

Many tasks

Zero / few-shot, new domains

• Examples: BERT, GPT, CLIP (image and text), Chronos (time series)

Key shift: from one model per task to one model, many tasks

Indian Institute of Technology Kharagpur

Foundation Models Across Fields

Source: Bian et al., "Foundation Models Defining A New Era In Sensor-based HAR", ACM IMWUT 10(2), 2026 (CC BY 4.0)

Indian Institute of Technology Kharagpur

A Definition for Sensor-based HAR

A pretrained, sensor-grounded model and its adapters that can solve diverse activity-understanding tasks across the sensing, temporal and context continuum, while generalizing across sensor modalities, body placements, users, devices and environments

Definition adapted from Bian et al., IMWUT 2026

• Different from CV and NLP: must generalize across sensors, bodies and deployments, not only tasks
• Typical recipe: self-supervised pretraining on large heterogeneous sensor corpora, optionally aligned with language or vision
• Stable interfaces: embeddings, prompts or small adapters enable zero / few-shot transfer and personalization

Three axes of variation: modality, time scale (micro-gesture to routine), context and embodiment

Indian Institute of Technology Kharagpur

How FMs Address the Five Challenges

Source: Bian et al., "Foundation Models Defining A New Era In Sensor-based HAR", ACM IMWUT 10(2), 2026 (CC BY 4.0)

Indian Institute of Technology Kharagpur

Six Criteria for a HAR Foundation Model

1. Corpus coverage

Many users, devices, placements, settings

2. Cross-domain generalization

Works on unseen users, devices, datasets; zero / few-shot

3. Modality extensibility

Add new sensors with little change; robust to missing channels

4. Label-efficient pretraining

Self-supervised or hybrid objectives on unlabeled streams

5. Adaptation surfaces

Adapters, LoRA, heads, prompts with small parameter budgets

6. Broad applicability

One core for many tasks; emergent skills on unseen tasks

Not a rigid checklist: a model should show these hallmarks in substance, not necessarily all at once

Indian Institute of Technology Kharagpur

When Is an LLM a HAR Foundation Model?

Encoder + LLM decoder

Sensor encoder conditions a language decoder via cross-attention

e.g., SensorLM, Sensor2Text

Sensor tokens into an LLM

Frozen or partly frozen LLM reads projected sensor tokens

e.g., SensorLLM, Time2Lang

Prompt-only, in-context

Fixed sensor-to-text pipeline; zero / few-shot across tasks

e.g., HARGPT

• Qualifies: reusable across tasks and datasets, with a reproducible sensor-to-language interface
• Does not qualify: a generic chatbot with hand-written prompts for one dataset; that is only an auxiliary reasoning tool

The test is reuse plus a stable interface, not the presence of an LLM

Indian Institute of Technology Kharagpur

A Lifecycle Taxonomy: Four Phases, Nine Axes

Axes 1 to 3: what the model sees. Axes 4 to 5: how it learns. Axis 6: how it adapts. Axes 7 to 9: how it is used.

Source: Bian et al., "Foundation Models Defining A New Era In Sensor-based HAR", ACM IMWUT 10(2), 2026 (CC BY 4.0)

Indian Institute of Technology Kharagpur

Axis 1: Modality Scope

Unimodal

One sensor type, e.g., accelerometer only

Scales easily; limited semantic grounding

Multimodal

Several sensors fused in pretraining, e.g., IMU + PPG + EDA

Richer context, robust to missing channels; imbalance, sync issues

Cross-modal

Align sensor with text or video (IMU ↔ text)

Zero-shot, explainable; needs paired data, more compute

• Trend: gradual shift from unimodal pretraining toward cross-modal integration, for semantic grounding and explainability

Our lab view: radar, CSI and air-quality sensing still sit mostly at the unimodal stage

Indian Institute of Technology Kharagpur

Representative Models by Modality Scope

Adapted from Table 1, Source: Bian et al., "Foundation Models Defining A New Era In Sensor-based HAR", ACM IMWUT 10(2), 2026 (CC BY 4.0)

Indian Institute of Technology Kharagpur

Axis 2: Data Landscape

• Collected in the wild
• IMU: PAMAP2, WISDM, RealWorld, Opportunity, UCI-HAR, HHAR
• Physiology: WESAD, Sleep-EDF, MIMIC, PTB-XL
• Smart home: CASAS
• RF / radar and egocentric video: smaller, growing
• Scale ranges from a few hours and tens of people to billions of records and over 100k people

• Generated data
• LLM writes activity scripts → text-to-motion → virtual IMU (IMUGPT)
• physics simulation of bodies and sensors
• pseudo-captions from sensor-language models
• Check before use: noise realism, drift, label fidelity, non-IID tests

A foundation model inherits its biases from its pretraining corpus

Indian Institute of Technology Kharagpur

Virtual Sensor Data From Language

LLM prompt

"Describe 20 ways a person climbs stairs"

Text description

"A person climbs stairs slowly, holding the rail"

Motion synthesis

Text-to-motion model creates 3D body motion

Virtual IMU

Place a virtual sensor on the wrist, compute acceleration

• Benefit: covers rare or sensitive activities (falls, medical events) without collection
• Risk: synthetic signals lack real sensor noise, placement shifts and individual style
• Practice: mix real and virtual data; always test on real users

Pipeline in the spirit of IMUGPT 2.0 (Leng et al., IMWUT 2024)

Indian Institute of Technology Kharagpur

Axis 3: Tokenization

Windows / patches

Fixed-length chunks (e.g., 128 samples); enable masking and attention

Statistical features

Minute or hour summaries (mean HR, steps); short sequences, interpretable

Spectrograms

STFT or wavelet patches; robust to noise, good for periodic signals

Quantized tokens

Codebook symbols; reuse NLP models, talk to LLMs

5. Multimodal alignment, positional encoding

6. Token fusion, cross-modal projection

Families 1 to 4 are parallel choices inside one modality; 5 and 6 come after, across modalities

SensorFM uses family 2: 34 one-minute aggregate features over a 24-hour window

Indian Institute of Technology Kharagpur

Tokenization Trade-offs

Spathis and Kawsar (2023): "The first step is the hardest". Poor tokenization limits what any LLM can do with time series.

Indian Institute of Technology Kharagpur

Axis 4: Four Base Architectures

Encoder-only

Sensor encoder → embedding → linear head

LIMU-BERT, RelCon, UniMTS

Dual encoder

Sensor encoder + text encoder, shared space, contrastive

IMU2CLIP, FM-Fi

Encoder-decoder

Sensor encoder + causal (language) decoder via cross-attention

SensorLM, Sensor2Text, LSM

Language-model stack

LLM itself is the backbone over sensor tokens

LLM4HAR, Time2Lang

Backbones (CNN, Transformer, Mamba, GNN, MoE) instantiate these graphs; they are not separate families

Adapted from Table 3, Source: Bian et al., "Foundation Models Defining A New Era In Sensor-based HAR", ACM IMWUT 10(2), 2026 (CC BY 4.0)

Indian Institute of Technology Kharagpur

Choosing an Architecture

In practice systems mix patterns with small projection heads, so tokenizers and backbones can be swapped. SensorFM: encoder-decoder for pretraining, encoder-only for downstream use.

Indian Institute of Technology Kharagpur

Axis 5: Pretraining Paradigms

Contrastive

Pull related views together, push others apart

Cross-domain alignment, retrieval

IMU2CLIP, RelCon, FM-Fi

Generative

Mask and reconstruct, or forecast the future

Temporal continuity, imputation

LIMU-BERT, LSM, SelfPAB

Hybrid

Masking + contrast + distillation, often with language

Best of both; dominant trend

SensorLM, NORMWEAR, CrossHAR

• Contrastive needs large batches and good augmentations; generative needs no negatives but may need alignment for semantics

Field is converging on hybrid, large-scale self-supervised pretraining

Indian Institute of Technology Kharagpur

Scaling: Does Bigger Help for Sensors?

• LSM (Google, ICLR 2025): wearable FM trained on 40 million hours from 165k people
• loss falls as data, compute and model size grow
• small datasets limit generalization
• Lesson: model size must match data size
• SelfHAR-700k: 700,000 person-days of wrist accelerometry
• SensorFM (Part 3): over 2 billion hours, 5 million people

40M

hours (LSM)

700k

person-days (SelfHAR)

2B+

hours (SensorFM)

Scaling laws seen in vision and language are now appearing for wearable data

Indian Institute of Technology Kharagpur

Axis 6: Adaptation Strategies

PEFT

Freeze backbone; train adapters, LoRA, soft prompts

Few labels, on-device, federated

Full / partial fine-tuning

Unfreeze deeper layers or all layers

Big domain shift, enough labels

Instruction tuning

In-context prompts (no update) or SFT on sensor-text pairs

Stable output format, multi-turn use

• These can be combined: e.g., in-context learning first, then LoRA instruction tuning on curated pairs
• PEFT has become the default when compute, labels or data access are limited

Indian Institute of Technology Kharagpur

LoRA in One Slide

W

(d × d)

frozen

+

B

(d × r)

A (r × d)

h = W x + B A x

rank r is much smaller than d (e.g., r = 8)

• Only A and B are trained
• Parameters per layer: 2dr instead of d²
• d = 768, r = 8: about 12k instead of 590k
• Store one small adapter per user or per task
• Population knowledge stays in W; personal shift goes into BA

Ideal for personalization on a phone and for federated learning

Indian Institute of Technology Kharagpur

Axis 7: Downstream Capabilities

Pair each claimed capability with its matching protocol and metric

Indian Institute of Technology Kharagpur

Axis 8: Deployment Settings

Cloud-scale

Large pretraining on servers

Heavy backbones, full fine-tuning

Shows what FMs can learn

On-device and mobile

PEFT, quantization, pruning

Compact LMs, streaming inference

Shows what FMs can sustain

• Likely future: cloud-scale pretraining, then light, privacy-aware adaptation on the device
• Report system costs (latency, energy, memory) on target hardware, next to non-IID accuracy

Edge-cloud split is a networking and systems problem as much as an ML one

Indian Institute of Technology Kharagpur

Axis 9: Application Domains

• General HAR / ADL: activity logs, routine profiling
• Health and wellbeing: remote monitoring, sleep staging, clinician summaries
• Smart home: sparse binary event streams; LLMs used for zero-shot cold start
• Agentic assistants: chat over personal logs, nudges, journaling

Source: Bian et al., "Foundation Models Defining A New Era In Sensor-based HAR", ACM IMWUT 10(2), 2026 (CC BY 4.0)

Indian Institute of Technology Kharagpur

Smart Homes: Event-based Sensing

• Binary, event-driven sensors: motion on/off, door open/close
• Sparse, asynchronous, low frequency
• Ambiguity: the same event pattern can mean different activities
• Needs long-horizon context and routines, not fine motion
• Current approach: LLMs turn event sequences into text for zero-shot recognition, then hand over to supervised models as labels arrive

Example event stream

07:02 Bedroom motion ON

07:05 Bathroom door OPEN

07:18 Kitchen motion ON

07:19 Fridge door OPEN

07:24 Stove ON

→ "preparing breakfast"?

Foundation models for ambient event streams are still at an early stage

Indian Institute of Technology Kharagpur

Three Development Directions

1. HAR-specific FMs from scratch

Pretrain on large sensor corpora

HAR-FM, MASTER, NORMWEAR, oneHAR, LSM, SensorFM

2. Adapt general FMs

Reuse time-series or vision-language FMs

IMU2CLIP, Time2Lang, FM-Fi 2.0, Chronos adapters

3. LLMs for HAR

Reasoning, annotation, interaction

HARGPT, DailyLLM, LanHAR, ContextLLM

• Complementary, not competing: native pretraining gives the base, adaptation gives reach, LLMs give semantics

Expected convergence: hybrid frameworks combining all three

Indian Institute of Technology Kharagpur

Direction 1: Sensor-native FMs

• Why: directly captures sensor noise, biomechanics and user variation. Barriers: few public datasets, privacy, long-sequence compute

Indian Institute of Technology Kharagpur

Direction 2: Adapting General FMs

Sensor window

IMU, PPG, RF

Small encoder / projector

Trainable bridge into FM space

Frozen general FM

CLIP, Chronos, LLaMA

Output

Label, retrieval, text

• IMU2CLIP: map IMU segments into CLIP space, so text and video queries find motion
• Time2Lang: Chronos time-series FM + LLaMA via fixed-length tokens; LLM stays frozen
• FM-Fi 2.0: distill a vision-language teacher into an RF encoder for zero-shot WiFi HAR
• Gain: low training cost, strong in low-label regimes. Gap: no motion-specific priors; multi-rate, noisy signals

Indian Institute of Technology Kharagpur

Direction 3: LLMs for HAR

• Roles of the LLM
• zero-shot recognizer (HARGPT)
• explainer: daily activity logs (DailyLLM)
• health interpreter (PhysLLM)
• reasoning engine across devices (ContextLLM)
• Shift: from label prediction to narrative, context-aware understanding
• Open issues: few paired sensor-text corpora, modality gap, edge cost, hallucination

Prompt-only example (HARGPT style)

"You are an expert in IMU-based activity recognition. Below are 3-axis accelerometer readings at 10 Hz from a wrist device:

x: 0.12, 0.85, -0.40, ...

Think step by step about the patterns, then answer with one of: sitting, walking, running, stairs."

Works surprisingly well for simple activities; fails on long or fine-grained signals

Indian Institute of Technology Kharagpur

Open Challenges

Scaling under fragmentation

Data split across devices, rates and owners; privacy limits pooling; how to use synthetic data safely

Temporal hierarchy and semantics

Encode seconds-to-days structure; align with language without overfitting to captions

Adaptation as a first-class problem

Few-shot personalization without drift; cloud + on-device + federated; co-design with hardware

Evaluation and responsibility

Standard cross-dataset and open-set protocols; calibration, uncertainty, hallucination, privacy

Next phase: better objectives, lifecycle-aware adaptation and rigorous evaluation, not only bigger backbones

Indian Institute of Technology Kharagpur

Think About It

Q1

A model scores 97% accuracy on a random split of 30 users. What would you ask before believing it?

Q2

You have 10 million unlabeled hours of mmWave radar data and 200 labelled hours. Which pretraining paradigm would you try first, and why?

Q3

Is a chatbot that reads your step count and gives advice a HAR foundation model? Use the survey's criteria.

Indian Institute of Technology Kharagpur

Part 3

Case Study: SensorFM

"Towards a General Intelligence and Interface for Wearable Health Data", Google Research, July 2026 (arXiv 2605.22759)

Data at population scale

Missing-aware pretraining

35 health tasks

Agents on top

Indian Institute of Technology Kharagpur

Why SensorFM: Beyond One Model Per Task

• Billions of wearables track heart rate, motion, temperature, SpO2 and sleep every day
• Problem 1: baseline physiology differs hugely between people
• Problem 2: health labels (diagnoses, lab tests, validated surveys) are slow and costly
• So most wearable health models are bespoke, supervised, single-endpoint
• SensorFM idea: learn one reusable representation of physiology from unlabeled data

Before

35 tasks = 35 pipelines, 35 label sets

SensorFM

One encoder + 35 light heads

Source: Liu, McDuff et al., "Towards a General Intelligence and Interface for Wearable Health Data", Google Research, 2026

Indian Institute of Technology Kharagpur

Pretraining Corpus: Population Scale

5M

consented participants

1T+

minutes (2B+ hours)

100+

countries, all 50 US states

20+

Fitbit and Pixel Watch models

Data captured September 2024 to September 2025; several weeks per person; de-identified

Source: Liu, McDuff et al., "Towards a General Intelligence and Interface for Wearable Health Data", Google Research, 2026

Indian Institute of Technology Kharagpur

Input: 34 Minute-level Features

• Five sensor modalities: PPG, accelerometer, EDA, skin temperature, altimeter
• 34 one-minute aggregates, grouped as
• cardiovascular: HR, RMSSD, SDNN, HRV spectra
• cardiopulmonary: SpO2
• sleep: stage durations
• motion: steps, jerk, energy
• skin surface: temperature, conductance
• Input: a full 24-hour window (1440 minutes × 34 features)

Tokenization family 2 (statistical features): short, interpretable, cheap

Indian Institute of Technology Kharagpur

Real Wearable Data Is Full of Holes

• Many channels are only on part of the day
• Causes: power cycling, off wrist, power-saving modes, sensors switching
• Conventional SSL assumes complete input

Impute (adds bias) or drop windows (wastes data)? SensorFM does neither

Indian Institute of Technology Kharagpur

AIM: Adaptive and Inherited Masking

HR

HRV

SpO2

Steps

EDA

Temp

Inherited mask: real gap in the data

Artificial mask: hidden for training

Visible to the encoder

• Builds on LSM-2 (Google, 2025)
• Real gaps are treated as inherited masked tokens
• Extra tokens are artificially masked for the reconstruction objective
• Both kinds are treated the same by the encoder
• Loss is computed only where true values exist
• Result: missingness-aware by construction

Incomplete recordings become useful training data instead of being thrown away

Schematic, not real data

Indian Institute of Technology Kharagpur

Architecture: Masked Autoencoder

• Pipeline: resample, normalize, remove outliers, mask
• Encoder: ViT-style transformer over visible tokens
• Decoder: lighter transformer rebuilds masked tokens
• Downstream: keep the encoder, pool the embedding, attach task heads

Figure from the SensorFM paper

Encoder-decoder for pretraining (generative paradigm), encoder-only for use (survey Axes 4 and 5)

Indian Institute of Technology Kharagpur

Model Family and Scaling

31%

lower reconstruction loss, B vs XXS

+9%

avg AUC, classification

+21%

avg Pearson r, regression

33/35

tasks won by SensorFM-B

Data scaled from about 2M to 2B sensor-hours, model from 100K to 100M parameters; no saturation seen

Indian Institute of Technology Kharagpur

Downstream Evaluation: 35 Health Tasks

• Three IRB-approved prospective studies, 13,985 participants in total
• Six categories: cardiovascular, metabolic, mental health, sleep, demographics, lifestyle
• Label sources
• SLF: self-reported
• LAB: lab tests (e.g., HbA1c, lipids)
• SCR: standard screeners (PHQ-8, GAD-7)
• 20 classification and 15 regression tasks

Protocol: freeze the encoder, train a linear head, compare with supervised models on engineered features

Indian Institute of Technology Kharagpur

Linear Probing on Frozen Embeddings

• Beats the feature-engineered baseline on 34 of 35 tasks
• No task-specific architecture
• Bigger gains on hard, person-specific outcomes like depression and anxiety

A single representation transfers across cardiovascular, metabolic, sleep and mental health

Indian Institute of Technology Kharagpur

Classification Results (F1)

• Bars grow with model size (light to dark blue)
• Red: agentic "classroom" heads, best on most tasks
• Demographics help less as models grow

Larger models seem to capture age- and sex-related physiology during pretraining itself

Indian Institute of Technology Kharagpur

Regression Results (Pearson Correlation)

• Age, BMI, weight: strong
• Cardiovascular risk scores (ASCVD, Framingham)
• Metabolic: HOMA-IR, HbA1c, triglycerides
• Mental health: PHQ-8, GAD-7, PSS

Lab values and screener scores predicted from passive wearable data alone

Indian Institute of Technology Kharagpur

Label Efficiency and Data Infilling

• Label efficiency: with a small fraction of labels, SensorFM passes demographic-only and feature-engineered baselines
• Generation: the decoder fills gaps, so daily metrics (time in SpO2 ranges, exercise zones, sleep stages, steps) can be estimated from incomplete days
• Figure: percent error in daily estimates, baseline vs SensorFM (lower is better)

Generative pretraining gives a second product for free: robust daily metrics

Indian Institute of Technology Kharagpur

What Do the Embeddings Capture?

Cosine similarity of task directions: related tasks group together

UMAP of embeddings coloured by condition: demographics, lifestyle and cardiovascular health separate

Indian Institute of Technology Kharagpur

A "Classroom" of Agents Builds the Heads

• LLM "students" write Python code for a prediction head on SensorFM embeddings
• Code runs, gets scored; a teacher LLM gives feedback
• Students share and compete across cycles
• More than 30,000 candidate solutions explored
• Beats linear probe on 16 of 20 classification and 12 of 15 regression tasks

Quality rises over the search and with stronger LLMs; collaboration helps weaker LLMs

Indian Institute of Technology Kharagpur

Grounding a Personal Health Agent

• User asks: "How can I improve my health?"
• SensorFM heads predict age, BMI, insulin resistance, sleep issues, anxiety
• Predictions go to Gemini as grounding context
• Answer is personal and physiological, not generic

Survey Direction 3 in action: the sensor FM supplies grounded numbers, the LLM supplies reasoning and language

Indian Institute of Technology Kharagpur

Clinician Evaluation of Health Summaries

• 31 real participant profiles × 3 conditions: wearable metrics only, + SensorFM predictions, + ground truth labs
• Blinded clinicians rated context, relevance, justifiability, personalization and harm (1 to 5): 1,860 ratings, 40+ hours
• SensorFM beats baseline on every dimension; no significant difference from using true lab values

Model inferences served the agent about as well as real measurements

Indian Institute of Technology Kharagpur

SensorFM Through the Survey's Lens

Indian Institute of Technology Kharagpur

Critical View and Why It Matters for Us

• Limitations to discuss
• minute aggregates, not raw signals: fine dynamics are lost
• one vendor, consumer devices; cohort bias
• closed data and model; hard to reproduce
• cloud-scale; not yet on the watch
• association, not diagnosis

• Lessons for new modalities
• mmWave, WiFi CSI and air-quality sensing have no FM yet
• design for gaps from day one
• co-scale data and model
• evaluate with frozen probes across many tasks
• plan the LLM interface early

Open question: what is the "trillion minutes" for ambient RF sensing?

Indian Institute of Technology Kharagpur

Summary

• Classical pipeline: acquire, preprocess, segment, extract features, classify, post-process; evaluation protocol decides trust
• Core bottleneck: labels and heterogeneity, not model capacity
• Self-supervision (contrastive, masked) turned unlabeled streams into a resource
• Sensor FMs: broad pretraining + stable adaptation interfaces; nine design axes from input to deployment
• SensorFM: population-scale, missing-aware masked pretraining gives one representation for 35 health tasks and a grounding tool for LLM agents

From one model per task to one model, many tasks, many sensors

Indian Institute of Technology Kharagpur

References and Further Reading

• S. Bian et al., "Foundation Models Defining A New Era In Sensor-based Human Activity Recognition: A Survey And Outlook", ACM IMWUT 10(2), 2026
• X. Liu, D. McDuff et al., "Towards a General Intelligence and Interface for Wearable Health Data" (SensorFM), arXiv 2605.22759, 2026
• A. Bulling, U. Blanke, B. Schiele, "A Tutorial on Human Activity Recognition Using Body-worn Inertial Sensors", ACM CSUR, 2014
• F. Ordóñez, D. Roggen, "Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition", Sensors, 2016
• H. Xu et al., "LIMU-BERT: Unleashing the Potential of Unlabeled Data for IMU Sensing Applications", ACM SenSys, 2021
• H. Haresamudram, I. Essa, T. Plötz, "Assessing the State of Self-Supervised HAR Using Wearables", ACM IMWUT, 2022
• G. Narayanswamy et al., "Scaling Wearable Foundation Models" (LSM), ICLR, 2025
• Y. Zhang et al., "SensorLM: Learning the Language of Wearable Sensors", arXiv, 2025
• S. Moon et al., "IMU2CLIP: Language-grounded Motion Sensor Translation", EMNLP, 2023
• S. Ji, X. Zheng, C. Wu, "HARGPT: Are LLMs Zero-shot Human Activity Recognizers?", FMSys, 2024
• Paper list: github.com/zhaxidele/Foundation-Models-Defining-A-New-Era-In-Human-Activity-Recognition

Indian Institute of Technology Kharagpur