1 of 15

Modelling Human Biases and Uncertainties� in Temporal Annotations

Taku Yamagata, Emma L. Tokin, Benjamin Arana Sanchez, Ian Craddock, Miquel Perello Nieto, Raùl Santos-Rodríguez, and Peter Flach

* taku.yamagata@bristol.ac.uk

WUML 2023

2 of 15

Overview

  • Background
      • SPHERE project / Annotations
  • Motivation
      • What is the issue?
  • Our approach
      • How do we tackle the issue?
  • Evaluation
      • How well it works?
  • Discussion
      • What are the open issues for this approach?

3 of 15

SPHERE (Sensor Platform for Healthcare in a Residential Environment)

Wearable device

Silhouette Sensor

Environmental sensors

Light

Temperature

Humidity

Barometric pressure levels

Passive InfraRed (PIR)

3-axis accelerometer

RSSI

Data collection

Appliance Monitor

Kettle

Microwaves

TV etc.

4 of 15

SPHERE (Sensor Platform for Healthcare in a Residential Environment)

Ground-truthing

  • Technician walkaround (location)

Technician walkaround a house, annotating each corner of each room.

  • Wearable scripted test (activity)

​

  • Self-reporting (activity)

Participants fill in a form - Start / End time for certain activities (sleep, meal, shower etc.)

  • NFC tag (location)

Installed two NFC tags on each door frame (going in and out). Then, each participant receives one annotation phone, and is asked to tag the NFCs when crossing every door.

  • Head camera (location, activity)

Participant walk around house with head camera. Post-hoc annotation

  • Etc.

​

​

5 of 15

Motivation

Histogram of annotations start/end time (minutes) for an activity

  • Most of them are on a certain grid – 30mins. 15mins. … intervals
  • Find out what time resolution was used, then infer the distribution of the true timing

6 of 15

Our approach (overview)

  1. Estimate which time resolution the annotator used
  2. Assume a probability distribution of the true timing
  3. Generate soft label

08.30

07.00

09.00

10.30

08.00

09.15

07.30

Tend to use 30 min. resolution.

15 min. resolution for 09.15.

​

1.

Annotations for

shower start timing

8.30

30 min.

2.

3.

8.30

Soft label

The probability of the event is happening

time

time

7 of 15

  • We prepared five annotation categories

​

​

​

​

​

​

Our approach (categories)

8 of 15

Our approach (model)

​

​

​

​

 

Annotator’s habit

Resolution category used for i-th annotation

True timing

Annotation

Annotation index

9 of 15

Posterior for H

Annotator’s habit

Resolution category used for i-th annotation

True timing

Annotation

Annotation index

10 of 15

Posterior for Ci

 

Annotator’s habit

Resolution category used for i-th annotation

True timing

Annotation

Annotation index

11 of 15

Soft label

 

Start timing

End timing

12 of 15

Evaluation (data / method)

 

annotated

start time

soft-label

hard-label

time

13 of 15

Evaluation (results)

14 of 15

Discussion (future works)

 

15 of 15

Thank you!