Eye Tracking and Gaze-based Interaction�Introduction and Current Research Trends
Human-Computer Interaction 2 - 10.01.2018
WS 2017/2018
1
Picture source: http://www.black.ninja
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
New Interaction Methods are Desirable
The amount of interaction with computer devices increases.
Consequently we look for interaction methods which are:
Eye tracking is a promising technology as the eyes are quick and we use them with ease.
2
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Things We Can Do with Eye Tracking
Text 2.0
A project from DFKI (German Research Center for Artificial Intelligence)
3
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
A Smart Computer Needs Awareness for Gaze
4
Eye gaze is very important for human-human interaction
... and human-computer interaction should be like human-human interaction.
Taken from Milekic: The More You Look the More You Get: Intention-based Interface using Gaze-tracking
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
What the Eyes Can Tell
5
In HCI Gaze Can Be
Eye trackers keep advancing and getting cheaper!
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Expectations for Eye Gaze Interaction
Hopes
Fears
6
As eye movements are part of our social protocol we are able to control our eyes
Normally we look at the interaction element for input
Our eyes move constantly even if we sleep
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Eye Tracking Applications
7
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Eye Tracking Applications
8
Gaze Monitoring
Implicit gaze-based Interaction
Gaze-supported (Multimodal) Interaction
Explicit gaze-based Interaction
Päivi Majaranta and Andreas Bulling, “Eye Tracking and Eye-Based Human–Computer Interaction,” in Advances in Physiological Computing, Human–Computer Interaction Series (Springer, London, 2014), 39–65, https://doi.org/10.1007/978-1-4471-6392-3_3.
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Gaze Monitoring
9
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Gaze Monitoring
Studying visual attention
10
Nicholas S. et al. (CHI 2015)
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Gaze Monitoring
Support in collaborative environments
11
Akkil et al. (CHI 2016)
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Gaze Monitoring
Optimizing Interfaces
12
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Implicit gaze-based Interaction
Detecting intention/need for assistance
13
Karolus et al. (CHI 2017)
Walber et al. (CHI 2014)
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Implicit gaze-based Interaction
Gaze-assisted Photo/Video editing
14
Santella et al. (CHI 2006)
Jain et al. (ACM Trans. Graph. 34, 2015)
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Gaze-supported (Multimodal) Interaction
15
Chatterjee et al. (ICMI 2015)
Khamis et al. (CHI 2016)
Kumar et al. (CHI 2007)
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Gaze-supported (Multimodal) Interaction
16
Pfeuffer et al. (UIST 2014)
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Gaze-supported (Multimodal) Interaction
17
Stellmach et al. (CHI 2013)
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Explicit gaze-based interaction
18
Gaze-based authentication�EyePassShapes - De Luca et al. (SOUPS 2009)
Microsoft Patent 2014
Source: http://www.winbeta.org/news/microsoft-patents-eye-tracking-keyboard-could-use-technology-future-devices
Eye-based interaction with displays - Zhang et al. (UbiComp 2013)
Gaze input for the disabled
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Explicit gaze-based interaction
19
Esteves et al. (UIST 2015)
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Explicit gaze-based interaction
Spontaneous Interaction with Displays
20
Zhang et al. (CHI 2013)
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Explicit gaze-based interaction
Manual And Gaze Input Cascaded (MAGIC) pointing
21
Zhai et al., (CHI 1999)
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
How Does Eye Tracking Work?
22
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Eye Anatomy
23
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
The Human Eye - Movement Control
24
Images from: https://droualb.faculty.mjc.edu/Lecture%20Notes/Unit%203/muscles%20with%20figures.htm
and https://de.wikipedia.org/wiki/Augenmuskeln#Entwicklungsgeschichte
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
The Human Eye - Vision
25
The eye works similar to a camera.
However there are some differences:
From Duchowski, T. D.: Eye Tracking Methodology: Theory and Practice
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Types of Eye Movement
26
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
The Human Eye - “Movement Types”
27
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Fixations
28
http://www.prweb.com/releases/2005/03/prweb213516.htm
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Fixations - Interaction example
Typical application: Eye typing for handicapped (since the 80ies)
Problem: Midas Touch effect,
slow
29
https://www.youtube.com/watch?v=oXBKXRqxVnU
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Saccades
30
http://www.web-solution-way.be/3-marketing-internet/19-eye-tracking-google.html
https://en.wikipedia.org/wiki/Eye_tracking
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Saccades - Interaction Examples
Looking at the corners of the screen in a certain order
A series of forward saccades and a long backward saccade indicate reading activity.
31
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Smooth Pursuits
32
https://www.andreas-bulling.de/fileadmin/docs/vidal13_ubicomp.pdf
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Interaction using Smooth Pursuits
33
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Vestibulo-ocular Reflex
Interaction method using the vestibulo-ocular reflex:
gaze-based head gestures
34
https://www.youtube.com/watch?v=j_R0LcPnZ_w
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Optokinetic Nystagmus
35
Jalaliniya and Mardanbegi. CHI 2016. EyeGrip: Detecting Targets in a Series of Uni-directional Moving Objects Using Optokinetic Nystagmus Eye Movements.
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Vergence Movements
36
Florian Alt, Stefan Schneegass, Jonas Auda, Rufat Rzayev, and Nora Broy. 2014. Using eye-tracking to support interaction with layered 3D interfaces on stereoscopic displays. IUI '14
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Pupil Dilation
37
Bastian Pfleging, Drea K. Fekety, Albrecht Schmidt, Andrew L. Kun, A Model Relating Pupil Diameter to Mental Workload and Lighting Conditions in Proc CHI `16
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Eye Tracking Technologies
38
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Eye Tracking Technologies
39
Remote Eye Tracking
Head Mounted Eye Tracking
Tobii eye trackers
Pupil labs eye tracker
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Eye Tracking Technologies
40
| Remote Eye Trackers | Head Mounted Eye Trackers |
Tracks gaze.. | on displays | in natural settings |
Building responsive systems requires... | developing a software using an SDK | labeling the surroundings (e.g. adding QR-codes to products) |
Setup | Easy to setup | Cumbersome to wear |
Flexibility | Allows very limited movements | Allows free head/body movements |
Can users see the gaze data? | Yes | No, unless you provide the user with another display |
http://www.research-results.de/
http://www.useeye.de/
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Eye Tracking Producers
41
Commercial
Open Source
Open Eyes project
http://thirtysixthspan.com/openEyes/
Pupil Eye Tracker (open source software)
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Eye Tracking Techniques
42
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Eye Tracking Techniques
43
EOG Goggles
Tobii Glasses Eye tracker
Tobii Eye Tracker (stationary)
IR-PCR Eye trackers
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Eye Tracking techniques - Video-based
44
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Eye Tracking techniques - IR-PCR
45
Majaranta and Bulling 2014
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Eye Tracking techniques - EOG
46
EOG Goggles
Ear-pads based [Manabe et al. 2013]�(only detects looks to the left/right)
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Challenges
47
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Challenges
Interaction
Data Interpretation
Technical
48
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Midas Touch
49
Should the class end earlier?
Accidently choosing “No” while reading
(Midas Touch)
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Involuntary eye movements
50
In other words:
What is relevant data and what is not?
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Data Interpretation
Gaze fixation point != Visual attention
Humans are not necessarily paying attention to what they look at
51
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Data Interpretation
Eye fatigue
Kosch et al. (CHI 2018)
52
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Accuracy
53
http://www.cns.nyu.edu/~david/courses/perception/lecturenotes/eye/eye.html
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Environmental Artefacts
54
Makeup
Interference (e.g. electromagnetic)
Lighting conditions
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Calibration
55
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Calibration
56
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Calibration
+ Results in more accurate data.
57
On-screen calibration using the Eye-tribe
Calibration using markers
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Calibration-free Eye Tracking
58
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Calibration-free eye tracking
59
SideWays�(Zhang et al. CHI 2013)
Smooth Pursuits�(Vidal et al. UIST 2013)
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Calibration-free eye tracking
60
Nagamatsu et al. PerDis 2014
Drewes et al. 2007
(can also be done without calibration)
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Take-home Messages
Gaze is a promising modality
Gaze technologies still have some limitations
61
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Our Research Interests
62
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Gaze-based Interaction on Public Displays
63
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Challenges of Gaze-based Interaction with Large Public Displays
64
1. Position
2. Movement
3. Calibration
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
EyeScout: Active Eye Tracking for Position and Movement Independent Gaze Interaction with Large Public Displays.
65
Project by Alexander Klimczack and Martin Reiss
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
GazeDrone: Using Drones as Mobile Remote Eye Trackers for Public Displays
66
Project by Anna Kienle
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Text-based Calibration of Eye Trackers
67
Project by Ozan Saltuk
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Text-based Calibration of Eye Trackers
Seamlessly integrating eye tracker calibration to public display application
68
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis
Our Research Interests
69
Proficiency Awareness through Gaze Features
Eye Tracking and Gaze-based Interaction - Heiko Drewes, Jakob Karolus, Mohamed Khamis