1 of 19

Optical Gaze Tracking�with Spatially-Sparse�Single-Pixel Detectors

Richard Li ^*1,2・Eric Whitmire ^*1,2・Michael Stengel ²・Ben Boudaoud ²・�Jan Kautz ²・David Luebke ²・Shwetak Patel ¹・Kaan Akşit ^2,3

¹ University of Washington・² NVIDIA Research・³ University College London

2 of 19

3 of 19

We are motivated by...

Towards practical and deployable gaze tracking

Accuracy

Refresh Rate

Power Consumption

Signal Processing

Cost

Form Factor

< 0.5°

> 250 Hz

< 50 mW

4 of 19

Related Work

Camera-based

Single-pixel detectors

Simulations

Other

Pupil Labs (camera)

M. Kassner, et al. Pupil: An open source platform for pervasive eye tracking and mobile gaze-based interaction

5 of 19

Related Work

Camera-based

Single-pixel detectors

Simulations

Other

Battery-free eye tracker on glasses

T. Li and X. Zhou

6 of 19

Related Work

Camera-based

Single-pixel detectors

Simulations

Other

InvisibleEye

M. Tonsen, et al. InvisibleEye: Mobile eye tracking using multiple low-resolution cameras and learning-based gaze estimation

7 of 19

Related Work

Camera-based

Single-pixel detectors

Simulations

Other

Jins Meme (EOG)

K. Kunze, et al. Quantifying reading habits: Counting how many words you read

Scleral Search Coils

D. A. Robinson. A method of measuring eye movement using a scleral search coil in a magnetic field

8 of 19

Single-Pixel Detectors for Gaze Tracking

Fewer pixels sensed

Fewer pixels processed

Less power consumed

Preserve privacy

First, I will introduce the concept of single-pixel detection of gaze.

>

Normally, camera-based gaze tracking systems obtain an image such as this.

Computer vision algorithms then extract relevant landmarks for estimating gaze orientation.

>

However, we hypothesize that not every pixel in that camera image is required to make that estimation.

In this image, the image of the eye is subsampled to 1% of the number of pixels, yet general features such as the pupil and eyelid are still visible.

>

Even when the image is subsampled by another half, the pupil remains clear.

>

The primary advantage of such a sensing technique is that since fewer pixels are sensed,

>

then fewer pixels need to be processed, reducing computational requirements,

>

then overall less power is consumed, addressing one of our main motivations for this work.

>

In addition, by using only the amount of pixels needed for inferring gaze orientation, we would potentially also remove the sensing of any biometric information, helping to preserve privacy at the hardware level.

9 of 19

Design Process

1

2

3

4

5

Initial test

Sensors have direct line-of-sight to the eyes

Blocks the user’s field of view

10 of 19

Design Process

1

2

3

4

5

Take the sensors out of the user’s field of view

Signals were not as clear anymore

11 of 19

Design Process

1

2

3

4

5

Previous work suggested that sensors below the eye are most effective

Realized we needed to do more groundwork ourselves

12 of 19

Interlude: Simulation

Design recommendations:

Maximize variety in perspective
Minimize emitters so detectors do not saturate
Co-locate emitter and detector to maximize signal

13 of 19

Design Process

1

2

3

4

5

Introduced sensors to the left and right sides

Included hot mirror and camera for debugging

14 of 19

Design Process

1

2

3

4

5

Final design for NextGaze

8 photodiodes (receivers)

3 LEDs (emitters)

One LED on at a time, surrounding photodiodes take measurement

15 of 19

NextGaze: Design & Implementation

Accuracy

Refresh Rate

Power Consumption

1.68°

400 Hz

16 mW

16 of 19

LED2Gaze: Design & Implementation

Duplex LEDs for both emitting and receiving

Accuracy

Refresh Rate

Power Consumption

1.57°

250 Hz

800 mW

Next, for reducing the number of components used, we leveraged a technique for duplexing LEDs such that they can be used for both emitting and receiving.

In doing so, we developed the LED2Gaze prototype, round robins through the LEDs, with one acting as an receiver while the remaining LEDs are used to make light measurements. This approach also increases the flexibility of sensing perspectives, since a single point can act as both.

At the same time, we also found that a simple Gaussian process regression model to be more effective than the previous model, while also requiring less training data.

This resulted in a slightly improved accuracy of 1.57 degrees.

However, while both refresh rate and power consumption were compromised, we believe that improved engineering quality could improve both of these factors significantly.

17 of 19

Conclusions

Accuracy

Refresh Rate

Power Consumption

Signal Processing

Cost

Form Factor

< 0.5°

> 250 Hz

< 50 mW

In conclusion, we have shown how the use of a

>

simulation framework was used to inform the design of two functional prototypes:

>

NextGaze and

>

LED2Gaze.

The latter is also, to the best of our knowledge, the first system to leverage the sensing capabilities of LEDs, reducing the requirement of an extra, separate component for sensing as well.

With respect to the initial motivating factors we presented at the beginning,

>

we did not meet our expectation for accuracy.

>

However, both systems did have sample rates of 250 Hz,

>

and NextGaze consumed less power than 50 milliWatts.

>

In using the Gaussian process regression model for LED2Gaze, we have offered a computationally lighter weight alternative to neural networks as a modeling technique.

>

By using less components can be saved,

>

And the use of discrete components allows for more flexibility in placement and topology, enabling more acceptable form factors.

18 of 19

Thank you for listening!

Find the paper and these slides at: https://Lichard49.github.io/nextgaze

Email with questions and comments at: Lichard49 @gmail.com

19 of 19

Discussion and Future Work

Deployability

User and Session Dependence
Synthetic Data
Wearable Accessories
Compensating for Slippage

Applications in User Interfaces

Human perception
Interaction techniques

Finally, we raise the following points as food for thought:

While we have made progress towards practical gaze tracking, a number of challenges remain before such systems can be deployed.

These include user and session dependence, the possible use of synthetic data for more robust training, offset the potential issues of wearable accessories such as mascara or eyelashes, and compensating for device slippage over time.

In addition, identifying more applications of gaze tracking that require different gradations of accuracy and sample rate can allow systems that are “good enough” to be deployed sooner.

Such applications might be found in human perception, such as animating a social avatar, or in foveated rendering, which has benefits both from an interaction side and from a computation side.