GOAL
To learn about how hardware and software can combine to solve the problem of people who can only communicate with their eyes
Biomechanical Systems
Assistive Technology with Eye Tracking
2025-08-18_v1.0
2
Lab development team
ALLIE SACK
College: Duke University (‘26)
Major: Biomedical Engineering
GRACE MARKOVICH
College: University of Notre Dame (‘25)
Major: Chemical Engineering
DIXON CLEVELAND
College: MIT & George Washington University
Major: Electrical Engineering
Industry Experience: Eyegaze Inc., Greenbriar Systems Inc, Adaptronics Inc.
BETSY FORTMAN
College: Purdue University
Major: Aerospace Engineering & MBA
Industry Experience: NASA, Sophic Synergistics
JOHN SEXTON
College: University of Notre Dame
Major: Electrical Engineering (‘23)
Industry Experience: Epic Systems, LifeDrive
3
Part 1
Using Hardware and Software to Communicate
As you go through the workbook check in the speaker notes for additional information!
4
Identifying the Problem
A primary goal of engineering is to contribute to the common good by improving people’s lives.
What Would You Fight For?
5
Medical Conditions Affecting Mobility and Speech
While these conditions affect muscular function, the movement of the eyes often remain accurate! How can we use this help these people communicate?
Amyotrophic Lateral Sclerosis (ALS)
Cerebral palsy (CP)
Multiple sclerosis (MS)
Rett syndrome
Spinal Muscular Atrophy (SMA)
Spinal Cord Injuries
6
Designing for Others
Engineers always need to consider the needs of their users when designing their solution
When you’re designing this type of technology, it’s crucial to remember the motivation for it’s creation
The technology is being created to improve the lives of real people
It’s important to get input from people who will actually be using this technology in order to produce the highest quality product
Thinking Like an Engineer
ANSWER ME! How can you obtain input commands from the user when physical movement is limited? What considerations need to be made when you are designing technology for someone with physical disabilities? (Timestamp 7:45 in the into video)
Type your answer here.
8
Developing a Solution: Eyegaze
Eyegaze uses eye tracking technology to allow users to control a device with their eyes
Components of the Eyegaze System:
How the system works:
What can people do with Eyegaze?
9
9
Using Eyegaze
What are three tasks that Eyegaze allows users to carry out?
How does this technology account for users with varying needs and abilities?
Type your answer here.
Type your answer here.
Watch the video on the right and answer the following questions
10
Eyegaze in Action
Watch as a person uses the Eyegaze system to communicate!
11
Part 2
Eye Tracking Technology
12
Look through this timeline of major achievements in eye tracking technology!
History of Eye Tracking
13
The Basic Terminology
Typical eye tracking device
The basic terminology of the eye gaze system
14
Light-emitting-diode (LED)
Aperture
Special Features of the Eye Tracking Camera
ANSWER ME!
Drag the terms in the word bank below to their correct label on the image.
Label the Parts of the Eye Tracking System!
Gaze Vector
Gaze Line
Eye Location
LED
Gaze Point
Video Camera
The Eye
The Pupil
Corneal Reflection
Terminology & Parts of the Eye Tracking System!
Glint Spot
Terminology & Parts of the Eye Tracking System!
Glint Spot
Pupil Center
Terminology & Parts of the Eye Tracking System!
What is a Vector?
An object that has both a magnitude and a direction.
Geometrically, we can picture a vector as a directed line segment, whose length is the magnitude of the vector and with an arrow indicating the direction. The direction of the vector is from its tail to its head.
Terminology & Parts of the Eye Tracking System!
Glint Spot
Pupil Center
Glint Pupil
Vector
Determining Where the Eye is Looking
21
Pupil-Center Corneal-Reflection (PCCR) Method
A pattern to note: As the eye rotates from the left of the camera to the right, the pupil location shifts from the left to right of the corneal reflection.
22
Try it Yourself!
Using the “insert” tool, place a colored-vector from the center of the glint spot to the center of the pupil. Draw the vector directly on the eye image.
Gaze: Type your answer here.
Gaze: Type your answer here.
Describe the direction the user is looking with respect to the camera (up, down, left, right, up-left, down-right, up-right, down-left, etc…)
23
Determining Gaze Point
How does a computer calculate the gaze point?
The image to the left is a plot of the brightness of a cross-section of an image of the eye
The bright pupil effect revisited:
Engineers use image analysis and complex algorithms to finding data peaks and calculate the gaze point
Determining Gaze Point
ANSWER ME! How can you account for the different geometries of people’s eyes? What are other examples from your everyday life of things that need to be calibrated? (Timestamp 14:10 in intro video)
Type your answer here.
25
Activity #1:
Creating your own 3D Eye Geometry Model
26
Visualizing the Eye and Eye Tracker Camera
Look at the images below to get a better understanding of the eye and eye tracker camera that you will be modeling
Anatomical Eye Cross Section
Eye Tracker Camera Cross Section
Camera Image of the Eye
27
Geometry Involved in Eye Tracking
Gaze Offset Angle (B):
28
3D Model
Thinking about the eye geometry in 3D space
Click the link to view the 3D geometry of the eye. https://www.geogebra.org/3d/bfgaeqgy
Move the pupil around to at least 2 different positions.
Take a screenshot of the geometry.
What do you notice about the glint vector? Did it get bigger or smaller from your first pupil placement to your second?
Place your first 3D diagram here
Place your second 3D diagram here
Insert your answer here:
ANSWER ME!
29
Your Task Today: Create a 3D Foam Model of the Eye Geometry
The instructions for this activity are intentionally vague. Be creative and use your resources to gain a firm understanding of eye geometry. There is no one correct way to construct your model!
INSTRUCTIONS
STUDENT KIT ITEMS | ||
1 kit: 3 students | ||
Item/Link | Quantity | Photo |
1 | | |
5 | | |
2 | | |
1 | | |
1 | | |
TEACHER’S KIT | ||
Materials will be distributed throughout the class. | ||
Item/Link | Quantity | Photo |
1 | | |
Materials You’ll Need
30
31
Building Your Model
INSTRUCTIONS
TIPS/TRICKS
Video instruction
*Please remember that this is one of many ways to create an eye model. Create a model the makes sense to you; the goal is to gain a deeper understanding of the geometry of an eye.
Other examples!
34
Your Final 3D Foam Models
Insert images of your final models here!
35
The Big Picture
Understanding eye geometry is essential to Eyegaze technology
Geometry forms the basis for modeling and algorithms:
Everyone’s eyes are unique
36
Applying the Glint Pupil Vector to Commands on the Screen
Workbook Activity: Motion
2. Draw the corresponding
wheelchair motion (path)
The computer stores a vector for each point on the screen. Imagine that the following vectors are stored along with the following commands.
Type your answers here:
Draw the expected wheelchair path here
ANSWER ME!
Workbook
Activity: Words
Type your answer here
What do these eye movements spell?
Now, imagine the vectors are tied to a letter instead of a movement.
Insert your answer here:
ANSWER ME!
39
Further Calculations
Now you’ve formulated the fundamental PCCR equations for computing the compound angle of the gaze line with respect to the camera’s sight line to the eye!
There are still many steps to compute the person’s gaze point on a computer screen:
How do you process an eye image to identify the glint and pupil?
How do you map out the pupil perimeter in the image and compute its center point?
Based on the image data, how do you compute the millimeter location of the eyeball in real space?
How do you compute the gaze line intercept on the computer screen?
40
Activity #2
Use Eye-Gaze to Determine Wheelchair Movements
IT’S YOUR TURN:
Now that you understand the technology, it’s time to interpret eye movements yourself in order to maneuver your wheelchair along a path
LifeDrive
Founder: John Sexton
Company’s Mission
LifeDrive Technology
42
Translating Eye Gaze into
Wheelchair Movements
Objective: To help a user interact with their surroundings, create a controller-template to be used to translate a user’s eye movements into actual outputs (wheelchair movements)
What you will be doing:
Materials
In this exercise, you will use a more primitive method of reading where someone is looking.
Experiment Set-up & Examples
User
Engineer
12 in
12 in
Recorder
Holding maze
(behind engineer)
Engineer
User
Template/
Controller
Maze
2) The Engineer reads the Eye Movements
3) Holds the Maze and Counts Correct & Missed Steps
45
Independent
Variables
As you work through this activity, consider the following variables that you can change
Distance between the user, engineer, and controller
Design of the controller template
Placement of the maze
Difficulty of the maze
CONSIDER
As you design various iterations of your set-up, how can you change each of these variables to get the best result?
46
Additional Considerations
Before you start this activity discuss the following
Dwell Time
Determine a ‘step’
What is your dwell time? |
|
How are you determining a ‘step’/when to stop? |
|
TEMPLATE
EXAMPLES
1
2
3
4
MAZE
EXAMPLES
Insert pictures of each of your templates here:
ANSWER ME!
User Distance to the Template | Engineer Distance to the Template | Total Moves | Total Correct Moves | Time through maze | Accuracy (%) (# correct moves) x 100 # of total steps | Notes - Type of Maze (easy, medium, hard) - Template description, number of buttons - How to know how many steps to move |
12” | 12” | 10 | 9 | 45sec | 9/10 x 100 = 90% | - Easy Maze - Eye Template had up/down/left/right/up-left/up-right/down-left/down-right/stop - Used dwell time to say go to more than 1 step in a given direction |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
Experiment Results
Interpreting Results
Insert your answer here.
Insert your answer here.
Insert your answer here.
Insert your answer here.
ANSWER ME!
58
The Big Picture
Considerations when creating Eyegaze technology
When developing eyegaze technology for a specific user, it is important to take all of the results from your experiment into account
In the real world, the typical parameters are as follows:
* These are averages for each parameter. It is important to remember that these values may vary for each user
ANSWER ME!
What are some possible limitations of eye tracking technology? How could they be addressed?
Type your answer here.
Limitations of Eye Tracking Technology
60
Additional Applications of Eye Tracking Technology
61
Additional Eye Tracking Uses
Eye tracking is not only used for communicating and driving, but can also be used for research, vision therapy, and diagnosing vision impairments.
EyE TRACKING GLASSES
RIGHT EYE
VISION THERAPY
62
Internet of Things
The Internet of Things (IoT) describes the network of physical items with embedded systems consisting of software and sensors that connect over the internet with other devices
Some examples of the IoT in everyday life are…
TO LEARN MORE…
Watch this short video to learn more about the Internet of Things!
Smart homes utilize the IoT by having various appliances connect to the internet so homeowners can control their homes using a smartphone or computer. If an eye gaze user can connect to the internet, and if they have smart devices in their home, then they can control the temperature, lights, and more using their eyes!
SMART WATCHES
SELF DRIVING CARS
TRACKING DEVICES
Using Eye Tracking Technology
ANSWER ME! List at least two possible other applications of eye tracking technology.
Type your answer here.
64
Reflect on Your Design and Results
ANSWER ME!
Write your answer here
What do you think went well when completing this activity?
Write your answer here
What is something you would do differently if you were to do this again?
Complete the mandatory 5-minute Exit Ticket by clicking HERE!
65
Continue to Explore
IF YOU LIKED TODAY’S BREAKOUT, �YOU MAY BE INTERESTED IN THESE TOPICS:
TYPES OF ENGINEERING RELEVANT TO TODAY’S EYEGAZE BREAKOUT:
66
Extension Activities
This section will provide an overview of the optional extension activities. These activities are an opportunity for students to dive deeper and ideate. The materials associated with the extension labs may not provide as many detailed instructions as the main lab activity.
67
Eye Geometry
Calculations
3D Cornea Model & Flashlight
Biomechanical:
Assistive Technology & Devices
E-TRAN Board
68
Extension #1:
Creating an E-TRAN Board
We are being the software algorithm today by taking a visual image and analyzing it for a given pattern to determine the necessary "letter/command" intended by the user's gaze.
What you’ll be doing:
Ideal option if you want to:
69
“Writing” With Your Eyes
Try this activity!
To the right is an actual keyboard from Eye gaze. Try writing out a sentence (“How are you doing today?”) with your eyes by looking at the letters, separating words using the spacebar, and then speaking it with the “SPK” key.
Time yourself to see how fast you can say the sentence, but be sure to pause briefly on each key so that the ‘computer’ knows you are pressing the key.
What did you learn when doing this? What could go wrong with the process? What does it feel like to have limited function?
70
Your Task Today: Create an E-TRAN Board
INSTRUCTIONS
What is an E-TRAN Board?
Materials
71
Creating Your Board
What you will be doing: use a clear plastic sheet and write out a keyboard (with an erasable marker), then create a procedure for interpreting what someone is saying
Things to consider when designing your board
72
Examples of E-TRAN Boards
73
Design and Build Your Board
1. Sketch a design for your board and insert a photo here.
2. Build your board and insert a photo here.
74
Testing Your Design
You will assign one person to be the ‘reader’ and one person to be the ‘writer’
Clear acrylic sheet & erasable markers
A third student can observe the interaction, monitor mistakes, and think of possible improvements
75
Collect Data
Have one partner use their eyes to spell out a phrase using your E-TRAN board while the other interprets what they’re saying. As you go through each word/phrase, record the number of guesses to correctly interpret each character.
Test # | Word or sentence being interpreted | Record the number of guesses for each character | Accuracy # of Correct / Total x 100 | Notes/observations |
Sample | Mary is happy | M A R Y I S H A P P Q 2 1 3 1 4 2 2 1 3 1 3 1 __ | 12/13 x100 = 92.3% | At end of each word, reader could guess next letter, middle letters were harder |
Test 1 | | | | |
Test 2 | | | | |
Test 3 | | | | |
Test 4 | | | | |
Analyze Your Results
ANSWER ME! Was it difficult to understand what your partner was trying to say/communicate? Was it harder than you thought it’d be? Easier?
Type your answer here.
Type your answer here.
ANSWER ME! Were your interpretations accurate? How could you improve the design of your E-TRAN board?
ANSWER ME! Did the reader find themself trying to guess the word before the writer finished? How could this be frustrating to a user if someone is looking over their shoulder?
Type your answer here.
77
Eye Geometry
Calculations
3D Cornea Model & Flashlight
Biomechanical:
Assistive Technology & Devices
E-TRAN Board
78
Extension #2:
PCCR Activity, Eye Tracker Model
What you’ll be doing:
Ideal option if you want to:
79
Experimenting with an
Eye Tracker Model
Objective: uncover a critical piece of an eye tracker’s optics geometry by modeling how the camera sees the eye
What you will be doing:
80
Materials:
Activity #1
(simulates corneal sphere)
Step 1: Build Your Model
81
82
For Reference:
Geometry Involved in Eye Tracking
Gaze Offset Angle (B):
83
Consider the following while testing your model eye.
In real life eye tracker hardware, the LED is mounted at the center of the camera lens
If the eye is too close to the camera, the glint spot will appear to be in the same spot no matter where the person is looking
Similarly, if you hold the flashlight too close to your ‘eye,’ the glint spot will always align with the center of the corneal sphere, no matter how you rotate it
Distance Between the Eye
and the Eye Tracker
The ‘eye’ should be looking directly at your eye when the glint spot is in the center of the pupil.
See if you should adjust the ‘eye’ model by rotating the ornament. When the glint is at the center of the pupil, your gaze line aligns with the eye model’s gaze line.
Thinking About Line of Sight
Can you begin to predict the eye’s gaze direction from the position of the pupil with respect to the glint?
Question to Consider
Reflect on Your Model
ANSWER ME! As you were experimenting with your eye model, what did you notice? Did anything stick out to you? Did you have any questions as you moved your eye around?
Type your answer here.
ANSWER ME! Insert a picture or drawing of what you observed with the cornea eye model (ornament & light). Be sure to label the center of the pupil, the center of the glint spot (reflection spot), and draw in the glint pupil vector.
Type your answer here.
85
Biomechanical:
Assistive Technology & Devices
E-TRAN Board
Eye Geometry
Calculations
3D Cornea Model & Flashlight
86
Extension #3:
Geometry Involved in Eye Tracking
What you’ll be doing:
Ideal option if you want to:
87
Determining where the gaze point is
Eye tracking technology uses complex geometric algorithms to determine exactly where the user is looking on the screen
Geometry helps relate the spatial orientation and location of the pupil center, the glint spot, the LED on the camera, and the gaze point using various angles and vectors
Information from the video camera central processing unit (CPU) processes the eye images, calculates the vectors/angles, and computes the gazepoint geometry
Geometry Involved in Eye Tracking
88
B
*
*
*
Gaze Offset Angle (B):
Angle (A):
Geometry Involved in Eye Tracking
89
Review of Right Triangles
In eye tracking, it is crucial to determine angles between vectors relating the eye, the camera, and the computer screen to each other
Pythagorean Theorem: a² + b² = c²
a = side of right triangle
b = side of right triangle
c = hypotenuse
These formulas are extremely helpful in determining missing side lengths or angles in right triangles
SOH CAH TOA: a mnemonic that helps one remember the meaning of the three common trigonometric functions - sine, cosine, and tangent functions
a
b
c
90
Inverse Trigonometric Functions
Using inverse notation will allow us to derive a formula to find the measurement of the angles in a triangle. The inverse equations can be seen to the right.
The inverse is the opposite of taking the sine, cosine, or tangent of an angle. The input is the side lengths, and the output is the angle.
91
Geometry Activity
Show your work here!
Show your work here!
Show your work here!
For each of the following triangles, determine the indicated angle or side length value
Find the values of d and θ
Find the value of θ and the missing side length
Find the 2 missing side lengths
92
Compound Angles
Thinking about geometry in 3D spaces
A compound angle describes a direction pointing in 3-dimensional space
It’s like pointing at something in the sky where you first specify a north/south/east/west ‘heading’ angle and then designate an ‘elevation’ angle specifying how high up to point above the horizon
In the sky pointing case, when you point straight up along a gravity line, (perpendicular to the ground) the ‘elevation’ angle (also known as the gaze offset angle) is 90°, and as you point further away from the gravity line, it gets smaller
This is important in eye tracking because the eyegaze system and the eye itself are 3D
CLICK HERE and watch this video to visualize compound angles
Find Glint Vector
& Calculate Magnitude
ANSWER ME!
Given two coordinates
Determine the Lengths
Δx = Horizontal Distance = (x₂-x₁)
Δz = Vertical Distance = (z₂-z₁)
Calculate
(4, 6)
(1, 3)
A =
Recall that:
tan(A) = value
A = tan⁻¹ (value)
A = tan⁻¹
ANSWER ME!
Tangent Equation: To Find Angle (A)
Enter values for the Δx & Δz on the diagram (from the previous page).
a = _____
b =_____
c
tan (A) =
3
3
3
3
3
3
(
(
45
Angle (A)
CLICK HERE to see how the equation was derived!
Gaze Offset Angle (B) =
7.8mm
B
ANSWER ME!
Given : Corneal sphere radius: 7.8 mm
4. Enter Δz on the diagram (from the previous pages).
b =
Enter here
Cosine Equation: To Find Gaze Offset Angle (B)
Enter & Calculate
sin (B) =
Enter here
Enter here
B = sin
-1
Enter here
Enter here
(
(
Enter value here
CLICK HERE
to see how the equation was derived!
97
Continue to Explore
IF YOU LIKED TODAY’S BREAKOUT, �YOU MAY BE INTERESTED IN THESE TOPICS:
TYPES OF ENGINEERING RELEVANT TO TODAY’S BREAKOUT:
Thank you!
Any text here?
Follow up info here?