HeadTrace: hardware-free head tracking
Tanmay Kulkarni, CREATE Community Day 2026
- Existing eye tracking solutions are often extremely expensive for people with disabilities (such as ALS or Duchenne muscular dystrophy)
- HeadTrace is an alternative webcam-based, external-hardware-free head tracking solution for people with mobility limitations
- Allows a user to operate a computer hands-free, using nose direction to control a cursor
- MSR Eye Tracking framework: Framework and drivers that makes it straightforward to integrate eye trackers into Windows - both through the Windows Eye Control and a set of standard APIs.
- Google MediaPipe: Framework for creating ML pipelines for tasks like face landmarking and segmentation (through a graph structure carrying out different subtasks).
Versions, testing, and prototyping
- The first version utilized a very overcomplicated projection algorithm on a "nose angle vector" derived from MediaPipe and camera matrices (without calibration)
- Through CWU’s SETC, one of their AAC (eye tracker) users and contributors tested it – however, his range of head motion was asymmetrical leading to poor performance
- Made me realize I needed calibration, so I tried an interpolation algorithm on the nose angle, which ran into poor quality data and projection math that didn’t help:
- I returned to this problem many months later, purely interpolating the plain nose coordinate (not any calculated angle) – yielding significantly better results
HeadTrace’s pipeline (see demo!)
Nose point per target (MediaPipe)
OLS + interpolation to solve nose-point to screen target formula
Metadata parameters calculate screen target
Screen point injected (Eye Tracking Stack)!
- Accuracy: Exploring alternate polynomials, different point selection, and changing the smoothing parameters to get maximum accuracy and require less repeated calibration
- Speed: running the FaceMesh every frame is expensive, not a huge latency at the moment but could be improved
- User experience: More user testing is absolutely necessary, and exploring new UI capabilities as well as forms for HeadTrace’s format (executable, app, website, etc.)
I would like to thank Central Washington University's SETC and their users and support with distributing the first version of HeadTrace; Ann Paradiso, Dwayne Lamb, and Chris O'Dowd for their endless support through my accessibility tech journey; Dr. Mar Gonzalez-Franco for her input; and of course, Harish Kulkarni for his extensive guidance and getting me started on all of this!