The Math of FaceID
Lesson 1: Images as Data
What is Machine Learning?
Why FaceID?
Take a look at your cell phone, and think for a moment about the information it holds.
From photos, contacts, your internet search history…lots of data is stored within your phone’s memory!
Furthermore, those data are extremely valuable. Technology companies are motivated to help people feel secure about storing information in their phones.
First came passcodes.
Next, fingerprint recognition.
And finally, FaceID.
Our goal is to understand how a recent innovation, FaceID, works. Although we don’t see it, much of FaceID relies on ideas from Statistics.
At its core, FaceID combines statistics and computing, which makes up a new-ish field called machine learning.
In machine learning, computers learn patterns that helps them to make decisions without being explicitly programmed.
Machine Learning
Computing
Statistics and Math
Motivation for machines to “learn” began more than a half century ago.
Scientists at the time were exploring ideas such as, “How to get a computer to beat a human player at Checkers.”
Samuel Arthur, in 1959, training a computer how to play checkers.
Now, machine learning is everywhere!
Many people aren’t even aware of machine learning, or just how heavily it uses math and statistics!
Goals of this Project Unit
Goals for Lesson 1
Images and Numbers
First, let’s make sure that we’re all on the same page when it comes to FaceID.
FaceID begins when a user sets up their phone. The phone takes initial photos called reference images of their face. The phone stores the reference photos in memory.
First, let’s make sure that we’re all on the same page when it comes to FaceID.
FaceID begins when a user sets up their phone. The phone takes initial photos called reference images of their face. The phone stores the reference photos in memory.
Then, each time the user wants to unlock the phone, the phone takes a current image and compares it to the reference image. If the photos are “similar enough”, the phone unlocks.
When FaceID unlocks a phone, it asks, “Is the current image the same as the saved reference images of that person?”
But, how does it actually do that with math?
How can you describe a face with numbers?
Idea 1: A list of facial features using categorical variables
Idea 2: Use numbers to describe distances on faces.
Idea 3: Use numbers to describe pixel brightness.
Part of the picture described with pixel brightness values
[Optional: What is a pixel?]
A pixel is the most basic a unit of length in a digital image.
Each one of the little squares here is a pixel!
Pixels are represented by numbers according to their brightness.
In a black-and-white image, each pixel is represented by a number from 0 to 255.
Pixels are represented by numbers according to their brightness.
In the picture above, each square is a pixel and is represented by a 0-255 number that the computer reads in as a list.
So, whereas we see images and pixels, computers just process long lists of numbers when fed images.
Color images take up three dimensions: red, green, and blue. Each dimension is represented by a 0-255 value.
The cube above reinforces the idea that every pixel in a color image is made up of three numbers.
See pixel breakdowns for yourself! Navigate to the pinetools webpage and see an image broken into its RGB components.
How to decide if numbers are similar?
What does FaceID need to decide?
“Is a new test image similar to the reference image?
Test image
Reference image
What does FaceID need to decide?
“Is a new test image similar to the reference image?
Test image (current photo of you)
[126, 155, 75, 22, 21, 90, 35…]
Reference image (stored in your phone)
[177, 123, 80, 36, 44, 80, 55…]
The phone doesn’t see faces. It uses a numerical representation (the variables that describe the face picture)
What does FaceID need to decide?
“Is a new test image similar to the reference image?
Test image (current photo of you)
[126, 155, 75, 22, 21, 90, 35…]
Reference image (stored in your phone)
[177, 123, 80, 36, 44, 80, 55…]
The phone doesn’t see faces. It uses a numerical representation (the variables that describe the face picture)
“Are the test numbers similar enough to the reference numbers?”
How could we use math to determine if the sets of numbers are similar?
What does FaceID need to decide?
In Statistics, we can think about error as a deviation that we expect to see. There have been several ways that we have thought about error in the past:
What does FaceID need to decide?
Let’s hone in on one measure, the mean-squared error, which is like the average amount of error (or, “difference” in our case) from what we’d expect.
What does FaceID need to decide?
Going back to this photo, we could, for example, use mean-squared error to think about the average amount of difference between this person’s test and reference photos.
What does FaceID need to decide?
Idea 1: Compare difference between each individual pixel.
Idea 2: Compare difference between facial features’ distances.
Using either method on the previous slide will produce some amount of difference - after all, this guy looks similar in both photos, but not exactly the same.
How might companies like Apple decide what a “good amount” of error is?
If we measure error numerically (say, with the mean-squared error), we need to determine how much error is acceptable so that the phone:
Saved image of user
Still the user, but different pose (should unlock)
Not the user (don’t unlock)
Experimenting with Image Similarity
Activity: Try it and see!
With a partner, you will test two simplified methods for FaceID: comparing pixel brightness and comparing distances on faces.
Using the widgets provided, you will capture reference and test images of yourself and a partner. Your goal is to find the lowest MSE that produces correct decisions for as many cases as possible. In this context, a “correct decision” means that the phone unlocks for its owner and remains locked for everyone else (in this case, your partner).
When you have found your best overall MSE, take a screenshot of your results and post them on this class powerpoint. Then summarize your findings in a few bullet points by responding to the prompts on the slides.
Whole-Group Discussion
Share out and Discussion
Which method worked better, and why?
What was the best MSE that you obtained?
Any interesting fail cases?
Wrapping Up
Today, we introduced FaceID and some statistical measures that can be used to answer the question, “Is this picture me or not me?” from a computer’s perspective.
IMPORTANT: neither of those methods are how FaceID actually works, but it’s good to start thinking about how computers can make comparisons. More on this next time!