CounterFace: A Synthetic Face Dataset

for Fine-Grained Counterfactual Evaluation

of Face Recognition Systems

Guruprasad Viswanathan Ramesh, Ashish Hooda, Shimaa Ahmed,

Harrison J. Rosenberg, Ramya Korlakai Vinayak, Kassem Fawaz

WI-PI Lab, University of Wisconsin-Madison

​

​

​

Wi-Pi Lab

Face Recognition is everywhere

Increasingly used for identity decisions with real consequences.

1

Face Recognition in Action

Face Image 1

Face Image 2

Face Recognition Model

Face Recognition Model

Embedding 2

Embedding 1

Similarity Metric

Same Identity

Different Identity

2

Face Recognition Failures

• Demographic Bias in Face Recognition has been widely reported

There is a need for fine-grained diagnostics.

• Factors like aging, lighting, pose, and facial hair have also shown to cause failure

3

Counterfactual Evaluation

Addition of attribute “mustache”

g_ai

Source x

Edited g_ai(x)

• Explainable evaluation paradigm to assess robustness, biases and algorithmic fairness

​

• Uses counterfactual examples obtained by making small human-interpretable changes to an input

4

Need for Synthetic Counterfactual Examples

• Real counterfactual examples are hard to obtain at scale.
• Synthetic counterfactual examples using modern generative models is the alternate
• However, synthesized counterfactuals are not always reliable, and they need to be verified

5

While prompting different editing techniques to add the attribute “facemask”

Limitations of Prior Work

​

• Human-in-the-loop → costly, does not scale

​

​

• Checked only the intended attribute — confounders (other unrelated attributes) were not considered

​

​

​

Verifier checks

✓ “is attribute a_i added or removed?”

6

Prior Work: [1] Balakrishnan et al. (ECCV 2020) [2] Liang et al. (ICCV 2023)

Our work: Automated Verifier + Controlling Confounders

Automated verifier

Verifier checks three requirements:

✓ Validity

✓ Correctness

✓ Specificity

​

• A validated automated verifier allows to scale to more attributes and demographics

​

• Also allows minimizing confounders while selecting counterfactuals

​

• We generate CounterFace, spanning 20 attributes and 8 demographics

​

​

Source face x

Edited face g_ai(x)

GAN Editor

7

1. Generate “Source Faces” for all 8 demographics (150 identities per demographic, 6 seeds per identity)

East Asian Male

White Male

Indian Male

Black Male

East Asian Female

White Female

Indian Female

Black Female

Generating CounterFace: Generating Candidate Face Pairs

8

2. Modify source face by applying or removing the intended attribute

Generating CounterFace: Generating Candidate Face Pairs

Attributes

or

…

…

Source Face

Attribute Detector

Remove

Add

Add

Modified Faces

9

1. Check Counterfactual requirement: Validity

Generating CounterFace: Rejecting Non-Compliant Candidates

Modified Faces

Artifact

Detector

10

2. Check Counterfactual requirements: Correctness and Specificity

Generating CounterFace: Rejecting Non-Compliant Candidates

Attribute Detector

Attribute

modified correctly

Other attributes change

Attribute

modified correctly

Other attributes do not change

Passes

Correctness

Fails

Specificity

Passes

Correctness

Passes

Specificity

Reject Candidate

Select Candidate

11

• 20 attributes & 8 demographics (160 attribute–demographic combinations)

• 11,821 face pairs (75 pairs for all but 7 combinations)

• 900-respondent post-hoc user study on a subset:
• 96.9% pairs retain identity
• 84.04% pairs meet all 3 counterfactual requirements

CounterFace: Dataset Details

12

• Metrics: FNMR at FMR=0.1% after tuning decision threshold for each model/system

​

Evaluation: Models & Metrics

• We evaluate four open-source models and two commercial systems

13

AWS & AdaFace are the most robust across all models

Results: General Performance Trends

14

Older open-source models, FaceNet and ArcFace show highest FNMR

​

Results: General Performance Trends

15

Results: Attribute-wise Performance

All models are less robust to rare attributes and attributes that occlude facial features

Models are fairly robust to non-occluding and peripheral attributes

16

Results: Attribute-wise Performance

Top2 models are also less robust to occluding attributes

17

Case Study: East Asian Male + Thick Beard

Even the top 2 models, AWS and AdaFace are less performant for East Asian demographic when thick beard is applied.

18

Ablation: Importance of Minimizing Confounders

All models lose performance with the relaxed dataset

Facial hair and occlusion attributes cause biggest FNMR delta on average

These results highlight the need for the Specificity Criterion that helps in minimizing impact of confounding factors in edits

​

19

Dataset available upon request

Thank You, Any Questions??

20

Conclusion