1 of 10

Beliefs, Relationships, and Equality:

An Alternative Source of Discrimination in a Symmetric Hiring Market via Threats

By: Serafina Kamp, Therese Nkeng, Vicente Riquelme, Ben Fish

University of Michigan – Ann Arbor

2 of 10

Motivation

Fairness metrics measure this

Problem Statement: What are the sources of discrimination exhibited in machine learning?

2

Group A

Group B

Testing

≠

3 of 10

Exogenous Discrimination

3

Definition: A classifier satisfies Equalized Odds if the TPR and FPR are equal across given groups.

Group A

Group B

Unconstrained

Constrained

Unconstrained

Discriminatory outcomes can be traced to exogenous sources of discrimination

One such metric is Equalized odds. A classifier is said to satisfy Equalized Odds if TPR and FPR are equal across groups.

So, suppose we train a model on positive and negative instances of group A and group B. we might get a classifier as follows since there are more instances of group A than group B that the classifier favors to classify correctly. The training distributions of the two groups are different in the unconstrained case which justifies the use of the Eq Odds to create the following constrained classifier (one for group A, one for group B). So, if the training distributions are the same, there is no need to “fix” the unconstrained classifier

This implies that the discriminatory outcome was a result of an exogenous source of discrimination (the dataset) the training distributions. In fact, other fairness metrics including individual, group, counterfactual and even statistic and taste based discrimination address some type of exogenous discrimination.

This is not to say that current fairness metrics are failing -- it is very important to address these sources of discrimination. Rather, we are interested in expanding their scope with the following question.

4 of 10

An Alternative Source

Problem Statement: Is it possible for an ML model to have endogenous discrimination?

4

Discriminatory Outcome

Testing

Training

Dataset

5 of 10

Equilibrium and ML

5

Learn strategies

Strategies in equilibrium

Current Analysis

Unlike exogenous sources, endogenous discrimination is the result of the interactions of the agents during the learning process. So, we are going to use game theory to demonstrate the possibility of endogenous discrimination. Here, ML can be used by agents to learn strategies in a game and it is considered good for algorithms to converge to strategies that are in equilibrium since this is where each agents’ strategy is the best response to all others’ strategies

Specifically, to answer our research question, we came up with a game theoretic model that controlled for exogenous sources of bias that still had a discriminatory outcome at equilibrium to show that discriminatory strategies could have been learned by an ML model endogenously. Notably, this type of discrimination can’t be prevented by any of the fairness metrics…

6 of 10

Approach

Goal: Demonstrate an unfair outcome with no exogenous bias in a hiring market

6

Firms

Candidates

Bargaining in the Market

Proposal Acceptance

Check for discrimination among the accepted proposals

Firm

Candidate

Firm,

Candidate

Proposes (y, 1-y)

Accepts or

rejects 1-y

Decides to opt out or not

Becomes the new proposer

…

Our model is a simple hiring market where agents (firms and equally all equally entitled to the surplus) in the market participate in rounds of a bargaining game to determine the split of a surplus generated by the candidates employment in terms of what percentage goes to the firm and what percentage goes to the candidate-- this is like wage negotiation.

Explain market with graphics -- we come up with strategies that tell agents how to act in the bargaining phase. In our work, we actually focus on showing the strategies are in a stronger type of equilibrium (SPE) that is particularly interesting because it does not allow for non-credible threats (no empty threats) which is key in our model for balancing power and allowing for a range of possible payoffs to the agents.

What we are interested in is examining the accepted proposals and check if there is discrimination. Specifically, we can check and see if it is possible for one group of candidates to receive more of the surplus than another group. Because they’re equally skilled, any difference in payoff can be considered discriminatory.

7 of 10

Results - Main Theorem

There exist strategies that are in equilibrium where there is a gap in the expected payoffs between two types of candidates

7

½

0

1

A Candidates

B Candidates

Expected payoffs

Firms

Time discount factor

Matching cost

Proportion of B candidates

Our main theorem says that READ STATEMENT

recall that we are working with percentage splits of the surplus, our payoff values are between 0 and 1.

So, say we have two types of candidates: A candidates and B candidates that are grouped in an arbitrary way. Then, it is possible to find strategies that are in SPE for the A candidates to receive less than half of the surplus while B candidates receive more than half of the surplus.

It may be tempting to use something like equalized odds to address the unequal outcome, but recall that the strategies are in equilibrium, so it’s not clear how to “correct” the outcome since everyone is already playing their best response to the other agents. Further, these strategies developed out of an equal starting point, so equalizing payoffs at any point in time would not prevent this unequal outcome from developing again.

8 of 10

Results - Main Theorem

There exist strategies that are in equilibrium where there is a gap in the expected payoffs between all candidates and firms

8

½

0

1

A Candidates

B Candidates

Expected payoffs

Firms

9 of 10

What Happened?

9

The strategies reveal endogenous discrimination is possible!

Future Work

Current Analysis

Learning

Strategy

A candidates accept < ½

B candidates accept > ½

Firms offer < ½ to A candidates and > ½ to B candidates

Dataset

Learning

what happened? we started with a market of candidates and firms where A and B candidates were equally skilled and all agents were equally entitled to the surplus. Then, each agent went through some learning process to learn how to interact in the market and we proved that their strategies were in a strong type of equilibrium. We found that these “good” strategies they learned can be discriminatory and did not rely on any exogenous factors! Thus, there’s no justification like we had in the first example to use equalized odds (or any other metric) to fix the discrimination. Thus, in this analysis, we showed that the discrimination exhibited in the strategies must be something endogenous to the learning process.

firms are able to credibly threaten some or all candidates and if all firms do that, then the candidates have no viable outside options (probably skippable)

or maybe their beliefs were different…

For future work, we hope to understand exactly how the learning process leads to discriminatory strategies.

10 of 10

Thank you

I look forward to any follow up discussions at the poster session!

contact email: serafibk@umich.edu