1 of 21

Tackling Algorithmic�Disability Discrimination�in the Hiring Process:� �An Ethical, Legal and Technical Analysis

Maarten Buyl*, Christina Cociancig*, Cristina Frattone*, Nele Roekens*

ACM FAccT 2022

Seoul 서울, June 21-24

* equal contribution

2 of 21

Background

Algorithmic discrimination�Unjust discriminatory manner in which automated decisions impact members of a certain protected group/minority

ACM FAccT 2022

2

Seoul 서울, June 21-24

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

adobestock.com

Okay so our focus is quite specific and probably requires some background.

When talking about algorithmic bias or discrimination, we mean the “Unjust discriminatory manner in which automated decisions impact members of a certain protected group or minority”.
Ofcourse, tackling this issue is usually referred to as AI fairness. AI fairness has been at the forefront of ethical AI research for years, and there is another presentation on it later today.
But if you are new to this topic, then it’s important to know that algorithms mostly discriminate because they learn from discrimination bias in the data. And the bias usually got into the data because the data came from biased humans making discriminatory decisions. So much of algorithmic discrimination is in fact a reproduction or magnification of human biases.

3 of 21

Background

Algorithmic discrimination�Unjust discriminatory manner in which automated decisions impact members of a certain protected group/minority
AI-driven automated hiring systems (AHSs)�e.g. CV screening, job interview analysis, etc.

ACM FAccT 2022

3

Seoul 서울, June 21-24

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

adobestock.com

4 of 21

Background

Algorithmic discrimination�Unjust discriminatory manner in which automated decisions impact members of a certain protected group/minority
AI-driven automated hiring systems (AHSs)�e.g. CV screening, job interview analysis, etc.
Focus on candidates with disabilities (PWDs)
Understudied in the literature on AHSs
Specific ethical, legal, and technical issues

ACM FAccT 2022

4

Seoul 서울, June 21-24

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

adobestock.com

5 of 21

5

“people with disabilities are on average ~41% less likely to receive a positive response to a job application”

An additional reason to study discrimination against people with disabilities comes from this very recent paper by researchers at Ghent University.

They conducted a meta-analysis of all forms of discrimination in hiring (so not just in automated systems, but in `classic’ hiring as well), and they indeed found that discrimination in hiring occurs quite often
But the most common form of discrimination in this analysis was against people with disabilities. If a job applicant reported a disability when applying to a job, they were 41% less likely to receive a positive response. It was slightly worse for physical disabilities like wheelchair use or deafness, but it also occurred often for mental disability, which here included former depression and autism spectrum disorder.��Addressing discrimination against people with disabilities is therefore both theoretically and empirically very relevant, despite the fact that it’s often overlooked.

6 of 21

Ethics of non-discrimination

ACM FAccT 2022

6

Seoul 서울, June 21-24

Moral obligation

Discrimination against PWDs is distinct compared to other groups:

Separation and segregation
People in direct surroundings seldomly share disability
Physical toll and economic cost

Anti-discrimination law does not suffice to protect PWDs

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

As a first step in our analysis, we looked at ethical implications of hiring with algorithms.

Prior research shows that discrimination against PWDs has several unique characteristics:

For one, PWDs are systematically separated and segregated. For example, this happens with the well-intentioned aids of separate entries, buses and bathrooms, which lead us to involuntarily ban PWDs from our sight and thus alienate them.
Another unique characteristic lies in their social environment: PWDs seldomly share their “otherness” with people they are involved with, like family and friends. They have to actively seek out a community to interact with people that know what they’re going through.
PWDs also tend to deal with pain, fatigue and muscle weakness. To manage those, assistive technology can help, but is expensive and therefore not accessible to everyone.

On top of this, we believe that anti-discrimination law may not suffice to adequately protect PWDs. We therefore have a moral obligation to look for additional ways to protect them

7 of 21

Ethics of non-discrimination

ACM FAccT 2022

7

Seoul 서울, June 21-24

Moral obligation

Information

Discrimination against PWDs is distinct compared to other groups:

Separation and segregation
People in direct surroundings seldomly share disability
Physical toll and economic cost

Anti-discrimination law does not suffice to protect PWDs

Disclosure of a disability requires trust
Employers are not aware of legal possibilities for offering reasonable accommodation

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

8 of 21

AI Ethics

ACM FAccT 2022

8

Seoul 서울, June 21-24

Area of research fails PWDs

AI aims to generalize “the norm” → PWDs cannot participate in the datafication that is required
PWDs are exposed to digital discrimination, e.g. failing voice recognition
PWDs are almost exclusively mentioned as therapeutic users of AI

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

9 of 21

AI Ethics

ACM FAccT 2022

9

Seoul 서울, June 21-24

Area of research fails PWDs

AI aims to generalize “the norm” → PWDs cannot participate in the datafication that is required
PWDs are exposed to digital discrimination, e.g. failing voice recognition
PWDs are almost exclusively mentioned as therapeutic users of AI

Quantitative literary study of 82 AI Ethics guidelines shows:

Only 27% of guidelines even mention “people with disabilities” or similar phrases
None of them acknowledge the opportunity to actively involve PWDs to progress ethical principles in AI

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

10 of 21

AI Ethics

ACM FAccT 2022

10

Seoul 서울, June 21-24

Area of research fails PWDs

AI aims to generalize “the norm” → PWDs cannot participate in the datafication that is required
PWDs are exposed to digital discrimination, e.g. failing voice recognition
PWDs are almost exclusively mentioned as therapeutic users of AI

Quantitative literary study of 82 AI Ethics guidelines shows:

Only 27% of guidelines even mention “people with disabilities” or similar phrases
None of them acknowledge the opportunity to actively involve PWDs to progress ethical principles in AI

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

Understanding of discrimination is consequentialist

EU Law considers the intent to discriminate irrelevant → consequentialist
Consequentialist theory can help re-calibrate our moral compass to AI ethics

11 of 21

EU Law: Employment Equality Directive (EED), General Data Protection Regulation (GDPR), AI Act Proposal (AIA)

ACM FAccT 2022

11

Seoul 서울, June 21-24

Risk of unlawful discrimination

Right reasonable accommodation applies to the entire hiring process (3(a), 5 EED)

Note: Violation of right to reasonable accommodation = discrimination

Presumption of discrimination → reversed burden of proof (10 EED)

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

In the legal analysis, we take the viewpoint of EU law, which also allows us to consider the impact of the AI Act Proposal that was also discussed yesterday.

Again, I most definitely do not have a legal background, so please check the paper for the full details.

But let’s give it a go.

We start by asking whether AHSs are discriminatory and the answer is yes.

In fact, people with disabilities have a specific right to (what is called) reasonable accommodation under the Employment Equality Directive (the EED!). Reasonable accommodation means that employers shall take appropriate measures, if needed, to enable a person with a disability to work for them. Only reasonable accommodation is expected, which means that it should be within the limit of what is not disproportionately burdensome for employers.

Now, many people know about reasonable accommodation and assume this is about helping an employee when they are already hired. However, the right to reasonable accommodation already applies during the hiring process. For example, while just calling an applicant with a disability, they have the right to reasonable accommodation. But, existing AI systems often do not provide any mechanism to offer reasonable accommodation, or even provide an opportunity for the applicant to mention they have a disability.

So, if an AI system does not help offer reasonable accommodation, it not only illegally discriminatory, but we argue that they can be presumed as discriminatory too. This difference seems subtle, but it’s a crucial legal distinction because the presumption of discrimination shifts the burden of proof onto the employer in anti-discrimination lawsuits, which is a tremendous procedural advantage.

12 of 21

EU Law: Employment Equality Directive (EED), General Data Protection Regulation (GDPR), AI Act Proposal (AIA)

ACM FAccT 2022

12

Seoul 서울, June 21-24

Some existing exceptions
for obligations in employment (9(2)(b) GDPR)
for public interest (9(2)(g) GDPR)
for debiasing (10(5) AIA)
Impact of non-personal data (e.g., proxies, anonymised data) → legal gap

Risk of unlawful discrimination

Processing of disability data

Right reasonable accommodation applies to the entire hiring process (3(a), 5 EED)

Note: Violation of right to reasonable accommodation = discrimination

Presumption of discrimination → reversed burden of proof (10 EED)

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

Second, there is the question of whether disability data can even be processed in automated systems. In principle, this is prohibited under GDPR. You are not allowed to process this personal, sensitive data.

Article 9 of the GDPR provides for some exceptions. For example, letter b) of this article allows the processing of personal data if it is done to fulfill obligations in employment, which is the case here. And there is also exception g) which allows processing personal data for reasons of public interest, such as preventing discrimination. But really, it is not very clear whether these arguments hold up. That’s why the AI Act proposal has a specific provision to allow the processing of sensitive data for debiasing.

Finally, we also note that non-personal data, such as proxy data or anonymised data, falls outside the scope of the GDPR. But as we know from AI fairness research, it is in practice possible to extract information on one’s disabilities from proxy data and then this can be a source of algorithmic bias. Clearly, the legal process is behind the technological advances here.

13 of 21

EU Law: Practical recommendations

ACM FAccT 2022

13

Seoul 서울, June 21-24

Partial automation

AHSs shall not be fully automated (22 GDPR)

Decisive role of humans (even though they can be biased as well…)

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

14 of 21

EU Law: Practical recommendations

ACM FAccT 2022

14

Seoul 서울, June 21-24

Partial automation

Rigorous DPIA

AHSs shall not be fully automated (22 GDPR)

Decisive role of humans (even though they can be biased as well…)

Mandatory data protection impact assessment (35 GDPR)

Identify risks for PWDs and implement reasonable accommodation

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

15 of 21

EU Law: Practical recommendations

ACM FAccT 2022

15

Seoul 서울, June 21-24

New obligations (for providers), e.g.:

Debiasing (Art. 10(5))
Human supervisor (Art. 14) can decide whether to use the AHS in the first place, or disregard its output considering the risk of bias against PWDs

EU central database (Art. 60) can assist advocacy bodies and national authorities in detecting discriminatory AHSs

Partial automation

Rigorous DPIA

Impact of the AIA: AHSs as high risk AIs

AHSs shall not be fully automated (22 GDPR)

Decisive role of humans (even though they can be biased as well…)

Mandatory data protection impact assessment (35 GDPR)

Identify risks for PWDs and implement reasonable accommodation

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

16 of 21

EU Law: Practical recommendations

ACM FAccT 2022

16

Seoul 서울, June 21-24

New obligations (for providers), e.g.:

Debiasing (Art. 10(5))
Human supervisor (Art. 14) can decide whether to use the AHS in the first place, or disregard its output considering the risk of bias against PWDs

EU central database (Art. 60) can assist advocacy bodies and national authorities in detecting discriminatory AHSs

However…

Naïve assumptions: e.g., data should be ‘free of error’ (Art. 10(5))

→ impossible thus ineffective

Lack of remedies for affected individuals (e.g., job seekers subject to AHSs)

Partial automation

Rigorous DPIA

Impact of the AIA: AHSs as high risk AIs

AHSs shall not be fully automated (22 GDPR)

Decisive role of humans (even though they can be biased as well…)

Mandatory data protection impact assessment (35 GDPR)

Identify risks for PWDs and implement reasonable accommodation

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

17 of 21

Technical understanding of disability

ACM FAccT 2022

17

Seoul 서울, June 21-24

People with disabilities are a heterogeneous group, e.g.

Require more time to complete a test
Avoid eye contact during interview
Gap in résumé from long-term illness

↔ homogeneous treatment of discrimination for other protected groups

Heterogeneity

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

This finally brings us to the technical point of view.

Here, the most important takeaway is that people with disabilities are a particularly heterogenous group. For example, someone may require more time for a test because they require assistive technology. Someone with autism spectrum disorder may be more likely to avoid eye contact during an interview, which is then seen as a lack of confidence. Someone who has recovered from a long-term illness may have a gap in their résumé. All these people could receive a worse assessment from an automated hiring system because of their disability, though for vastly different reasons.

Now, for other protected groups, there exist technical tools to adjust data or algorithms such that discrimination is reduced. However, these tools rely on simplistic assumptions that view bias as something that affects all members of a protected group in a homogenous way. Clearly, such assumptions are especially misplaced when it comes to people with disabilities.

18 of 21

Technical understanding of disability

ACM FAccT 2022

18

Seoul 서울, June 21-24

People with disabilities are a heterogeneous group, e.g.

Require more time to complete a test
Avoid eye contact during interview
Gap in résumé from long-term illness

↔ homogeneous treatment of discrimination for other protected groups

Construct technical model of the limitations that may result from a disability

Are there any limitations we should consider?
To which extent do they actually matter for the assessment?
How can we accommodate them?

Risks of poor technical model

Too vague to be relevant
Too rigid to allow for nuance
Ethical concerns with standardisation

Heterogeneity

Modeling disability

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

We should therefore, carefully, explore technical solutions to handle this heterogeneity. For example, we can try to build a technical model of disability by focussing on the limitations that may result from a disability. After all, to provide reasonable accommodation, we don’t really need to know why someone has a certain limitation. We mainly need to know: “are there any limitations we should consider?”, “to which extent do they actually matter for the assessment that we are making?”, and “how can we accommodate them?”.

I think we have people from Techwolf here, who focus on distilling people’s skills and figuring out how to leverage and improve them. It would be interesting to also find out what is limiting people with disabilities from carrying out their potential to the fullest extent.

However, I should also stress the need to be very careful here, because a poor technical model of disability, and the limitations that result from it, could do more harm than good.

19 of 21

Providing reasonable accommodation

ACM FAccT 2022

19

Seoul 서울, June 21-24

Automated adjustment of decisions is difficult due to heterogeneity�e.g. depression
Human intervention is costly and not necessarily better
Reasonableness is hard to estimate and not clearly defined

How much investment should be made?
How much does a limitation matter during assessment?

Obstacles

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

Now, we motivated that there are moral and legal obligations to provide reasonable accommodation. But the question remains whether this is possible with automated hiring systems. Of course, it is difficult.

Every disability can have very different impacts on the assessments of candidates. For example, a candidate that suffers from depression has the right to reasonable accommodation. Should this excuse gaps in their resume when they could not work because their depression was worse? How strongly does their depression affect their work today? Some types of depression medication make people more tired in the morning. Is this a problem for the current job or can we accommodate a shifted work schedule?

If a fair, automated assessment can’t be made, then we could decide to have a human intervene instead. But this can obviously be costly, and we then have to make sure that the human is not more biased than the automated system already was.

Now, solutions may exist to all these technical obstacles, and many limitations could be addressed with some adjustments. However, we should still question how reasonable these accommodations are. The idea of “reasonableness” is a legal concept, but there is no clear legal threshold on which accommodations are reasonable or not. We should therefore wonder: How much investment should be made, for example, into accommodating different ways of communication? How strongly should we take a specific limitation into account when assessing someone’s profile? These questions are clearly difficult to answer.

�

20 of 21

Providing reasonable accommodation

ACM FAccT 2022

20

Seoul 서울, June 21-24

Automated adjustment of decisions is difficult due to heterogeneity�e.g. depression
Human intervention is costly and not necessarily better
Reasonableness is hard to estimate and not clearly defined

How much investment should be made?
How much does a limitation matter during assessment?

Recommend accommodations to human recruiter
Audit AI system for patterns of bias

Obstacles

Opportunities

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis

21 of 21

ACM FAccT 2022

21

Seoul 서울, June 21-24

Maarten Buyl

Ph.D. Candidate in AI

maarten.buyl@ugent.be

Christina Cociancig

Ph.D. Candidate in AI Ethics

chrcoc@uni-bremen.de

Cristina Frattone

Ph.D. Candidate in EU Private Law

cristina.frattone@uniroma3.it

Nele Roekens

Legal Officer at Unia

nele.roekens@unia.be

Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis