Module 4 : Machine Learning
Session 4C : Ethics in AI
Dr Daniel Chalk
Guru Meditation
With Thanks
Big thanks to my colleague Mike Allen for developing the content on which much of this session is based.
Many of Mike’s materials are drawn from two excellent (and easy to read) books (also available as audio books) :
Health Warning
Some people may find some of the content I am going to present in this session to be offensive and / or shocking. Please be assured that it is not our intention to offend, but to talk about real occurrences where AI has had negative and sometimes severe consequences for peoples’ lives.
The purpose of this session is to expose you to this reality, and ask you to consider how, as you develop as AI engineers, that you can take measures to avoid this kind of negative consequence.
It is our responsibility as tutors of AI to ensure that you go into the world not only equipped with the ability to develop AI algorithms, but also an awareness of what can go wrong and how this can affect people.
A thought from Brian Christian
“There is a growing sense that more and more of the world is being turned over, in one way, or another, to mathematical and computational models. It is as if we are consumed by the task of putting the world - figuratively and literally - on autopilot”
Representation
When we train Machine Learning models we use vast amounts of data. Our models find the inherent patterns in this data to try to get better at coming to the correct conclusion given a set of input data.
But are our data sets fit for purpose? How representative are they?
Representation
On Sunday evening of June 28th 2015, Jacky Alcine got a notification that a friend had uploaded a photo to Google Photos. It had created a new group for it and, using an AI algorithm, placed the photo in this new group which it titled “Gorillas”.
Representation
It should be said that Google were equally horrified by what had happened, and took corrective measures.
Nobody is implying that Google did this deliberately.
But it did happen and it had a real impact on real people.
And the machine had learned this. So how good was the data used to train it…?
Representation
When Joy Buolamwini was a computer science undergrad at Georgia Tech in the early 2010s, she worked on an assignment to recognise emotion in faces.
The problem was that the face recognition library she was using would not detect her face… until she held up a white mask in front of her face…
Perhaps she shouldn’t have been surprised given the data that was used to train such libraries at the time - a database known as Labelled Faces in the Wild. That database had twice as many images of George W Bush as all black women combined.
This dataset was used to train leading face detection algorithms. And yet, the data is clearly not representative.
Representation
Research by Joy Buolamwini and Timnit Gebru into automated gender recognition showed that the error rate was 100x HIGHER for dark-skinned women than white men.
So just how reliable is such an algorithm?
Representation
It seems we have problems of representation in terms of race and gender. And there’s more…
Word embeddings is an approach in Natural Language Processing that encodes words in a common space, such that an algorithm groups words that are thought to be semantically similar close to each other. This allows us to do some rather cool things, like perform mathematical operations on words by applying them to the word vectors!
For example, if we subtract the vector for male from king, and add the word female we get the vector for the word queen.
But these embeddings are learned from vast amounts of human text. Which means we also get things like this…
If we subtract the vector for male from doctor, and add the word female, can anyone guess what word we get….?
Oh and btw, these kinds of word embeddings have been used in all sorts of things in our daily lives - internet searches, translations, analysis of sentiment of text…
Representation
Large Language Models (such as GPT, atop which sits the ChatGPT interface) have become a huge part of our lives now. You’ll learn a bit more about how they work in the Natural Language Processing module (and how they’re a great excuse to try to convince your IT department you really need a Geforce RTX 4090…) but basically they are trained on unfiltered text data from the internet. This allows them to build up significant experience in how humans communicate.
I say again, everything on the internet. How humans communicate.
Warts and all… (and by warts, I mean bias, prejudice, toxicity, all floating in a soup consumed en masse by these huge AI models now taking over the world…)
Representation
So what can we learn from all this?
So what should we do?
Transparency
We’ve talked about how important it is that our models are as representative as possible.
Let’s now talk about Transparency - do we understand why our models make certain predictions?
Transparency
An important concept to understand in Machine Learning is that correlation is not the same as causation.
In the mid 90s, a group of researchers led by Tom Mitchel produced state-of-the-art neural network models for predicting the risk of death from pneumonia.
They were surprised to see in a rules based model that pneumonia patients who had a history of asthma were associated with a lower risk of death…
Can anyone think why that might be the case?
Transparency
In 2015 dermatologists Justin Ko and Robert Novoa used a Google image analysis network and trained it on 130,000 skin lesion images to recognise melanoma and other conditions (see example). In a 2017 paper published in Nature, they reported that it out-performed 25 human dermatologists.
But a year later, they had to issue a warning. Can anyone guess what that warning was…?
Transparency
Machine Learning models - particularly Neural Networks (which you’ll learn about later in the module) are infamous for being difficult to understand why the model comes to the conclusions it does. They “just work”.
However, in recent years there is a growing field of Explainable AI in which we try to tease out how our models made their predictions.
One key method that you’ll learn about in this module is the use of Shapley Values - which allow us to plot the extent and direction of influence of individual features on an individual classification in a Machine Learning model.
Exercise 1
You will now break into your groups for 45 minutes (+ 10 minute comfort break at the end) to discuss the following :
You work for a local police force. A local academic group has developed a piece of cutting-edge facial recognition software, which is 90% accurate in automatically identifying patients turning up to a GP surgery, to save them signing in at reception. They have tweaked the algorithm, and run some extensive tests on the new software, which has been shown to be 99% accurate at identifying sexual offenders from a database.
Your area has experienced a number of late night sexual assaults taking place around a local train station. Your force is considering installing the software at the train station, such that any people flagged up as a sexual offender who are entering the station area after 8pm trigger an officer attending to monitor the individual’s movements at the station and potentially question them. Typically the station sees around 200 people attending between 8pm and midnight.
You have been asked to provide feedback of your thoughts on the implementation of this system. You should also consider issues of representation and transparency and suggest questions you would want to ask of the developers, as well as key points that should be considered by the force if the system were to be implemented (or reasons why it shouldn’t).
When we come back, I’ll ask a few groups to share the outputs of their discussion.
Consequences
We’ve talked about how things can go wrong with AI models.
Our data might not be representative and might lead to a model that is biased.
We might not understand why our model is making its predictions, and consider the model to be performing better than it is.
But what happens when things do go wrong? And they can and will.
Consequences
Steve Talley was asleep at home in South Denver in 2014 when he heard a knock at the door. He opened it to find a man apologising for accidentally hitting his car. The stranger asked Talley to step outside and take a look. He obliged.
As he crouched down to assess the damage to his driver’s door, a flash grenade went off. Three men dressed in black jackets and helmets appeared and knocked him to the ground. One man stood on his face. Another restrained his arms while another started repeatedly hitting him with the butt of a gun.
Talley’s injuries would be extensive. By the end of the evening he had sustained nerve damage, blood clots, and a broken penis. “I didn’t even know you could break a penis” he later told a journalist. “At one point I was screaming for the police.”
Consequences
“Then I realised these were cops who were beating me up.”
Consequences
Steve Talley had been misidentified as a bank robber who assaulted a police officer in the course of one the robberies by an AI face recognition system looking at CCTV footage.
A model got it wrong, and changed a man’s life.
Consequences
Consequences
Consequences
Consequences
People make mistakes.
But to make mistakes and promote bias at scale takes automation.
Summary
We have only skimmed the surface of the ethics of working with AI here. Consider reading extensively around this area if you’re developing AI-based models.
Your models should generate positive impact. Not a broken penis, or someone’s phone telling them they look like a gorilla, or getting unknowingly excluded because of their skin colour or their gender, or…
Exercise 2
You’ll now work in your groups again. You will have 45 minutes to undertake the following task.
You have been asked to come up with a project idea that uses machine learning. This idea might be related to a real problem of one of your organisations (and may even be a project idea for HSMA) or it might be fictitious and unrelated to your work. As a group, you should discuss some potential ideas and decide on one. Then, for your chosen idea :
- come up with a list of the benefits that this project could provide
- identify any potential issues of representation and transparency that the proposed project might bring, and for each issue, identify how you would mitigate against this issue
- identify any potential negative consequences (intentional and unintentional) of the implementation of the proposed project. For each, consider the group(s) that would be affected, how they would be affected, and how you might mitigate against (or prevent) these negative consequences
At the end of the task, I’ll ask a number of groups to present what they came up with. Sammi will judge the top three for PSG Points!