1 of 32

Public Sector use of Algorithmic Decision Making�20 questions to answer before using algorithmic

decision making in a live

environment

@EddieACopeland

Director of Government Innovation

2 of 32

Introduction

About this document

In February 2018, I invited feedback on my draft

Code of Standards for Public Sector use of Artificial Intelligence in Algorithmic Decision Making.

Thank you to everyone who shared their comments!

With the aim of getting the right balance between a code that offers protection, whilst helping organisations to innovate, I’ve adapted, refined and rephrased the principles into a set of questions.

These questions incorporate many of the points of feedback I received, which seem better geared towards a more practical tool.�

3 of 32

Introduction

About this document

The following slides therefore outline the questions that public sector organisations should be able to answer before using algorithmic decision making in a live environment - whether in-house or in a service they procure.

The questions cover:

The algorithm’s purpose and appropriate circumstances of use
The outcomes the algorithm is intended to make possible and whether they are ethical
The algorithm’s function
The algorithm’s limitations and biases
The actions that will be taken to mitigate the algorithm’s limitations and biases; and
The layer of accountability that will be put in place

4 of 32

Introduction

About this document

Organisations must use their own professional judgement - and the requirements of relevant legal frameworks - to decide how detailed their responses need to be, what ethical models are most appropriate, and who needs to see their answers, based on the specific context in which the algorithm will be used.

If it’s a very simple algorithm used in a non-sensitive area, the questions should be quick and easy to answer. If the algorithm and its context are more complex, the questions will, rightly, take longer to address.

If you’re interested in why I think these questions are necessary at all, you might like to read this blog.

5 of 32

Introduction

Each slides consists of four parts:

The question the organisation using the algorithm should be able to answer
An example response - based on an example of a city authority using an algorithm to find unlicensed Houses in Multiple Occupation (HMOs)
An explanatory comment
The Assessment Criterion - the question is rephrased into a statement that can be marked on a scale of, say, 1-5, where 1 means ‘completely disagree’ and 5 means ‘completely agree’. Best practice is to be able to score 4-5 on every answer.

Completely Disagree	Somewhat disagree	Unsure	Somewhat Agree	Completely Agree
1	2	3	4	5

6 of 32

The algorithm’s purpose and appropriate circumstances of use�

7 of 32

Why has the algorithm been created?

Question 1

Example Answer
The volume of unlicensed HMOs and the complexity of the factors linked to predicting their location are too numerous for traditional processes to target the highest risk properties promptly. The algorithm is needed to support building inspectors to make better informed decisions about which properties to prioritise for inspection.
Comment
The reason for using the algorithm should be made explicit, including why it is needed at all.
Assessment Criterion
We can clearly state the specific rationale for using the algorithm.

8 of 32

In what processes and circumstances is the algorithm intended and appropriate to be used?

Question 2

Example Answer
The algorithm is intended to be used to create a weekly, prioritised list of properties that are statistically likely to be unlicensed HMOs, for consideration by council building inspectors as they plan their inspection schedules.
Comment
The algorithm should not be considered approved for use in any process not listed in this answer.
Assessment Criterion
We can clearly state the processes and circumstances in which the algorithm is appropriate and intended to be used

9 of 32

To what extent is human judgement required before acting on the algorithm’s insights?*

*It’s worth remembering GDPR’s requirement that an individual must be notified before any automatic decision making affecting them occurs.

Question 3

Example Answer
The algorithm is intended to complement and help inform, not replace, the professional judgement of council building inspectors. For example, they understand that the algorithm only assesses physical characteristics of a property, and not details about the landlord.
Comment
Organisations must ensure that those expected to use an algorithm fully appreciate the extent to which they should act automatically on its insights.
Assessment Criterion
All staff using this algorithm understand its function, appropriate circumstances of use, limitations and potential impacts, and are clear on how and when they should use their own judgement

10 of 32

The outcomes the algorithm is intended to make possible and whether they are ethical

11 of 32

What outcome(s) is the algorithm intended to enable?

Question 4

Example Answer
To enable councils to take action to protect more vulnerable tenants, issue more licences, and take legal proceedings against more rogue landlords who fail to comply after initial warnings.
Comment
This should be a list of all the outcomes to be achieved as a result of using the algorithm. This question is therefore not about what insight the algorithm shows, but what is done with that information.
Assessment Criterion
We can clearly state the intended outcome(s) of using the algorithm

12 of 32

What negative impacts could the algorithm have on individuals or groups, and how severe and likely are they?

Question 5

Example Answer
A detailed assessment of potential impacts on landlords, tenants and building inspectors has been conducted, highlighting 10 potential negative impacts, of which 2 are of significant severity.
Comment
It is recommended that the process of identifying potential impacts is as inclusive as possible, to ensure that groups likely to be affected by the algorithm’s use have been fully consulted.
Assessment Criterion
We can clearly explain the nature, severity and likelihood of the negative outcomes the algorithm could have on a given individual or group.

13 of 32

What assessment has been made of the ethics and acceptability of these impacts, and by whom?

Question 6

Example Answer
An assessment of each potential impact listed in question 5 has been conducted. This involved our ethics committee, information governance team, frontline workers and consultations with those affected by the algorithm’s use. Once the mitigating actions specified in the next section are in place, we deem that the impacts are both ethical and acceptable.
Comment
It is for each organisation to use its considered professional judgement on what process or framework is appropriate for determining the ethics and acceptability of using an algorithm to produce or optimise for certain outcomes.
Assessment Criterion
We have assessed the potential impacts of using this algorithm and deemed them to be both ethical and acceptable

14 of 32

How, and by what criteria, will the effectiveness of the algorithm be measured and assessed?

Question 7

Example Answer
The algorithm will be deemed to be successful if it results in at least a 5% increase in the number of unlicensed rental properties successfully identified during its trial period compared with a control group, as measured during a 3 month Randomised Control Trial.
Comment
Organisations will need to ensure they record sufficient data before and during the testing of a new algorithm so that its effectiveness can be fairly measured and evaluated. The required standards of evidence should be proportionate to the algorithm’s potential impacts.
Assessment Criterion
We can clearly state how the effectiveness of the algorithm will be measured and assessed

15 of 32

The algorithm’s function

16 of 32

How does the algorithm work?�

Question 8

Example Answer
The algorithm determines the probability that a particular rental property is unlicensed based on an analysis of ten physical characteristics, and reported incidents linked to known past cases of HMOs in the city recorded between January 2014 and December 2017. Those characteristics are X, Y and Z, etc.
Comment
This description should be in non-technical such that its basic function is comprehensible by anyone who will be expected to use or oversee the use of the algorithm. It is not acceptable for public sector staff to be expected to use an algorithm without having a basic understanding of what factors it is taking into account and how it evaluates them.
Assessment Criterion
We can state in plain language how the algorithm functions

17 of 32

What datasets is / was the algorithm trained on?�

Question 9

Example Answer
A complete list of the exact datasets used to train the algorithm is provided on a separate spreadsheet. This includes UPRNs of every address in the city; the status of the property, such commercial, owner occupied, privately rented and social housing; council tax band; whether the property has been late on payments; whether the bill is sent to a different address or to the property; the number of occupants, number of surnames; noise complaints, etc.
Comment
A complete list of the exact datasets used to train the algorithm should be provided in a spreadsheet.
Assessment Criterion
We have a complete list of all the datasets on which the algorithm is / was trained

18 of 32

On what assumptions is the algorithm based?��

Question 10

Example Answer
The algorithm assumes that: Future cases of unlicensed HMOs can be predicted from a subset of characteristics observed in past known cases. That unlicensed HMOs will have similar characteristics in different parts of the city Properties that are owner occupied cannot be HMOs etc.
Comment
A complete list of the assumptions used to train the algorithm should be provided in a spreadsheet.
Assessment Criterion
We have a detailed list of the assumptions on which the algorithm is based

19 of 32

What inputs are used by the algorithm to make a decision?��

Question 11

Example Answer
A complete list of the exact datasets used by the algorithm to make a decision is provided on a separate spreadsheet. This includes the property’s UPRN; status (commercial, owner occupied, privately rented or social housing); council tax band; payment history; number of occupants etc.
Comment
Listing the inputs is important to ensure that the algorithm is not discriminating on inappropriate grounds (e.g. based on a person’s ethnicity or religion) or using a proxy measure to infer personal details from other data (e.g. guessing someone’s religion based on their name or country of origin).
Assessment Criterion
We have a complete list of the inputs used by the algorithm to make a decision

20 of 32

The algorithm’s limitations and biases

21 of 32

What are the limitations and potential biases of the datasets on which the algorithm was trained?�

Question 12

Example Answer
Details of the limitations and potential biases are provided on a separate sheet. These include: Data on past known unlicensed HMOs only includes those discovered in the city between January 2014 and December 2017. The model does not take into account any data about the Landlord of a property. etc.
Comment
It is important to outline both the limitations of the datasets on which the algorithm was trained, but also the limitations of the algorithm based on what it was not trained on. In this example, no data about the Landlord of a property was included.
Assessment Criterion
We have catalogued the limitations of the datasets used to train the algorithm

22 of 32

What potential errors or biases could follow from the assumptions on which the algorithm is based?�

Question 13

Example Answer
The potential errors and biases that could follow from each assumption are outlined on a separate spreadsheet. These include: The model may flag properties which meet the known risk factors but do not meet the legal definition of an HMO etc.
Comment
A list of the potential errors and biases that could follow from the assumptions should be provided in a separate spreadsheet.
Assessment Criterion
We have catalogued the errors and biases that could follow from the assumptions on which the algorithm is based

23 of 32

What are the limitations and potential biases of the inputs used by the algorithm?

Question 14

Example Answer
Details of the limitations and potential biases are provided on a separate sheet. These include: For some addresses we do not have the property type No aata relating to the ownership of the property is factored in. etc.
Comment
It is important to outline both the limitations of the inputs used by the algorithm, but also the limitations of the algorithm based on what inputs it is not able to use. In this example, no data about the Landlord of a property was included.
Assessment Criterion
We have catalogued the limitations and potential biases of the inputs used by the algorithm

24 of 32

The actions that will be taken to mitigate the algorithm’s limitations and biases�

25 of 32

How will the impact of the limitations and potential biases in the training datasets be mitigated?

Question 15

Example Answer
For example, in response to one limitation, a mitigating statement could be: “Building inspectors will be trained to ignore results that suggest flats are unlicensed HMOs, as they do not meet the legal definition of an HMO”.
Comment
For each limitation or bias outlined in Question 12, a statement should be given on how it will be mitigated. This might be a specific action, or simply confirmation that those using the algorithm will be made fully aware of the limitation so they can use their own judgement.
Assessment Criterion
We have implemented measures to mitigate the impact of the limitations and potential biases in the training datasets

26 of 32

How will the impact of the potential errors and biases in the assumptions used by the algorithm be mitigated?

Question 16

Example Answer
In response to one bias, a mitigating statement could be: “Building inspectors will be asked to immediately give feedback to the design team if they find the algorithm does indeed focus on certain property types to the exclusion of other relevant types.”
Comment
For each error or bias outlined in Question 13, a statement should be given on how it will be mitigated. This might be a specific action, or simply confirmation that those using the algorithm will be made fully aware of the error or bias so they can use their own judgement.
Assessment Criterion
We have implemented measures to mitigate the impact of the potential errors and biases in the algorithm’s assumptions

27 of 32

The layer of accountability that will be put in place�

28 of 32

Who has been consulted in the course of creating this algorithm?

Question 17

Example Answer
We have consulted service managers, frontline building inspectors, landlord and tenants groups to invite their feedback on the algorithm's design and use.
Comment
The amount of consultation used in creating an algorithm should be proportionate to its sensitivity and potential impact. As with any agile design phase, engaging users during discovery, alpha and beta stages is highly recommended.
Assessment Criterion
We have, at minimum, consulted individuals who will need to use the algorithm, and those who will be affected by its decisions

29 of 32

Who is responsible for ensuring the proper use of this algorithm, and addressing any appeals that result from its use?�

Question 18

Example Answer
The Data Privacy Officer and Building Inspectors Service Manager will be held jointly responsible for ensuring the proper use of this algorithm, and have put in place a clear process to address appeals from its use.
Comment
This is a powerful way to ensure that the leadership of each organisation has a strong incentive only to deploy algorithms whose functions and impacts on individuals they sufficiently understand.
Assessment Criterion
Named roles within our organisation are responsible for ensuring the proper use of the algorithm and addressing appeals that result from its use

30 of 32

On what schedule and by whom will the algorithm be reviewed, evaluated and updated?

Question 19

Example Answer
The algorithm and its impacts will be reviewed quarterly by the Building Inspections Service Manager, and a committee of frontline building inspectors. Their assessment will be shared with the Senior Management team and design team.
Comment
An algorithm may be effective for an initial period, but its continued usage may change facts about the external environment in which it operates, and some of the initial assumptions and data on which it was trained could become out of date, requiring updates.
Assessment Criterion
We have a clear schedule for the review, evaluation and updating of the algorithm by specified roles

31 of 32

Who will see and review the answers in this pack?

Question 20

Example Answer
This pack has been shared for review by the senior management team, ethics team, and frontline workers.
Comment
In cases of high sensitivity, it may be desirable to have an independent panel assess the answers to this pack, since those who implement algorithms are invested in having their work approved. Sometimes the answers should be made public, but this may be infeasible, for example in cases that seek to tackle crime / fraud.
Assessment Criterion
We have, at minimum, shared the results of this pack for review by those who will use and be held responsible for the algorithm, and anyone else who could reasonably expect to be informed

32 of 32

Public Sector Use of Algorithmic Decision Making�20 questions to answer before deploying

an algorithm in a live

environment.

@EddieACopeland

Director of Government Innovation