1 of 32

Rethinking AI Governance

Justice, Sustainability, and the Future of AI

Blair Attard-Frost (she/they)

University of Toronto

Faculty of Information

OPS Finance Day

September 18, 2024

1

2 of 32

What is an AI system?

3 of 32

3

Crawford, K. (2021). Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence. Yale University Press.

“Each way of defining artificial intelligence is doing work, setting a frame for how it will be understood, measured, valued, and governed.”

4 of 32

4

Resources, activities, and impacts aggregated from Attard-Frost, B. & Widder, D. G. (2023). The ethics of AI value chains. https://arxiv.org/abs/2307.16787�

Diagram components are intended to illustrate a selection of significant �ethical concerns, not a comprehensive mapping

“AI” is an ambiguous object of governance.
AI systems consist of many interdependent activities, resources, and societal and environmental impacts that are now very extensively documented in the research literature.
In “The Ethics of AI Value Chains”, a paper I co-authored with David Gray Widder at Cornell Tech, we review 67 sources on various ethical concerns in AI systems along with many other sources from academia, industry, and government on AI supply chains
We find many different categories of resources that are needed to build and maintain AI systems: it’s not just software resources such as data, algorithms, neural networks, machine learning models.

AI systems also require hardware resources, human resources, various types of financial resources, along with legal and policy resources

As well, we find many different types of activities are involved in moving resources through AI systems across their lifecycles, and many different types of impacts that AI systems have on society & the environment.
The vast range of activities & policy issues implicated in this diagram presents a complex, global scale challenge for AI governance, as AI governance practices can apply to potentially all of these resources, activities, and impacts, or potentially only a narrow slice of them.

5 of 32

5

Some examples of incidents from the AI Incident Database: https://incidentdatabase.ai/

Some examples of harmful AI incidents from the AIAAIC Repository: https://www.aiaaic.org/aiaaic-repository/ai-algorithmic-and-automation-incidents

I want to talk about how we build governance around all of these issues shortly, but first:
Important to note that I’ll be using the word “harm” a lot here. When we talk about harms, we’re not just talking about hypothetical risks of AI or abstract moral wrongs - harm refers to real-world instances in which AI systems cause observable, measurable damage.
There are now 1000s of real-world, documented, empirically observable harms to individuals, groups, and the environment that are continuing to happen as we speak.
There are many types of harms caused by AI systems, some researchers have created very detailed taxonomies of these harms. For example, we might think of physical harms, psychological harms, economic, social, environmental harms
There are now entire databases dedicated to cataloguing incidents in which different types of AI use cases cause different types of harm, including autonomous driving, robotics, automated decision-making, chatbots, content generation, and facial recognition use cases
AIAAIC Repository & AI Incident Database are two of the most notable of these databases

6 of 32

6

Williams, A., Miceli, M., & Gebru, T. (2022). The exploited labor behind artificial intelligence. Noema. https://www.noemamag.com/the-exploited-labor-behind-artificial-intelligence/

Appel, G., Neelbauer, J., & Schweidel, D. A. (2023). Generative AI has an intellectual property problem. Harvard Business Review. https://hbr.org/2023/04/generative-ai-has-an-intellectual-property-problem

Beyond harms caused during deployment & use of AI systems, harms also occur during data preparation & model development
For example, we might think of issues related to labor exploitation: Outsourcing of data annotation and content moderation work to historically marginalized communities and regions with less stringent labor regulations
Another area with labor concerns is the development of generative AI systems: Text, images, audio, video, data used to train generative AI models/large foundation models is often scraped from the web and used without consent from or compensation to the creators

There’s been some research done on how these practices are impacting the lives of artists and it’s very clear these practices are causing significant psychological & economic harm and in many cases are impacting employment and livelihoods.

7 of 32

7

Human factors in AI systems

Utility
Ethics
Power
Responsibility
Safety
Bias
Fairness
Justice
Accountability
Transparency
Explainability
Usability
Accessibility
Privacy
Security
Sustainability
etc.

Images from Better Images of AI. https://betterimagesofai.org/

8 of 32

8

Material factors in AI systems

Computer hardware
Minerals & mines
Fuel & transportation
Factories & assembly plants
Data centres
Energy & carbon emissions
Water
Land
etc.

We also need to remember that AI systems are not just abstract software systems that exist in some purely virtual space - these systems are grounded in material reality, and as such, there are material factors involved in the AI lifecycle as well

Computer hardware on which software resources like data, algorithms, machine learning models actually operate
Minerals & mines required for the raw materials in that hardware
Fuel & transportation and factories & assembly plants required to produce and distribute and integrate all of these hardware resources
Data centres which consume water and energy & produce carbon emissions
The land on which these data centres are built

Without these material resources, AI systems would cease to operate, so it is very important we consider these material factors as being part of an AI system too

9 of 32

9

O’Brien, M. & Fingerhut, H. (2023, September 9). Artificial intelligence technology behind ChatGPT was built in Iowa – with a lot of water. AP. https://apnews.com/article/chatgpt-gpt4-iowa-ai-water-consumption-microsoft-f551fde98083d17a7e8d904f8be822c4

St John., A. (2024, July 2). Google falling short of important climate target, cites electricity needs of AI. AP. https://apnews.com/article/climate-google-environmental-report-greenhouse-gases-emissions-3ccf95b9125831d66e676e811ece8a18

On the subject of material factors in AI systems:
Energy & water usage of the data centers used to train and operate AI models presents a particularly significant sustainability challenge across industry right now
Google announced this year that the electricity consumption of their AI systems was causing them to fall short of emissions reduction targets
As generative AI has become popular, there’s also been sharp growth in water consumption - water is needed to cool computer hardware in the data centers, as well as to cool power plants that supply electricity to the data centers
Last year, Microsoft reported a sharp increase in water consumption that researchers have attributed to an expansion of AI projects
The companies operating these data centers are aware of these issues and working to address them through more energy-efficient machine learning methods and hardware, and by increasing the use of renewable energy in data centers
But in the interim, these environmental impacts of AI becoming increasingly a concern in emerging regulatory frameworks and policy initiatives

10 of 32

10

A technological system that processes data and other resources into predictions, recommendations, decisions, and/or content within a socio-material context and with some degree of autonomy from human actors.

AI system

11 of 32

What is AI governance?

12 of 32

12

AI governance is a set of practices intended to �maximize benefit and minimize harm caused by AI systems.

13 of 32

13

Scales & contexts of AI governance

AI governance is practiced across many scales and contexts, including:

State-led AI governance

International AI governance
National AI governance
Subnational AI governance

Provincial/territorial AI governance
Regional AI governance
Municipal AI governance

Industrial/sectoral AI governance
Corporate AI governance
Organizational AI governance
Community-led AI governance
Worker-led AI governance

14 of 32

14

From Attard-Frost, B., Brandusescu, A., & Lyons, K. (2024). The governance of artificial intelligence in Canada: Findings and opportunities from a review of 84 AI governance initiatives. Government Information Quarterly, 41(2), 101929. https://www.sciencedirect.com/science/article/pii/S0740624X24000212

15 of 32

15

How is AI governance organized?

From Attard-Frost, B., Brandusescu, A., & Lyons, K. (2024). The governance of artificial intelligence in Canada: Findings and opportunities from a review of 84 AI governance initiatives. Government Information Quarterly, 41(2), 101929. https://www.sciencedirect.com/science/article/pii/S0740624X24000212

One of the first things we found is that AI governance is commonly organized through different types of organizational initiatives

We identify 5 types of AI governance initiatives that frequently appear across the Canadian landscape:

Policies, programs, strategic plans, standards, ethics statements

Policy: A set of statements of principles, values, and intent that provide a basis for consistent decision-making and resource allocation
Program: A collection of projects managed together with shared strategic intent
Strategic Plan: A detailed action plan to address key strategic objectives
Standard: A detailed document established by a consensus of experts and approved by a recognized body that provides technical guidance
Ethics statement: Broad statements and declarations of principle that prescribe high-level direction for the development and use of AI systems

16 of 32

16

Pan-Canadian AI Strategy

Commissioned by Innovation, Science and Economic Development Canada, launched by the Canadian Institute for Advanced Research (CIFAR) in 2017 with $125 million budget

Phase One programs included National AI Institutes, AI Research Chairs, AI Futures Policy Labs, Solution Networks, and training activities

https://cifar.ca/ai/

Let’s take a brief tour of some notable AI governance initiatives that we analyzed in this paper
Beginning with the Pan-Canadian AI Strategy:

The Strategy was commissioned by Innovation, Science and Economic Development Canada (ISED) to CIFAR, launched in 2017 with a $125 million budget for phase 1 of the strategy, which ran from 2017-2022
Phase 1 programs included:

National AI Institutes: Three centres for AI research and innovation: the Vector Institute in Toronto, Mila in Montréal, and Amii in Edmonton.
AI Research Chairs: A program that aims to recruit and retain the world’s top AI researchers
AI Futures Policy Labs: Workshops held across Canada to chart out emerging policy implications of AI development
Solutions Networks for supporting development of AI solutions tailored to specific sectors and use cases
Training activities for developing AI talent and expertise across the country

17 of 32

17

Pan-Canadian AI Strategy

Phase Two began in 2022: �$440 million for programs including National AI Institutes, �AI commercialization, �AI standardization, �research & training, and computing infrastructure

Greater emphasis on targeted sectoral development than Phase One

18 of 32

18

Bill C-27

“Digital Charter Implementation Act” - Originally tabled in Parliament as Bill C-11 in 2020, re-worked & re-tabled as Bill C-27 in 2022 following federal election of 2021

https://www.parl.ca/DocumentViewer/en/44-1/bill/C-27/first-reading

Three main purposes:

New protection measures for consumer privacy (e.g., data transfer & use requirements)

Creation of new Data Protection Tribunal to enforce those measures

New comprehensive responsibilities for persons & companies using AI systems

19 of 32

19

Artificial Intelligence and Data Act (AIDA)

Innovation, Science and Economic Development Canada (2023). The Artificial Intelligence and Data Act (AIDA) – Companion document. https://ised-isde.canada.ca/site/innovation-better-canada/en/artificial-intelligence-and-data-act-aida-companion-document

That third purpose of Bill C-27 - general regulation of AI systems - is addressed through the Artificial Intelligence and Data Act (AIDA)
AIDA is Canada’s largest effort to regulate AI. It is a cross-sectoral and cross-application framework: it applies across all sectors except the public sector, and across all application areas of AI that are expected to have a high impact on individuals or organizations.
AIDA is currently undergoing a parliamentary study, so it is still an in-progress legislative effort, but if passed it will establish a regulatory framework with three main pillars:
Building on existing Canadian consumer protection and human rights law to set requirements for design and accountability on what the legislation refers to as “high-impact AI systems”
Ensuring the Act can be enforced, primarily through the development of a new regulator who would report to the Minister responsible for AIDA known as the “AI and Data Commissioner”
Prohibit reckless and malicious uses of AI through the creation of new monetary penalties and criminal offences.

20 of 32

20

Directive on Automated Decision-making (DADM)

Put into force by the Treasury Board of Canada Secretariat (TBS) in 2019, specifies operational requirements for automated decision-making in federal institutions

Accompanied by the Algorithmic Impact Assessment tool: divides system risk into multiple categories that require various mitigation measures based upon a broad set of expected system impacts

https://www.tbs-sct.canada.ca/pol/doc-eng.aspx?id=32592

- While AIDA aims for cross-sectoral and cross-application AI regulation, the Directive on Automated Decision-making (or DADM for short) is a more targeted policy instrument

- The DADM applies only to automated decision-making systems (a particular type of AI applications) in production in federal institutions

- The DADM specifies requirements for developing and operating automated decision-making systems

- The DADM also requires institutions to conduct an impact assessment using an Algorithmic Impact Assessment tool prior to putting an automated decision-making system into production

- The AIA tool is used to assess the system’s potential risk across multiple categories of social, economic, and environmental impact, and also requires various mitigation measures based on the expected impacts of the system

21 of 32

21

Ontario Beta Principles for Ethical Use of AI

Government of Ontario (2023). Principles for ethical use of AI [Beta]. https://www.ontario.ca/page/principles-ethical-use-ai-beta

The Principles have since expanded into Ontario’s Trustworthy AI Framework: https://www.ontario.ca/page/ontarios-trustworthy-artificial-intelligence-ai-framework

22 of 32

22

AI & Data Governance Standardization Collaborative

From Standards Council of Canada (2024). AI and Data Governance Standardization Collaborative. https://scc-ccn.ca/areas-work/digital-technology/ai-and-data-governance-standardization-collaborative

23 of 32

23

What is being governed?

From Attard-Frost, B., Brandusescu, A., & Lyons, K. (2024). The governance of artificial intelligence in Canada: Findings and opportunities from a review of 84 AI governance initiatives. Government Information Quarterly, 41(2), 101929. https://www.sciencedirect.com/science/article/pii/S0740624X24000212

In our study, we also sliced those initiatives along with the other initiatives from our sample of 84 initiatives into 8 different categories we call “governance areas”
These 8 governance areas are:

(1) AI Research: Initiatives that respond to perceived impacts of AI by attempting to intervene in the production of scientific research on AI.
(2) AI Education & Training: Initiatives that intervene in education or training on AI design, development, or use.
(3) Technology Production & Use: Initiatives that intervene in the design, development, use, or adoption of AI technologies.
(4) Industry & Innovation: Interventions in economic activities related to AI innovation or AI-enabled industrial development.
(5) Data Governance: Interventions that respond to the impacts that new advancements in AI are having on existing data governance frameworks.
(6) Digital Infrastructure: Intervention in the development or operation of computing infrastructure and hardware.
(7) Social & Workforce: Intervention in social programs and workforce development
(8) Public Administration: Initiatives that aim to intervene in AI development practices & AI adoption specifically in public administration contexts.

A couple of important notes on the data we present in the study:

These governance areas are not mutually exclusive: the Pan-Canadian AI Strategy, for example, contains strategic priorities and projects that cross many of these areas.
There are some interesting patterns & gaps in the data we see here: much more attention has been devoted to AI governance initiatives that focus on ensuring AI will beneficially impact areas such as industry and innovation & technology production and use compared to initiatives that aim to intervene in broader impacts on social wellbeing and workers, which there are comparatively few of.

24 of 32

24

State-led AI governance often prioritizes industry needs

From Attard-Frost, B., Brandusescu, A., & Lyons, K. (2024). The governance of artificial intelligence in Canada: Findings and opportunities from a review of 84 AI governance initiatives. Government Information Quarterly, 41(2), 101929. https://www.sciencedirect.com/science/article/pii/S0740624X24000212

Looking across these 84 initiatives, we find a strong prioritization of industrial development, innovation support, and technology �production & adoption.

These initiatives often assume that technological diffusion and economic �gains from AI adoption in industry will cascade down into broad-based benefit for all of society.

There is no clear evidence indicating that widespread adoption of AI technologies will result in broad-based societal benefit.

A stronger focus on public participation in AI governance is needed to counterbalance the prioritization of industry needs

So, there are some urgent implications that our study raises for justice, sustainability, and the future of AI
There’s one trend that we discuss in this paper that I think is especially relevant to these issues:

State-led AI governance (in this case, initiatives led by federal & provincial government organizations) often prioritizes the needs of industry over broader societal needs
Looking across these initiatives, we found a strong prioritization of industrial development, innovation support, and technology production & adoption
An underlying assumption we found in the documentation we reviewed across many of those initiatives is that technological diffusion and economic gains from industrial AI adoption and commercialization will somehow cascade down into broad-based benefit for all of society.
In other words, societal benefit is assumed to be a by-product of economic benefits accrued through government stimulation of AI innovation, AI markets, and AI implementation in industry
Important to be mindful that this assumption rests on some long-contested political & economic theories and a series of speculative leaps about the potential efficacy of state-led AI technology pushes and technology regulation: there is no clear empirical evidence showing that widespread adoption of AI technologies will result in broad-based societal benefit.
In fact, there are many studies of the politics and economics of AI that we discuss in our paper that indicate the exact opposite: without cross-cutting and equity-seeking interventions in industries, technologies, communities, workforces, and infrastructure development, a stimulation approach to AI governance might instead cause negative societal impacts.

25 of 32

25

Who is included in AI governance? Who is excluded?

From Veale, M., Matus, K., & Gorwa, R. (2023). AI and global governance: Modalities, rationales, tensions. �Annual Review of Law and Social Science, 19, 255-275. https://www.annualreviews.org/doi/10.1146/annurev-lawsocsci-020223-040749

From Wilson, C. (2022). Public engagement and AI: A values analysis of national strategies. Government Information Quarterly, 39, 101652.

This trend of industry prioritization & civil society exclusion from AI governance isn’t just limited to Canada
Other studies have found similar tendencies in the AI strategies & policies of other nations, as well as in international AI governance initiatives.
In “AI and Global Governance”, the authors look at how international initiatives to govern AI often structurally exclude many nations and regions from the Global South.
In “Public engagement and AI”, the author looks across 16 national AI strategies and finds little evidence that there is much meaningful public engagement in the development of these countries’ AI governance systems, with speculative economic benefits of AI and other policy concerns often taking priority over public debate and participatory approaches to AI governance.

26 of 32

Government & industry often work closely to develop AI strategies, policies, and �other governance mechanisms.

Impacted communities & workers are often pushed to the margins of AI governance.

How can this be counteracted?

27 of 32

27

Recommendations for researchers

Recommendation 1: Researchers should study the outcomes of Canada’s AI governance initiatives.

Recommendation 2: Researchers should study challenges to public trust in Canada’s AI governance initiatives.

Recommendation 3: Researchers should study the effects of AI impact representation on the outcomes of Canada’s AI governance initiatives.

Recommendations for practitioners

Recommendation 4: Policymakers and public servants should specify success measures for initiatives and routinely publish information on the outcomes of Canada’s AI governance initiatives.

Recommendation 5: Policymakers and public servants should collaborate more directly with the public on designing and implementing Canada’s AI governance initiatives.

Recommendation 6: Policymakers and public servants should account for a greater variety of AI impacts when designing and implementing Canada’s AI governance initiatives.

Recommendation 7: Policymakers and public servants should launch a new initiative to cultivate a more unified national approach to AI governance in Canada.

From Attard-Frost, B., Brandusescu, A., & Lyons, K. (2024). The governance of artificial intelligence in Canada: Findings and opportunities from a review of 84 AI governance initiatives. Government Information Quarterly, 41(2), 101929. https://www.sciencedirect.com/science/article/pii/S0740624X24000212

Turning back to our study, we offer some practical recommendations that I think offer a solid start for correcting these issues and creating a more just, equitable, and sustainable future for AI
We group these into 3 recommendations for researchers & 4 recommendations for practitioners

1) Researchers should study the outcomes of Canada’s AI governance initiatives.
2) Researchers should study challenges to public trust in Canada’s AI governance initiatives.
3) Researchers should study the potential effects of creating a large volume of initiatives that represent some types of AI impacts over other types of AI impacts
4) For policymakers and public servants, we recommend specifying success measures for initiatives and routinely publishing information on the outcomes of initiatives to improve transparency, accountability, and public trust in AI & public trust in government’s ability to regulate AI
5) Policymakers and public servants should collaborate more directly with the public on designing and implementing Canada’s AI governance initiatives.
6) Policymakers and public servants should account for a greater variety of AI impacts when designing and implementing Canada’s AI governance initiatives.
7) Policymakers and public servants should launch a new initiative to cultivate a more unified national approach to AI governance in Canada across departments, governments, sectors, and into civil society organizations, communities, associations, and unions.

There are significant barriers to collaboration in Canadian AI governance that are often preventing stakeholders from coordinating their efforts effectively, sharing resources, and understanding each others’ needs.
To diminish these barriers, we can look to initiatives like the US National AI Initiative or the European AI Alliance that have successfully brought together a diverse range of many different stakeholders across sectors to share resources and to formulate policy together

28 of 32

Expand the periphery of “AI”

Expand the periphery of “AI governance”

Imagine futures for AI governance outside of industry & the state

In addition to new initiatives and specific instruments and practices of governance, I want to leave you with three high-level action items for rethinking what the future of AI governance ought to look like
1) Expand the periphery of “AI” - AI is about much more than data, algorithms, neural networks, and software

It’s often said that “data is the new oil” or “data is the fuel of AI.”
Data certainly does help to fuel AI systems, but we should also remember that AI systems are also fueled by literal fuel
These systems have a material reality, and the development & operation of these systems has material impacts on people & the environment.
When we think about what “AI” is and what its governance therefore ought to entail - when we have discussions about what the scope of an AI policy should be - we need to be re-asserting the social & material reality of these systems.

2) Expand the periphery of “AI governance”

- We are doing AI governance right now by having this conversation - you are participating in an act of AI governance by listening to this, by leaving a question or comment in Slido
Together we are exploring & setting frames for how AI can be understood, we’re thinking about which people & impacts might need to be included in or excluded from our frames, which interventions and design requirements might need to be developed
Just like there is no one right way to understand “AI”, there is no one right way to understand “AI governance”
AI governance is a dynamic and fluid field, AI systems and their governance impact virtually every sector of society. As a result, AI governance includes a diversity of values, perspectives, and practices
If we want a more just and sustainable future for AI, we need to make spaces and governance systems that encourage participation from more diverse perspectives, not just small enclaves of experts

- 3) On the subject of bringing more perspectives into our understanding of AI governance, a final thought I want to share with you:

- We need to imagine futures for AI governance outside of industry & the state

- In the absence of effective regulation & government intervention in harmful AI systems, communities and workers are taking AI governance into their own hands

- As just one example, we might think about last year’s labor strikes by the WGA & SAG-AFTRA (the Writers Guild of America & Screen Actors Guild).

- These strikes were prompted by a wide range of many different labor concerns, but a significant issue in the strike negotiations was the training of generative AI on union-protected creative works and worker control over the use of generative AI in their work processes.

- We might also think about broader forms of resistance in creative communities both online and physically against generative AI tools that are trained on copyrighted works

- These communities have structures and rules and norms about how AI should be developed and used, and engage in many of their own AI governance practices:

- Many creatives are now using and contributing to data masking tools such as Glaze and Nightshade to protect their work from being scraped or used to train ML models

- Open letters, petitions, and community knowledge resources have been created, and media engagement has been used to raise awareness about these issues

- What we can learn from these examples is that AI governance can exist and does exist outside of industry and the state:

- Collective organizing of communities and workers, community-led policy, direct action, small-scale resource-sharing and capacity-building initiatives, and public protest have all been directed toward maximizing benefits and preventing harms caused by AI systems.

- Therefore, these are all legitimate practices of AI governance.

- If we want to design AI systems & AI policies that more effectively prevent harm and serve a public good, we need to think about the future through the eyes of those who are being harmed at this very moment by unjust and unsustainable AI systems.

- We need to imagine alternative futures that are better fitted to their hopes and their fears.

29 of 32

For more on advancing equity & justice in AI governance, see my essay in Midnight Sun Magazine entitled “AI Countergovernance”: https://www.midnightsunmag.ca/ai-countergovernance/

30 of 32

This presentation draws on research supported by the Social Sciences and Humanities Research Council of Canada and the University of Toronto Faculty of Information

31 of 32

Big AI Governance Reading List:

https://docs.google.com/document/d/1MF3HctFi_dsijKE8Wa-tQwbfnUVgu_dyeUp5Mr_o5CM/edit?usp=sharing

32 of 32

Thank you

Website: blairaf.com

Email: blair@blairaf.com

LinkedIn: https://www.linkedin.com/in/blairaf/