1 of 67

Fostering open ecosystems around data

The role of data standards, infrastructure and institutions

November 2021 | @timdavies | tim@practicalparticipation.co.uk

2 of 67

3 of 67

Drawing on

Participatory practice�
Information Infrastructure Studies�
Data Feminism�
Indigenous Data Sovereignty �
& other critical data practice

4 of 67

This talk also draws particularly on some work in progress developed through a residency at the Rockefeller Bellagio center looking at the intersection of standards and Artificial Intelligence: a point I won’t labour - as I fear a focus on ‘AI’ - in inverted commas - can distract us from looking at many more ‘mundane’ (also in inverted commas) uses of data: but I will say at this point that when we think about the datasets and data infrastructures our work might create, we need to keep in mind that these will likely end up being used to feed machine learning models in the future, and so what gets encoded, and what gets excluded from shared datasets is a powerful driver of bias before we even get to the scope of the training sets, or the design of the AI algorithms themselves.

Standards work is AI work.

5 of 67

6 of 67

Talk outline

Standards 101�
Data ecosystems�
Data infrastructures�
Data Institutions�
Reflections & challenges

7 of 67

Standards 101

8 of 67

12-10-2021

2021-10-12

Oct 12th 2021

der 12. Oktober 21

1633993200

October 12th 2021

Seshhanbeh 1400 Mehr 20

Mon Oct 11 2021 23:00:00 GMT+0000

12/10/2021

9 of 67

12-10-2021

2021-10-12

Oct 12th 2021

der 12. Oktober 21

1633993200

October 12th 2021

Seshhanbeh 1400 Mehr 20

Mon Oct 11 2021 23:00:00 GMT+0000

12/10/2021

US Date: 10th December

European: 12th October

10 of 67

ISO8601

2021-10-12T00:00:00+00:00

11 of 67

ISO8601

2021-09-12T00:00:00+00:00

2021-10-12T00:00:00+00:00

2022-01-01T00:00:00+00:00

2023-03-05

2024-10-12T09:00:00+05:45

12 of 67

ISO8601

ISO = International Organisation for Standards

13 of 67

ISO8601

There are a lot of (data) standards

14 of 67

ISO8601

Standards build on other standards: it’s a system

15 of 67

ISO8601 RFC3339

“...an Internet profile of the ISO 8601 [ISO8601] standard for representation of dates and times using the Gregorian calendar.”

People follow implementations rather than specifications

16 of 67

ISO8601 RFC3339

Standards for interchange not data entry or storage

{ 2021-11-16 }

17 of 67

ISO8601 RFC3339

The event took place in early November�
2021-11 fails validation.�
To enter data, user adds arbitrary day: 2021-11-01�
Data from multiple sources can be analysed (all dates standardised) but the data might mislead: “The 1st of the Month is the best day to run events.”

Standards enable and constrain

18 of 67

19 of 67

20 of 67

Essentials of a data standard schema

Fieldnames (or paths)�
Definitions�
Validation rules

21 of 67

site_code	commodity_name	commodity_code	quantity	unit

22 of 67

site_code - (string) a unique identifier for the site�
commodity_name - (string) the given name for a primary agricultural product that can be bought and sold. Note: this is used for labelling only, and might be over-ridden by values taken from the commodity_code reference list.�
commodity_code - (string; enum) a value from the approved codelist that uniquely identifies the specific commodity.�
quantity - (number) the number of unit of the commodity�
unit - (string;enum) the unit the quantity is measured in, from the list kg for kilograms or tonne for metric tonne. If quantities were collected in an unlisted unit, they should be converted before representation.

site_code	commodity_name	commodity_code	quantity	unit

site_code- (string) a unique identifier for the site
commodity_name- (string) the given name for a primary agricultural product that can be bought and sold. Note: this is used for labelling only, and might be over-ridden by values taken from the commodity_code reference list.
commodity_code- (string; enum) a value from the approved codelist that uniquely identifies the specific commodity.
quantity- (number) the number of unit of the commodity
unit- (string;enum) the unit the quantity is measured in, from the list kg for kilograms or tonne for metric tonne. If quantities were collected in an unlisted unit, they should be converted before storage.

And we need validation rules to say that if we find, for example, a non-number character in the quantity column - the data should be rejected - as it will be tricky for systems to process.

23 of 67

site_code - (string) a unique identifier for the site�
commodity_name - (string) the given name for a primary agricultural product that can be bought and sold. Note: this is used for labelling only, and might be over-ridden by values taken from the commodity_code reference list.�
commodity_code - (string; enum) a value from the approved codelist that uniquely identifies the specific commodity.�
quantity - (number) the number of unit of the commodity�
unit - (string;enum) the unit the quantity is measured in, from the list kg for kilograms or tonne for metric tonne. If quantities were collected in an unlisted unit, they should be converted before representation.

site_code	commodity_name	commodity_code	quantity	unit

24 of 67

Apples =

080801

c_541

25 of 67

For example: The Open Apparel Registry has developed unique production site identifiers by combining different existing datasets

26 of 67

site_code	commodity_name	commodity_code	quantity	unit

27 of 67

Data standards are technical specifications of how to exchange data

28 of 67

Designing standards is a technical task

29 of 67

Designing standards goes beyond the technical tasks

30 of 67

Creator

Intermediary

User

31 of 67

Creator

Intermediary

User

Creator

User

32 of 67

Intermediary

User

Creator

33 of 67

Creator

Intermediary

User

Creator

User

34 of 67

Creator

Intermediary

User

Creator

User

35 of 67

Creator

Intermediary

User

Creator

User

$

36 of 67

Creator

Intermediary

User

37 of 67

Data standards could be a powerful tool for the projects you are working on

38 of 67

Data ecosystems

39 of 67

Standards can support decentralisation and innovation

40 of 67

Approaches open to decentralisation can support greater generativity, freedom and resilience

41 of 67

Let many flowers bloom?

Publish once. Use anywhere.

42 of 67

43 of 67

To give a more concrete example - a data ecosystem around the International Aid Transparency Initiative has grown up over the last decade - with hundreds of aid donors big and small sharing data on their projects in a common data format: the IATI Standard. ‘Keystone species’ such as D-Portal, which visualises the collected data, have helped create an instant feedback loop for data publishers to ‘see’ their data, whilst behind the scenes, a Datastore and API layer feeds all sorts of specific research and analysis projects, and operational systems - that one their own would have had little leverage to convince data publishers to send them standardised data.

However, elements of this ecosystem are fragile: much data enters through a single tool - AidStream - which over time has come to be the tool of choice for many NGOs and if unavailable would diminish the freshness of data. Many users accessing data rely on 'the datastore' which aggregates published IATI files and acts as an intermediary so users don't need to download data from hundreds of different publishers. If the datastore is down, many usage applications may fail. Recently, when new versions of the official datastore hit technical trouble, an old open source software was brought back initially by volunteers.

Ultimately, this data ecosystem is more resilient than it would otherwise be because it’s built on an open standard. Even if the data entry tool, or datastore become inaccessible, new tools can be rapidly plugged in. But that they will can’t just be taken for granted: data ecosystems need careful management just as natural ones do.

44 of 67

Support standards over apps

45 of 67

Building data ecosystems through standardisation can crowd-in investment: having a significant multiplier effect.

For example, most of the implementations of the Open Contracting Data Standard have not been funded by the Open Contracting Partnership which stewards it’s design - yet they incorporate particular ideas the standard encodes, such as providing linked information on tender and contract award, and providing clear identifiers to the companies involved in procurement.

For the low millions of dollars invested in maintaining OCDS since it’s first half-million dollar year-long development cycle - many many more millions from a myriad of sources have gone into building bespoke and re-usable tools, supported by for-profit and non-profit business models right across the world.

And innovation has come from the edges, such as the adoption of the standard in Ukraine’s corruption-busting e-procurement system, or the creation of tools using the standard to analyse paper-based procurement systems in Nigeria.

46 of 67

Support ≠ build�

Adopt, adapt, engage, shape

47 of 67

48 of 67

Infrastructures

49 of 67

50 of 67

Some components of data infrastructure

Schema & documentation�
Validation tools�
Reference implementations & code�
Reference data�
Data registries�
Aggregators and APIs

51 of 67

52 of 67

See https://www.stateofopendata.od4d.net/ infrastructure chapter for references

53 of 67

Institutions

54 of 67

Standards adoption requires trust�

Ownership, stewardship and institutions matter

55 of 67

56 of 67

57 of 67

58 of 67

Industry (data) standards are set by those who turn up.

For example, over the last few weeks we’ve heard how a key mechanism to meet the goals of the COP26 Climate Conference will be in the actions of finance and investment - and a raft of new standards, many likely backed by technical data standards - are emerging for corporate Environmental, Social and Governance reporting. There are relatively few groups out there like ECOS, the NGO network engaging in technical standards committees to champion sustainability interests - and I’ve struggled to locate any working on data standards - yet, in the current global system of ‘voluntary standardisation’, standards get defined by those who can afford to follow the meetings and discussions and turn-up. Too often, that restricts those shaping standards to corporate and developed government interests alone.

If we are to have a world of technical and data standards that supports social change, we need more support for the social change voices in the room.

59 of 67

Reflections & challenges

60 of 67

As I was preparing this talk, I looked back at The State of Open Data - the book I worked on with IDRC a few years ago to review how open data ecosystems had developed across a wide range of sectors. One of things that struck me when editing the collection was the significant difference between how far different sectors have developed effective ecosystems for standardised, generative, and solution-focussed, data sharing.

Whilst there are some great examples out there of data standards impacting policy, supporting smart data sharing and analysis, and fostering innovation - there are many places where we see dominant private platforms and data ecosystems developing that do not seem to serve the public interest - including, I’d suggest (although this is not my field of expertise) in a number of areas of agriculture.

So I asked myself why? What stops us building effective open standards? I have six closing observations, and in that, challenges, to share.

61 of 67

What causes data standards to fail?

(1) Underinvestment

62 of 67

What causes data standards to fail?

(2) Stopping short of version 2

63 of 67

What causes data standards to fail?

(3) Tailoring to the dominant use case

and

(4) Trying to meet all use cases

Standards are often developed because their author or sponsor has a particular problem or use-case in mind. For GTFS, the General Transit Feed Specification that drives many of the public tranport directions you find in apps like GoogleMaps, that problem was ‘finding the next bus’ in Portland Oreagan. That might be the question that 90% of data users come to the data with; but there are also people asking: “Is the bus stop and bus accessible to me as a wheelchair user?” or “Can we use this standard to describe the informal matatu routes in Nairobi where we don’t have fixed bus stop locations?”

A britle standard that only focusses on the dominant use case will likely crowd out space for these other questions to be answered. But a standard that was designed to try and cater for every single user need would likely collapse under it’s own weight. In the GTFS case, this has been handled by an open governance process that has allowed disability information to become part of the standard over time, and an openness to local extensions.

There is an art and an ethics of standardisation here - and it needs interdisciplinary teams. Which brings me to recap my final learning on where things can go wrong, and what we need to do to design standards well

64 of 67

What causes data standards to fail?

(5) Treating standards as a technical problem

and

(6) Neglecting the technical details

65 of 67

12-10-2021

2021-10-12

Oct 12th 2021

der 12. Oktober 21

1633993200

October 12th 2021

Seshhanbeh 1400 Mehr 20

Mon Oct 11 2021 23:00:00 GMT+0000

12/10/2021

US Date: 10th December

European: 12th October

66 of 67

Closing remarks

67 of 67

Thankyou!

E-mail: tim@practicalparticipation.co.uk

Twitter: @timdavies