1 of 44

Frictionless Data for Reproducible Research

Serah Rono

FORCE 2018. OCTOBER 10, 2018

2 of 44

Open up all essential, public interest information and see it used to create insight that drives positive change

Build communities, tools and skills to empower individuals and organizations to use open information to create insights that drive change.

3 of 44

OpenSpending

okfn.org

Here are a few examples to give you a better idea:

Through our School of Data program we’ve trained thousands of people and dozens of organizations in how to use and understand data.

Open Trials an open, online linked database of information about the world’s clinical research trials.

The Open Data Index provides one way for us to measure open data on a national level. Each year we produce an annual ranking of countries, which is crowdsourced and peer-reviewed

we’ve created an Open Knowledge network of people and organizations in more than forty countries, all working to make information open and see it made used and useful. This includes individuals, organisations as well as governments.

And open spending, built using the fiscal data package from frictionless data, for helping governments to publish fiscal data.

4 of 44

5 of 44

Open Data Publishing Best Practices

6 of 44

Frequency

Author

License

Contact point

Date of publication

Description

7 of 44

8 of 44

9 of 44

http://frictionlessdata.io/data-packages

10 of 44

11 of 44

Standards

12 of 44

13 of 44

O pendefinition.org

14 of 44

Open Contracting Data Standards

Governments spend $9.5 trillion every year on contracts, yet very little of this information is available for public scrutiny;
The Open Contracting Data Standard is a community driven standard for publishing this data;
It models the process of contracting (Planning > Formation > Performance > Completion)
It includes fields that should be included in all contracting data releases and records (e.g. contract value, supplier, period, currency etc)
It is currently a work in progress funded by the Omidyar and Web foundations.

http://ocds.open-contracting.org/

15 of 44

Fiscal Data Package Specification

frictionlessdata.io/specs/fiscal-data-package/

Data on budgets and spending is being made available in unprecedented quantities, we need standards to help citizens work with this data;
Data is supplied as a CSV file. Transaction data requires the fields amount and id and recommended fields description. Aggregated data must contain information on the way it was aggregated using UN and IMF classifications;
The metadata is supplied in a JSON file with required and suggested fields such as name, resources (links to actual data), currency, granularity (transactional or aggregated), type (expenditure or revenue), status (proposal, approved, adjustment, execution), licenses;
There is documentation on how to convert your budget data into the standard using easy to use and open source tools.

The Fiscal Data Package is a lightweight and user-oriented format for publishing and consuming fiscal data. Fiscal Data Packages are made of simple and universal components, are extremely flexible, can be produced from ordinary spreadsheet software and used in any environment.

This specification started about five years ago with a first version (then known as the “Budget Data Package”). Since then we’ve made quite a few iterations, until a fairly stable version was reached, which we name ‘version 0.3’. This version was field-tested in various use cases and scenarios – most prominent among them is the Government of Mexico, who adopted Fiscal Data Package to be used for publishing their official budget data.

In the first half of 2018, the team was hard at work in reshaping this specification to make it simpler to use and easier to adopt, while improving its flexibility and extensibility – thus making it relevant for more users.

A main concern we wanted to tackle was the ability to work with data as it currently exists, without forcing publishers to modify the contents or structure of their current data files in order to “adapt” them to the specification. This is a big deal, as publishers often publish data that’s the output of existing internal information systems, and requiring them to do any sort of data cleaning or wrangling on the data prior to uploading in a major source of friction for adoption.

16 of 44

http://standards.theodi.org

17 of 44

Frictionless Data

18 of 44

Legal barriers

(open data, sharing agreements etc)

Data Quality

Hard to find

Interoperability

No tool integration

So what specific issues do people face while working with data?

Data that we are interested in working with may not be available. On your Left, you can see a summary of all the things that get in the way of publishing data publicly. The reasons for this might be economic. Or political. We’ve made huge progress here in the last decade. These are all the things to do with openness. From my experience this feels like a battle which has largely already been won. If data’s available, people are generally up for sharing it.

The right side of the chart has all the more worrisome barriers that we face, because even when data is available, it is just hard to find, or stored in poor or inconsistent structures. Settling on a proper tool for analysis can also be nightmarish.

All these things are what we call “friction” as they prevent you from being able to get data from where it is generated, to the point where it is used to draw insight with ease.

19 of 44

Introduce a significant, measurable improvement in how data is shared, consumed, and analyzed.
Make it easier to maintain and improve data quality.

20 of 44

Data Package

21 of 44

Data Package

Tabular Data Package

Table Schema

CSV

Tool

e.g. data checking with Goodtables

Tool

e.g. import to R, SQL, BigQuery, etc

(datapackage.json + CSV)

So how do these Data Packages look like. Here’s an example of a Tabular Data Package, which is used to package tabular data in CSV. We have the actual data file or files, in CSV. These generally don’t require any change in themselves. Next to them we have the Data Package descriptor, which is the datapackage.json file.

The datapackage.json file contains basic metadata about the data, things like title, keywords,license contact info and of course pointers to the data resources. For each resource, there is another really powerful specification that can be used to describe them: Table Schema.

This describes what we expect to see in the CSV file, what are the field names, what are the field types (text, number, date), are there any constraints, eg this numeric field can only go from 1 to 100, or this field must contain one value from this predefined list, things like that.

The combination of all this information on the data package descriptor and the data itself is what forms a Tabular Data Package. And that’s pretty much it, you can create these descriptors by hand, or using an online helper, or more likely using some integration with another tool.

What’s exciting of course is what this simple standard way of packaging your data enables now, all the tools that can take advantage of this common specification.

There are libraries for importing Data Packages into R, SQL databases, BigQuery, etc. There is the datapackage-pipeline which is a framework for processing tabular data, and more importantly for the topic of this talk, tools for data validation to help improve the quality of your data.

22 of 44

Frictionless Data Lab

23 of 44

create.frictionlessdata.io

24 of 44

25 of 44

try.goodtables.io

26 of 44

“Can I please get this done in Terminal?”

27 of 44

pip install goodtables

28 of 44

For organisations with big datasets, one time data validation or validation in command-line can be quite hectic. What if we had round the clock, automated validation of datasets for everyone?

We need automatic, continuous data validation.

Data publication can benefit much in the same way as software projects have long benefited from services like Travis CI or Circle CI.

For those who are not familiar with the concept, theses services work by running a series of automatic tests that the developers have created to ensure that the software works as expected. These tests are run every time code is pushed to a shared repository, and the service report back the result to the repository. This allows developers to find errors quickly and be confident that their code works as expected.

These same principles can be applied to data publication. If there is a problem with the data at any stage, checks should fail and publishers should get a report indicating what went wrong. This kind of regular feedback allows to build better, more quality data by allowing data errors to be flagged and fixed early on.

And also crucially helps ensuring that this quality is maintained over time, even if different people or tools are involved in the publication the data.

On the other end it gives consumers the reassurance that data is well structured and can be relied on.

29 of 44

30 of 44

frictionlessdata.io/field-guide/

31 of 44

Recap

Data Packages and Data Package Creator
Data validation in your terminal with goodtables-cli
One-time data validation on the web with try.goodtables.io
Continuous data validation with goodtables.io

32 of 44

Frictionless Data

&

FAIR Research

33 of 44

Findability

Unique identifier 👍
Rich metadata 👍
Identifier as part of metadata 👍
Metadata indexed in a central searchable resource ?

34 of 44

Accessibility

Metadata is accessible, independent of associated data 👍
Metadata can be retrieved by the unique identifier 👍

Protocol allows for an authentication and authorization procedure
Protocol is open, free and can be adopted widely 👍

35 of 44

Interoperability

Provisions for metadata to reference other metadata 👍
Metadata should also be FAIR - findable, accessible, interoperable, reusable 👍
Metadata should use accessible and shared language 👍

36 of 44

Reusability

Metadata should include a wide range of relevant attributes

Metadata should include a data license 👍
Metadata should use standardised formats - that are easily understood and widely applicable across domains 👍
Metadata should link back to original sources (provenance) 👍

37 of 44

Frictionless Data Research

Pilots & Case Studies

38 of 44

http://frictionlessdata.io/articles/cmso/

39 of 44

http://frictionlessdata.io/articles/ukds/

Data quality and reuse of archived data

40 of 44

http://frictionlessdata.io/articles/open-power-system-data/

Packaging Energy Data

41 of 44

http://frictionlessdata.io/articles/pacific-northwest-national-laboratory-active-data-biology/

Validating metadata associated with ADBio repositories

42 of 44

What next?

Adopt our specs (frictionlessdata.io/specs)
use software (frictionlessdata.io/software)
Pilots and collaborations
GitHub issues and PRs
Join our community

43 of 44

Join our community

http://github.com/frictionlessdata/

discuss.okfn.org/c/frictionless-data

gitter.im/frictionlessdata/chat

youtube.com/user/openknowledgefdn

Twitter: @okfn . @okfnlabs

44 of 44

Serah Rono . @serahrono . serah.rono@okfn.org