Shape Expressions aka ShEx!

Data, you better get in shape...

What are we trying to do?

We want to enable humans and machines to find items that do not fit a certain shape in order to find mistakes and omissions in our data.

We want to increase the confidence in our data.

Who are we (not) doing this for?

Concentrate on:

• Niels - active editor on Wikidata
• Alex - data user in a small start-up

​

Also:

• Wiki projects caring about particular data like WikiCite, GeneWiki, ...

Keep in mind:

• Eloise - occasional Wikidata editor
• Liam - large data re-user

Usage scenarios (non-exhaustive list)

• I'd like to find existing schemas that are relevant to me
• I'd like to understand what an existing schema does
• I’d like to discuss an existing schema and form modelling consensus
• I’d like to adapt an existing schema
• I'd like to be able to store a new schema
• I’d like to be able to test a set of entities against an existing schema

I'd like to find existing schemas that are relevant to me

1. Go to wiki project that represents your interest
2. Find a list of schemas they care about

​

• Later: add categories to schemas to make it easier to maintain a larger number of schemas

I'd like to understand what an existing schema does

1. Go to schema
2. Read short description
3. Read longer description free-text

I’d like to discuss an existing schema and form modelling consensus

1. Go to schema
2. Go to schema talk page
3. Open new discussion

I’d like to adapt an existing schema

1. Go to schema
2. Click edit on schema field
3. Make edit
4. Click publish/save

I'd like to be able to store a new schema

1. Go to SpecialPage Special:NewSchema
2. Enter Label and Description
3. Click publish/save
4. Be redirected to new schema page
5. Edit to add actual schema

I’d like to be able to test a set of entities against an existing schema

1. Go to schema
2. Click “test data against this schema”
3. Go to schema checking service
4. Enter SPARQL query or list of entity IDs
5. Click check
6. Get list of entities with issues

What’s the current state?

• Shape Expressions are used by some projects (GeneWiki and Digital Preservation especially)
• There are competing standards but they are not used around Wikidata
• External services exist to check a Shape Expression against a query result

Milestones/breakdown

1. Now: Allow storing of schemas
• New namespace to hold schema entities (called "Schema:")
• Identified by unique ID
• Has Label, Description and Aliases
• SpecialPage to create a new schema entity
• Link to external tool to check a set of entities against the current schema
2. Next: Allow storing of explanatory text and categorizing schemas
• Store some explanatory text to clarify what the schema is about
• Categorize schemas to manage the growing number of schemas
• Indicate the type of schema it is since we probably don't want to be bound to ShEx forever (using MCR?)
3. Later: Allow checking the current entity against a schema
• Check the current entity against a schema on user-demand

What do we want to measure and learn for the first iteration?

• Do Shape Expressions work for our editors?
• Are Shape Expressions helping wiki projects keep their data in shape more easily?
• How would our editors like to see it improved and why?
• Is there a lot of conflict around defining the schemas?
• Do the schemas lead to an unintended lack of diversity in our knowledge?

Additions

Clarifications after first bonfire meeting

Example schema

See for example here for a Shape Expression for file formats.

What’s the current state?

• Shape Expressions are used by some projects (GeneWiki and Digital Preservation especially)
• There are competing standards but they are not used around Wikidata
• External services exist to check a Shape Expression against a query result
• Tool by Lucas exists to infer a Shape Expression from a given set of Items

Shape vs. schema

Schema:

• Consists of one or more shapes
• (plus a few other declarations – prefixes, default shape, etc.)
• Examples:
• Human, incl. shapes for “human”, “place” (of birth/death), “country” (of citizenship), and “name” (given/family)
• Book, incl. shapes for “book”, “edition”, “organization” (publisher, printer), and “human” (author, translator, illustrator)

Shape:

• Basic building block
• Examples:
• Shape to describe a human as having human parents and children, a date and place of birth and optionally death, and a country of citizenship (where “place” and “country” could be other shapes)
• Shape to describe a “date of birth” statement, requiring at least one reference (where another shape could describe what the reference looks like)

Do we store schemas or shapes?

We could go either way. Both have benefits and drawbacks:

• Schemas:
• fits the mental model our users expect (and currently use)
• lends itself well to have discussions around it
• can’t easily avoid duplication of similar shapes between schemas (think all painters and authors require similar checks to make sure they are humans)
• Shapes:
• Very detailed/small units
• Easily reusable in different schemas
• Easy referencing
• Difficult to see the “full picture”

-> We go with Schemas as the central unit of our data model because the mental model is crucial and not re-using huge shapes that specify everything to the smallest detail might actually solve the potential issue of enforcing too strict schemas on everything

Data model/mockup

Data model/mockup

Do we validate the input or not?

• We generally assume our editors put in valid Shape Expressions
• We do some basic regex checking to make sure that the shapes in a schema have the right IDs
• Initially, schemas must be self-contained: their shapes must only refer to other shapes from the same schema (to validate this: a schema reference always begins with the character "@", so we reject any schemas containing an "@" not followed by the schema ID).
• We can do more validation later if it turns out to be necessary

Schemas vs. constraints

Schemas:

• Class-centric (kinda – can be anything-centric)
• Apply to a definable subset of all entities
• Violation are not shown (yet) on individual statements
• Result is a violation or compliance on an entity (but can often be traced back to an individual statement or a chain of statements)

Constraints:

• Property-centric
• Apply to all uses of the property on all entities
• Violations are shown to logged-in users next to the individual statement causing the violation
• Result is a violation or compliance on a statement