DDI-CDI: Optimising Your Data Description for Integration and Reuse

24 March 2023

RDA Plenary 20

Gothenburg

Topics

• General Introduction
• Model Features
• DDI-CDI Specification: Model-Driven Development
• User Communities: Implementation Guides
• Use Case: DDI-CDI and the UK Data Service
• Use Case: ESS: Integration with Environmental and Air Quality Data

Data Documentation Initiative Cross-Domain Integration (DDI-CDI): General Introduction

Background

• The Data Documentation Initiative has produced metadata specifications for research data in the social, behavioural, and economic (SBE) sciences for two decades
• DDI Codebook (after-the-fact description of data in XML)
• DDI Lifecycle (metadata for all aspects of data production, management and dissemination in XML)
• With a demand for new types of data and support for new technologies, a new specification was needed: DDI-CDI
• Not specific to the SBE domain
• Model-driven: support for different syntaxes/technologies
• To be used as a complement to other domain standards to help with data reuse across domain/infrastructure boundaries: FAIR

A New DDI Specification

• DDI-CDI is a new specification
• Currently in revision after public review
• Release in Summer 2021
• DDI-CDI is an implementation of the “DDI 4”/”DDI Moving Forward” model
• Specific focus on cross-domain data integration
• Model-based standard
• XML and other syntax representations supported
• Designed to be machine-actionable
• Complementary to other DDI specifications
• Works with DDI Codebook and DDI Lifecycle
• Extends metadata coverage to support integration with other domain data
• Can work with other (non-DDI) domain metadata specifications

Why a New Specification?

• The volume of research data is increasing exponentially
• New sources
• New formats/structures
• The use of data across domain boundaries is increasing
• “Grand challenges” (e.g., COVID-19, climate change)
• New technologies and new approached (e.g., AI, machine learning)
• Problems of scale demand machine-actionability
• For metadata harvesting
• For navigating data at all levels and across domain boundries

• Cross-domain (and cross-platform) specification
• Formal UML model (Canonical XMI)
• Multiple syntax representations (XML, RDF/OWL, etc.)
• Supports a variety of data production environments:
• traditional research projects
• modern data platforms
• data science and analytics
• interoperable metadata-driven solutions
• Enables the development of object libraries that can be used in microservices
• Integrate/aligns with a variety of standards, including other DDI products
• Includes:
• Multiple data structures and content, e.g. wide/long data files, aggregate multidimensional data, unit and anonymized micro-data, data cubes, and semi-structured key-value pairs, multimedia content.
• A process model capable of describing complex data pipelines and provenance.
• Conceptual aspects of activities and their resulting data, e.g. variables, classifications, concepts.

​

​

DDI-CDI main aspects

DDI-CDI Functionality

• Structural description across diverse sources/types of data
• Wide (rectangular) data
• Tall (long), event data, sensors
• Key-vale data (big data, No SQL data)
• Multi-dimensional data cubes, indicators, time series
• Describes provenance of data between different structures/forms
• Processing framework
• Relies on other standards (PROV-O, SDTL, etc.)
• Describes data at an atomic level
• Variables
• Datums/cells

DDI-CDI at-a-glance

Data Organization

Concepts & Representations

Process Description

Variable Cascade

• Variable descriptions at a high level, e.g. conceptual domains
• Early design data capture/ intake
• Broad search and discovery
• Least specific/Most reusable

DDI-CDI variable cascade – Conceptual

• Variable descriptions at a detailed level, e.g. value domains
• Advanced design for all stages of data lifecycle
• Specific search and discovery
• More specific/Less reusable

DDI-CDI variable cascade – Representation

• Physical data description, e.g. physical data types
• Use of a variable in specific data instances
• Data search and discovery
• No reusable

DDI-CDI variable cascade – Instance

Legalmaritalstatus�(conceptual variable)

MARITALB

2008�(variable)

MARITAL�(represented variable)

MARITAL

2004�(variable)

MARITALB

2018�(variable)

MARITALB�(represented variable)

Represented variable

Common variable specification with a code representation

Conceptual variable�Common variable specification without a representation

Instance Variable

Variable specification within a dataset context

Example: comparability and traceability

• Two variables in different data sets might:
• Measure the same concept differently
• Measure the same concept in the same way with different physical representations
• Exist identically in two data sets, but with no formal link
• In all of these cases, understanding the variables at each level (conceptual, representational, and actual) provides a strong basis for programmatically identifying them as potential points for joining data sets

Application: comparability and traceability

• Integrating data across domains involves both dealing with different kinds of discipline’s structures and vocabularies
• Sensor data streams in tall structures
• Survey data in wide structures
• Administrative summary data in cubes
• New types of data (social media, etc.) in “data lakes”
• Event-related data
• A standard also needs to be discipline agnostic.
• Vocabularies need to be referenced, not built in
• Semantics for defining variables, concepts, categories, universes, populations, units, etc.
• Structures reference external controlled vocabularies/ontologies
• A standard needs to be able to at least reference metadata in other disciplines/standards.
• This, of course, presents challenges for machine actionability.

15

Application: cross domain integration

Describing Individual “Datums”

• DDI-CDI tracks how variables can be re-packaged in different structure
• It also can describe the reuse of individual values: datums
• This is important for tying the data to the higher-level structures in DDI-CDI which are employed in data management, dissemination, and integration

​

17

Datums – Holding data, variables and concepts together

• DDI-CDI can describe four different data structures
• Wide – as with unit records
• Long – as with event or stream data
• Key value – as in a key-value store
• Dimensional – as with aggregate data

​

​

​

DDI-CDI data description – Data structures

(Unit)

Identifier

Component

Measure

Component

Attribute

Component

Example 1: data in wide form

IdentifierComponent

VariableValueComponent

AttributeComponent

VariableDescriptorComponent

The Variable Descriptor Component has values taken from the list of non-Unit Identifiers in the wide data set.

​

The “key” for each value is composed from

the Identifier and the Variable Descriptor, and may include non-transposed components, e.g. DateTime.

Key members

Example1: data in long form

Aggregate fisheries statistics, Year 2019 – Norwegian boats – Round weight in tons

Long table (micro-data set)

Multi-dimensional “keys” are identifiers formed from the set of dimension values (variables in long table).

Example: data in dimensional form

Wide

Key-Value

Semantic Sensor Network Ontology (SSN) example

Example: data in key-value form

The DDI-CDI Process Model

• Describes the use of individual processes, and how they fit together
• Supports standard descriptions (SDTL, VTL) and specific languages (SQL, R, STAT, SPSS, Python, SAS, etc.)
• Three “modes”:
• Procedural: Step-wise, with decision points
• Declarative: “Black box” multi-threaded, uses a “playbook” and configurations
• Hybrid approaches of the two

Simple Diagram

Activity

Step

Sub-Step

Control Logic

Inputs

Outputs

PROCEDURAL PROCESS

DECLARATIVE PROCESS

Process

Engine

Parameters

Playbook

(Functions)

Inputs

Outputs

Process Model (High Level)

Datum: Bringing it Together

Detailed Documentation

• There is an integrated browser for seeing the DDI-CDI model and the integrated syntax representations:

https://ddi-alliance.bitbucket.io/DDI-CDI/DDI-CDI_2022-10-06_rdf/doc/_build/DDICDILibrary/index.html

Current Status

• Final draft of version 1.0 nearing completion
• Public review and voting starts June 2023
• Publication in early Fall 2023

User Communities: Implementation Guides and Syntaxes

Using the DDI-CDI Model

• DDI-CDI is of necessity a complex model
• The portion needed to support specific applications/communitiues is much smaller
• To enable user communities to more easily implement the specification, we will use implementation guides:
• Subset of the classes and properties in the model
• Specification of controlled vocabularies/ontologies to be employed for domain semantics
• Specification of syntax representation
• Integration with other standards

UK Data Services subset: Wide Data

UK Data Services subset: Long Data

Choosing Syntax Representations

• The model can be implemented in a variety of syntaxes
• RDF and XML provided as reference syntaxes
• Other possibilities being explored (Python, R, etc.)
• User Communities can specify their own syntaxes
• Subsets of reference syntaxes or entirely different ones
• Can leverage other standards which are limited to specific syntaxes (XML standards, RDF, standards, etc.)
• Reference syntaxes ensure more interoperability at the technical level

​

DDI-CDI and Other Standards

There are a lot of standards!

• DDI-CDI was explicitly designed to fill gaps in the available standards for cross-domain data use at a detailed level
• We have ontologies and classifications/concepts (OWL, SKOS, XKOS, etc.)
• We have standards for provenance (PROV-O)
• We have standards for domain data description (GSIM, OMOP CDM, Darwin Core, etc.)
• DDI-CDI is intended to fill two important gaps:
• Domain-neutral description of structure and processing flows
• “Glue” to connect other standards into a meaningful, complete picture

​

​

36

DDI-CDI and other standards

GSBPM

GSIM

Information Capabilities

(CSDA)

DCAT

other implementation Standards…

implements

enables

enables

integrates

integrates

integrates

integrates

DDI-CDI

integrates

integrates

PROV-O

SKOS

Features for standards integration

• Support for external identification systems
• URNs (URLs, URIs)
• Other formal systems (DOIs, ARKs, UUIDs, etc.)
• Other system-specific systems
• Use of implementation guides to specify points of interaction
• References to other standards for unsupported metadata
• “Study”-level metadata/cataloging information
• Detailed data collection information (DDI Lifecycle Survey Instrument, etc.)
• “Superset” of features in other standards
• Can generate PROV-O from process description
• Can generate DCAT/Schema.org from data description (as appropriate)
• Can reference additional metadata from CDI content expressions

A few examples…

• Schema.org
• DCAT
• WorldFAIR: The Cross-Domain Interoperability Framework (CDIF)

Schema.org (Example)

• WorldFAIR WP 7 (Public Health) is using Schema.org to describe experiments
• An extended version of Schema.org is used
• “Hand off” to DDI-CDI to describe wide variety of data at all levels/in all forms

Example: DCAT and DDI-CDI (1)

Slide credit: Arofan Gregory

Example: DCAT and DDI-CDI (2)

DCAT Instance

(Distribution, Service, etc.)

DDI-CDI Instance

(Proposed additional property)

Metadata

What is the Cross-Domain Interoperability Framework (CDIF)?

• A set of recommended practices for using a coordinated set of domain-agnostic standards – most often as specific subsets or profiles of those standards – to support a core set of functions for cross-domain FAIR reuse.
• Addresses the “agreed information” requirement for cross-domain FAIR
• Suggests a list of needed functions for supporting these exchanges
• Provides a foundation for the creation of registries/applications/services
• Does not provide a classification of domains/infrastructures

Many-to-Many or Many-to-One?

FAIR Activities and Standards

• Foundational: FAIR Digital Object Framework (FDOF)
• Find
• Discover FAIR resources and explore/evaluate their utility prior to access (coverage, etc.)
• Schema.org, DCAT
• Access
• Negotiate access to non-public data
• Enhance efficiencies through automation
• ODRL, DPV, DUO
• Assess/Integrate
• Understand data structure (DDI-CDI)
• Understand semantics (SKOS/XKOS, OWL, SSSOM)
• Determine origination/context (PROV-O, I-ADOPT/O&M)
• Establish connections to higher-level research and management processes
• Track citation/use
• Connect to ROI assessments
• CERIF, GAMSO

CDIF and DDI-CDI

• Provides detailed, granular description of data
• Structural “roles” for semantics
• Provides description of processes to connect data and processing
• Acts as glue between other standards
• Creation of DDI-CDI from domain standards is possible in an automated fashion