Vladimir Alexiev, PhD, PMP

Ontotext Corp, Sofia, Bulgaria

Generation of Declarative Transformations from Semantic Models

Outline

• Building Knowledge Graphs in 10 steps
• Semantic Modeling vs Ontology Engineering; Polyglot Modeling
• rdfpuml: semantic models from turtle examples
• rdf2sparql: tabular transforms from turtle examples
• rdf2rml: R2RML transforms from turtle examples
• Backup slides: Semantic ETL desiderata
• Next time: tsv2owl, tsv2soml, owl2soml: ontologies and Semantic Object models from tabular data

​

​

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

• I’m Chief Data Architect at Ontotext, working at the Knowledge Graph Solutions group. We build KGs for a living
• Above is a simple 10-step recipe for KG building
• Webinar: Knowledge Graphs: 5 Use Cases and 10 Steps to Get There
• Video: Building Knowledge Graphs in 10 Steps
• This presentation is about steps 4 (models) and 5 (semantic ETL)

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

Semantic Modeling

4

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

Semantic Modeling vs Ontology Engineering

• Ontology engineering:
• Know methodologies, modeling principles, top-level ontologies
• Create ontologies mostly from scratch (green-field)
• Semantic modeling:
• Research existing data standards and assets, harmonize them
• Know and reuse existing ontologies, understand tradeoffs
• Add classes and props when needed
• Make holistic models that show how ontologies are used together
• Document (human readable) and make machine readable artifacts (application profiles, RDF shapes, APIs, etc)
• For me semantic modeling is more than ontologies
• Also, we like to do it in a very pragmatic way

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

Polyglot Modeling

• Simple, technology-independent models
• Understandable by domain experts: increases participation
• Generate various tech artefacts: semantic web doesn’t (yet) rule the world!
• Diagrams, documentation
• RDFS, OWL, SHACL, SHEX, JSON-LD Context and Frame, 
• JSON schema, Elastic object model, etc, etc
• "Write once. Use many. Creative laziness encouraged" [RAML]
• Examples:
• FHIR: domain model to XML, JSON, JSONLD; XSD, JSON Schema, SHEX
• LinkML: YAML-based models to various technical artefacts incl. JSON Schema, ontology, SHEX
• Ontotext SOML: YAML-based model to SHACL and GraphQL (exposes KG parts to GraphQL)
• Schema Salad (part of CWL): YAML-based model to JSON Schema and RDFS
• A.ML and CloudInformationModel: YAML to RDFS, SHACL, SQL, R2RML
• RAML (RESTful API Modeling Language): YAML to APIs (ala OpenAPI specifications)
• Links:
• Many of these are in YAML: YAML-LD work group started (part of JSON-LD CG), yaml-ld#19 is about polyglot modeling
• JSON-LD, YAML-LD and Polyglot Modeling. Alexiev, V. presentation, October 2022. slides   link
• Data Spaces vs Knowledge Graphs: How to Get To Semantic Data Spaces? Alexiev, V. In Data Spaces & Semantic Interoperability Workshop, Vienna, Austria, July 2022. Paper   slides   blog   bibbase

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

Semantic Models from Turtle Examples

• https://github.com/VladimirAlexiev/rdf2rml: several open source tools. The first tool is:
• rdfpuml: generates model diagrams from examples in turtle. Lots of information and precision:
• Ontologies, specific classes and properties
• URL patterns
• Datatypes and language tags
• Simple data cleaning (e.g. SPLIT, URLIFY, FIX_DATE)
• Use of source data (in parentheses)
• Combining fields into the same property (e.g. alias1,2,3 mapped to multi-valued cb:alias)
• Data of some nodes comes from multiple tables
• This is an actionable model, no hand-waving here
• RDF by Example: rdfpuml for True RDF Diagrams, rdf2rml for R2RML Generation. Alexiev, V. Semantic Web in Libraries 2016 (SWIB 2016), Bonn, Germany, November 2016. slides   html   video   bibbase 

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

Crunchbase Semantic Model

• Crunchbase is a popular company dataset
• 18 CSV tables (with uuid and updated_at), 10M records, updated daily
• 2M companies, 929k detailed descriptions, 22k org parent links, 429k event appearances
• 1.5M people, 848k descriptions, 2.2M job positions, 495k degrees (education records)
• 223k investors, 488k funding rounds, 783k investments (participation in funding round), 137k acquisitions, 48k IPOs
• 750 industry classifications
• (Full image in slightly different style)

8

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

Focusing on Initial Public Offerings (IPOs)

Simple custom ontology:

• Exchange (exchangeSymbol) and ticker (stockSymbol) (e.g. NASDAQ:MSFT)
• Who (organization), when (wentPublicOn)
• Financial: moneyRaised, sharePrice, valuationPrice (in local currency and USD)
• cbPermalink, cbUrl (e.g. Microsoft IPO), CB rank
• Bookkeeping: uuid, createdAt, updatedAt (allow daily update)

Model elements:

• URL patterns
• Datatypes (e.g. ^^xsd:integer)
• Simple fixes (e.g. fixDate)

​

9

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

Generating Tabular Transforms

10

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

Tabular Transforms From Turtle Examples

• rdf2sparql: generates SPARQL queries for transforming tabular data
• UPDATE for Ontotext Refine
• ☺ Update directly to semantic repository: faster
• ☺ Can join tabular to RDF data in a repository.
• ☺ “Named-graph per table" for updates (locality and idempotence)
• ☺ For CB we used “named graph per row” for fast daily update: GraphDB handles 10M named graphs just fine
• CONSTRUCT for Ontotext Refine.
• ☹ Generates intermediate RDF file: slower
• ☺ Can join tabular to RDF data in a repository
• CONSTRUCT queries for TARQL.
• ☹ Generates intermediate RDF file: slower
• ☹ Cannot join tabular to RDF data in a repository
• Handles datatypes, templated URLs, simple data cleaning (functions/macros)
• Chains functions applied to the same field �(see example on the right)

​

• Turtle model using 2 functions:

cb:employeeCount <cb/employeeCount/urlify(ifNotNull(employee_count))>;

• Macros that define the functions:

#define urlify(x) bind(LCASE(REPLACE(REPLACE(REPLACE� (x, "[^\\p{L}0-9]", "_"), "_+", "_"), "^_|_$", "")) as x##_URLIFY)

#define ifNotNull(x) bind(if(x in� ("other","not provided","unknown"),?UNDEF,x) as x##_IFNOTNULL)

• SPARQL CONSTRUCT/UPDATE:

cb:employeeCount� ?cb_employeeCount_employee_count_IFNOTNULL_URLIFY_URL;

• SPARQL WHERE binds to compute the variable:

bind(if(?employee_count in� ("other","not provided","unknown"),?UNDEF, ?employee_count)� as ?employee_count_IFNOTNULL) bind(LCASE(REPLACE(REPLACE(REPLACE(?employee_count_IFNOTNULL,� "[^\\p{L}0-9]", "_"), "_+", "_"), "^_|_$", "")) � as ?employee_count_IFNOTNULL_URLIFY) bind(iri(concat("cb/employeeCount/",� ?employee_count_IFNOTNULL_URLIFY)) as� ?cb_employeeCount_employee_count_IFNOTNULL_URLIFY_URL)

11

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

CB IPOs: From Model to SPARQL UPDATE

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

Could Go a Lot More Complex: IPOs in FIBO

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

IPOs in FIBO: Turtle model (165 lines)

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

IPOs in FIBO: SPARQL (220 lines, 56 binds, 21 prefixes)

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

Generating R2RML Transforms

16

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

R2RML Transforms From Turtle Examples

• R2RML: W3C language to map RDBMS to RDF
• rdf2rml: generates R2RML transforms from Turtle examples
• Table/join in node comment; carried to “children” nodes
• Field in attribute/URL
• Both the model and R2RML are RDF, so implemented as some SPARQL transformation
• TODO: implement functions
• Example: Exhibitions from J.P.Getty Museum
• Then can use any R2RML implementation:
• ONTOP for RDBMS Virtualization (OBDA)
• GraphDB Virtual Repositories for Virtualization or Materialization
• Various implementations, see awesome-semantic-web: r2rml:�Antidot d2btriples, Morph-KGC, RMLmapper, RocketRML, carml…

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

Getty Exhibitions: Turtle model

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

Getty Exhibitions: Generated R2RML

• R2RML is quite verbose:
• subject map,
• predicateObject map,
• Templates and details about each URL and literal, etc.
• The one circled node is expanded to 15 nodes

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

Getty Exhibitions: Model (60 lines) vs R2RML (315 lines)

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

Insurance Customer Example: Model

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

Insurance Example: Model (52 lines) vs R2RML (502 lines)

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

Backup slides

Will present them if there is time

Semantic ETL Desiderata

Main requirements for semantic ETL�(green: yes or positive, orange: maybe or warning, red: not or negative)

24

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

ETL and Modeling

• Declarative approach: conversion scripts or queries should be written in a high-level language (such as SPARQL, XSPARQL, YARRML, LIXR). This ensures correctness and easier maintainability.
• Generation: conversion scripts should be generated from semantic models, which are also used to generate validation scripts. This saves further effort, ensures correctness and easier maintainability. 
• Polyglot Modeling: use the same set of models to generate a variety of technical artefacts, ensuring they are consistent and up to date: conversions, validations, JSON-LD contexts and frames, UI forms, etc.

​

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

ETL Performance

• Streaming/chunking:
• Tools that collect RDF output to a memory RDF model and then output it only work for thousands of records.
• To work for millions records, the tool must output each piece of RDF as soon as it's ready, otherwise memory requirements and execution time grow dramatically. OntoRefine and TARQL do.
• One pass: ETL tools should read the input data only once.
• OntoRefine does (on simple tabular data).
• R2RML mappers should process all maps applying to the same query/table at once for each row, instead of iterating over the same resultset/table multiple times.
• Direct update:
• ETL tool should be able to output directly to a semantic repository (e.g. GraphDB), both for ingest and update. OntoRefine does, TARQL doesn’t.
• This also allows to join to existing semantic data, e.g. to resolve keywords to nomenclature nodes.

​

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

ETL Logistics

• SRGO Methodology
• Steps, Reasoning, Repos/Graphs, Ordering, Ontologies, Reasoning.
• Strictly describe the What, Where, When, How of a pipeline.
• Prerequisite for:
• Making the pipeline reproducible
• Making the KG building process sustainable in the long run
• What: are the steps of the pipeline process.
• RDF (usually Turtle): obtained from LOD, written by hand (small), converted once (static)
• Non-RDF data: where to get them from, what credentials are needed
• Ingestion scripts (declarative or procedural)
• Virtualization transforms: ONTOP or R2RML mappings; MongoDB JSON connector
• Transformation/correction: SPARQL Updates that can transform data, derive new data, add missing data, correct fields
• Validation steps: SHACL, SHEX or SPARQL validations

​

​

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

ETL Logistics (2)

• Where the steps take place:
• Several Repositories for data separation or independent update cycles
• GraphDB Internal Federation eliminates network overhead and improves security
• Named Graphs for:
• Unit of Work
• Idempotence
• Unit of Transfer
• Provenance
• When each step is executed:
• The graph nature of RDF and GraphDB's incremental inference ensure that RDF steps are compositional, i.e. can be done in any order
• Other steps have ordering requirements, which are usually sparse. This partial order needs to be resolved to a total order for step execution
• How: what inference is needed by the pipeline
• Reasoning ruleset: RDFS, RDFS+, OWL-Horst, OWL RL, OWL QL, etc
• Ontologies loaded. Many ontologies come with heavy inference (part of heavy "ontological commitment") that you may not need

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

ETL Predictability and Transactionality

• Locality:
• Each ETL step should operate in its own graph and should not disturb any data that is the responsibility of other steps.
• Depends on ETL logistics and using named graphs
• Predictability and Idempotence:
• The results of each step should not depend on the previous state of its own graph (but can depend on the data of previous steps), in order to be predictable and reproducible.
• Running the same step twice should produce the same result (idempotence). Both satisfied by OntoRefine with UPDATE queries.
• Transactionality:
• Each step should leave the total KG operational and with full data, no matter how long it takes.
• ETL should use transactional (ACID) semantics or use a staging data store, in order to avoid prolonged down-time of the KG.

​

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023

ETL Virtualization and Automation

• NoETL: if source data is very large (Volume) or changes very frequently (Velocity), it may be better to use data in its original form and provide SPARQL querying over it, rather than converting it to RDF
• Relational Virtualization using ONTOP
• JSON "virtualization" using MongoDB Integration
• Orchestration: for complex ETL pipelines or when one needs to track multiple ETL instances
• Use orchestration tools like CWL, Airflow, Apache HOP, etc
• Automation: to ensure reproducibility in different developer and production environments, package up complex steps and whole pipelines
• Using automated deployment tools like Docker, Kubernetes, Helm.

​

Generation of Declarative Transformations from Semantic Models. ENDORSE 2023