Monthly OpenLineage

TSC meeting

​

Aug/11/2021

Recording of calls

​

Reminder:

The meeting is recorded and archived on the wiki

https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

​

2

Roll Call

TSC voting members:

​

Julien Le Dem

Mandy Chessell

Daniel Henneberger

Drew Banin

James Campbell

Ryan Blue

Willy Lulciuc

Zhamak Dehghani

Michael Collado

Maciej Obuchowski

​

3

Communication

​

• Slack: http://bit.ly/OpenLineageSlack
• Mailing list: https://lists.lfaidata.foundation/g/openlineage-tsc
• Confluence: https://wiki.lfaidata.foundation/display/OpenLineage/OpenLineage+Home

​

​

4

Agenda

​

• Coming in OpenLineage 0.1
• OpenLineage 0.2 scope discussion
• Facet versioning mechanism (Issue #153)
• OpenLineage Proxy Backend (Issue #152)
• Kafka client
• Roadmap
• Open discussion

​

5

Coming in OpenLineage 0.1:

https://github.com/OpenLineage/OpenLineage/projects/3

​

• Marquez integrations imported in OpenLineage
• OpenLineage spec versioning
• Clients

​

​

6

OL 0.1: Marquez integrations imported in OpenLineage:

https://github.com/OpenLineage/OpenLineage/tree/main/integration

​

• Apache Airflow:
• BigQuery
• Postgres
• Snowflake
• Redshift
• Great Expectations
• Apache Spark
• dbt

​

7

OL 0.1: spec versioning and clients

​

• OpenLineage spec versioning

"$id": "https://openlineage.io/spec/1-0-0/OpenLineage.json"

follows Schemaver

• Java client generated from the spec
• Python client

​

​

​

​

8

OpenLineage 0.2 scope:

https://github.com/OpenLineage/OpenLineage/projects/4

​

• Version facets separately
• Consumer spec
• Proxy backend

​

9

OL 0.2: Version facets separately

https://github.com/OpenLineage/OpenLineage/issues/153

​

• The core spec is also the consumer spec:

"$id": "https://openlineage.io/spec/1-0-0/OpenLineage.json"

It only defines LineageEvent, Run, Job and Dataset

​

• Each facet is versioned independently:

"$id": "https://openlineage.io/spec/1-0-0/DatasetSchemaFacet.json"

10

OL 0.2: Proxy Backend Proposal

11

Roadmap:

​

• OpenLineage 0.1
• OpenLineage 0.2
• OpenLineage Roadmap

​

12

Open Discussion

13