Monthly OpenLineage

TSC meeting

​

July/14/2021

Recording of calls

​

Reminder:

The meeting is recorded and archived on the wiki

https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

​

2

Roll Call

TSC voting members:

​

Julien Le Dem

Mandy Chessell

Daniel Henneberger

Drew Banin

James Campbell

Ryan Blue

Willy Lulciuc

Zhamak Dehghani

Michael Collado

Maciej Obuchowski

​

3

Communication

​

• Slack: http://bit.ly/OpenLineageSlack
• Mailing list: https://lists.lfaidata.foundation/g/openlineage-tsc
• Confluence: https://wiki.lfaidata.foundation/display/OpenLineage/OpenLineage+Home

​

​

4

Agenda

​

• Finalize the OpenLineage Mission Statement
• Review OpenLineage 0.1 scope
• Roadmap
• Open discussion

​

5

OpenLineage Mission Statement:

https://github.com/OpenLineage/OpenLineage/issues/84

"The mission of the Project is to enable the industry at-large to collect real-time lineage metadata consistently across complex ecosystems, creating a deeper understanding of how data is produced and used"

​

6

OpenLineage 0.1 scope:

https://github.com/OpenLineage/OpenLineage/projects/3

​

• OpenLineage spec versioning
• Marquez integrations import in OpenLineage
• Finalize 0.1 client spec

​

7

OpenLineage Spec versioning

https://github.com/OpenLineage/OpenLineage/issues/63

​

The spec is versioned independently of the libraries.

"$id": "https://openlineage.io/spec/0.1.0/OpenLineage.json"

​

8

Marquez integrations import in OpenLineage:

​

• marquez-airflow -> openlineage-airflow
• marquez-spark -> openlineage-spark

​

9

Finalize 0.1 client spec:

​

• job naming convention
• parent job
• run uuids

​

10

Roadmap:

​

• OpenLineage 0.1
• OpenLineage 0.2
• OpenLineage Roadmap

​

11

Open Discussion

12