Monthly OpenLineage

TSC meeting

​

Dec/08/2021

Recording of calls

​

Reminder:

The meeting is recorded and archived on the wiki

https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

​

2

Roll Call

TSC voting members:

​

Julien Le Dem

Mandy Chessell

Daniel Henneberger

Drew Banin

James Campbell

Ryan Blue

Willy Lulciuc

Zhamak Dehghani

Michael Collado

Maciej Obuchowski

​

3

Communication

​

• Slack: http://bit.ly/OpenLineageSlack
• Mailing list: https://lists.lfaidata.foundation/g/openlineage-tsc
• Confluence: https://wiki.lfaidata.foundation/display/OpenLineage/OpenLineage+Home

​

​

4

Agenda

• SPDX headers [Mandy Chessel]
• Azure Purview + OpenLineage [Will Johnson, Mark Taylor]
• Logging backend (OpenTelemetry, ...) [Julien Le Dem]
• Open discussion

​

​

5

Software Package Data Exchange (SPDX)

• SPDX is an open standard for creating software bill of materials
• SPDX includes a set of short identifiers that identify the different open source licenses.
• Both human readable and machine processable
• Easy to maintain and validate
• Full license is added in LICENSE file at top of git repository
• Each file includes the SPDX-License-Identifier tag

6

Replaces

Azure Purview + OpenLineage

​

• Spark lineage in Azure Purview
• Spark integrations for Azure connectors to Azure data sources like synapse and cosmos db

7

OpenLineage on Azure PaaS

8

More to Come!

​

• I: 333 | PR: 375 (Init Script)✅
• I: 407 | PR: 425 (url param)❓
• I: 181 Extensibility ❓
• Databricks Environment Data❓
• Cosmos DB (DataSourceV2)❓
• Synapse (SqlDWRelation)❓

Logging backends

​

• OpenTelemetry

9

Roadmap:

​

• OpenLineage Roadmap

​

10

Open Discussion

11