Monthly OpenLineage

TSC meeting

​

Nov/10/2021

Recording of calls

​

Reminder:

The meeting is recorded and archived on the wiki

https://wiki.lfaidata.foundation/display/OpenLineage/Monthly+TSC+meeting

​

2

Roll Call

TSC voting members:

​

Julien Le Dem

Mandy Chessell

Daniel Henneberger

Drew Banin

James Campbell

Ryan Blue

Willy Lulciuc

Zhamak Dehghani

Michael Collado

Maciej Obuchowski

​

3

Communication

​

• Slack: http://bit.ly/OpenLineageSlack
• Mailing list: https://lists.lfaidata.foundation/g/openlineage-tsc
• Confluence: https://wiki.lfaidata.foundation/display/OpenLineage/OpenLineage+Home

​

​

4

Agenda

• OL Client for Iceberg use cases
• Proxy Backend progress update (Issue #152) [Mandy]
• OpenLineage last release overview (0.3.1)
• Facet versioning
• Airflow 2 / Spark 3 support, dbt improvements
• OpenLineage 0.4 scope review
• Proxy Backend (Issue #152)
• Spark, Airflow, dbt improvements (documentation, coverage, ...)
• improvements to the OpenLineage model
• Open discussion

​

​

5

Iceberg requirements

​

• Ref: OpenLineage/issues/167
• adding facets through the client

6

OL 0.4: Proxy Backend Progress

7

Complete

Egeria Implementation Complete

8

Example Output

9

Egeria’s OpenLineage log store

Suggested new facets

• InputDataFacet
• RequestFacet covering input parameters that control job behavior
• OutputDataFacet
• ResultsFacet covering calculated results passed to caller rather than stored in a data source
• DataFacet
• ResourceTypeFacet
• Type information for the input/output data source
• RunFacet
• CheckpointFacet
• For long running processes to log stats and status (including errors and warnings) - part of an “OTHER” event

10

OpenLineage last release overview (0.3.1):

​

• Facet versioning
• Spark improvements
• Spark 3.x
• datasource coverage
• Airflow 2.0 support
• dbt coverage improvements

​

See: 0.2.3...0.3.1

​

11

OpenLineage 0.4 scope:

https://github.com/OpenLineage/OpenLineage/projects/6

​

• Proxy backend progress
• Spark, Airflow, dbt improvements (documentation, coverage, ...)
• Improvements to the OpenLineage model
• Timestamps
• Field lengths
• Treatment of unrecognized facets

​

12

Roadmap Discussion:

​

• Iceberg requirements

13

Roadmap:

​

• OpenLineage Roadmap

​

14

Open Discussion

15