1 of 14

How Intuit contributes to OpenLineage

Athitya Kumar, Senior Software Engineer & OSS Tech Community Lead

Sept 7th 2024

1

2 of 14

MISSION

Intuit Confidential and Proprietary

2

3 of 14

opensource.intuit.com

Athitya

3

4 of 14

Agenda

Lineage What is Data Lineage, and why is it needed?
Context Technical background of Big Data (like Spark, Flink, etc), Listeners & how OpenLineage works.
OpenLineage @ Intuit Story of integrating OpenLineage with Intuit Big Data: Impact, Challenges, Solutions & OSS Contributions!

Athitya

4

5 of 14

We assumed that no one is using this table and we wanted to save cost of any unused tables, so we dropped them without intimation.��We didn’t know that someone is using this topic and we had to evolve the schema, so we made a breaking change without intimation.��We follow a spray-and-pray method; by doing @here on 5 different Slack groups, while making any changes.

�— DATA PRODUCERS

We are building ML models, but we don’t understand the underlying data.��We are building critical dashboards and there are some fields missing/incorrect in the underlying data, but we don’t even know whom to reach out to!

�— DATA CONSUMERS

We don’t know what’s the impact of this data pipeline failure.

�We don’t have any visibility into the criticality of a table.

��— INCIDENT MANAGERS

5

Athitya

5

6 of 14

Why?

�We want:�

End-to-end visibility

Easier discovery
Easier to reachout to respective owners

Better Change Management

Impact reports
Proactive workflows

Better Incident Management

Impact/severity of incidents
Easier to involve the right stakeholders

��(Narrator: Enter, Lineage)�

Lifecycle of Data

From the first source
Through many intermediate hops
To the final destination

Transformations

Column level lineage

Over time

Versioning by execution/time
Stale vs fresh lineage

Experience

Capture & Persist
Visualize

Athitya

6

7 of 14

What’s OpenLineage?

OpenAPI spec interface, for modelling Data Lineage
Default implementations to capture lineage

Spark
Flink
Airflow
DBT

Event-driven & real-time
Extendable & customisable “facets”
Thriving & active OSS community!
Known issues & limitations: GitHub tracker

7

Athitya

7

8 of 14

How does Spark + OpenLineage work?

8

Athitya

8

9 of 14

Integrating OpenLineage

9

Athitya

Add spark-confs...

... and get Lineage events!

9

10 of 14

Issue 1: Jobs running longer

OpenLineage Spark Listener is a separate Async thread, which can keep running even after the Spark job has completed

This Async thread holds up the infrastructure from getting terminated, giving an illusion of the job running longer

Action items

Limit blast radius by adding a Timeout circuit breaker to the OpenLineage listener
Debug root-cause

OpenLineage exporting to Kafka / HTTP can take time and be long-running
Column Lineage created huge payloads, leading to API GW timeout
Use OpenLineage’s conf�--conf “spark.openlineage.facets.disabled=[spark_unknown;schema;Schema;columnLineage;spark.logicalPlan;]”

GitHub Issue

Changelog for contribution by Athitya Kumar (@athityakumar)

10

Athitya

10

11 of 14

Issue 2: Support for Drop Table commands

Hive Drop Table events are usually captured by OpenLineage Spark Listener…

Except Spark versions 3.4.2 and 3.5.0
Causing inconsistent behaviour for jobs using different Spark versions

Action items

Debug root-cause

Log the SparkListener events published by Spark Bus for different Spark versions
Find out where the inconsistency stems from (Hive, Spark, or OpenLineage?)
Brainstorm with OpenLineage maintainers to contribute a patch

GitHub Issue

Changelog for contribution by Savan Navalgi (@savannavalgi)

11

Athitya

11

12 of 14

Future Scope

Extend OpenLineage’s adoption at Intuit, by expanding capture of processing metrics along with Lineage

Use OpenLineage’s concept of ServiceLoader to inject any custom Listener logic in runtime
Utilise OpenLineage’s custom Facets capability to capture custom metadata

Experiment with capture of column-level lineage for Flink

12

Athitya

12

13 of 14

Q&A

13

14 of 14

Q&A

#slido-name

14