1 of 12

End-to-End Data Lineage Across EDAP

Keanu Ventura

Summer Associate – EDA

2 of 12

Problem: No End-to-End Visibility

In the Enterprise Data and Analytics Platform (EDAP), data flows through multiple systems:

Fivetran – Ingestion

Snowflake – Storage

dbt – Transformation

ThoughtSpot – Reporting

Objective: Create a repeatable framework for linking metadata across EDAP systems

3 of 12

EDAP Data Flow

S3 Files

Fivetran

Snowflake

dbt

ThoughtSpot

Challenge: How can metadata be linked across EDAP systems?

4 of 12

Metadata Exploration:�Notebook Demo

What metadata can we discover through system APIs?

5 of 12

Key Insights: Common Join Keys

Fivetran ↔ dbt

dbt Source Schema + Source Table

=

Fivetran Destination Schema + Destination Table

ThoughtSpot ↔ dbt

Thoughtspot Table Name

=

Published dbt Model

Towards End-to-End Lineage

ThoughtSpot → Published dbt Models → dbt Transformations → Source-Fed Models

Achievable by traversing dbt dependency metadata.

The exploration ultimately led to three main findings.

First, I found that dbt source schema and table values aligned with Fivetran destination schema and table values, providing a common way to connect those two systems.

Second, I found that ThoughtSpot table names aligned with published dbt model aliases, providing a way to connect reporting assets back to dbt models.

Finally, the Mortgage Account Fact example showed that the path doesn't stop at a single dbt model. Instead, it can continue through multiple layers of dbt transformations before reaching a source-fed model.

Together, these findings suggest that end-to-end visibility may be achievable through dbt dependency information.

Once these relationships were identified, the next step was to verify them directly within the systems themselves.

6 of 12

Validation: Lineage Linkages Confirmed

Identifiers matched across systems ✅

Metadata validated in Snowflake✅

dbt dependencies validated✅

Validation Example: Mortage Account Fact

Located and verified in

Fivetran ✓

Snowflake ✓

dbt ✓

Thoughtspot ✓

Metadata relationships discovered through API exploration were confirmed directly within EDAP systems.

After identifying these relationships, I wanted to make sure they actually existed in the systems themselves.

With access provided by Aaron, Nicole and Francois, I was able to go directly into Fivetran, Snowflake, dbt, and ThoughtSpot and verify what I was seeing through the APIs.

I confirmed that the identifiers matched across systems, validated the corresponding information in Snowflake, and verified the dbt dependencies from the lineage example.

For example, I used Mortgage Account Fact and was able to locate and trace the related assets across Fivetran, Snowflake, dbt, and ThoughtSpot.

This gave me confidence that the relationships discovered during exploration reflected real connections across EDAP.�With those relationships confirmed, the next step was to make the process repeatable through automation.

7 of 12

Automation: �Python SDK Connectors

Metadata Connectors Built:

Fivetran Connector
dbt Connector
Thoughtspot Connector

Replaces manual CSV exports with automated metadata ingestion connectors

Metadata collection now repeatable across EDAP systems.

8 of 12

SQL Validation: DuckDB Demo

Validating discovered relationships using connector outputs and SQL

9 of 12

Future Opportunities

Persist connector outputs into Snowflake
Build a centralized metadata inventory
Group assets by business domain (Q2, Loans, Deposits, etc.)
Track freshness and sync status across systems
Enable cross-system reporting and analysis through a common API layer

While this project focused on exploration, validation, and automation, I think the most exciting part is where this work could go next.

The connectors and relationships identified throughout this project create a foundation that could support a number of future use cases across EDAP.

For example, connector outputs could be persisted into Snowflake and used to build a centralized inventory of assets across systems.

Those assets could then be organized by business domain, making it easier to understand ownership and relationships across areas such as Q2, Loans, and Deposits.

The same information could also support monitoring use cases, such as tracking freshness and sync status across systems.

And ultimately, a common API or reporting layer could make it easier to answer questions that today require navigating multiple systems individually.

While a complete end-to-end lineage solution remains a future opportunity, I believe the work completed in these past several weeks helped establish some of the foundational pieces needed to move in that direction.

10 of 12

Appendix

Additional Thoughtspot metadata discovered:

Joins and join conditions
Formulas
Columns
Dependencies
Ownership metadata

11 of 12

Q&A

AI-generated images of me,

courtesy of (Ai)ron Dodd →

12 of 12

Mahalo

“You can think of the Ferrari, but most people only need the shopping cart” – Aaron Dodd