Gerard Toonstra| Data Discovery with Amundsen | 27-11-2019

Amundsen who?

​

• Roald Amundsen (1872-1928)

​

• Norwegian arctic explorer

​

• First to explore South Pole

​

• First to traverse NW passage by sea

​

Decentralization and democratization of data is not productive unless you use tools that allow you to make sense of the plethora of data products you never knew you were going to have.

Data discovery issues

​

Analysis of price elasticity for MacBooks

​

​

Data discovery issues

​

→ Question #1: Where is the data for sales and product prices by date?

​

Data discovery issues

​

→ Question #1: Where is the data for sales and product prices by date?

​

→ Question #2: Is the data current and actively monitored?

​

Data discovery issues

​

→ Question #1: Where is the data for sales and product prices by date?

​

→ Question #2: Is the data current and actively monitored?

​

→ Question #3: Does the “price” column include or exclude VAT?

​

Data discovery issues

​

→ Question #1: Where is the data for sales and product prices by date?

​

→ Question #2: Is the data current and actively monitored?

​

→ Question #3: Does the “price” column include or exclude VAT?

​

→ Question #4: Is it the actual “sales price” or is it list price?

​

Data discovery issues

​

Redesigning, relocating and deprecating data products

​

Data discovery issues

​

Finding other users of the same data (inspiration, innovation, sharing)

​

Side effects

​

• No way to know & understand trusted data
• Created channels & oncalls for data questions

Lots of queries like:

SELECT

*

FROM

default.my_table

WHERE ds=’2018-01-01’

LIMIT 100;

​

• Does data exist?
• Prior work?
• Source of truth?
• Who owns it?
• Who uses it?

uncertainty

Increased load

and query cost

interruptions

Metadata is key to next bigdata wave

​

• Discover new data sources
• Identify end users to notify them of changes
• Understand the popularity and trustworthiness of data
• Network with other people on their data analysis
• Investigate/monitor the magnitude of protected data exposure
• Know what your boss or colleagues are using
• Talk to upstream producers
• +30% productivity for data scientists

It’s all about relationships

​

ABC of metadata

​

Application Context

Metadata needed by humans or applications to operate

• Where is the data?
• What are the semantics of the data?

​

​

ABC of metadata

​

Application Context

Metadata needed by humans or applications to operate

• Where is the data?
• What are the semantics of the data?

​

Behavior

How is data created and used over time?

• Who’s using the data?
• Who created the data?

​

ABC of metadata

​

Application Context

Metadata needed by humans or applications to operate

• Where is the data?
• What are the semantics of the data?

​

Behavior

How is data created and used over time?

• Who’s using the data?
• Who created the data?

​

Change

Change in data over time

• How is the data evolving over time?
• Evolution of code that generates the data

Terminology: Ground paper

from Berkeley

What’s out there?

​

• Vendors - Alation, Collibra
• Existing open source projects (e.g. Apache Atlas, Marquez)
• LinkedIn’s data portal - Wherehows & DataHub (blog, code)
• Twitter’s data discovery (blog)
• Netflix’s metacat (code, blog)
• Airbnb’s data portal (blog, video)
• Big Query SQL Web UI & data catalog (blog)
• Goods: Organizing Google’s Datasets (paper)
• Data Warehousing and Analytics Infrastructure at Facebook (paper)
• Ground (RISE Lab): https://rise.cs.berkeley.edu/projects/ground/

What do we need?

​

Other requirements

​

What do we need?

​

Other requirements

​

What do we need?

​

Other requirements

• Leverage as much data automatically as possible
• Preferably, open source and healthy community
• API availability
• Easy to set up

What’s out there?

​

Important things for the journey

​

• Trust embodied in the solution itself
• Sort results in order of relevance and popularity

​

Important things for the journey

​

• Trust embodied in the solution itself
• Sort results in order of relevance and popularity
• Very low amount of manual curation
• Automated curation as much as possible

​

Important things for the journey

​

• Trust embodied in the solution itself
• Sort results in order of relevance and popularity
• Very low amount of manual curation
• Automated curation as much as possible
• Slight preference for open source
• Driving the OS community forward with this new tool

​

​

Important things for the journey

​

• Trust embodied in the solution itself
• Sort results in order of relevance and popularity
• Very low amount of manual curation
• Automated curation as much as possible
• Slight preference for open source
• Driving the OS community forward with this new tool

​

• Targeted user experience not available in other tools

​

​

Amundsen - landing page

Amundsen - search

Amundsen - table detail page

Amundsen - column details

Amundsen - preview

Amundsen - people search

Amundsen - people page

Amundsen services working together

​

Amundsen services working together

​

Amundsen services working together

​

Amundsen services working together

​

Amundsen services working together

​

Relations rather than entities

​

SQL vs Neo4J queries

​

SQL vs Neo4J queries

​

SQL vs Neo4J queries

​

ElasticSearch for search and relevance

​

• Normal search: match records based on relevancy

​

• Category search: match records first based on data type, then relevancy
• column: warehouse_cost

​

• Wildcard search:
• event_*

​

Metadata extractors

​

Scheduling extractor jobs ⇒ airflow

​

Amundsen uses Apache Airflow to orchestrate Databuilder jobs

​

What to expect...

• Customizing UI (logo, colors, etc.)

​

What to expect...

• Customizing UI (logo, colors, etc.)
• Security / OpenID Connect for authentication

​

What to expect...

• Customizing UI (logo, colors, etc.)
• Security / OpenID Connect for authentication
• Look at available extractors:
• Hive, BigQuery, Athena, Snowflake, Postgres

​

What to expect...

• Customizing UI (logo, colors, etc.)
• Security / OpenID Connect for authentication
• Look at available extractors:
• Hive, BigQuery, Athena, Snowflake, Postgres
• Build your dags

​

Roadmap

​

• UI/UX redesign
• Email notifications system
• Indexing dashboards
• Lineage integration
• Granular access level control
• More metadata sources
• Indexing services and teams

​

​

From Discovery towards Governance

​

Metadata

Compliance (GDPR/CCPA)

Data Discovery

Downstream impact analysis

. . . . .

Data Quality

Other users

​

57

Prominent users

Active community

Where to go for more

​

https://www.github.com/lyft/amundsen

​

amundsenworkspace.slack.com #amundsen @ slack

(link on github page)

​

​

Gerard Toonstra| Data Discovery with Amundsen | 27-11-2019 | https://www.careersatcoolblue.com

Join at slido.com with #bigdata2019