Welcome to Data Engineering!

Data 101, Fall 2025 @ UC Berkeley

Aditya Parameswaran https://data101.org/sp25

​

1

Enrollment is Ongoing, Room Capacity Announcements

If you cannot find a fixed seat to sit in, for safety reasons we will need to ask you to leave. Recording posted on bCourses as soon as it is available.

​

Enrollment: (also see FAQ on class website)

• This course will not expand beyond 400
• The waitlist (of about 70 people currently) is auto-managed
• No CE students will be admitted
• No audits will be permitted

2

Aditya Parameswaran

• PhD in Computer Science from Stanford (2013)
• Associate Prof in EECS
• Research focus: building better (scalable, usable, intelligent) data tools
• Open-source tools downloaded 10s of millions of times
• More recently: LLM-powered tools for processing text

​

​

• Fourth time teaching this class!
• Joe H and I originally developed this class in 2021
• Still evolving
• Random fact about me:
• I have three creatures who ensure I don’t get quality sleep at night: two foster cats, and my hyperactive toddler

​

4

Data Engineering: What? Why?

Course Trajectory

Course Logistics

Data System Paradigms

Outline

Data Science: The Conventional View

Data Science: The Conventional View�

A data scientist operating alone, on one static dataset at a time, with a clean “rectangular” shape and fitting in main-memory, employing various statistical and ML algorithms on predefined objectives.

5

• From Data 100
• Also the view reinforced by “popular” Machine Learning, e.g., leaderboards, Kaggle, …

Data Science Data Engineering

Data Science: The Conventional View�

A data scientist operating alone, on one static dataset at a time, with a clean “rectangular” shape and fitting in main-memory, employing various statistical and ML algorithms on predefined objectives.

Nowadays, Data Science also involves Data Engineering:

A set of activities that include collecting, collating, extracting, moving, transforming, cleaning, integrating, organizing, representing, storing, and processing data.

6

• From Data 100
• Also the view reinforced by “popular” Machine Learning, e.g., leaderboards, Kaggle, …

• Messy (often non-rectangular), dynamic, and large datasets
• One team generates the data, another team consumes it
• Unclear and ill-defined objectives
• Precursor to real-world DS & ML

​

A lot of data engineering must happen to support the conventional view!

Why Learn Data Engineering?

Data engineering is an essential ingredient�of real-world DS projects.

A set of activities that include collecting, collating, extracting, moving, transforming, cleaning, integrating, organizing, representing, storing, and processing data.

7

The backbone, plumbing, or infrastructure that supports data science.

Data engineering is as essential as plumbing!

• When it works well, you don’t realize it exists.
• When it doesn’t, you’ll really know.

[1 / 3] Why Learn Data Engineering?

Data Science/AI largely focuses on Data Engineering

8

• Most time spent in real-world data science/AI projects involve data engg:
• cleaning, moving, restructuring, processing, …
• (Often underappreciated compared to other activities, e.g., ML.)

[1 / 3] Why Learn Data Engineering?

“Under the strong influence of the current AI hype, people try to plug in data that’s dirty & full of gaps, that spans years while changing in format and meaning, that’s not understood yet, that’s structured in ways that don’t make sense, and expect those tools to magically handle it.”

9

Monica Rogati, 2017 [blog].

[2 / 3] Why Learn Data Engineering?

Data engineer roles >> data scientist roles.

10

Typical job categorization:

• Data scientist: Use various techniques�in stat & ML to process & analyze data.
• Data engineer: Making DS workflows run at scale; manage data infrastructure

New! ML Engineer: a specialization of data engineer focused on operationalizing ML.

[3 / 3] Why Learn Data Engineering?

New skills! Balance your data techniques with a systems perspective.

11

As a Data Science major, you are likely familiar with techniques: statistics/ML concepts & algorithms…

…but you are likely less familiar with systems.

• You will learn about systems/infra that enables these techniques.
• You’ll start thinking about efficiency and scalability, esp. on large datasets.
• Various “plumbing analogies”: data pipelines, data flows, …

So…what is Data 101 about?

Essentially, it’s all about data systems.

​

Unfortunately there are many, many data systems – too many for us to cover.

• We will try to cover the key categories and underlying principles.
• So you can make informed decisions about when to use what type of system.

​

​

12

2023 MAD (ML/AI/Data) Landscape: blog, interactive

Why Learn Data Engineering?

Data engineering is an essential ingredient�of real-world data science projects.

A set of activities that include collecting, collating, extracting, moving, transforming, cleaning, integrating, organizing, representing, storing, and processing data.

Understanding these skills will help you…:

• Apply skills from Data 100 to messy, large real-world datasets;
• Get your datasets to the point where you can apply AI/ML;
• Explore new, exciting, & specialized roles, e.g., data engineer/ML engineer;
• Make informed decisions within the vast landscape of data systems; and
• Start worrying about efficiency :-)

​

​

13

The backbone, plumbing, or infrastructure that supports data science.

14

Data Engineering: What? Why?

Course Trajectory

Course Logistics

Data System Paradigms

Outline

Roots and Foundations

Data Systems has a long history of academic and industrial interplay.

🎢 Academic jargon meets industry buzzwords!

🤝 Formal foundations meets best practices!

15

Roots and Foundations

Data Systems has a long history of academic and industrial interplay.

🎢 Academic jargon meets industry buzzwords!

🤝 Formal foundations meets best practices!

16

Two foundational approaches

Code-centric

Query-centric

Main Storage API is files.

• AWS S3, Azure�File Storage,�Google Cloud Storage,�HDFS, …

Libraries in general-purpose programming languages, various flavors

• Batch: Spark (Scala/Java)
• Interactive/exploration:�Ad hoc code (Python/pandas)
• Metadata tracked in a separate store

17

Two foundational approaches

Code-centric

Query-centric

Main Storage API is files.

• AWS S3, Azure�File Storage,�Google Cloud Storage,�HDFS, …

Libraries in general-purpose programming languages, various flavors

• Batch: Spark (Scala/Java)
• Interactive/exploration:�Ad hoc code (Python/pandas)
• Metadata tracked in a separate store

Main Storage API is tables.

• Snowflake, BigQuery,�Redshift, Azure Synapse,
• Teradata (founded 1979,�still relevant!!)

One language/paradigm for (almost) everything

• Batch: SQL
• Interactive: SQL�
• Metadata auto-tracked in database
• (Which in turn could use file storage, e.g., S3)

18

Our approach: Query-centric but Open-Minded

Structured Query Language (SQL): A domain-specific language for data

• Same for batch and interactive queries
• Declarative complement to (often imperative) general-purpose languages
• Abstraction: No “overfitting” of code to the task at hand
• Huge plus for cloud envt:�dynamically changing workloads, hardware, data

• Code-centric libs increasingly include SQL-like interfaces (e.g., SparkSQL)
• NoSQL movement of the 2000s? Pendulum is swinging back to SQL!
• Decades of extensions, tools, and support

Based on Relational Algebra and Relational Calculus

• Formal theory!
• Decades of research and development

19

Our approach: Query-centric but Open-Minded

• We’ll learn concepts using PostgreSQL (a powerful database system).
• Originally Berkeley-built!
• Open-source, four decades old, still developing!
• You can then apply these concepts to a variety of data systems.

…but nothing’s perfect!

• In practice, you will need to go beyond SQL.
• But: the lessons will apply, and most recent alternatives follow similar declarative paradigms.

​

20

Class Journey (note: out of order)

SQL review

Advanced SQL queries (views, subqueries, window functions, …)

DML, DDL, Referential integrity, index selection, performance tuning

Data transformation and preparation,�Data wrangling and cleaning

Non-relational data models (Tensors, Spreadsheets, etc.), semistructured data (and mongoDB)

Relational Model and Algebra, ER and normalization, Spreadsheets, Transactions, BI and OLAP, parallel computing, data pipelines, …

21

Project 0

If time permits, security/privacy, LLMs-meet-data engg, cloud computing, …

22

Data Engineering: What? Why?

Course Trajectory

Course Logistics

Data System Paradigms

Outline

Our wonderful Fall 2025 Course Staff: Head TAs

23

Sahil Bhatia

Bing Concepcion

Our wonderful Fall 2025 Course Staff: TAs

24

Vicky

Huang

Joshua

Chuang

Brian

Sui

Pranav Perumandla

Elizabeth

Fiske

Our wonderful Fall 2025 Course Staff: Tutors

25

Thanh

Tran

Sofia

Garcia

Jiajun

Liu

Sydney

Tung

Shashwat

Bansal

Syllabus Walkthrough

Syllabus

In particular: Course Culture, and Lecture Attendance

​

26

Beginning-of-Semester Logistics

Discussion Sections

Stay tuned. Start next week!

Not recorded, but handouts/solutions will be posted.

• If you plan not to attend regularly, please yield your spot so we can meet room capacities.

Office Hours

Stay tuned. Start next week!

​

Pre-Semester Survey:

Google Form [Ed Welcome]

Submit by Friday 29th 5pm.�

27

We are in this class together!

Some of the content will experimental! Please bear with the hiccups.

• i.e., not taught in typical "database" classes.
• Wherever possible we will emphasize underlying concepts…�…but some of what we say will also be practical advice.
• Software and tools always evolve, always a WIP to keep things relevant

​

With all of that said:

• You will have plenty of opportunities to demonstrate success
• Our goal is for you to learn the material,�not to stress you out.
• If you are feeling lost, please reach out.�It is much better to do so than to violate our trust.

28

Use the Extenuating Circumstances form!

We welcome feedback at any time about the course. Contact course staff at data101@berkeley.edu or stop by office hours.

One important note

There will be no class next week (at VLDB in London)

​

I will instead teach tomorrow (Friday) at Pimentel 1 (note new location) from 7-10pm - the only time available 😭. Please attend! As always, lectures will be recorded.

​

And we’ll be back to regular location/time the week after that.

29

TODO by Tomorrow: Brush up on SQL!

• As data 100 is a prerequisite, we expect basic familiarity with SQL.
• We will go briskly through SQL basics next time
• If you’ve forgotten, please plan to look at those classes at https://ds100.org/fa24/
• Specifically, all of Week 11

30

31

Data Engineering: What? Why?

Course Trajectory

Course Logistics

Data System Paradigms

Outline

Demystifying Industry Jargon

32

(the same VC who made the MAD Landscape diagram)

Data systems are tools that support data engineering.

What you mostly learn at Berkeley (eg. DATA 100)

Raw Data

Transactions

Sensors

Log Files

Experiments

Use-Case-Specific

Fit for purpose

Self-Service

Data Preparation

Data preparation example: Research experiments

“Experts are close to the data and should be the ones extracting / analyzing”

Alternative picture, but more traditional enterprise

Raw Data

Transactions

Sensors

Log Files

Experiments

Use-Case-Specific

Fit for purpose

Self-Service

Data Preparation

Source of Truth

Governed

Secure

Audited�Managed

Data Integration

Data Integration example: UC Berkeley data (contracts, student info, grants, etc…) – must be centrally managed

“Compute is expensive, data is precious”

How this actually happens? E, T, and L

Extract: Scrape raw data from all the source systems, e.g., transactions, sensors, log files, experiments, tables, bytestreams, …

Transform: Apply a series of rules or functions, wrangle data into schema(s)/format(s)

Load: Load data into a data storage solution

35

Traditional Single Source of Truth: Data Warehouses - through ETL

Data Integration

Extract

Transform

Load

Extract or scraping from API or log file, transform into common schema/format, load in parallel to “data warehouse”

Raw Data

Transactions

Sensors

Log Files

Experiments

Source of Truth

Governed

Secure

Audited�Managed

Entire organizations centered around this ETL process!

ELT for Data Warehouses: A Newer Picture (e.g. Original Snowflake)

Data Integration

Load

Transform

Extract

Load without doing a lot of transformation, with transformations done in SQL

Faster to get going, and more scalable, but requires more data warehousing knowledge (& may be more expensive).

Data Warehouse

Raw Data

Transactions

Sensors

Log Files

Experiments

Source of Truth

Governed

Secure

Audited�Managed

From Warehouses → … Lakes??? 💦

​

Data Warehouses are expensive

• Warehouses expect some degree of structure
• Transformation is costly, not necessarily just computing but engineering time

​

What about skipping the “data warehouse” entirely?

No Loading! Just “dump” the data in

Let’s be …agile…

Enter the data lake

[Editorial note: Try not to make too much sense of why these terms came to be. Often just marketing…]

ET? For Data Lakes?

39

Extract

Transform

Data Preparation

Use-Case-Specific

Fit for purpose

Self-Service

(joke)

Raw Data

Transactions

Sensors

Log Files

Experiments

No need to “load/manage” data

Data is dumped in cheaply and massaged as needed for various use-cases

Usually code-centric (Spark)

Why go through all this trouble?

Once data is “lost” (i.e. not saved, deleted, etc) it cannot be recovered. So record everything.

​

Can’t predict when a particular measurement will be crucial to understanding some situation.

​

How do you know something improved if you cannot measure change?

40

The Two Extremes

Data Lake, ~2010s

• Emerged during Hadoop/Spark revolution
• “Landing zone”: unconstrained storage for any and all data
• Data is then analyzed on demand
• Extract into files/storage
• Load into storage (easy!)
• Transform on demand for any use.
• Create new files in the lake, catalog files as they go for reuse
• Often code-centric

Data Warehouse, ~1990s

• “Single source of truth”: Central, organized repository of data for analytics throughout an enterprise.
• Design the uber-schema up-front of all tables you’d ever want.
• Extract from trusted sources
• Transform to warehouse schema using custom tools
• Load data warehouse
• Old school ETL solution: Informatica

Modern solution is likely Many-to-Many, ETLT

42

Extract

Load

Transform

Transform

Transform

Data Preparation /

Integration

Data Warehouse

Raw Data

Transactions

Sensors

Log Files

Experiments

Source of Truth

Governed

Secure

Audited�Managed

Use-Case-Specific

Fit for purpose

Self-Service

Sometimes start with a data lake

Empower data scientists to work on ad-hoc use cases

Allow for datasets that “graduate” to a carefully managed warehouse

Some datasets may directly be loaded into a data warehouse

This class will focus a lot on T: Transform

A Modern Buzzword for the Modern Solution: A Data Lakehouse (2020)

43

44

…but that was just the beginning….

​

As we move away from a “managed” data warehouse, there are other considerations we need to worry about …

Important considerations

Data Discovery, Data Assessment

• When your data is in a data lake, how do you find a dataset? How do you tell if it is useful for your needs?
• Think crawling+search (like web pages)

​

Metadata

• Data about your data: for what’s stored, how it relates to each other
• Who used the data, and for what, and how?
• What changed from one version to the next?

​

Dynamic Aspects: Operationalizing, Feedback

​

​

Data Quality & Integrity

• When data is messy, need to monitor over time and tell if data quality has degraded

​

​

Modern solutions

46

Data Preparation /

Integration

Data Warehouse

Raw Data

Transactions

Sensors

Log Files

Experiments

Source of Truth

Governed

Secure

Audited�Managed

Use-Case-Specific

Fit for purpose

Self-Service

Data Discovery & Assessment

Data Quality�& Integrity

TL;DR: Governance vs. Agility

Returning to the warehouse/lake/lakehouse question (or alt., the code/query-centric question)

There is always a tradeoff: Governance vs. Agility

• Between accuracy/accountability and empowerment/expediency

​

Varies across organizations, people, …

​

Many tools focus on one point in this spectrum, but most mature tools can support any philosophical mix.

​

​

Finally, themes for the semester

Data Models: found and designed

• (Non-)Equivalences and transformations

Data Assessment, Quality, Integrity

• Structural, distributional, constraint-based

Data Transformation

• Operations, languages, systems, interfaces
• Efficiency and scale

Serving multiple tasks and personas

• From creative to statistical to industrialized
• Governance and Agility

Automation

​

​

​

​

​