Lessons from reluctant data engineering

Yanir Seroussi

linkedin.com/in/yanirseroussi | yanirseroussi.com

Disclaimer #1: I am not a data engineer

Data engineering: More valuable than data science?

Why am I here today?

Disclaimer #2: Lessons may be obvious or better learnt the hard way

Snippet 1/5

2012

My first data science job

“Big Launch” prep with shiny new tech

• Wild old days of the Facebook API
• Giveable used API to recommend gifts
• Data Science work included plenty of engineering
• Shiny new tech: MongoDB
• Hit many early MongoDB snags: DB-level locks + various bugs
• Some issues were self-inflicted (sharding...)

Lesson: Shiny tech ain’t always shiny

We should forget about small efficiencies, say about 97% of the time:

premature optimization�is the root of all evil.

Yet we should not pass up our opportunities in that critical 3%.

– Donald Knuth, 1974

Snippet 2/5

2013

My first head of data science job

Real launch leads to real scaling problems

• Head of Data Science work included plenty of engineering
• Hynt started getting real traffic in Dec 2013
• Database recommendation tracking meant imminent explosion
• Good timing:
• Nov 2013: AWS Kinesis went live
• Mid-Dec 2013: Jay Kreps published The Log (aka Kafka principles post)
• Jan 2014: Crisis averted with event logger based on MongoDB + S3

Lesson: Shiny tech can be transformative; but principles beat tools

As to methods, there may be a million and then some, but principles are few.

The person who grasps principles can successfully select their own methods.

The person who tries methods, ignoring principles, is sure to have trouble.

– Harrington Emerson, 1911

Snippet 3/5

2015

My first enterprise consulting stint

Enterprise-scale blockers; human-scale workarounds

• Data Science with team of 20+ DS & 100+ DE
• ...still, plenty of engineering needed to get things done
• Getting nothing done was also fine:
• First month: No data access
• Subsequent months: More managers than workers
• Then: New project, meetings, and experiment live in production (!!!)
• Some DEs seemed to favour functional over useful programming
• Great people; bad incentives & structures
• Workaround: Build my own Python pipeline

Lesson: Solve problems; don’t be the problem

Focus on the user and all else will follow.

– Google, ~2004

Snippet 4/5

2017

My first remote data science job

Automattic: Remote company; normal data problems

• 4.5 years of “data science” with choose-your-own-adventure titles
• Analytics engineering?
• Machine learning engineering?
• Causal inference tech lead?
• Being able to speak & do eng/ops helped get sci to prod
• Went deep into many data rabbit holes

Lesson: Go deep; trust but verify

Given enough eyeballs, all bugs are shallow.

– Eric S. Raymond, 1999

Snippet 5/5

2022

My first committed climate and biodiversity moves

Recent highlights

• 90% engineering?
• Recently: Lead DS / Tech Lead with renewable energy startup
• Techie / DE volunteer: Reef Life Survey & Work on Climate
• Currently: Data & AI freelancing / own product work
• Luxury of exploration

Lesson: Tech & titles are tools; focus on what matters

You are not obliged to complete the work, but neither are you free to desist from it.

– Rabbi Tarfon, ~100

Why am I here today?

Takeaway:

Data problems have human roots – and human solutions

Recap: Data problems have human roots & solutions

• Humans get excited by shiny tech
• ...and produce transformative tech
• Humans optimise prematurely
• ...and when it makes sense
• Humans can act as unreasonable blockers
• ...and as the users we serve
• Humans generate messy data
• ...and clean it up
• Humans get distracted by tools
• ...and use them for beneficial ends

Questions?