Laying the Foundation for Successful Model Deployment
Hiranmayi Duvvuri | 03.04.2020
Hiranmayi Duvvuri
Transitioned from using python in an academic research setting to industry.
Started as an analyst then about a year went into data science. Been in DS for about 1.5 years.
I enjoy crafting and hanging with my dog Pepper in my free time :)
Data Scientist @ Vacasa
Content Sections
TABLE OF CONTENTS
Why Plan?
Lessons Learned - What to Look Out For
Compromise / Collaboration - Building Trust
Why Plan?
A goal without a plan is just a wish
—Antoine de Saint Exupéry
What We Want to Avoid
Starting Out
A Learning Experience
Starting Out
Only worked independently previously
My first big project
First time working together, not sure how to integrate
Brand new team!
Learning to communicate how model worked. Touched folks with different backgrounds.
Communication
Decrease risk of misunderstanding
Know software requirements up front
Confirm business case being solved
Confirm required resources available
Have a backup plan!
So, Why Plan?
Starting Out
Lessons Learned
Time Spent as a Data Scientist
Breakdown
EDA
Data Cleaning
Modeling
Production
Time Spent as a Data Scientist
Breakdown
Stakeholder communication
EDA
Data Cleaning
Modeling
Production
Foundation
Business impact +
actionable results
Awareness of limitations
Model interaction +
data ingestion
Consistent communication
+
building trust
Getting on the Same Page
Data Contract
Getting on the Same Page - Product
Data Contract
What’s the cost of over / under predicting
Acceptable Errors
Edge cases vs Big picture
Granularity
Who owns what after production?
Post-production
Stakeholders don’t always know the risks / trade offs involved with a data science model.
Getting on the Same Page - Product
Data Contract
What are the actionable results?�Who’s the audience?
Business Impact
What’s being predicted / does everyone agree
y-value
Using consistent communication during modeling / EDA process
Building Trust
Getting on the Same Page - Engineering
Data Contract
What can they expect�What data do they need
Inputs / Outputs
What kind of tool, model delivery
Interaction
What will they need to run your model
Software requirements
Engineering teams don’t always know how to work with a data science model and vice versa
Getting on the Same Page - Engineering
Data Contract
Latency / runtime / size
Model limitations
Could be limited by team size / resources
Outlier / Error monitoring
Is this needed, will there be a UI?
Manual intervention
Compromise / Collaboration
Push and Pull
Compromise
Keep it Simple and Straightforward
KISS
What can be done now to inform long term plan?
Long term vs Short term
What are the timelines here?
Rollout and Testing
Teaching Moments
Compromise
What can we do with data we have now
What’s Possible
What are the benefits to collecting better / more data
What Could be Possible
Maybe for now, something simple works the best
Buzzwords aren’t Always the Answer
Cool Things Can Happen!
Collaboration
Working with Domain Experts
Collaboration
Listen to their feedback, ideally they know the audience
Be Open
Each are experts in your own space, work with that in mind
Building Mutual Respect
Help keep bounds on project and getting too granular
Keep Big Picture
Data Science at Vacasa
Resources
Worked on a lot of trust building as data science grows
Diverse Stakeholders
Each project has come with own challenges, trust helps with iteration
Diverse Projects
Trust has helped with product wanting to keep exploring new projects
Success with Production!
HIRANMAYI DUVVURI | HIRANMAYI.DUVVURI@GMAIL.COM
Thank You
Questions?