Leveraging Statistical Experiment Designs in the Rideshare Industry
Danni Lu
DAE 2024
Danni Lu
Sr. Applied Scientist, Uber. San Francisco, CA.
Ph.D. student, Virginia Tech, Blacksburg, VA
DAE 2024
01 Uber and road safety
02 Experimentations
03 Testing a product
3
DAE 2024
Uber
“ Uber Technologies, Inc., or Uber, is an American multinational transportation company that provides ride-hailing services, courier services, food delivery, and freight transport. ”
From :
4
DAE 2024
Rides and beyond
In addition to helping riders find a way to go from point A to point B, we're helping people order food quickly and affordably, removing barriers to healthcare, creating new freight-booking solutions, and helping companies provide a seamless employee travel experience. And always helping drivers and couriers earn.
Stand for safety
Whether you’re in the back seat or behind the wheel, your safety is essential. We are committed to doing our part, and technology is at the heart of our approach. We partner with safety advocates and develop new technologies and systems to help improve safety and help make it easier for everyone to get around.
5
Road Safety
6
DAE 2024
01 Uber and road safety
02 Experimentations on the platform
03 Testing a product
7
02 Experimentations on the platform
8
02 Experimentations on the platform
While the statistical underpinnings of A/B testing are a century old, building a correct and reliable A/B testing platform and culture at a large scale is still a massive challenge!
Core experiment structure
9
02 Experimentations on the platform
Core experiment structure
The experiment units (eg. riders, drivers, eaters) are randomized by hashing their identifiers with a salt determined by the experiment key in a given experiment. Experiment keys are unique, which ensures that all experiments are randomized independently of each other and anything else,
10
02 Experimentations on the platform
Core experiment structure
The treatment plan specifies what is done in each context to each treatment group.
11
02 Experimentations on the platform
Other experiment structures
12
02 Experimentations on the platform
13
02 Experimentations on the platform
Sometimes, the experiment we want to test naturally introduce competition/interference between experiment units and network effect is inevitable.
14
02 Experimentations on the platform
In cases where network effect is inevitable.
Switching between control (incumbent algorithm) and treatment (challenger algorithm) that impact all users in the marketplace with a given frequency over a predetermined time period.
A switchback with a switch frequency of thirty minutes or less. Used principally for dispatch experimentation to get a quick read on what changes are likely to do.
A time-randomized switchback with 80% of time treated and remaining 20% plus prior observations as control.
15
DAE 2024
01 Uber and road safety
02 Experimentations on the platform
03 Testing a product
16
Percent of Intersection Related Traffic Fatalities occurred at unsignalized intersections
67.3
Percent of Traffic Fatalities in the US are intersection related
27.2
The idea
According to the Federal Highway Administration Safety Report:
Headroom analysis
Estimate the impact of the product
Product design
Experiment plan
Experiment type
Country/City selection
Duration
Stakeholders
Metrics
Patent application
Model R&D
Product Launch
Model research, development, and production
How to surface the product to App users
Experiment
XP deployment
Monitor
Data analysis
The idea → product
Improving intersection safety by making app users aware of the upcoming non-all-way-stop intersections.
The product:
Uncontrolled Intersection Alert
Hypothesis
Product design: Navigation reminders
Geography
Experiment type: Driver A/B
Uncontrolled Intersection Alert
Experiment monitor & analysis
Launch decision
DAE 2024
Thank you!
And… we’re hiring! 😄
Opportunities to:
See more in career website.
21