Mastering Experimentation

Deepak Singh

Why should you listen to me?

Mission - Helping PMs create impact in their roles

9k copies sold in Print

10k Subscribers

Growth for PMs

Agenda

1. Need for experimentation
2. Necessary conditions for experimentation
3. Experiments come in all shapes and sizes
4. Lifecycle of an experiment
5. Common mistakes
6. Q&A

Why do we need experimentation?

1

Q: Which experimentation tool does your company have right now?

• Inhouse
• VWO
• Optimizely
• Don’t have an experimentation program
• Others

​

​

#Assessment

Need for experimentation

Why do we need experimentation?

Need for experimentation

Evidence

Necessary Conditions at Org Level

• Enough units (users, new users)

Why do we need experimentation?

• Key metric (north star metric) and success criteria for experimentation program is agreed upon and can be measured accurately

• Changes are easy to make
1. Regulation don’t prohibit you
2. Neither does the HiPPO

Necessary Conditions at Org Level

• Data-driven

Why do we need experimentation?

• Willingness to invest in infrastructure

• Okay with failure

Comfortable with failure?

Experiments come in all shapes and sizes

Google’s 41 shades of Blue / 2009 / +$200 million

Bing longer headline of ads / 2012 / +$100 million

Airbnb - URL level testing for SEO

Airbnb - URL level testing for SEO

So what? Take a toolbox approach as a PM because experiments come in all shapes and sizes

Lifecycle of an Experiment

Start by setting the goal for experimentation

• For accumulated learnings

• To avoid goal shifting in-between

• To create meaningful impact by having focus

• To ensure cross-functional support and prioritization

• To get buy-in from leadership

Why do we need to define goal for experimentation?

2

Q: What would be a good goal for an eCommerce platform search team experimentation program?

1. CTR on search results
2. User conversion (search → payment)

#Assessment

Start by setting the goal for experimentation

Determine if it’s a server side or client side experiment

Why do we need to define goal for experimentation?

Start by setting the goal for experimentation

Determine if it’s a server side or client side experiment

Determine the sample size

Why do we need to define goal for experimentation?

Sample size depends on p-value

• p-value is the measure of how much the result is a matter of chance/luck

• p-value < 0.05 means there is a >95% chance that the result isn’t a matter of chance/luck. 1-p is known as confidence level.

The result has to be statistically significant

Sample size depends on MDE

​

• Minimum detectable effect (MDE) is the smallest improvement you want to detect.

• Suppose you want to test a new drug for dogs that whether it can be used for increasing their height or not. And let’s say there are two cases, where you are saying that:
• the increase in height should be 1cm(MDE).
• the increase in height should be 10cm(MDE).

Calculate the sample size to reach significance in advance

3

Q: Where would you need higher sample size?

• MDE = 1 cm
• MDE = 10 cm

#Assessment

Why can’t we set MDE as low as possible?

​

#Assessment

Sample size depends on Power

• Power is the probability that an experiment will flag a real change as statistically significant.
• 80% power → 8 of 10 changes detected
• 60% power → 6 of 10 changes detected

Power

• Across industries, 80-90% power is considered reasonable

4

Q: What would be the sample size for p=0.01, baseline conversion = 10%, MDE = 10% relative, power = 95%?

• 25,681
• 19,826
• 20,923
• 32610

https://www.evanmiller.org/ab-testing/sample-size.html

#Assessment

Mistakes in Life Cycle of an Experiment

Mistake #1 is not calculating sample size in advance

​

Calculating the sample size and # of days needed to reach sample size after the experiment starts

Don’t calculate the sample size to reach significance in advance and # of days required

Mistake #2 is not evaluating non-binomial metrics differently

​

​

Non-binomial: https://www.analytics-toolkit.com/tools/stats/sample-size.php

​

Mistake #3 is peeking

​

The more the peeking, the higher the chances of false positives as the experiment often dips into significance before coming back out

​

Don’t calculate the sample size to reach significance in advance and # of days required

Mistake #4 is multiplicity

​

Multiplicity refers to the “potential inflation of type I error rate (false positive) as a result of multiple testing, for example because of multiple subgroup comparisons, analysis of multiple outcomes, and multiple analyses of the same outcome at different times.”

​

Don’t calculate the sample size to reach significance in advance and # of days required

Q&A