1 of 14

Lecture 22

A/B Testing

DATA 8

Fall 2017

Slides created by John DeNero (denero@berkeley.edu) and Ani Adhikari (adhikari@berkeley.edu)

2 of 14

Announcements

3 of 14

Comparing Samples

4 of 14

A/B Testing

  • Two random samples:
    • Sample A
    • Sample B

​

  • Question: Are they drawn from the same underlying distribution?

​

  • Answer by A/B testing

5 of 14

The Hypotheses

​

  • Null:
    • The two samples are drawn from the same underlying population distribution; they look like two random draws from the same set.

​

  • Alternative:
    • The samples are drawn from different distributions; they don’t look like random draws from the same set.

6 of 14

Smoking and Birth Weights

  • Measured the birth weights of many babies and whether the mom smoked.�
  • Null: The distribution of birth weights is the same for both smoker and non-smoker moms.�
  • How to simulate the distribution of the test statistic, if the null hypothesis is true?

7 of 14

Hypothesis Test

  • Null: The distribution of birth weights is the same for both smoker and non-smoker moms.

​

  • If the null is true, we can model each birth as:
    • randomly choose a birth weight
    • randomly choose whether the mom is a smoker or non-smoker�
  • How do we know the distribution of birth weights?

8 of 14

Resampling

9 of 14

The Test Statistic

  • If the samples are categorical, then a natural test statistic is the total variation distance. It measures the difference between the distributions in the two samples.
  • If the samples are numerical, often a simpler statistic is just fine, such as the absolute difference between the two sample means.

​

(Demo)

10 of 14

The Test Statistic

  • If the samples are numerical, the absolute difference between the two sample means might be a reasonable measure.

​

(Demo)

11 of 14

Permutation Test

  • Null: The two samples are drawn randomly from the same underlying distribution.

​

  • If the null is true, all rearrangements of the variable values among the two samples are equally likely. So:
    • compute the observed test statistic
    • then shuffle the attribute values and recompute the statistic; repeat; compare with the observed statistic

12 of 14

Attendance

bit.ly/attend8

13 of 14

Randomized Controlled Experiments

14 of 14

Causality

  • Sample A: control group
  • Sample B: treatment group

​

  • If the treatment and control groups are selected at random, then you can make causal conclusions.

​

  • Any difference in outcomes between the two groups could be due to
    • chance
    • the treatment

(Demo)