Bayesian Reasoning

In Data Science

22/11/2022 - Lectures 21

Outline

2

• Bayesian A/B Testing

Bayesian A/B Testing

• Bayesian A/B Testing gained a lot of popularity in the last few years
• Simple and easy to understand
• Allows to calculate the probability that a “treatment” is better than a “control” (A/B testing)
• It performs better on small sample size compared to frequentist approaches: see [2], where experiments show the required sample size to make the “right” decision can be reduced by 75%

​

[1] https://towardsdatascience.com/bayesian-a-b-testing-and-its-benefits-a7bbe5cb5103

[2] https://towardsdatascience.com/exploring-bayesian-a-b-testing-with-simulations-7500b4fc55bc

[3] https://exp-platform.com/top-challenges-from-first-practical-online-controlled-experiments-summit/

• A/B testing
• Consists of a randomized experiment that usually involves two variants

• (Airbnb, Amazon, Booking.com, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, and Stanford University)

3

Bayesian A/B Testing

[1] https://towardsdatascience.com/bayesian-a-b-testing-and-its-benefits-a7bbe5cb5103

[2] https://towardsdatascience.com/exploring-bayesian-a-b-testing-with-simulations-7500b4fc55bc

• Main steps (see Notebook):
• Select distribution based on your metric of interest.
• Calculate prior
• Run experiment with Monte Carlo simulations
• Bayesian Statistics (see Notebook):
• Probability of being best
• Treatment Lift
• Expected Losss

4

Considerations

[1] https://towardsdatascience.com/bayesian-a-b-testing-and-its-benefits-a7bbe5cb5103

[2] https://towardsdatascience.com/exploring-bayesian-a-b-testing-with-simulations-7500b4fc55bc

• If you choose an effective prior, Bayesian A/B testing requires a smaller sample size so you can get results faster.
• There is a tradeoff though: Bayesian methods are more computationally intensive than frequentist methods.

​

5

From Class

Uniform

6

Backup

Useful References

8

• These lectures largely based on
• C. Davidson-Pilon, Bayesian methods for hackers, 2015
• https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers
• Other references
• https://towardsdatascience.com/bayesian-a-b-testing-and-its-benefits-a7bbe5cb5103

​

​

References of our course

9

https://cfteach.github.io/brds/referencesmd.html