1 of 11

Simpson's paradox: how I got deceived by statistics

​

​

July 2022

Vivek Pandey

2 of 11

Outline

  • A real life incident
    • How I got surprised
  • A cricket example
  • A medicine example
  • Reason

3 of 11

A real life incident

  • I was working in an ad-tech company
  • My job was to show ads that got more clicks
    • There was a default algo
  • I was given some clients on some fraction of whose traffic to run my algo
    • And demonstrate that my algo is better

4 of 11

A real life incident

​

Default algo(D)

New algo(N)

​

impressions

clicks

impressions

clicks

Client 1

i1

c1

i4

c4

Client 2

i2

c2

i5

c5

Client 3

i3

c3

i6

c6

My observations:

  • c4 / i4 > c1 / i1
  • c5 / i5 > c2 / i2
  • c6 / i6 > c3 / i3

i1, i2... = number of impressions, i.e. number of ads shown

​

c1 c2... = number of clicks

​

c1/i1, c2/i2 etc are known as click through rate

5 of 11

A real life incident

​

Default algo(D)

New algo(N)

​

impressions

clicks

impressions

clicks

Client 1

i1

c1

i4

c4

Client 2

i2

c2

i5

c5

Client 3

i3

c3

i6

c6

My observations:

  • c4 / i4 > c1 / i1
  • c5 / i5 > c2 / i2
  • c6 / i6 > c3 / i3

But then:

(c1 + c2 + c3) / (i1 + i2 + i3) > (c4 + c5 + c6) / (i4 + i5 + i6)

i1, i2... = number of impressions, i.e. number of ads shown

​

c1 c2... = number of clicks

​

c1/i1, c2/i2 etc are known as click through rate

6 of 11

So which algo is better?

Argument 1:

New algo is better because it is better on each individual client

​

Argument 2:

Default algo is better because overall, it is better on the aggregate

7 of 11

A question on Cricket

​

Sachin Tendulkar

Rahul Dravid

​

runs

innings

batting average

runs

innings

batting average

2001-2005

1200

48

25

10400

411

25.3

2006-2010

18300

582

31.4

4500

140

32.1

Total

19500

630

31.0

14900

551

27.0

  • Rahul has better average than Sachin in both the years
  • But overall Sachin has better average
  • Who is the better batsman?

*These are made up numbers, not real cricket stats

8 of 11

Which treatment is better?

  • Treatment A is better on small stones
  • Treatment A is better on large stones
  • But overall, treatment B is better

9 of 11

The reason: confounding variable

Size of the stone is "confounding variable"

​

Large stones in general have lower success rate.

​

Treatment A has been done more often on large stones

10 of 11

The reason: confounding variable

Size of the stone is "confounding variable"

​

Large stones in general have lower success rate.

​

Treatment A has been done more often on large stones

Correct way to evaluate:

If we apply treatment A to all stones, success rate = [0.93 * (87 + 270) + 0.73 * (263 + 80)] / (87 + 270 + 263 + 80)

If we apply treatment B to all stones, success rate = [0.87 * (87 + 270) + 0.69 * (263 + 80)] / (87 + 270 + 263 + 80)

11 of 11

Q&A

Cricket and medicine examples are copied from wikipedia article.