1 of 13

statistical claims

Evaluating

source: xkcd

CC-BY-NC 2.5

2 of 13

Evaluate a statistic

What was the sample size?

How representative was the sample?

What is the margin of error?

Is it a direct measurement or a proxy?

Here are four key questions to ask yourself when encountering a statistic.

Sample size: In theory, you can draw valid conclusions about a population of ANY size from a sample of around 30, IF AND ONLY IF your sample is representative.

Representative sample: This is much more important than sample size (and much harder to evaluate). Note that the sample doesn’t have to be representative in ALL ways, just in the ways that matter for the question you are researching. So, for example, if you are looking at the link between education and employment, it probably doesn’t matter whether your respondents are representative of the whole U.S. population in terms of age.

Margin of error: This is a mathematical estimate of how close a statistic probably is to the real value. It is a reminder that ALL statistics are estimates. If the margin of error isn’t reported, that is cause for suspicion. If the difference or change you see in the statistic is less than the margin of error, you shouldn’t have much confidence in it.

Proxy: Very often, we can’t directly measure the thing we care about, so we measure something else instead, something we think is an indicator. For example, student standardized test scores are often used as a measure of teacher effectiveness, but they are not a direct measurement. What problems might arise from drawing conclusions about teacher effectiveness from standardized test scores? (students respond and discuss)

3 of 13

Evaluate a statistical claim

Causation or only correlation?

Data being cherry-picked?

Are these the right data? The best data?

Are the data being reported accurately?

Is terminology clearly and appropriately defined?

And here are four common pitfalls to look for when evaluating a statistical claim. (discuss)

Causation vs. correlation: Just because two variables go up or down at the same time doesn’t mean one causes the other. It could be random chance, or it could be that there’s a confound, a third variable affecting both of the first two, the way that weather affects both ice cream consumption and crime rates.

Cherry-picking: This is the name for using the part of the data that supports your claim and ignoring the rest. You shouldn’t trust cherry-picked results.

The best data: This is a big philosophical and methodological question that often requires serious expertise to answer, but it needs to be asked every time.

Accurate reporting: Sometimes mistakes or misunderstandings get passed down a chain. If one source cites an earlier source which cites an earlier source, the best practice is to track down the original source of the information and check to see that it actually says what people claim it says.

Terminology: People often reword claims in ways that benefit their rhetorical goals. Anytime a claim is being reworded, it’s being subtly changed.

4 of 13

Who sounds more credible?

Bilbo: The world is getting hotter every year.

Frodo: Global temperatures have increased

by about 0.14°F each decade.

Who is more likely to be wrong?

5 of 13

What about now?

Bilbo: Global temperatures have increased

by about 0.14°F each decade.

Frodo: Global temperatures have decreased

by about 0.14°F each decade.

6 of 13

Evaluate a statistical claim

Malala finds that on days when she gets to school early, she does

better on her exams. Thus, she advises her younger sister to show

up to school early.

The sister doesn’t know Malala used this time to study, so she

just hangs out with her friends and her grades do not improve.

What was the sisters’ mistake?

7 of 13

Evaluate a statistical claim

Crime is increasing. Violent crime reports increased nearly 15% between August and September of this year.

reports of violent crime

8 of 13

Evaluate competing statistics

You find five different studies about grocery shopping.

Four studies say 10% of people buy candy in the checkout

aisle, but the fifth says 60% of people buy candy in the

checkout aisle.

How would you figure out which number to trust?

9 of 13

Evaluate

I want a reliable dataset about the number of crimes committed by individuals.

Which number is more trustworthy, their number of arrests? Or their number of convictions?

10 of 13

Evaluate

Today, 70% of kids in high school are cool.

Ten years ago, it was only 30%.

11 of 13

Evaluate

Support for abolishing the minimum wage stands at 15% among Depublicrats and 10% among Remmicans, according to a poll of 750 likely voters.

Can we conclude that Depublicrats are 5% more likely to support abolishing the minimum wage?

adapted from Milo Schield (2021), Statistical Literacy

12 of 13

Evaluate

Original study:

“69% of respondents who were violent by the age of 34 years reported that they ate confectionary nearly every day during childhood, compared with 42% who were non-violent.”

Moore et al. (2009). Confectionary consumption in childhood and adult violence. The British Journal of Psychiatry 195: 366-367

News headline:

Study says too much candy could lead to prison

“Of the children who ate candies or chocolates daily at age 10, 69 percent were later arrested for a violent offense by the age of 34.”

Associated Press, Sept 30, 2009

adapted from Milo Schield (2021), Statistical Literacy

13 of 13

Evaluate

a statistical claim from your own project and write a brief assessment based on the nine questions:

What was the sample size?

How representative was the sample?

What is the margin of error?

Is it a direct measurement or a proxy?

Causation or only correlation?

Data being cherry-picked?

Are these the right data? The best data?

Are the data being reported accurately?

Are terms defined appropriately?