HOW TO AVOID COMMON MISTAKES WITH STATISTICS

Hints and free tools

OVERVIEW

1. Why am I doing this?
2. The free tools & workshop download
3. Common mistake 1: Choosing the wrong test�→ Not considering data types�→ Misunderstanding p > .05
4. Common mistake 2: Not reporting effect size�→ Not realizing the importance
5. Common missed extra step: Not considering association / correlation

WHY AM I DOING THIS?

• I review a lot of papers
• I see a lot of the same mistakes
• I used to make these mistakes (and more)
• We need to do better as a field (Plonsky, 2023)
• I am not a jerk (Wife, 2022)

I FEEL YOUR PAIN

1. Most examples not from our field�→ Most textbooks: examples from medicine or economics�
2. Mix of different types of studies in our field�(sometimes old tests, styles, etc.)�→ Copying previous studies’ stats methods might be dangerous

SO!

I made a free online stats tools that:

1. Helps ensure you have the right test
2. Helps ensure you are justifying / reporting correctly
3. Gives examples in the field of linguistics / language learning

​

https://springsenglish.online/stats

​

→ Partially to help me learn programming

SPSS’ STRANGLEHOLD IS GONE

R is difficult, but…

FOR THIS WORKSHOP!

• Please go to my website and download today’s example data sets

springsenglish.online/stats

(my name)

(my site)

(what I teach)

(because I’m poor)

(to go to the stats part)

.com is REALLY expensive!!!

COMMON MISTAKE 1:�CHOOSING THE WRONG TEST

​

Part 1:

Considering Data Types

COMMON MISTAKE 1:�CHOOSING THE WRONG TEST

Consideration 1: Is my data normal and continuous?

1. Average in the middle

2. Most data close to average

1. Many values
2. + / - “equal”

COMMON MISTAKE 1:�CHOOSING THE WRONG TEST

Normal data might look like:

COMMON MISTAKE 1:�CHOOSING THE WRONG TEST

Why does data normality matter?

​

・Beware the AVERAGE TRAP!!

WHY DOES THIS MATTER?

If the data isn’t normal / parametric:

The average doesn’t mean much�

M = 70; students knew about 70%? YES!

M = 70; students knew about 70%? NO!

WHY DOES THIS MATTER (2)?

t-test = What is the chance there is so much difference in averages? (Are these curvatures from the same normalized sample?)

​

​

​

Mann-Whitney = What is the chance that so many high scores are in one of these groups? (Do these rank distributions come from the same sample?)

Calculations are ENTIRELY DIFFERENT!!!

​

COMMON MISTAKE 1:�CHOOSING THE WRONG TEST

Consideration 2: Are the data sets paired?

OR

WHY DOES THIS MATTER?

INDEPENDENT t-test = Is there a difference in these groups’ averages? (Are these curvatures from the same normalized sample?)

​

​

​

DEPENDENT t-test = Is the average of these individual’s differences large? (Are these averages from the same normalized sample indicative of a change?)

Calculations are ENTIRELY DIFFERENT!!!

​

LET’S TRY USING THE TOOL!

• Get data from the first tab “ParametricOrNot”

​

1. Is there a significant difference in Parametric TestScore(A) and TestScore(B)?
2. How about the Non-Parametric sets?
3. How about between TestScore(B) Parametric and TestScore(B) Nonparametric? �(try saying that both sets are parametric and see what happens!)

COMMON MISTAKE 1:�CHOOSING THE WRONG TEST

Consideration 3: How many data sets do I have?

2 ONLY

-T-test

-Mann-Whitney

-Wilcoxon�-etc.

3+

-Anova

-Kruskall-Wallis

-Friedman

-etc.

WHY DOES THIS MATTER?

LET’S TRY USING THE TOOL!

• Get data from the first tab “3Sets”
• Where are the differences in the classes?�→ If these are different students and the data is parametric?�→ If these are the same students and the data is non-parametric?

COMMON MISTAKE 1:�CHOOSING THE WRONG TEST

​

Part 2:

Misunderstanding p >.05

COMMON MISTAKE 1:�CHOOSING THE WRONG TEST

GENERALLY…

p < .05 means there IS a difference (effect / etc.)

But…

p > .05

The groups are the same

We don’t KNOW if there is a difference

WHY DOES THIS MATTER?

(e.g.) Two classes of 10 students: 1~5 Likert Scale question response

p = .09

BUT!!!!

• Same trend for 11 students each p = .05�
• Potential type 1 error�
• You should take more data, NOT NECESSARILY give up on this idea

WHY DOES THIS MATTER?

p = .06

p = .02

Many people think:�Group 2 improved more than Group 1

BUT!!!!

Look at the data:�Clearly they improve about the same amount

Pre-Post of 2 groups is FOUR GROUPS

​

No double t-tests!

​

• There is a separate page for this in my tool
• Multiple pieces of data (groups, times, etc.) MUST be put into a combined model
• Let’s try the data in my example set

WHY DOES THIS MATTER?

COMMON MISTAKE 2:�NOT REPORTING EFFECT SIZE

Effect size:

• How meaningful the relationship is
• Do we care?
• Does the trend hold for most points in the data set?

​

WHY DOES THIS MATTER (1)?

HELPS TO INTERPRET RESULTS BEYOND P VALUE

​

-P value is affected by sample size

-Effect size is not*

-Prevents “p-hacking”

-P value is becoming less admissible by American Statistical Association (Wasserstein & Lazar, 2016)

REMEMBER THIS EXAMPLE?

(e.g.) Two classes of 10 students: 1~5 Likert Scale question response

p = .09

• The effect size is MODERATE�
• Alerts you that more data would likely give you p < .05

PRACTICAL EXAMPLE (2)

Which year had the best learning outcomes?

More students improved overall; more even spread of improvement

With large sample sizes, p value becomes uninformative.

PRACTICAL EXAMPLE (3)

Differences in Males/Females to survey questions

If you did this randomly with no theory for N=40~90, how many times would you expect to see p<=.05 for 50 questions?

A: depends on sample size, but about 2 times

Effect sizes were all about the same

WHY DOES THIS MATTER (2)?

-Effect size is more directly comparable across various studies

-By including it, your study can be included in META – ANALYSES

-Very difficult to include without effect size reporting

​

​

COMMON MISSED OPPORTUNITY:�NOT CONSIDERING CORREL./ASSOC.

1. Starting point for analyzing survey data
2. Data triangulation
3. Finding factors involved in improvement

WHY DOES THIS MATTER (1)?

Students who like movies liked the lesson much more than those who don’t like movies

Example: �Four Likert-Scale Questions

WHY DOES THIS MATTER (2)?

Definite Improvement:

t(19) = 11.92, p < 0.01, d = 2.67

What activity pushed improvement?

Multiple Correlation with Relative Importance: F = 10.810, p < .001, R2 = 0.730

LET’S USE THE TOOL!

• Check the Correl1 tab
• Compare the correlation between Q1 and Q2
• Make a correlation matrix

​

​

• Check the MultiCorrel tab
• Check my calculations on the previous slide!
• See how different the calculations are if you predict the post-test scores instead of “improvement”

QUESTIONS?

Email:

Spring.ryan.Edward.c4@tohoku.ac.jp

​

Homepage:

www.springsenglish.online

​

Please cite or work with me!�(I have many other free tools as well)

​