1 of 50

���Big little lies: a compendium and simulation of p-hacking strategies� ������

Biostatistics Journal Club

Presenter: Kaylen Wei

Written by: Angelika M. Stefan & Felix D. Schönbrodt

2 of 50

Content

  • Self introduction
  • Reason of why I picked this article
  • Discussion on the selected article
    • Definition of p-hacking
    • P-hacking strategies
    • Evaluation on potential solutions for avoiding p-hacking
    • Conclusions

2

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

3 of 50

3

About the presenter

Kaylen Wei

  • A first-year MS student in Biostatistics
  • Completed undergraduate in Statistics
  • Interested in clinical trials
  • A big fan of Friends

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

4 of 50

4

Ethics

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

5 of 50

5

Ethics in the science field

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

6 of 50

6

Ethics in the science field

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

7 of 50

7

Ethics in the science field

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

8 of 50

8

P-hacking

  • It blew my mind when I first heard about it
  • P-hacking does happen in real world
  • P-hacking can have a bad and significant impact on the statistical result ☹
  • Increase our awareness
  • Hear what you guys think

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

9 of 50

9

P hacking definition

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

10 of 50

10

P hacking definition

data

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

11 of 50

11

P hacking definition

data

non-significant hypothesis testing results

(not expected ☹)

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

12 of 50

12

P hacking definition

data

non-significant hypothesis testing results

(not expected ☹)

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

13 of 50

13

P hacking definition

significant hypothesis testing results (expected ☺)

data

non-significant hypothesis testing results

(not expected ☹)

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

14 of 50

14

P-hacking strategies

  1. Selective reporting of the dependent variable
  2. Selective reporting of the independent variable
  3. Optional stopping
  4. Outlier exclusion
  5. Controlling for covariates
  6. Scale redefinition
  7. Variable transformation
  8. Discretizing variables
  9. Exploiting alternative hypothesis tests
  10. Favourable imputation
  11. Subgroup analyses
  12. Incorrect rounding

​

​

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

15 of 50

15

1. Selective reporting of the dependent variable

  • Only report the models with significant results by trying different dependent variables
    • An analysis of Cochrane reviews: 46.8% publications changed their dependent variables after submitting their protocol
    • Issue with conducting multiply hypothesis testing
      • Increase false-positive rates

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

16 of 50

16

Issue with conducting multiply hypothesis testing

Goal: Need to test 100 locations of the brain to see which part of brain responses to a cognitive task

​

​

​

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

17 of 50

17

Issue with conducting multiply hypothesis testing

Goal: Need to test 100 locations of the brain to see which part of brain responses to a cognitive task

Method: Conduct t-test at each of these 100 locations

​

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

18 of 50

18

Issue with conducting multiply hypothesis testing

 

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

19 of 50

19

Issue with conducting multiply hypothesis testing

 

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

20 of 50

20

Inflate false positive rate

Figure 1. Impact of selective reporting of the dependent variable on false-positive rates in a t-test. Number of dependent variables indicates how many hypothesis tests were conducted (at maximum) to obtain a significant result. The solid grey line shows the nominal α-level of 5%.

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

21 of 50

21

  • Only report the models with significant results by trying different independent variables
    • Issue with conducting multiply hypothesis testing
      • Increase false-positive rates

2. Selective reporting of the independent variable

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

22 of 50

22

3. Optional stopping (“data peeking”)

​

  • Stop collecting data once get expected significant result
  • Affect data collection process
  • Theoretically, collecting infinite data will be guaranteed to give a statistically significant result

​

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

23 of 50

23

Optional Stopping Example

​

Step 1: Sample (simulate) 100 data from the population

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

24 of 50

24

Optional Stopping Example

​

Step 1: Sample (simulate) 100 data from the population

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

> 0.05

25 of 50

25

Optional Stopping Example

​

Step 2: Continue sampling (simulating), and get another 100 data

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

26 of 50

26

Optional Stopping Example

​

Step 2: Continue sampling (simulating), and get another 100 data

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

> 0.05

27 of 50

27

Optional Stopping Example

​

Step 3: Continue sampling (simulating), and get another 100 data

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

28 of 50

28

Optional Stopping Example

​

Step 3: Continue sampling (simulating), and get another 100 data

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

< 0.05

29 of 50

29

4. Outlier exclusion (“data trimming”)

​

  • 39 different outlier identification techniques listed in a literature review (Aguinis et al.)
  • Vague about outlier identification techniques
  • Not report outlier identification techniques
  • Self define the threshold of outlier
  • Increase false positive rate

Extreme

values

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

30 of 50

30

5. Controlling for covariates

  • Only leave significant covariates/predictors in the model
  • 32% survey respondents (Chin et al) admitted to dropping covariates selectively based on p values
  • Increase false positive rate

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

31 of 50

31

`

6. Scale redefinition

​

  • Redefine measurement scale
  • Use composite variables to fit model
  • SPSS allows users to recalculate reliability coefficients every time an item is deleted from the score
  • Increase false positive rate

​

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

32 of 50

32

7. Variable transformation

​

  • Apply opportunistic transformation on variable (i.e. compute its logarithm, reciprocal, or change its underlying measurement scale)
  • Normality assumption for linear regression refers to residuals, not refers to dependent/independent variables
  • Increase false positive rate without assumption check

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

33 of 50

33

7. Variable transformation

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

34 of 50

34

8. Discretizing variables

​

​

​

  • Split continuous variables into categories
  • Decide cut off points based on p values
    • median split
    • tertile split
    • cut-the-middle split
  • Increase false positive rate

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

35 of 50

35

9. Exploiting alternative hypothesis tests

​

​

  • Use the same variables and try different statistical models/hypothesis tests to answer the same research question
  • Only report the statistically significant result
  • Increase the false positive rate

​

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

36 of 50

36

10. Favourable imputation

​

​

  • Choose imputation method for missing data based on statistical significance
  • Surveys on ethical research: Slightly less than 10% of researchers hid their imputation methods
  • Increase false positive rate

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

37 of 50

37

10. Favourable imputation

​

​

Simulation on 10 different imputation methods

​

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

38 of 50

38

11. Subgroup analyses

​

​

  • Change inclusion criteria based on statistical significance
  • 34% of the published protocols made major changes to participant inclusion criteria (Silagy et al.)
  • Increase false positive rate

​

​

​

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

39 of 50

39

12. Incorrect rounding

​

​

  • Round p value incorrectly to obtain statistically significant findings
    • P = 0.0499, P = 0.0501
  • psychology literature (1965 – 2005): 36 out of 93 p-values that were larger than 0.05 rounded as 0.05 (Leggett et al.)
  • Among 2470 reported p-values = 0.05, 67.45% of them were incorrectly rounded down towards significance (Hartgerink et al.)

​

​

​

​

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

40 of 50

40

P hacking sounds bad, so any solutions?

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

41 of 50

41

Protentional solutions & evaluations

​

​

​

  1. Larger sample sizes
  2. Redefine statistical significance
  3. Shifting the focus to effect sizes
  4. Shifting the focus to Bayes factors
  5. Correct for p-hacking
  6. Preregistration and registered reports

​

​

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

42 of 50

42

1. Larger sample sizes

​

​

  • Sample size is large
  • Can NOT reduce p-hacking effects when p-hacking presents
  • Affect incentives for p-hacking
    • Might be impossible to reconduct a study with a large sample to obtain statistically significant results
    • Report the null result
    • Increase researchers’ willingness of trying different hypothesis tests -> further empirical investigation

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

43 of 50

43

2. Redefine statistical significance

​

​

​

 

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

44 of 50

44

3. Shifting the focus to effect sizes

​

​

​

​

  • Abandon p-values and use effect size estimation instead
    • Mean differences
    • Correlations
    • Odds ratio
  • Effect sizes are overestimated with the presence of p-hacking
  • Other p-hacking and reporting strategies can influence the effect size distributions

​

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

45 of 50

45

4. Shifting the focus to Bayes factors

​

​

​

​

​

  • Frequentist hypothesis testing -> Bayesian statistical framework
  • Use Bayes factors
    • Bayes factor hacking strategies
    • Impacted by prior distribution

​

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

46 of 50

46

5. Correct for p-hacking

​

​

​

​

​

  • Apply p-hacking detection and correction method
    • P curve
    • Fisher’s method
    • Excess test statistics
  • Require multiply p-values/studies to make inference about p-hacking
  • Can not capture all the p-hacking techniques and reporting strategies

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

47 of 50

47

6. Preregistration and registered reports

​

​

​

​

​

  • Decide hypotheses, methods, and analyses before data collection
  • Preregistrations are often vague
    • How to handle missing values
    • Still leave the room for applying p-hacking strategies
  • Be specific and constrain the number of analysis options

​

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

48 of 50

48

Conclusions

​

Summary

  • P-hacking strategies can increase false positive rate
  • Potential solutions have their own shortages
  • Recommend to report Bayes factor & p value

Further Steps

  • Investigate whether certain p-hacking strategies are more fallible in certain fields
  • Develop mathematical models for p-hacking strategies

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

49 of 50

49

  1. basil848. (2019, November 7). Friends confused Gif - friends confused looking - discover & share gifs. Tenor. https://tenor.com/view/friends-confused-looking-staring-what-gif-15497617
  2. CANVA: Visual suite for everyone. (n.d.). https://www.canva.com/
  3. Encyclopædia Britannica, inc. (n.d.). Reproductive cloning. Encyclopædia Britannica. https://www.britannica.com/science/cloning/Reproductive-cloning
  4. Flandra. (2023, September 10). Medium. https://medium.com/@flandrab14/do-u-really-know-u-have-real-friends-heres-how-they-should-be-real-877c92f66d5d
  5. History of the human brain. Cardiff University. (n.d.). https://www.cardiff.ac.uk/news/view/425889-history-of-the-human-brain
  6. Hussein2022-06-08T07:41:00+01:00, L. (2022, June 8). 6 steps to successful science lab groups. RSC Education. https://edu.rsc.org/ideas/6-steps-to-successful-science-lab-groups/4015758.article
  7. Molnar, C. (2023, May 9). Bayesian modeling from First Principle and memes. Bayesian modeling from first principle and memes. https://mindfulmodeler.substack.com/p/bayesian-inference-from-first-principles
  8. Nope try again! - Futurama Fry. Make a Meme. (n.d.). https://makeameme.org/meme/nope-try-again-845dad4218
  9. Rabin, N. (2022, May 10). I’m obsessed with this stupid meme. Nathan Rabin’s Happy Place. https://www.nathanrabin.com/happy-place/2022/5/9/im-obsessed-with-this-stupid-meme
  10. Tolliday, B. (2022, February 3). Feature: Animal research saves lives. so why do opponents say it is ineffective?. EARA. https://www.eara.eu/post/feature-animal-research-saves-lives-so-why-do-opponents-say-it-is-ineffective
  11. Wow. you blow my mind. Yarn. (n.d.). https://getyarn.io/yarn-clip/6e259218-3225-435d-9e33-6c2f9945a646

​

​

​

​

​

​

​

​

Introduction

Reason of picking this article

Definition of p-hacking

P-hacking strategies

Evaluation on potential solutions

Conclusions

References

​

50 of 50

50

Thank you!