Some Questions
For a future blog post, it would be really cool if you could answer the questions below.
You have performed three studies, all testing the same idea in a similar manner, of which two turn out to be significant, and one is not significant. Before this line of studies, you judged it was equally likely that the null-hypothesis is true, as that it is false (a uniform prior). The significance level in all studies was 0.05 (analyses were pre-registered to guarantee the Type 1 error rate). You designed the study to have 80% power if there is a true effect (assume you succeeded perfectly). Based just on this pattern of significant and not-significant results, how likely do you think it is you observed these significant and non-significant outcomes when there is a real effect?
Answer a percentage (from 0 to 100%)
You are planning a new study (in a different research line than in the question above). Beforehand, you again judge it is equally likely that the null-hypothesis is true, as that it is false (a uniform prior). You set the significance level at 0.05 (and pre-register this single confirmatory test to guarantee the Type 1 error rate). You design the study to have 80% power if there is a true effect (assume you succeed perfectly). What do you expect is the most likely outcome of this single study?
How likely do you think it is that you will observe the outcome of this single study that you have chosen above as most likely?
Answer a percentage (from 0 to 100%)
Assume you performed the single study described above, and have observed a statistical difference (p < .05, but you don’t have any further details about effect sizes, exact p-values, or the sample size). Simply based on the fact that the study is statistically significant, how likely do you think it is you observed a significant difference because you were examining a true effect?
Answer a percentage (from 0 to 100%)