1 of 7

Does HWA work?

How We Argue: A Quasi-Randomized Controlled Trial of a Pedagogical Intervention to Improve Critical Thinking

All 1^st yr Honors students (n=1,000, 2 yrs) quasi-randomly assigned to 1 of 3 groups:

Control Do not complete How We Argue
HWA Complete How We Argue
HWE Complete How We Argue and How We Evaluate

All 3 groups complete pre-, mid-, & post-tests (LSAT logical reasoning + Evaluate Essay)

3. How effective is How We Argue when taught in a large classroom (n=250)?

Gary Comstock gcomstock@ncsu.edu Preliminary data, not for citation or publication /10/2024

2 of 7

Timeline

The “Evaluate” Test: Read a 1,200 word argumentative essay and answer 20 multiple choice questions to identify its main claim; the author’s reasons for it; and rebuttals to objections. Assess the soundness of the argument.

Pre-Test

Test

Post-Test

HWA

HWE

Control

HWA only

HWE

Aug. 2023

Sep. 2023

Nov. 2023

Jan./Feb. 2024

March 2024

Gary Comstock gcomstock@ncsu.edu Preliminary data, not for citation or publication /10/2024

3 of 7

Does HWA work?

Preliminary: Yes!

Among the students who completed both assessments in the Fall, the HWA group (n=306) improved significantly more than the Control group (n=305) from the pre-test, F(1,609) = 45.4, p<.001, 𝜂²=.07.

Control +2.1%
HWA +10.7%

HWA

Control

Beginning Fall

End Fall

Mid Spring

Gary Comstock gcomstock@ncsu.edu Preliminary data, not for citation or publication /10/2024

�F(1,609) = 45.4 (the Fisher or F-test) is the ratio of variance that can be explained by the corresponding effect to error. It is determined by dividing the Explained variance by the Unexplained variance.

p<.001 the P value describes how likely you would be to find this particular set of observations if there really was no association between HWA and gains in critical thinking. P values are used in hypothesis testing to help decide whether to reject the null hypothesis. The smaller the p value, the more likely you are to reject the null hypothesis. In the social sciences, a p value of <.05 is routinely taken as evidence that there are statistically sufficient grounds to believe there is a relationship between two phenomena. Therefore, a p value of <.001 is a significant indicator that HWA and CT gains are related.�

𝜂2=.07 (eta squared) refers to the magnitude of the effect sizes. In the social sciences, a rule of thumb [Cohen and Cohen (1983)] is that an effect size of .06 is described as a medium effect, distinct from small (.01) and large effects (.14). Therefore, our eta squared of .07 is a medium effect.

4 of 7

Is the effect stable?

Preliminary: Yes!

Among the students who completed all THREE assessments, the HWA only group (n=131) maintained their gain over the Control group (n=143) from the pre-test, F(2,271) = 12.4, p<.001, 𝜂²=.084.

Control -2.0%
HWA +6.1%

HWA

Control

Beginning Fall

End Fall

Mid Spring

Gary Comstock gcomstock@ncsu.edu Preliminary data, not for citation or publication /10/2024

5 of 7

Does HWE work?

Preliminary: Yes!

Among the students who completed all THREE assessments, the HWE group (n=34) outperforms both, HWA only (n=131) and Control (n=143), F(4,612) = 11.2, p<.001, 𝜂²=.068.

Control -2.0%
HWA +6.1%
HWE +16.0%

HWE

HWA

Control

Beginning Fall

End Fall

Mid Spring

Gary Comstock gcomstock@ncsu.edu Preliminary data, not for citation or publication /10/2024

6 of 7

Does HWE work?

Selection effect?!

Interestingly, students who enrolled in HWE were students who tended to be doing better in the Fall semester.

HWE

HWA

Control

Beginning Fall

End Fall

Mid Spring

Gary Comstock gcomstock@ncsu.edu Preliminary data, not for citation or publication /10/2024

1 of 7

2 of 7

3 of 7

4 of 7

5 of 7

6 of 7

7 of 7