1 of 40

Public Peer Review Motivates Higher Quality Feedback

Xu Wang¹, Yali Chen¹, Amanda Godley², Carolyn Rosé¹

1

2

2 of 40

3 of 40

Peer Review

Hard for instructors to grade all students’ work
4000+ peer reviews on Coursera every day

3

4 of 40

4

Feedback quality is inconsistent

5 of 40

5

Feedback quality is inconsistent

Students do not feel accountable
Students lack a sense of community
Students may lack feedback generation skills

6 of 40

6

Feedback quality is inconsistent

Students do not feel accountable
Students lack a sense of community
Students may lack feedback generation skills

7 of 40

7

Student Assignment

Review 1

Review 2

Review 3

8 of 40

Potential Benefits of Public Environment

Students enjoy reading each other’s work (Kulkarni et al., 2015)
Motivation is important (Tinapple et al., 2013; Staubitz et al., 2016)
Meta-reviewers are helpful (Joyner et al., 2016)
Collaboration is improved after community engagement (Wen et al., 2018)

8

9 of 40

Potential for Synergy

Technology foundation for facilitating peer feedback at scale from Learning@Scale and Learning Analytics communities (Kulkarni et al., 2015; Hick et al., 2016; Joyner et al., 2016; Staubitz et al., 2016)�
Knowledge about what makes effective feedback from the Learning Sciences community (Nelson & Schunn, 2009; Ferris, 1997; Bitchener et al., 2005)

9

10 of 40

Will public environment work?

Will public environment increase feedback quality?

Will public environment increase student revision?

10

11 of 40

Methods

Random Assignment Lab Experiment

Mini MOOC unit on environment science

11

12 of 40

12

13 of 40

13

14 of 40

14

15 of 40

15

16 of 40

16

17 of 40

17

18 of 40

18

19 of 40

19

20 of 40

20

21 of 40

Participant Recruitment

From Amazon Mechanical Turk
65 participants

32 in the private condition
33 in the public condition

21

22 of 40

Measures

Measure of Feedback Quality

Expert coding of feedback

Adapted from Chris Schunn’s feedback coding manual

22

23 of 40

23

Summary	“The main point that I took away from this recommendation is that Plan B would be the best option due to the decrease in environmental risks associated with Wind Power.”
Problem	“Overall there was a lack of coherence and I don't feel there is a substantial enough argument here.”
Problem Localized Writing	“There were too many 'and' words that made it very difficult to follow what the writer was saying in the proposal.”
Problem Localized Idea	“The only problem I see is the fact that having wind turbines and the main source of energy will require quite a large amount of wind turbines.”
Solution	“I would only suggest maybe going more in depth on why these other alternatives are more useful.”
Solution Localized Writing	“I think it would be easier for me to get into your proposal if you stated that you're proposing Plan 2 in the beginning of your argument. Then, giving us reasons why the other plans are not the best would make an argument that is easy to follow.”
Solution Localized Idea	“I think that a solution to this could be to emphasis their strengths and maybe sell some of there energy from nuclear plants to adjacent towns to make up the difference.”
Praise	“The proposal is clear, concise and very well written.”
Praise Localized	“Plenty of factors and perspectives were added the tradeoffs were also considered. E.g. tourism, safety and lack of waste.”

24 of 40

Measures

Measure of Feedback Quality

Expert coding of feedback

Adapted from Chris Schunn’s feedback coding manual

Subjective rating

24

25 of 40

Measures

Measure of Feedback Quality

Expert coding of feedback

Adapted from Chris Schunn’s feedback coding manual

Subjective rating

Measure of Assignment Quality

Clarity, Supporting arguments, tradeoff/comparisons

25

26 of 40

Measures

Measure of Feedback Quality

Expert coding of feedback

Adapted from Chris Schunn’s feedback coding manual

Subjective rating

Measure of Assignment Quality

Clarity, Supporting arguments, tradeoff/comparisons

Measure of Revision

Revise or not
Revision quality

26

27 of 40

Results - 1a

Will public peer review increase feedback quality measured by expert coding?

27

28 of 40

Will public peer review increase feedback quality measured by expert coding?

28

Dependent variables
Summary
Problem
Problem Localized Writing
Problem Localized Idea
Solution **	p<0.05 effect size = 0.71
Solution Localized Writing
Solution Localized Idea **	p<0.05 effect size = 0.71
Praise
Praise Localized

Main Effect: Condition

Covariates:

Receiver Assignment Quality

29 of 40

Assignment Quality Negatively Correlated with Feedback Quality

Students show more specific and substantive feedback when the assignment quality is lower. [consistent with prior work]

29

30 of 40

Results - 1b

Will public peer review increase feedback quality measured by subjective rating?

30

31 of 40

Will public peer review increase feedback quality measured by subjective rating?

Please Rate how helpful the feedback comment is:

1: Very unhelpful; 2: Somewhat unhelpful; 3: Neutral; 4: Somewhat helpful; 5: Very helpful

Feedback generated in the public condition received higher subjective rating. (p < 0.001, effect size = 3.06)

31

32 of 40

Results - 2

Will public peer review increase student revision?

32

33 of 40

Will public peer review increase student revision?

There is no difference in the propensity to revise

Assignment text difference

Public condition showed a trend of more revision on tradeoff points. (marginally significant, p = 0.08)

33

34 of 40

Feedback Constructs Predicts Revision

The presence of Problem and Solution in feedback predicts higher propensity to revise. (p<0.05, p<0.05) [consistent with prior work]
The presence of Summary in feedback negatively predicts revise or not. (p<0.05) [inconsistent with prior work]

34

35 of 40

Summary of Results

Students in the public condition provided more Solutions and Localized Solutions about substantive ideas
Students in the public condition found the feedback they received to be more helpful.
Though we did not observe a difference in students’ propensity to revise.
We see a trend showing students in the public condition demonstrated higher revision quality.

35

36 of 40

Discussion and Implications

Public peer review increases the prevalence of certain dimensions of feedback, but not all

The intervention isn’t strong enough
Elevating the effort may not be the same as increasing the ability to offer feedback

36

37 of 40

Discussion and Implications

Public peer review increases the prevalence of certain dimensions of feedback, but not all

The intervention isn’t strong enough
Elevating the effort may not be the same as increasing the ability to offer feedback

Feedback to revision

More solutions don’t mean better solutions
Support on appropriation of feedback in revision process

37

38 of 40

Discussion and Implications

Public peer review increases the prevalence of certain dimensions of feedback, but not all

The intervention isn’t strong enough
Elevating the effort may not be the same as increasing the ability to offer feedback

Feedback to revision

More solutions don’t mean better solutions
Support on appropriation of feedback in revision process

Implications for feedback research

Subjective rating isn’t correlated with expert coding
Feedback construct [summary] didn’t predict higher revision
Feedback quality needs to be associated with assignment quality

38

39 of 40

Limitations and Future Work

Lab experiment → deployment study
Practical concerns (cheating, students’ privacy, etc.)
Future work: training on feedback, increasing students’ ability.

39

40 of 40

Questions?

xuwang@andrew.cmu.edu

www.iamxuwang.com

Public Peer Review Motivates Higher Quality Feedback

Xu Wang¹, Yali Chen¹, Amanda Godley², Carolyn Rosé¹

1

2