AI Advancements for Literacy Learning and Beyond
Presenters:
Tamara Tate, PhD
Youngsun Moon
Jiali Wang
This material is based on work supported by the National Science Foundation under Grant No. 23152984 and the Institute of Education Sciences, U.S. Department of Education, through Grant R305C190007 to University of California, Irvine.
Paper 1:
Can ChatGPT Provide Useful
Holistic Essay Scoring?
Paper 2:
Comparing the Quality of Human and ChatGPT Feedback on Student Writing
Tate, Steiss, Warschauer, Bailey, Ritchie, Tseng, Moon, and Graham
Steiss, Tate, Cruz, Graham, Wang, Moon, Tseng, Warschauer, and Hebert
Outline
3
Section | Topic | Time |
Introduction | What is ChatGPT and how does it work? | 3 |
Why AI might have promise in classrooms | ||
Method | Participants and Measures | 3 |
Paper 1 Results | Scoring: Consistency & Validity | 8 |
Paper 2 Results | Feedback: Qualities of Good Feedback | 5 |
Human vs. AI Feedback | 8 | |
Discussion | Contextualizing Results & What’s next | 5 |
Digital Learning Lab
Introduction
4
Digital Learning Lab
Digital Learning Lab
What is ChatGPT and how does it work?
5
Digital Learning Lab
Cycle of Scoring & Formative Feedback
6
Human scoring and feedback is resource intensive.
Generative AI tools might improve this cycle:
Digital Learning Lab
Cycle of Scoring & Formative Feedback
7
But first, need to understand:
Digital Learning Lab
Method
8
Digital Learning Lab
Digital Learning Lab
Student Essay Samples for Paper 1
9
Paper 1 (Scoring)
Subset (n = 30) for RQ1
Digital Learning Lab
Sample writing prompt--WRITE Center, History
10
Digital Learning Lab
Paper 1:
Can ChatGPT Provide Useful
Holistic Essay Scoring?
Paper 1: Can ChatGPT Provide Useful Holistic Essay Scoring?
12
Digital Learning Lab
Internal Consistency of Scoring
13
Subset (n = 30) for RQ1
Digital Learning Lab
Human & AI Population Level Scores Similar
14
| Mean Score | SE | Confidence Interval |
Sample 1: Human | 3.16 | .06 | 3.05-3.28 |
Sample 1: AI | 2.69 | .06 | 3.32-3.55 |
Sample 2: Human | 2.69 | .05 | 2.60-2.78 |
Sample 2: AI | 2.71 | .04 | 2.63-2.79 |
Sample 3: Human | 3.38 | .04 | 3.30-3.46 |
Sample 3: AI | 3.36 | .03 | 3.30-3.42 |
Digital Learning Lab
Individual Scores
15
| Human-Human | Human-AI (Sample 1) | Human-AI (Sample 2) | Human-AI (Sample 3) |
Weighted Kappa | .79 | .52 | .23 | .52 |
Agreement w/ 1 point | 74% | 76% | 83% | 89% |
Digital Learning Lab
Individual Scores
16
| Human-Human | Human-AI (Sample 1) | Human-AI (Sample 2) | Human-AI (Sample 3) |
Weighted Kappa | .79 | .52 | .23 | .52 |
Agreement w/ 1 point | 74% | 76% | 83% | 89% |
Digital Learning Lab
Regression-No Significant Difference AI v. Human
17
| B | SE | t | p |
Constant | 3.36 | .07 | 51.18 | 0.000 |
AI scoring | .07 | .11 | 0.65 | 0.584 |
Corpus 1 | -.09 | .02 | -4.97 | 0.038 |
Corpus 2 | -.51 | .12 | -4.46 | 0.049 |
English learner | -.28 | .26 | -1.11 | 0.382 |
AI x EL | -.03 | .12 | -.25 | 0.829 |
Digital Learning Lab
Difference by English Language Status
18
| Non-English Learner | English Learner | ||
Wtd. Kappa | Agreement w/ 1 | Wtd. Kappa | Agreement w/ 1 | |
Sample 1: Human-Human | .82 | 73% | .81 | 80% |
Sample 1: Human-AI | .51 | 77% | .40 | 75% |
Sample 3: Human-AI | .52 | 89% | .43 | 92% |
Digital Learning Lab
Difference by English Language Status
19
Digital Learning Lab
Implications of AI for Scoring
20
Digital Learning Lab
Limitations
21
| Non-English Learners | English Learners |
GPT-3.5 | .52 | .36 |
GPT-4 | .58 | .37 |
Digital Learning Lab
Paper 2:
Comparing the Quality of Human and ChatGPT Feedback on Student Writing
Example of ChatGPT & Human feedback for Student Writing
Human Feedback: Thank you for writing this essay on how the boycott succeeded! I think your claim, “Well the reason the boycott was successful was because Rosa Parks started to give flyers out to black people to stop using the buses” is interesting. But did Parks distribute the flyers? Better check that source again (I think it was Jo Ann Robinson and her students).
Next time, you might use this sentence starter when adding evidence to support your claim, “According to the Bus Boycott Timeline,___________.” This will help you support your claim with clear evidence directly from the source. Your reader might also be convinced because you get your evidence from a source too.
You are on the right track to becoming a great writer!
ChatGPT Feedback: Great start! You have identified some key factors that contributed to the success of the Montgomery Bus Boycott. However, your essay could benefit from more detail and explanation. For example, you mention that Rosa Parks gave out flyers, but you don't explain what was on the flyers or how they helped the boycott. Try to provide more specific examples and evidence to support your argument.
Additionally, it's important to note that Rosa Parks' refusal to give up her seat was not the only reason for the boycott's success. Can you think of any other factors that contributed to the boycott's success?
Keep up the good work!
23
?
?
Digital Learning Lab
Student Essay
Why was the Montgomery Bus Boycott successful? Well the reason the boycott was successful was because Rosa Parks started to give flyers out to black people to stop using the buses. It did work because after a while the buses were not getting the money they used to get when the white and black people were there. Only the white were on the buses.
The second way they got the boycott to be successful is, Rosa Parks was in the front of the bus but black people were in the back but she sat on the front which they were not supposed to be on. She…
24
Digital Learning Lab
Example of ChatGPT & Human feedback for Student Writing
Human Feedback: Thank you for writing this essay on how the boycott succeeded! I think your claim, “Well the reason the boycott was successful was because Rosa Parks started to give flyers out to black people to stop using the buses” is interesting. But did Parks distribute the flyers? Better check that source again (I think it was Jo Ann Robinson and her students).
Next time, you might use this sentence starter when adding evidence to support your claim, “According to the Bus Boycott Timeline,___________.” This will help you support your claim with clear evidence directly from the source. Your reader might also be convinced because you get your evidence from a source too.
You are on the right track to becoming a great writer!
ChatGPT Feedback: Great start! You have identified some key factors that contributed to the success of the Montgomery Bus Boycott. However, your essay could benefit from more detail and explanation. For example, you mention that Rosa Parks gave out flyers, but you don't explain what was on the flyers or how they helped the boycott. Try to provide more specific examples and evidence to support your argument.
Additionally, it's important to note that Rosa Parks' refusal to give up her seat was not the only reason for the boycott's success. Can you think of any other factors that contributed to the boycott's success?
Keep up the good work!
25
?
?
Digital Learning Lab
Providing Feedback for Student Writing with ChatGPT
26
Introduction: Why AI might have promise in classrooms
Digital Learning Lab
27
Research Questions
Digital Learning Lab
Coding Scheme for Rating the Quality of Feedback (on a scale of 1-5)
28
Standard | Quality of feedback ranges from… |
Criteria-based | does not explicitly reference criteria (generic) → explicitly references criteria of source-based argumentative writing (e.g., claim, evidence) |
Clear directions for improvement | vague → always offers clear directions |
Accuracy | inaccurate → accurate description of student essays |
Prioritization of essential features | nonessential → essential, attainable, and reasonable for students’ immediate next step |
Supportive tone | unsupportive and directive → affirming and supportive |
Digital Learning Lab
Example of Human vs. ChatGPT feedback
Human Feedback: Thank you for writing this essay on how the boycott succeeded! I think your claim, “Well the reason the boycott was successful was because Rosa Parks started to give flyers out to black people to stop using the buses” is interesting. But did Parks distribute the flyers? Better check that source again (I think it was Jo Ann Robinson and her students).
Next time, you might use this sentence starter when adding evidence to support your claim, According to the “Bus Boycott Timeline,”___________. This will help you support your claim with clear evidence directly from the source. Your reader might also be convinced because you get your evidence from a source too.
You are on the right track to becoming a great writer!
ChatGPT Feedback: Great start! You have identified some key factors that contributed to the success of the Montgomery Bus Boycott. However, your essay could benefit from more detail and explanation. For example, you mention that Rosa Parks gave out flyers, but you don't explain what was on the flyers or how they helped the boycott. Try to provide more specific examples and evidence to support your argument.
Additionally, it's important to note that Rosa Parks' refusal to give up her seat was not the only reason for the boycott's success. Can you think of any other factors that contributed to the boycott's success?
Keep up the good work!
29
clear directions & prioritizes essential features
criteria-based
Digital Learning Lab
Coding Scheme for Rating Feedback
30
Digital Learning Lab
Analysis Methods
31
Digital Learning Lab
Comparing Scores for Human vs ChatGPT Feedback
32
Results
***
***
***
***
*
Digital Learning Lab
Coding Scheme for rating Feedback (on a scale of 1-5)
33
Standard | Quality of feedback ranges from… |
Criteria-based | does not explicitly reference criteria (generic) → explicitly references criteria of source-based argumentative writing (e.g., claim, evidence) |
Clear directions for improvement | vague → always offers clear directions |
Accuracy | inaccurate → accurate description of student essays |
Prioritization of essential features | nonessential → essential, attainable, and reasonable for students’ immediate next step |
Supportive tone | unsupportive and directive → affirming and supportive |
Digital Learning Lab
34
Digital Learning Lab
Whether the Quality of Feedback from ChatGPT varied for High- and Low-Scoring Essays
35
Digital Learning Lab
Whether the Quality of Feedback from Humans varied for High- and Low-Scoring Essays
36
Digital Learning Lab
Does the quality of feedback vary for essays of different levels of writing quality?
37
Digital Learning Lab
Does the quality of feedback vary for students with different language statuses?
Feedback given to students who were either initially fluent or English-only speakers (collectively, EO/IFEP) was not statistically different compared to that given to students classified as English learners (ELs) and reclassified fluent speakers (RFEP) using Scheffe correction for multiple comparisons.
38
Results
Digital Learning Lab
Does the quality of feedback vary for students with different language statuses?
39
Results
Digital Learning Lab
Discussion: Implications of AI for formative feedback
40
Digital Learning Lab
Contextualizing Results & What’s Next?
41
Conclusion
Digital Learning Lab
Questions?
Acknowledgements
43
Digital Learning Lab
Thank you!
This material is based on work supported by the National Science Foundation under Grant No. 23152984 and
the Institute of Education Sciences, U.S. Department of Education, through Grant R305C190007 to University of California, Irvine.
45
Digital Learning Lab
46
BE SURE TO USE CALL OUTS FOR SPECIFIC AREAS OF FIGURES, USE MULTIPLE SLIDES IF NEEDED
Digital Learning Lab
47
Digital Learning Lab
48
Pretend you are a secondary school teacher. Provide 2-3 pieces of specific, actionable feedback on each of the following essays written to this rubric that highlight what the student has done well and what they could improve on. Use a friendly and encouraging tone. If needed, provide examples of how the student could improve the essay.
49
Great job on providing a brief background on the Montgomery Bus Boycott and highlighting the key players involved. You have also provided some reasons why the boycott succeeded, such as the loss of revenue for the bus company and the strong leadership of King, MIA, NAACP, WPC, and churches. To improve your essay, consider providing more specific examples to support your points. For instance, you could mention how much revenue the bus company lost during the boycott or provide specific actions taken by the leaders and organizations involved. Additionally, you could expand on the impact of the boycott beyond Montgomery and how it influenced the Civil Rights Movement as a whole. | You did a wonderful job explaining what the Montgomery Bus Boycott was historically and you have a strong claim toward the end of your first paragraph.: “The Bus Boycott has succeeded because most of the people who rode Montgomery's buses were African Americans and this made the company lose a lot of business and revenue from the boycott so this left them with no easy solutions to fix.” Next time, I would love to see some evidence from the sources to help back up your claim. This will show your readers that your claims are supported! Keep on practicing your writing and you’ll keep improving. |
50
Great job on providing a brief background on the Montgomery Bus Boycott and highlighting the key players involved. You have also provided some reasons why the boycott succeeded, such as the loss of revenue for the bus company and the strong leadership of King, MIA, NAACP, WPC, and churches. To improve your essay, consider providing more specific examples to support your points. For instance, you could mention how much revenue the bus company lost during the boycott or provide specific actions taken by the leaders and organizations involved. Additionally, you could expand on the impact of the boycott beyond Montgomery and how it influenced the Civil Rights Movement as a whole. ChatGPT | You did a wonderful job explaining what the Montgomery Bus Boycott was historically and you have a strong claim toward the end of your first paragraph.: “The Bus Boycott has succeeded because most of the people who rode Montgomery's buses were African Americans and this made the company lose a lot of business and revenue from the boycott so this left them with no easy solutions to fix.” Next time, I would love to see some evidence from the sources to help back up your claim. This will show your readers that your claims are supported! Keep on practicing your writing and you’ll keep improving. Human |
Scoring the Feedback
51
52
All were within 1 point