Generative AI & Writing
Tamara Tate
Digital Learning Lab
This material is based on work supported by the National Science Foundation under Grant No. 23152984
Have you used GenAI?
Mentimeter: Menti.Com 6504 9325
How are you feeling about generative AI?
Digital Learning Lab
The foundation
Digital Learning Lab
What is generative AI?
ChatGPT and other large language models (LLMs) are a type of artificial intelligence that are trained to generate text similar to human-generated text. They are called “large” because they are typically trained on a corpus of text data ranging from millions to billions of words.
Digital Learning Lab
Digital Learning Lab
What is generative AI?
In spite of its startling capacities, generative AI’s serious flaws are also evident, including the risk of data privacy breaches, unclear intellectual property protection, algorithmic biases that replicate biases in the data, and ”hallucinations” when generative AI may give completely wrong answers.
Digital Learning Lab
Where to start?
Need help? See “Setting up ChatGPT”
Digital Learning Lab
Other thoughts?
Mentimeter: Menti.Com 6504 9325
Where does generative AI fit into writing?
Digital Learning Lab
Where does generative AI fit in?
Generate prompt
Topic Selection
Summarizing Content
Review/Feedback
Outlining
Digital Learning Lab
But use of AI is filled with contradictions
Digital Learning Lab
The “rich get richer” contradiction
AI is incredibly powerful for assisting communication
To get the most out of it, you need to know how to prompt it well, critically evaluate its output, and edit and incorporate it into your work
Digital Learning Lab
Digital Learning Lab
The “with or without” contradiction
If students never learn to use AI, they will be at a disadvantage in their study and careers.
If they use AI too much and too early, they will also be at a disadvantage as they will be robbed of foundational skills necessary to use it well.
Digital Learning Lab
Digital Learning Lab
Imitation contradiction�
English learners and developing writers are constantly told they need to imitate native speakers’ or expert writers’ language use.
But when they borrow exact phrases from sources, they can be accused of plagiarism. Sometimes even paraphrasing is frowned upon.
Digital Learning Lab
Digital Learning Lab
June/July contradiction�
Digital Learning Lab
Digital Learning Lab
What other issues and concerns do you have?
…….
Mentimeter: Menti.Com 6504 9325
Digital Learning Lab
Disclaimer in every assignment that uses AI
What Students Need to Know |
Students should have reviewed the Understand: How LLMs work curriculum and have an understanding of the Biases and Limitations of generative AI. Remember:
|
Digital Learning Lab
Why should I bother?
Digital Learning Lab
Post-Secondary Students’ Use and Perspectives of ChatGPT
Clare Baek, Tamara Tate, & Mark Warschauer (In progress)
Survey of U.S. Students in Post-Secondary Education (N=1001)�September 2023
�
Use for Writing? | Native Speakers | English learners |
At least weekly | 19.2% | 39.5% |
At least daily | 4.5% | 13.2% |
Several times a day | 2.3% | 5.3% |
Best available human standard
Would
do a better job solving a problem than the
Digital Learning Lab
Other thoughts?
Mentimeter: Menti.Com 6504 9325
What do we know?
BUT: Prior technology not as powerful or rapidly diffused
Digital Learning Lab
Understand
Digital Learning Lab
Understand
Digital Learning Lab
Access & navigate
{shout out to the IT team!}
Digital Learning Lab
3 lessons learned (so far)
Digital Learning Lab
Prompt
Critical new skill or flash in the pan?
Digital Learning Lab
Corroborate
FYI: Quality of AI feedback is very close to the quality of human feedback. It appears to be similarly good at scoring writing on a rubric (in process research).
Digital Learning Lab
Incorporate
Digital Learning Lab
GPT Detectors are Biased against Non-Native English writers (Liang et al, 2023)
Used 7 GPT Detectors on
(All pre ChatGPT release)
How to talk about generative AI to your class, team, department ….
Adapted from Daniel Stanford
Digital Learning Lab
How to talk (& think) about generative AI
Adapted from Daniel Stanford
Digital Learning Lab
Disability considerations
Digital Learning Lab
How to talk (& think) about generative AI
Adapted from Daniel Stanford
Digital Learning Lab
And that is the really hard part
While balancing all the contradictions
Digital Learning Lab
And now the really hard part
Digital Learning Lab
Implementing generative AI in a course
Process:
Researchers came up with types of AI activities based on what we had learned about course, goals, AI
Digital Learning Lab
Implementing generative AI in a course
Process:
Then team aligned AI w/ appropriate assignments where the use supported learning goals
Pesky Professor; executive summary
Project 1 and 2
Pesky Professor; executive summary
Digital Learning Lab
We always start with the course goals
Digital Learning Lab
What this looks like: Sample mini lessons
Digital Learning Lab
What this looks like: Sample mini lessons
Digital Learning Lab
UCSD Slide Deck
Image source: UCSD
Digital Learning Lab
How to talk (& think) about generative AI
Adapted from Daniel Stanford
Digital Learning Lab
Interdisciplinary uses for Instructors
Consider whether your use aligns with the uses allowed for your students. Why or why not? Are you transparent about this?
Digital Learning Lab
Interdisciplinary uses for students
Feedback: Our research shows within 1 point of humans
Digital Learning Lab
Research in process
Generative AI -- Holistic Scoring of Essays
Research Questions
Holistic Scoring
Holistic Scoring/ Grading
Digital Learning Lab
Research in process
Highlights
Holistic Scoring
Holistic Scoring/ Grading
Digital Learning Lab
Research in process
Generative AI -- Holistic Scoring of Essays
Agreement within 1 point: Close enough?
Holistic Scoring
Note: Sample 3 was done on a newer version of GPT 3.5
| All | Non English Learner | English Learner |
Sample 1: Human v. Human | 74% | 73% | 80% |
Sample 1: Human v. AI | 76% | 77% | 75% |
Sample 2: Human v. AI | 83% | NA | NA |
Sample 3: Human v. AI | 89% | 89% | 92% |
Holistic Scoring/ Grading
EL better
Digital Learning Lab
Research in process
Generative AI -- Feedback on Essays
Feedback
Digital Learning Lab
Research in process
Generative AI -- Feedback on Essays
Feedback
Digital Learning Lab
Pretend you are a secondary school teacher. Provide 2-3 pieces of specific, actionable feedback on each of the following essays written to this rubric that highlight what the student has done well and what they could improve on. Use a friendly and encouraging tone. If needed, provide examples of how the student could improve the essay.
Great job on providing a brief background on the Montgomery Bus Boycott and highlighting the key players involved. You have also provided some reasons why the boycott succeeded, such as the loss of revenue for the bus company and the strong leadership of King, MIA, NAACP, WPC, and churches. To improve your essay, consider providing more specific examples to support your points. For instance, you could mention how much revenue the bus company lost during the boycott or provide specific actions taken by the leaders and organizations involved. Additionally, you could expand on the impact of the boycott beyond Montgomery and how it influenced the Civil Rights Movement as a whole. | You did a wonderful job explaining what the Montgomery Bus Boycott was historically and you have a strong claim toward the end of your first paragraph.: “The Bus Boycott has succeeded because most of the people who rode Montgomery's buses were African Americans and this made the company lose a lot of business and revenue from the boycott so this left them with no easy solutions to fix.” Next time, I would love to see some evidence from the sources to help back up your claim. This will show your readers that your claims are supported! Keep on practicing your writing and you’ll keep improving. |
Great job on providing a brief background on the Montgomery Bus Boycott and highlighting the key players involved. You have also provided some reasons why the boycott succeeded, such as the loss of revenue for the bus company and the strong leadership of King, MIA, NAACP, WPC, and churches. To improve your essay, consider providing more specific examples to support your points. For instance, you could mention how much revenue the bus company lost during the boycott or provide specific actions taken by the leaders and organizations involved. Additionally, you could expand on the impact of the boycott beyond Montgomery and how it influenced the Civil Rights Movement as a whole. | You did a wonderful job explaining what the Montgomery Bus Boycott was historically and you have a strong claim toward the end of your first paragraph.: “The Bus Boycott has succeeded because most of the people who rode Montgomery's buses were African Americans and this made the company lose a lot of business and revenue from the boycott so this left them with no easy solutions to fix.” Next time, I would love to see some evidence from the sources to help back up your claim. This will show your readers that your claims are supported! Keep on practicing your writing and you’ll keep improving. |
Mentimeter: Menti.Com 6504 9325
Great job on providing a brief background on the Montgomery Bus Boycott and highlighting the key players involved. You have also provided some reasons why the boycott succeeded, such as the loss of revenue for the bus company and the strong leadership of King, MIA, NAACP, WPC, and churches. To improve your essay, consider providing more specific examples to support your points. For instance, you could mention how much revenue the bus company lost during the boycott or provide specific actions taken by the leaders and organizations involved. Additionally, you could expand on the impact of the boycott beyond Montgomery and how it influenced the Civil Rights Movement as a whole. ChatGPT | You did a wonderful job explaining what the Montgomery Bus Boycott was historically and you have a strong claim toward the end of your first paragraph.: “The Bus Boycott has succeeded because most of the people who rode Montgomery's buses were African Americans and this made the company lose a lot of business and revenue from the boycott so this left them with no easy solutions to fix.” Next time, I would love to see some evidence from the sources to help back up your claim. This will show your readers that your claims are supported! Keep on practicing your writing and you’ll keep improving. Human |
Scoring the Feedback
All were within 1 point
What would a feedback assignment look like?
Let’s take a look at the assignment and the reflection after use.
Digital Learning Lab
Want to learn more?
If you only read a few things on generative AI in education, start with these:
Digital Learning Lab
Need curricular resources?
Foundational Resources
Curricular Resources
Digital Learning Lab
Let’s try it
Act as a kind, helpful tutor with an understanding of the applicable subject matter area and good research and writing skills, who is helping a student begin their research; provide useful, specific advice. I’m writing a research paper for [course topic] and I need help coming up with a topic. I’m interested in topics related to [subject]. [Consider stating additional suggestions, requirements of the assignment, things you are interested in and areas to not include.] Please give me a list of 10 topic ideas.
My research question is going to be this: [input topic or question]. Please list some keywords I can use when searching library databases.
and even:
Please construct a few Boolean search strings I can use when researching this topic in library databases.
and:
Now I'd like you to recommend 2 or 3 library databases that would be good to search for this topic.
Digital Learning Lab
What did you notice?
Mentimeter: Menti.Com 6504 9325
What would this assignment look like?
Digital Learning Lab
Mark Warschauer
Tamara Tate
Clare Baek
Sharin Jacob
Daniel Ritchie
Jacob Steiss
Beth Harnick-Shapiro
Michael Dennin
Waverly Tseng
Sabrina Look
Questions?
Thank you!
Tamara Tate
This material is based upon work supported by the National Science Foundation under Grant No. 23152984.
© 2023 The Regents of the University of California
tptate.com
Instructional Framework
Digital Learning Lab
GenAI for professionals
Noy & Zhang, 2023
Dell’Acqua et al., 2023
Digital Learning Lab
Patterns: GenAI for professionals
Digital Learning Lab
What types of things is generative AI good at?
Digital Learning Lab
Interdisciplinary uses for Instructors
Consider whether your use aligns with the uses allowed for your students. Why or why not? Are you transparent about this?
Digital Learning Lab
Interdisciplinary uses for students
Feedback: Our research shows within 1 point of humans
From Tim Brodsky:
Digital Learning Lab
Research in process
Feedback
Digital Learning Lab