Writing Instruction Across the Disciplines: Evidence-Based Practices in Grade 6-12

​

Leveraging Generative AI to Improve Secondary Writing Instruction

�Mark Warschauer and Tamara Tate�University of California, Irvine

July 16, 2024

​

​

​

​

This material is based on work supported by the National Science Foundation under Grant No. 23152984

Slide Deck

(plus extra resources)�

​

​

DigitalLearningLab.org

​

Mark Warschauer

Tamara Tate

Daniel Ritchie

Beth Harnick-Shapiro

Michael Dennin

Waverly Tseng

UCI Team

Project Lead

Lead Instructor

Tool Lead

Curriculum, Measures

Kristi Werry

Software Engineer

Based on funding provided�by NSF #23152984

digitallearninglab.org

​

Thanks for applying to our PhD in Education Program. I wanted to share with you a couple of our recent grant proposals to get your thoughts on them.

Mark

digitallearninglab.org

​

I hope this email finds you well. I am writing to express my profound interest in two groundbreaking research initiatives that I believe are reshaping the landscape of AI and education - a field I am deeply passionate about.

​

First, I came across the "Converse to Learn” project, which focuses on integrating AI dialogue education. As an AI enthusiast, I am fascinated by the potential of AI in enhancing educational processes and am eager to learn more about how this project aims to balance technological advancement with pedagogical effectiveness, particularly for young children.

Thanks for applying to our PhD in Education Program. I wanted to share with you a couple of our recent grant proposals to get your thoughts on them.

digitallearninglab.org

​

I hope this email finds you well. I am writing to express my profound interest in two groundbreaking research initiatives that I believe are reshaping the landscape of AI and education - a field I am deeply passionate about.

​

First, I came across the "Converse to Learn” project, which focuses on integrating AI dialogue education. As an AI enthusiast, I am fascinated by the potential of AI in enhancing educational processes and am eager to learn more about how this project aims to balance technological advancement with pedagogical effectiveness, particularly for young children.

Thanks for applying to our PhD in Education Program. I wanted to share with you a couple of our recent grant proposals to get your thoughts on them.

digitallearninglab.org

​

In recent years, the transformer architecture has gained more momentum in both natural language processing(NLP) and computer vision(CV) fields particularly because of its ability to capture complex, long-distance relationships in sequence data. Meanwhile, researchers discover that LLM is capable of learning new tasks with very few samples and could break complicated tasks into smaller ones by applying innovative approaches like Chain-Of-Thoughts.

digitallearninglab.org

​

In recent years, the transformer architecture has gained more momentum in both natural language processing(NLP) and computer vision(CV) fields particularly because of its ability to capture complex, long-distance relationships in sequence data. Meanwhile, researchers discover that LLM is capable of learning new tasks with very few samples and could break complicated tasks into smaller ones by applying innovative approaches like Chain-Of-Thoughts.

digitallearninglab.org

​

In recent years, the transformer architecture has gained more momentum in both natural language processing(NLP) and computer vision(CV) fields particularly because of its ability to capture complex, long-distance relationships in sequence data. Meanwhile, researchers discover that LLM is capable of learning new tasks with very few samples and could break complicated tasks into smaller ones by applying innovative approaches like Chain-Of-Thoughts.

In recent years, transformer architecture has gained significant attention in merging natural language processing (NLP) and computer vision (CV) fields, notably for its ability to understand complex, long-range relationships in sequential data. Simultaneously, research has shown that Large Language Models (LLMs) can learn new tasks with only a few samples and break down complicated problems using innovative approaches like Chain-Of-Thoughts.

digitallearninglab.org

​

digitallearninglab.org

​

David Autor, MIT Economist

digitallearninglab.org

​

558 Strategy Consultants

​

Writing tasks: Conceptualize and�develop new product ideas

• Creativity
• Analytic skills
• Persuasive skills
• Writing skills

1. No AI
2. With ChatGPT
3. With ChatGPT and�prompt engineering overview

​

Work graded by evaluators

Harvard Study

Dell’Acqua et al., 2023

digitallearninglab.org

​

558 Strategy Consultants

​

Writing tasks: Conceptualize and�develop new product ideas

• Creativity
• Analytic skills
• Persuasive skills
• Writing skills

• No AI
• With ChatGPT
• With ChatGPT and�prompt engineering overview

​

Work graded by evaluators

Harvard Study

Dell’Acqua et al., 2023

digitallearninglab.org

​

digitallearninglab.org

​

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486

digitallearninglab.org

​

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486

This is not surprising. We allow students to use Gen AI in our intermediate programming course and many of them just use it and don't even check the results and then can not perform without it in the midterms.

​

Dr. Barbara Ericson

Associate Professor, School of Information

University of Michigan

​

digitallearninglab.org

​

digitallearninglab.org

​

Scaffolding for Writing, or Scaffolding for Learning to Write?

​

​

digitallearninglab.org

​

Scaffolding for Writing, or Scaffolding for Learning to Write?

​

• Word Processors
• Spell Check
• Grammar check
• Thesauruses
• Do-what charts
• KWL charts
• Sentence starters
• Writing organizers
• Revision planners
• Peer feedback
• Teacher feedback and coaching

​

digitallearninglab.org

​

The “with or without” contradiction

If students never learn to use AI, they will be at a disadvantage in their study and careers.

digitallearninglab.org

​

The “with or without” contradiction

If students never learn to use AI, they will be at a disadvantage in their study and careers.

If they use AI too much, too early, and in the wrong ways, they will also be at a disadvantage as they will be robbed of foundational skills necessary to use it well.

digitallearninglab.org

​

Human-centered use of Gen AI brings opportunities for implementing best practice research

​

​

digitallearninglab.org

​

The really hard part

• When?
• Who?
• How much?
• In what ways?

While balancing all the contradictions

​

​

digitallearninglab.org

​

Implementing generative AI in a course

Process:

1. Identifying key takeaways for students to learn about GenAI and understanding what GenAI can and can’t do

Researchers came up with types of AI activities based on what we had learned about course, goals, AI

​

​

digitallearninglab.org

​

Implementing generative AI in a course

Process:

1. Identifying key takeaways for students to learn about GenAI; understanding what GenAI can and can’t do
2. Connecting with existing course learning objectives; understand course assignments, tone, content

Then team aligned AI w/ appropriate assignments where the use supported learning goals

Pesky Professor; executive summary

Project 1 and 2

Pesky Professor; executive summary

​

​

digitallearninglab.org

​

What is generative AI good at?

• Brainstorming/thinking partner
• Topics
• Counter arguments
• Evidence
• Perspectives
• Rubber duck debugging

Think first!

Often part of planning, pre writing

​

​

digitallearninglab.org

​

Planning

• Break down the assignment
• Brainstorm the topic
• Research

​

​

digitallearninglab.org

​

What is generative AI good at?

• Writing text
• if truth doesn’t matter or
• you know enough to gauge accuracy AND
• writing the text is not part of the learning objective

​

Hallucinations

Learning objectives

​

​

digitallearninglab.org

​

Writing text

• Where does “truth” not matter?
• Where are students experts or have access to the source material so they can corroborate the accuracy?
• Summarizing a source
• Creating a quiz on content (textbook, instructor slides) the student needs to know, for help reviewing
• Summarizing the student’s writing

​

​

digitallearninglab.org

​

Drafting

• Understanding sources
• Text organization

​

​

digitallearninglab.org

​

What is generative AI good at?

• Feedback on text
• On its own or
• In conjunction with peer review

​

Hallucinations

Learning objectives

Opportunity to motivate student revision

​

​

digitallearninglab.org

​

Feedback, Revision

• Terrific in conjunction with peer review
• Helps get additional, different input--critically reflect on difference
• Also helps teach reviewer some additional things they might do differently
• Research shows quality of feedback is close to human feedback quality (Steiss et al., 2024)
• Research shows holistic scoring moderately agrees with human scores, so has some low stakes value (Tate et al., under review) and per Grimes & Warschauer, may have some value encouraging students to [continue to?] revise

​

​

digitallearninglab.org

​

Revision

​

​

digitallearninglab.org

​

What is generative AI good at?

• Style
• Change genre (poetry, song, limerick….)
• Tone (academic, colloquial, sarcastic….)
• Mimic style of author, media outlet (like a newscaster, like a horror movie, in the style of Taylor Swift . . . .)

​

​

digitallearninglab.org

​

What opportunities have we seen so far?

​

​

digitallearninglab.org

​

What are the opportunities?

• Practice critical thinking
• Engage in perspective taking
• Reflection leads to discussion--and ownership--of their own process
• Push on corroboration, evidence use, sourcing: this is critical in the current environment
• It opens up the opportunity to discuss the writing process in ways we haven’t been doing
• More author agency and awareness of importance of their own voice

​

​

digitallearninglab.org

​

While you are at it …

​

Evaluate learning objectives and assessments

• What skills might be lost if students rely too heavily on AI?
• What skills might be essential for students in the future?
• Should course/program learning objectives be updated? What might that process/timeline look like?
• Which assessments are solid as is? Which might need minor adjustments or a major redesign?
• What strengths and weaknesses can we identify for specific assessment methods?

Adapted from Daniel Stanford

UCSD

​

​

digitallearninglab.org

​

So I can just let students use generative AI?

Nope, sorry.

• Basic facts.

​

​

digitallearninglab.org

​

Understand

• Only the basics, BUT enough to know
• It’s built on prediction & language
• It doesn’t think
• It has no ground truth
• It fabricates things rather than say “I don’t know”
• It was trained on a specific set of texts & by humans, so very real biases

• Impacts prompting
• Creates bias
• Need for corroboration

​

​

digitallearninglab.org

​

• PapyrusAI can’t _________________
• It can only ________________
• Sometimes PapyrusAI has pretty words but they are not ________
• PapyrusAI can have __________________
• Your job is to always ______________ what PapyrusAI writes before you use it. That’s part of what it means to be an author.
• When you give PapyrusAI information, make sure you don’t give it anything _____________.

When using PapyrusAI, always remember:

true

think

bias

check

predict

private

Are there other things I should think about when I have students use generative AI?

• Understand
• Think first
• Access--which tool does what, how to find it and log on, etc.
• Prompt
• Corroborate--critical skill, opportunity to reiterate this, practice it
• Incorporate--citation (NO), transparency (Yes)
• Reflect

​

​

digitallearninglab.org

​

Prompt

• Technical, prompting-specific knowledge
• Personas
• Emotion
• Details
• Content knowledge
• What question to ask
• Key words and phrases
• (Also helps recognize fabrications)
• Does not need to be perfect, aim for good enough, then push back and force the AI to revise

Mini lesson

Critical new skill or flash in the pan?

​

​

digitallearninglab.org

​

Mini Lesson

Important points to keep telling students

• AI can’t think
• AI makes stuff up, you have to check it
• The best way to keep stuff private is to not put it into the AI
• You are the author, you decide what to implement
• You are the author, you have to make sure your work is accurate
• What bias or limitations might be impacting the AI output?

​

​

digitallearninglab.org

​

To do

1. Address ChatGPT and similar technology in your class rules
2. Discuss expectations with your class and establish community norms for ethical use
3. Run your assignments through generative AI to understand potential output (and your assignment)--consider the value of process writing, your learning goals, alignment with assessments
4. Consider how it might help you
5. Consider how it might help your students

​

​

digitallearninglab.org

​

Where should I start?

1. Try generative AI for your own purposes, something low stakes like vacation ideas, and put one of your assignments in the tool and see what the output looks like.
2. Before using generative AI in your class, make sure that students learn the foundational knowledge about what AI is and its limitations.
3. Then pick two or three places where generative AI could be used in a lesson in support of existing learning objectives. Look for the “low hanging fruit,” modular, smaller pieces within the writing process, not some large stand-alone assignment. In each case, have the students turn in the text of their conversation with the generative AI tool as an appendix to the assignment.

​

​

digitallearninglab.org

​

Possible starting places ….

• Summarize. If the students are to take notes on readings or research, after they do so, have them use the AI to do the same and then have them revise their initial notes and write a short paragraph reflecting on the differences (both positive and negative) of both their version and the AI’s draft.
• Brainstorming. Have the student brainstorm on their own, then with a peer, then with the AI tool. Have them discuss with the peer what was helpful about each stage of the process.
• Feedback. After the students have written an initial draft of a text, have them revise the text. Then have them provide peer feedback to one another. Finally, use the generative AI to provide feedback. A short reflection on what kinds of feedback was most helpful and for what would complete the assignment.

​

​

digitallearninglab.org

​

Possible starting places ….

• Perspective taking. For an assignment in which counterargument is appropriate, have them try to come up with and rebut a counterargument. Then have them try the generative AI tool to do the same. Have them pick one or more counterarguments to include in their text and rebut the counterargument(s) as well. Have them reflect on their use of the tool.
• Bad example. Have them prompt the generative AI tool to create a bad example of something you are focusing on, perhaps tone or run on sentences. Have them share with other students in a gallery walk the final resulting text and the prompt(s) that got them there. This assignment will also build AI literacy in prompting and could be combined with instruction on prompting.
• Reverse outline. Have students take their writing and have the AI create a reverse outline of what they have written. Students can then examine the outline for logic, flow, and missing pieces.

​

​

digitallearninglab.org

​

Interdisciplinary uses for Instructors

• Idea generation, refinement--thinking partner, PLN
• Generate metaphors, culturally diverse examples of concepts
• Refine, drafts of lesson plans, quiz questions, writing prompts
• Creation of models: e.g., an essay that uses mostly passive voice
• Draft emails, administrative reports
• Feedback on writing

Consider whether your use aligns with the uses allowed for your students. Why or why not? Are you transparent about this?

​

​

digitallearninglab.org

​

​

​

A Walled Garden to Teach Writing and AI Literacy

�Educational Web Portal to GPT-4�

• Guaranteed Free Student Access
• Use in Evidence-Based Ways
• Improve Student Writing
• Teach AI Literacy
• Protect Student Privacy
• Leveraging Teachers’ Time

​

​

digitallearninglab.org

​

PapyrusAI Student Interface

• Students log in to class�PapyrusAI page
• Select among options to
• Help plan writing
• Feedback on thesis, rhetorical�strategy, essay structure
• Help revise writing
• Feedback on content,�organization, and style
• Numerical score
• Type or paste in their text
• Entry is sent via “API” to GPT-4 for immediate response

digitallearninglab.org

​

PapyrusAI Teacher Page

Choose assignments (pre-set prompts or new)

​

View reports on

• Which student logged into PapyrusAI
• Start and end time
• Functions they chose
• Texts they inputted and received

digitallearninglab.org

​

Papyrus AI Instructional Resources

• Integrated login with LMSs
• Teacher- and Student-Facing Instructional Resources
• Slides, Videos, Brief Readings
• Accessing and Using PapyrusAI
• Ethical Use of AI for Learning

​

​

​

digitallearninglab.org

​

digitallearninglab.org

​

I used to be super against using GenAI because I thought it was cheating. But now after the course I think it can be used as a tool if you do it responsibly.

PapyrusAI was a fantastic tool for my writing journey. I typically struggle in the early stages of writing because the ideas in my brain are always scattered and hard to put on paper. The AI helps me create outlines to organize these ideas in a simple, clear narrative.

My internship interview went great after I shared with the recruiter about my experience working with AI.

digitallearninglab.org

​

digitallearninglab.org

​

PapyrusAI

Sign Up Sheet to Get Updates on K-12 Rollout!

​

​

digitallearninglab.org

​

Topic Prompt

You are a kind and helpful middle school teacher with strong subject knowledge and good research and writing skills. You want to help a student brainstorm ideas for a topic and provide useful, specific advice. Follow the steps below and ask the questions one at a time. Wait for a response before moving on. Do not deviate from the step-by-step instructions. Use language and speak in a way that is appropriate for the student's grade level. Here are the steps. (1) First, ask the student about what subject the writing is for. Wait for input. Then ask for any relevant details of the assignment, such as topic or genre. Wait for input. Then ask what they might be interested in writing about. (3) Once they have responded, help them come up with a specific topic for the assignment by giving them a list of 5 topic ideas that align with the subject and the student’s areas of interest. Ask guiding questions to help them generate more ideas or expand on ideas. (4) Provide an organized summary of the student responses. Then ask the student if they would like to choose one topic and narrow down or would like to explore other related topics. Proceed accordingly until the student is satisfied with their topic.

​

​

digitallearninglab.org

​

Create a course

​

​

digitallearninglab.org

​

Questions?

Additional resources

To read:

• Large language models, explained with a minimum of math and jargon (Lee & Trott, 2023).
• What makes A.I. chatbots go wrong? (Metz, Apr. 4, 2023) is a simplified overview of issues. This Science article provides information on how AI can distort beliefs.
• Mollick (March 4, 2024).
• And this on interrogating.

To watch:

• But what is a GPT? Visual intro to transformers (video, 27 min.) by 3Blue1Brown.
• Mollick & Mollick, Part 3.
• Mollick & Mollick v. 5

​

​

digitallearninglab.org

​

Additional resources

Foundations of Generative AI Curriculum (higher education)

To help students with understanding how generative AI tools work, we have created several introductory mini-lessons that we recommend before students use these tools in the course:

​

UNDERSTAND: How LLMs work

Basic background information on generative AI.

UNDERSTAND: LLMs’ inherent limitations and biases

Background information on the limitations and biases of LLMs.

Prompt Generation

Tips for prompting generative AI.

Interrogating AI

Sentence starters for refining desired output.

​

​

​

digitallearninglab.org

​

Limitations & biases

Resources:

• Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models;
• These women tried to warn us about AI;
• Wan (gender bias in letters of rec by AI);
• Tao (cultural bias in LLMs);
• Why LLMs hallucinate;
• NYTimes, hallucinations.

​

​

digitallearninglab.org

​

Instructional Framework

Tate, T., Doroudi, S., Ritchie, D., & Xu, Y. (2023). Educational research and AI-generated writing: Confronting the coming tsunami.

digitallearninglab.org

​

https://www.sciencedirect.com/science/article/pii/S1060374323001091

digitallearninglab.org

​

Human-Driven Generative AI Use

This material is based upon work supported by the National Science Foundation under Grant No. 23152984.

© 2023 The Regents of the University of California