1 of 56

What is ChatGPT and Why Should I Care?

Tamara Tate

Digital Learning Lab

This material is based on work supported by the National Science Foundation under Grant No. 23152984

2 of 56

Digital Learning Lab

3 of 56

Have you used GenAI?

Tell us in the chat yes/no and how you’ve used it

4 of 56

Post-Secondary Students’ Use and Perspectives of ChatGPT

Clare Baek, Tamara Tate, & Mark Warschauer (In progress)

Survey of U.S. Students in Post-Secondary Education (N=1001)�September 2023

�

Use for Writing?	Native Speakers	English learners
At least weekly	19.2%	39.5%
At least daily	4.5%	13.2%
Several times a day	2.3%	5.3%

Digital Learning Lab

5 of 56

How are you feeling about generative AI?

Digital Learning Lab

6 of 56

The foundation

Digital Learning Lab

7 of 56

What is generative AI?

ChatGPT and other large language models (LLMs) are a type of artificial intelligence that are trained to generate text similar to human-generated text through text prediction. They can not think or reason. They are called “large” because they are typically trained on a corpus of text data ranging from millions to billions of words.

Want to know more?

A mini lesson for your students (or you)

What is ChatGPT doing ... and why does it work? This is a detailed explanation by Stephen Wolfram (Feb. 14, 2023).

Digital Learning Lab

8 of 56

Digital Learning Lab

9 of 56

So I can’t quite picture that….

OK, so let’s say you’re teaching Shakespeare’s Hamlet, and the students should have all read it but you feel like they haven’t synthesized it well. For homework, you tell them to go home and write a paragraph summary of the basic plot.

OR

Using (or not) generative AI, make something creative that tells the story of Hamlet, it can be visual, textual, video, any genre from a comic to a song.

The next day they share with classmates and look for things that were missed, wrong, etc. by the AI (not mistakes by a classmate)

Good UDL too!

Digital Learning Lab

10 of 56

Verse 1:

In Elsinore, the castle walls

Echo with the stories of it all

A father's murder, a son's regret

A love that's lost, I can't forget

Chorus:

Oh, Hamlet, Hamlet, what did you do?

Revenge was sweet, but it ruined you

Verse 2:

Ophelia, innocent and kind

Caught in a love that was one of a kind

But now you're gone and I'm alone

Wondering if it was worth the stone

Chorus:

Oh, Hamlet, Hamlet, what did you do?

Revenge was sweet, but it ruined you

Bridge:

The crown, the power, the royal show

A battle for the throne, but it's all for show

In the end, what does it mean?

When all is said and done, and the curtain falls between

Outro:

In Elsinore, the castle walls

Echo with the memories of it all

A love, a life, a death, a reign

But in the end, it was all in vain.

Note: This song is written in a way that tries to capture the essence of the play, but also in a style that resembles Taylor Swift's way of storytelling in her music, with focus on personal emotions and experiences.

So it didn’t really know Billy Joel, but Taylor …..

Notice how it knows the genre/form

Added on its own; it’s chatty

11 of 56

Interdisciplinary uses for Instructors

Idea generation, refinement--thinking partner, PLN
Generate metaphors, culturally diverse examples of concepts
Refine, drafts of lesson plans, quiz questions, writing prompts
Creation of models: e.g., an essay that uses mostly passive voice
Draft emails, administrative reports
Feedback on writing

Consider whether your use aligns with the uses allowed for your students. Why or why not? Are you transparent about this?

Digital Learning Lab

12 of 56

Interdisciplinary uses for students

Studying: create self-testing, plan for studying, practice questions
Feedback: use like peer feedback for revision suggestions (emphasize that they remain the author and make active choices)
Generate metaphors, examples to explain difficult concepts
Generative AI as a thinking partner (example of what it can do with prompting, simulated middle school student)
Topic generation, refinement

From Tim Brodsky:

Tutor for AP Macro

Digital Learning Lab

13 of 56

Let’s try it

Topic

Act as a kind, helpful tutor with an understanding of the applicable subject matter area and good research and writing skills, who is helping a student begin their research; provide useful, specific advice. I’m writing a research paper for [course topic] and I need help coming up with a topic. I’m interested in topics related to [subject]. [Consider stating additional suggestions, requirements of the assignment, things you are interested in and areas to not include.] Please give me a list of 10 topic ideas.

Want some keywords for your library research? You can use this prompt:

My research [question] is going to be this: [input topic or question]. Please list some keywords I can use when searching library databases.

and even:

Please construct a few Boolean search strings I can use when researching this topic in library databases.

and:

Now I'd like you to recommend 2 or 3 library databases that would be good to search for this topic.

https://chat.openai.com/

Digital Learning Lab

14 of 56

What did you notice?

15 of 56

Research

Feedback: Our research shows quality within 1 point of humans

Steiss, J., Tate, T., Graham, S., Cruz, J., Hebert, M., Wang, J., ... & Tseng, W. (2023). Comparing the Quality of Human and Chatgpt Feedback on Students’ Writing.

Digital Learning Lab

16 of 56

Limitations & biases

In spite of its startling capacities, generative AI’s serious flaws are also evident, including the risk of data privacy breaches, unclear intellectual property protection, algorithmic biases that replicate biases in the data, and ”hallucinations” when generative AI may give completely wrong answers. It was built to chat.

Resources:

Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models;
These women tried to warn us about AI;
Wan (gender bias in letters of rec by AI);
Tao (cultural bias in LLMs);
Why LLMs hallucinate;
NYTimes, hallucinations.

Digital Learning Lab

17 of 56

Understand

Digital Learning Lab

18 of 56

Hannigan, Timothy and McCarthy, Ian P. and Spicer, Andre, BEWARE OF BOTSHIT: HOW TO MANAGE THE EPISTEMIC RISKS OF GENERATIVE CHATBOTS (December 28, 2023). Business Horizons, Forthcoming, Available at SSRN: https://ssrn.com/abstract=

Digital Learning Lab

19 of 56

Disclaimer in every assignment that uses AI

What Students Need to Know
Students should have reviewed the Understand: How LLMs work curriculum and have an understanding of the Biases and Limitations of generative AI. Remember: The AI doesn’t think or understand. It does not have opinions and cannot make value judgements. It’s just predicting the next bit of text given what you told it and what it has been trained on. Do NOT put sensitive personal information or confidential information into ChatGPT. Hallucinations, biases, and other concerns: You are responsible for the accuracy and appropriateness of anything you use or incorporate into your own text.

Digital Learning Lab

20 of 56

Concerns beyond bias

State assessment results
Classroom management
Academic integrity
Preparation for 4-year college & career
Protecting privacy
Meeting learning goals & standards
Time & resource pressures

Digital Learning Lab

21 of 56

Use of AI is filled with contradictions

The rich get richer
With or without
Imitation
June/July

Digital Learning Lab

22 of 56

The “rich get richer” contradiction

AI is incredibly powerful for assisting communication

To get the most out of it, you need to know how to prompt it well, critically evaluate its output, and edit and incorporate it into your work

Digital Learning Lab

23 of 56

The “with or without” contradiction

If students never learn to use AI, they will be at a disadvantage in their study and careers.

If they use AI too much and too early, they will also be at a disadvantage as they will be robbed of foundational skills necessary to use it well.

Digital Learning Lab

24 of 56

Imitation contradiction�

English learners and developing writers are constantly told they need to imitate native speakers’ or expert writers’ language use.

But when they borrow exact phrases from sources, they can be accused of plagiarism. Sometimes even paraphrasing is frowned upon.

Digital Learning Lab

25 of 56

June/July contradiction�

Digital Learning Lab

26 of 56

Should we just ban generative AI in schools?

Impossible: Some students can access it without you
Equity issues:

Need to make sure all students have access & education, not just privileged few
Very real affordances for English learners, disabled

Prepare for 4-year college & career

Digital Learning Lab

27 of 56

What do we know?

Prior research on digital technology integration into schools

Laptops/1-1 devices
Automated writing evaluation
Online learning

BUT: Prior technology not as powerful or rapidly diffused

Digital Learning Lab

28 of 56

Instructional Framework

Tate, T., Doroudi, S., Ritchie, D., & Xu, Y. (2023). Educational research and AI-generated writing: Confronting the coming tsunami.

Digital Learning Lab

29 of 56

Understand

Only the basics, BUT enough to know

It’s built on prediction & language
It doesn’t think
It has no ground truth
It fabricates things rather than say “I don’t know”
It was trained on a specific set of texts & by humans, so very real biases

Impacts prompting
Creates bias
Need for corroboration

Digital Learning Lab

30 of 56

Access & navigate

Literally, able to connect to the tool

{shout out to the IT team!}

Know which tool to use for what
Know how to navigate the tool
Know how to protect the privacy of information when using the tool

Digital Learning Lab

31 of 56

Prompt

Technical, prompting-specific knowledge

Personas
Emotion
Details

Content knowledge

What question to ask
Key words and phrases
(Also helps recognize fabrications)

Mini lesson

Critical new skill or flash in the pan?

Digital Learning Lab

32 of 56

Corroborate

AKA sourcing (history), good research skills (librarians)
Critical habit to get into in the current environment, not all “facts” are equal

FYI: Quality of AI feedback is very close to the quality of human feedback. It appears to be similarly good at scoring writing on a rubric (in process research).

Digital Learning Lab

33 of 56

3 lessons learned (so far)

Mini lesson

Digital Learning Lab

34 of 56

Incorporate

Transparency
Citation
Academic integrity

No reliable detector

Not even humans

Easy to evade detectors w/ prompting
Especially unreliable for non-native English speakers

Presumably dialects would also be problematic

Digital Learning Lab

35 of 56

GPT Detectors are Biased against Non-Native English writers (Liang et al, 2023)

Used 7 GPT Detectors on

Real TOEFL Essays
Real US 8th-grade essays

(All pre ChatGPT release)

A special note regarding English language learners and generative AI:

Warschauer, M., Tseng, W., Yim, S., Webster, T., Jacob, S., Du, Q., & Tate, T. (2023). The affordances and contradictions of AI-generated text for second language writers. Available at SSRN.

Digital Learning Lab

36 of 56

Best available human standard

Would

the best available AI
in a particular moment,
in a particular place,

do a better job solving a problem than the

best available human that is actually able to help
in a particular situation?

--Ethan Mollick, Oct. 22

Digital Learning Lab

37 of 56

Where to start?

Try something personal: “Suggest 5 recipes for dinner tonight using chicken and broccoli” Stir fry it is!
Try something for your class: “What are 5 different metaphors for explaining logarithms to undergraduate students in math class?” Hmmmm, maybe the password decoder?
Try out one of your assignments to see what generative AI would produce: “Write a 5 word poem on what you did this summer.” I’m a machine; I pondered.

Need help? See “Setting up ChatGPT”

Digital Learning Lab

38 of 56

Where to start?

If you only read a few things on generative AI in education, start with these:

Digital Learning Lab

39 of 56

And now the really hard part

When?
Who?
How much?
In what ways?

While balancing all the contradictions

Digital Learning Lab

40 of 56

And now the really hard part

When?
Who?
How much?
In what ways?

Digital Learning Lab

41 of 56

Researcher
Instructor
V.P. Teaching & Learning
Researcher

Digital Learning Lab

42 of 56

Implementing generative AI in a course

Goals:

Focusing on what we can do now to build up to more long-term solutions
Supplementing what is already being done in the classroom
Leveraging instructors’ agency and knowledge of their students

Centering students’ learning in and beyond the classroom

Digital Learning Lab

43 of 56

Implementing generative AI in a course

Process:

Identifying key takeaways for students to learn about GenAI and understanding what GenAI can and can’t do

Researchers came up with types of AI activities based on what we had learned about course, goals, AI

Digital Learning Lab

44 of 56

Implementing generative AI in a course

Process:

Identifying key takeaways for students to learn about GenAI; understanding what GenAI can and can’t do
Connecting with existing course learning objectives; understand course assignments, tone, content

Then team aligned AI w/ appropriate assignments where the use supported learning goals

Pesky Professor; executive summary

Project 1 and 2

Pesky Professor; executive summary

Digital Learning Lab

45 of 56

We always start with the course goals

Digital Learning Lab

46 of 56

What this looks like: Sample mini lessons

Digital Learning Lab

47 of 56

What this looks like: Sample mini lessons

Digital Learning Lab

48 of 56

How to talk (& think) about generative AI

Providing support and managing risk

What support do you need to learn more about AI and feel empowered?
What obstacles need to be removed?
What might backfire?
What risks would you be taking on?

Adapted from Daniel Stanford

Digital Learning Lab

49 of 56

How to talk (& think) about generative AI

Evaluate learning objectives and assessments

What skills might be lost if students rely too heavily on AI?
What skills might be essential for students in the future?
Should course/program learning objectives be updated? What might that process/timeline look like?
Which assessments are solid as is? Which might need minor adjustments or a major redesign?
What strengths and weaknesses can we identify for specific assessment methods?

Adapted from Daniel Stanford

Digital Learning Lab

50 of 56

UCSD Slide Deck

Digital Learning Lab

51 of 56

How to talk (& think) about generative AI

Address ethical concerns

How can we support AI literacy in a way that is ethical and equitable?
When might AI use be forbidden, discouraged, encouraged, or required?

Why/how might this vary for specific courses, assignments, students, or instructors?

How might we address concerns around data privacy, unpaid labor, and equal access?

Adapted from Daniel Stanford

Digital Learning Lab

52 of 56

So what do educators need to be successful?

Technical support
Access to training and resources to support their work
Time -- to think, collaborate, plan--integrating AI into existing learning objectives and assessments requires a great deal of expertise, creativity, and thinking
An atmosphere that encourages innovation and allows for mistakes

Digital Learning Lab

53 of 56

To do

Address ChatGPT and similar technology in your syllabus
Discuss expectations with your class and establish community norms for ethical use
Run your assignments through generative AI to understand potential output (and your assignment)--consider the value of process writing, your learning goals, alignment with assessments
Consider how it might help you
Consider how it might help your students

Digital Learning Lab

how we can use these things as tools where language is purely functional boilerplate

as a tool, it’s most useful in the hands of people who already have the skills and knowledge to write well. The reason you could work with that tool is because you recognize what effective communication is in the context you were using it in. If you didn’t, you would have no idea if what it produced was good, bad, or otherwise.

What it emphasizes it’s the experience of writing. That is the thinking that is associated with the act of writing as opposed to the product that we assess at the end of a experience of writing. One of my mantras about writing is that writing is thinking. When we write, we are both expressing and exploring an idea. This is one of the experiences I want students to have as they learn to write is that while they are in the act of writing they are uncovering new ideas or new aspects of what they thought about what they’re writing about. Anybody who has spent any time writing has experienced this.

54 of 56

Mark Warschauer

Tamara Tate

Clare Baek

Sharin Jacob

Daniel Ritchie

Jacob Steiss

Beth Harnick-Shapiro

Michael Dennin

Waverly Tseng

Sabrina Look

Digital Learning Lab

55 of 56

Questions?

56 of 56

Thank you!

Tamara Tate

This material is based upon work supported by the National Science Foundation under Grant No. 23152984.

tptate.com