JavaScript isn't enabled in your browser, so this file can't be opened. Enable and reload.

1 of 30

Getting Started with AI in Teaching and Learning: Assessments

UNIVERSITY OF FLORIDA

CENTER FOR TEACHING EXCELLENCE

April 12, 2024

https://tinyurl.com/GSWAI-4

Getting Started with Artificial Intelligence in Teaching and Learning: Assessments © 2023-2024 by University of Florida, Center for Teaching Excellence is licensed under CC BY-NC-ND 4.0

2 of 30

INTRODUCTIONS TO TODAY'S PRESENTERS

JOHN JORDI, PHD

Instructional Assessment Coordinator

Center for Teaching Excellence

JENNIFER PARKER, ED.D.

Faculty Development Coordinator

Center for Teaching Excellence

LEOTA O'MALLEY, M. Ed.

Online Course Quality Specialist

Center for Teaching Excellence

MICHAEL BARBER, PHD

Assistant Director

Center for Teaching Excellence

KIMBERLY HEAL, M.S.

Graduate and Post-Doc Development Coordinator

Center for Teaching Excellence

2

3 of 30

Key Questions

What do I need to know about AI detectors?
How can I use AI to create assessments?
How can AI help students prepare for assessments?
What AI tools can I use for grading?

An alligator sitting at a desk in a college classroom taking an exam, looking happy

Content credentials

Generated with Copilot ∙ March 28, 2024 at 11:10 AM

3

4 of 30

Session 4�GSWAI: Assessment

AGENDA

Agenda:

What you need to know about AI Detectors
Using AI tools to create assessments
Multiple Choice
Higher Order Questions
AI and Assessments - Hands-On
Authentic Assessments
Case Studies
AI tools to help students prepare for assessments
AI tools for grading
AI and Assessments - Hands-On

5 of 30

UF Preferred AI Tool

5

6 of 30

WHAT DO I NEED TO KNOW ABOUT AI DETECTORS?

7 of 30

AI Detectors

AI detectors:

Attempt to distinguish between human-written and AI-generated text.
Are not reliable in detecting more advanced AI models.
ESL users more likely to be suspected of using AI
Not reliable enough to make accusations

Elkhatat, Elsaid, & Al‐Meer (2023); Perkins, Roe, Postma, McGaughran, & Hickerson, (2023).

A robot grading papers

Content credentials

Generated with Copilot ∙ April 11, 2024 at 3:20 PM

7

Research findings from the studies—will not share all of these but including them in the notes in case there are questions:

The study evaluated the performance of AI content detection tools (OpenAI, Writer, Copyleaks, GPTZero, and CrossPlag) in distinguishing between human-written and AI-generated text from ChatGPT models 3.5 and 4.
The tools were generally more accurate in identifying text generated by GPT 3.5 compared to GPT 4 or human-written content, indicating the increasing challenge of detecting more advanced AI models.
The OpenAI classifier showed high sensitivity but low specificity, while CrossPlag had high specificity but struggled with AI-generated text, especially from GPT 4.
The varying performance highlights the inconsistencies of these tools and the need for a more holistic approach that combines AI detection with manual review and contextual considerations, rather than relying solely on AI tools.
As AI text generation capabilities advance rapidly, continuous development and evaluation of AI detection tools are crucial to maintain their effectiveness and reliability in upholding academic integrity.

8 of 30

HOW CAN I USE AI TO CREATE ASSESSMENTS?

9 of 30

AI and Assessments

In this section, we will:

Use AI to create multiple choice questions and case studies
Review criteria for evaluating the quality of multiple- choice questions and case studies

9

10 of 30

Benefits and Limitations of Multiple Choice Questions

Limitations

Rely on student memorization
Misleading
Easy to guess

Benefits

Cover a wide range of topics
Provide instant feedback
Efficient for assessing a large number of students

10

Kim
Multiple choice questions are the most commonly used questions for assessments because they can be quick and easy to score and can cover lots of content on a single exam and still be answered in a class period. However, if not written carefully, unprepared students have the opportunity to guess and if their guess is correct, it is not a true assessment of their knowledge. The same goes for poorly written questions that can be misleading.

Good multiple choice questions take time and skill to construct- we want them to be a true assessment of whether or not students have mastered learning objectives, and we want questions to be more than just recall- we want them to have a balanced level of difficulty- move student from recall to applying or analyzing information.

Today, we will show you a few ways that Copilot can help you save time and effort in creating a repository of questons that cover a wide range of topics and cognitive skills.

11 of 30

Structure of a Multiple Choice Question

What is chiefly responsible for the increase in the average length of life in the United States from 1970 to 2020?

*A. The reduced death rate among infants and young children.

B. Compulsory health and physical education courses �in public schools.

C. The greatly reduced accidental deaths due to the �safety movement.

D. Increased mechanization which reduces the �need for manual labor.

Stem

Answer

Alternatives

11

Kim

Before we jump in, let's talk about the anatomy of a multiple choice question and a few best practices for evaluating each part.

Stem- question, problem, or scenario we want students to address.

Clear, should not contain irrelevant information, should not be stated negatively ( "which of the following is not true") students have a difficult time understanding negative, and should be a question or partial sentence (filling in the blank increases cognitive load)

2. Answer- have only 1 answer choice, similar length as the distractors

3. Alternatives-serve as distractors and should be plausible and stated clearly and concisely. "All of the above or none of the above" should be avoided because students could use partial knowledge to arrive at the correct answer.

For more in-depth criteria on evaluating each part of the multiple choice question, we have provided a link to an evaluation guide on the link document provided at the beginning of today's session.

12 of 30

Your Turn

Step 1:

Prompt: Generate 5 college level multiple choice questions that cover the water cycle and include correct answers with rationale.

Step 2:

These questions should align with the following student learning objectives:

define the terms transportation, evaporation, precipitation, runoff, groundwater, and condensation
describe the stages of the cycling of water through the water cycle
identify the main source of energy in the water cycle,
explain the importance of the water cycle for living organisms
describe the methods and the importance of producing potable water
interpret and explain the processes in diagrams of the water cycle

12

Kim
Step 1: For demonstration, I'm going to switch to Copilot and ask it to generate 5 questions- something to consider with prompts- we must be precise in what we are looking for.
After generating the 5 questions, remind participants to check for accuracy, look over the stem and distractors to make sure they meet the criteria we discussed.
Demonstrate in copilot- make sure participants understand they should check for accuracy, look over stem and distractors to meet criteria discussed on previous slide.

Step 2: We know the importance of making sure questions align with our learning objectives, so lets use the same prompt but include our learning objectives. After generating, once again, check for accuracy, etc.

Allow 5 minutes for participants to generate their own questions – if they have learning objectives available, use those as well.
Debrief with participants- who would like to share their experience- Explain that Microft is rolling out the Paperclip feature- with this, we will be able to attach our lecture notes, powerpoints, etc,

13 of 30

Your Turn

Prompt: Generate 5 college level multiple choice questions based on the image included. Include correct answers with rationale.

13

14 of 30

Bloom’s Taxonomy

14

15 of 30

Your Turn: Higher Order Multiple Choice Questions

Prompt

Write 5 higher order multiple choice questions that test a college psychology student’s ability to critically evaluate, analyze, and apply information about psychological disorders, not just recall information.

A realistic cartoon of a brain with an aura that looks like a galaxy

Content credentials

Generated with Copilot ∙ April 11, 2024 at 3:27 PM

15

16 of 30

Best Practices for Multiple Choice Questions

Revised Taxonomy of Multiple-Choice Item-Writing Guidelines

A realistic cartoon of an excited professor holding up an exam created using best practices

Content credentials

Generated with Firefly ∙ April 11, 2024 at 3:47 PM

16

17 of 30

HOW CAN I USE AI TO CREATE AUTHENTIC ASSESSMENTS?

18 of 30

Benefits of Using Case Studies as Assessments

Facilitate higher order thinking- analysis, synthesis, evaluation
Bridge theory to practice within a safe environment
Paired with a reading or lecture
On an assessment/exam

18

19 of 30

Criteria for Evaluating Case Studies

Real-world scenario
Supporting data
Open-ended problem

19

20 of 30

Your Turn: Case Study

Step 1: Medical

Prompt: Create a case study scenario for medical students to practice diagnosing shingles and include 5 questions for students.

Step 2:

Prompt: Regenerate the response and include the patient's lab work results.

Step 1: Art History

Prompt: Create a case study scenario for college level art history students to evaluate.

20

21 of 30

HOW CAN AI HELP STUDENTS PREPARE FOR ASSESSMENTS?

22 of 30

AI Tools for Assessment Preparation

AI tools can be used for interactive studying and can play the role of:

Tutors
Debate partner
Professors
Test prep coach for state and national exams

22

23 of 30

Your Turn: Debate Partner

Prompt: I want to practice debating the viewpoints of language development. You are playing the role of a nativist. I am playing the role of a behaviorist. This is a friendly debate, and you should be open to discussing the topic, asking questions to understand my viewpoint, and be willing to change your mind regarding this topic when presented with good arguments. We will begin after I introduce myself.

AI Prompt Cookbook

A realistic cartoon of a college student debating with a robot

Content credentials

Generated with Firefly ∙ April 11, 2024 at 3:53 PM

23

24 of 30

WHAT AI TOOLS CAN I USE FOR GRADING?

25 of 30

AI Tools for Grading

25

26 of 30

Packback

26

27 of 30

Gradescope

27

28 of 30

Perusall

28

29 of 30

Q AND A

CONTACT US:

CTE@aa.ufl.edu

30 of 30

References

Elkhatat, A. M., Elsaid, K., & Al‐Meer, S. (2023). Evaluating the efficacy of AI content detection tools in differentiating between human and AI-generated text. International Journal for Educational Integrity, 19(1). https://doi.org/10.1007/s40979-023-00140-5

Perkins, M., Roe, J., Postma, D., McGaughran, J., & Hickerson, D. (2023). Detection of GPT-4 generated text in higher Education: combining academic judgment and software to identify generative AI tool misuse. Journal of Academic Ethics. https://doi.org/10.1007/s10805-023-09492-6