1 of 26

Reimagining Assessment in the World of AI

February 1, 2024

Reimagining Assessment in the World of AI by Paula Demacio at Centennial College is licensed under CC BY-NC 4.0 unless otherwise noted

2 of 26

Today’s conversation

The current realities
The current opportunity - the Redesign Approach

Make GenAI irrelevant or difficult to use
Integrate into assignment

How do we decide?
How can we make it happen?
Things to remember…

Image by Gerd Altmann from Pixabay

3 of 26

Traditional AI versus GenAI

Artificial Intelligence (AI)

Spell checkers
Grammar checkers
AI driven design recommendations (e.g. PPT)
Using Google Translate to translate text from one language into another
Using a voice-to-text app on a smartphone
Non-human assessors (eg. ACCUPLACER)
AI driven similarity checkers (Turnitin)
AI proctoring services

Generative Artificial Intelligence (GenAI)

AI that is trained on a dataset and generates new content—in response to prompts by a user
text, images, audio, and video
ChatGPT, DALL·E, Bing, Bard, etc

4 of 26

The Current GenAI Realities

Faculty and students are using the “new” GenAI tools (i.e ChatGPT, DALL·E)
This is the beginning of an evolving conversation. GenAI tools are not going away. They will continue to evolve…
GenAI literacy is critical (for you and your students)
GenAI is and will continue to be incorporated into the jobs of the future - how will we help prepare our learners?
There are no simple solutions to complex technologies

5 of 26

Assessment concerns

Elephant in the room by Bene Riobó is licensed under CC BY-SA 4.0

6 of 26

Avoid Getting Caught Up in the Arms Race

7 of 26

Our Current Opportunity - Redesign Approach

Trust that your students want to learn and give them something meaningful to do…..

Practical approach �

Make AI-proof*
Embrace/Incorporate GenAI tools
authentic/alternative assessment
There’s no one size fits all approach
GenAI literacy

Philosophical approach�

Meaning and meaningful
Connection
Relevance
Motivation

Food for thought:

Why do students cheat?
The trust gap

Why do we assess
What’s in it for the student
We can design assessments and use teaching strategies that help students build investment in their learning by connecting it authentically to their lived experiences, career goals, professional contexts, and positionality as learners.
Design assessment opportunities with meaningful connections to students’ lives and/or future careers, where students can see the relevance of the assessment outside of getting a grade.
Create an assignment that holds value for people beyond your classroom, and give your students the opportunity to openly publish their work. Working on an assignment that has a larger audience, a longer life and a greater impact often makes the assignment process more meaningful for the student and discourages academic misconduct.
The trust gap - an illustration depicting the difference between what we say we want and what we show we want…..and the student response to this

8 of 26

GenAI literacy

Starting Place – your own GenAI literacy �

What is it?
How does it work?
Limitations
How is it impacting your discipline?
Ways to use it

Not all are academic misconduct

Prompting
Critical analysis
Resource for educators at Centennial College

Image by Gordon Johnson from Pixabay

9 of 26

Assessment - a Cake Metaphor!

As an educator, which one do you want/need? What are some of the factors you will use to help you decide?

From scratch

From a box

From the bakery

Image by Noshin Naz from Pixabay

10 of 26

This slide is not openly licensed. Copyright Matt Miller, “Classroom AI Use: What's Cheating? What's OK?”, Ditch that Textbook

11 of 26

How will you decide?

Learning Outcomes*

Course learning outcomes (CLOs)
Essential employability skills (EESs)
New Essential Skills (NESs)
*may need to be adjusted

Future skills needed in the workplace?

Strengths and limitations of GenAI �
Test it out - how does GenAI handle your prompt?

Like other educational technologies, the use of generative AI in teaching or learning does not imply a disregard for academic integrity or the promotion of academic misconduct.

(Eaton & Anselmo, 2023).

12 of 26

Bloom’s Taxonomy Revisited by Oregon State University is licensed under CC BY 4.0

Bloom’s Taxonomy Revisited

13 of 26

Make your Assessment AI-proof*

Be clear about your expectations around GenAI use, and talk to your students about why
Make AI irrelevant or difficult to use by understanding and exploiting GenAI’s limitations
Be aware that GenAI is becoming integrated into many mainstream tools (Microsoft, Google, etc) so will be hard to separate

How can we do it?

14 of 26

Make your Assessment AI-proof*

Be clear about expectations around GenAI use, and talk to your students about why
Make AI irrelevant or difficult to use by understanding and exploiting GenAI’s limitations

How can we do it?

Make it current
Make it personal
Build in connections
Make it authentic
Make it renewable
Scaffold components of the assignment
Focus on process over product
Give choice

15 of 26

Make your Assessment AI-proof* - for you to read later

How can we do it?

Current: Ask your students to connect their writing to current events, recent data or new information (the current version of ChatGPT does not consult the internet and only contains information up to 2021).
Personal: Design very specific writing prompts that ask students to make connections to work or experiences in the course and to their personal experiences (reflective)
Connections: to course-work, community, local contexts, etc
Authentic: Authentic assessments give learners an opportunity to meaningfully apply their knowledge to realistic, real-world situations that reflect the type of task/scenario the learner will be faced with in the workplace (relevant, meaningful) Designing, Implementing and Evaluating Online Authentic Assessments, A Guide to Alternative Assessments, Beyond the Exam
Renewable (an assignment that holds value for others beyond the class. They have a larger audience, a longer life and a greater impact), Encouraging Academic Integrity Through Intentional Assessment Design - Renewable Assessments
Scaffold (break down your larger assignment into smaller steps that build upon one another) to create opportunities for feedback, revision and reflection.
Emphasize process over product (and ask students to show evidence of their learning process)
Choice (Return to your course learning outcomes to determine if you can offer any choice within your assignment. Can your students submit the assignment in any other format (e.g., audio, video, infographic, etc.) and/or can they have choice in the topic?

16 of 26

Scaffold

Cheating escalates when stress levels go up
Learn, fail, try again
Opportunity for feedback
Allow resubmission

This image on this slide is licensed under CC BY-NC-SA 4.0

Article: Contextualizing learning using scaffolding

17 of 26

GenAI and Faculty Use

How can you leverage GenAI as you design/develop your assessment?

Rubric development
Case study creation
Simulation creation
Formative test questions
Clear instructions
Create a checklist
etc

*Be sure to properly cite any GenAI-generated content you share with your learners

18 of 26

Embrace/Incorporate AI tools (ethically and responsibly)

Designing Authentic Assessment in the Age of AI

19 of 26

Embrace/Incorporate AI tools (ethically and responsibly)

Things to consider:

Align with your learning outcomes
For all or part of your assignment?
During what phase of the assignment?
Process (documenting)
Product (citing)

20 of 26

Examples

Designing Authentic Assessment in the Age of AI - video (start at 13:40) - mission statement example
Debate with GenAI (slide 23)
Visualize a concept (Slide 44)
History - Medieval Plague Simulator
Design a quiz, multiple choice - example question set for automotive maintenance (are answers correct? Are explanations correct?)
Create an output - critique it, improve on it
Ideation and brainstorming, initial research, outline creation, improving grammar and other aspects of writing, adding creative elements to assignments, etc

Assessment Strategies

Assessment Ideas for an AI-enabled world

21 of 26

How will learners acknowledge GenAI use?

Citation

“Citation refers to the practice of referencing the GenAI used when including its outputs in your finished work through quotation, paraphrase, or summary. When you use generated content from GenAI, you must reference it in both the body of your text and in your bibliography.” (University of Waterloo, AI and the Writing Process – Documenting and Citing)

Documenting

“Documenting refers to keeping track of your activities with GenAI and your corresponding actions. These actions may be related to idea generation and drafting, so documentation is not limited to what ends up in your finished document.” (University of Waterloo, AI and the Writing Process – Documenting and Citing)

Example of a documentation table
Copy and paste text
Screenshots

22 of 26

Talking to your students about GenAI

Talk to your students about GenAI and academic integrity

Don’t assume your students know everything they need to know about GenAI tools, their potential and their limitations
Support their knowledge around academic integrity and the importance of not representing the work of others as their own in their academic work

Citing/documenting
Support your students as they work on their own GenAI literacy

Talking to your Students About GenAI - slidedeck

23 of 26

GenAI literacy - students

Image by Gordon Johnson from Pixabay

Things learners need to know/understand:

What is GenAI?
How does it work?
Limitations, risks
How is it impacting your discipline/industry?
How to use it ethically and responsibly (including citation)

Things learners need to practice:

Prompting
Critical analysis of output

ChatGPT and GenAI library guide�

GenAI Learner Guide - coming soon

24 of 26

Communication & Support

Provide transparent communication on each assignment about when/how you can use GenAI and how/when to cite/document its use (GenAI Sample Assignment Level Statements)
Provide alternatives to address concerns and/or access issues (students will not be mandated to use AI)

Eg: Byte (no account required)

Provide support around effective use of the tools (GenAI literacy)
Share College support resources
Don’t depend on these tools working in a live class (at the moment)

25 of 26

Grading the Assignment

AI detection software is NOT reliable
DO NOT put student work in ChatGPT or other GenAI tool/technology
Ask for feedback from your students

26 of 26

Assessment Resources

Assessment Ideas

Upcoming sessions

Navigating College Writing in the Age of GenAI (Monday, Feb 5, 12-1pm)

Assessment (Re)Design Incubator - engagement week (Mon. Feb 26, Thurs. Feb 29), spaces are limited