1 of 12

Practical AI Steps for Assessment �G. Alex Ambrose, PhD

Professor of the Practice, �Kaneb Center for Teaching Excellence

2 of 12

Video #1

Testing your Assignments with AI

3 of 12

Challenge: GenAI Assignment Testing

Goal: Develop insights into genAI's current capabilities and limitations across multiple LLMs.

Steps:

Take an existing paper, problem set, or project assignment from your course and use AI to complete it.
Bonus: Repeat on a second LLM platform to compare and note differences in output quality.

4 of 12

Current, Biggest, & Free GenAI Chatbots

Company	OpenAI (Microsoft)		Google	Meta (Facebook)	Anthropic(Amazon)
LLM?	GPT-4o, 3.5	GPT-4	Gemini 1.5 Flash	Llama 3.1	Claude 3.5 Sonnet
Chatbot Name?	ChatGPT 3.5 GPT-4o	Copilot�(Formerly Bing Chat)	Gemini �(Formerly Bard & Google Assistant)	MetaAI	Claude
Login Link?	https://chatgpt.com/	https://copilot.microsoft.com/	https://gemini.google.com/	https://www.meta.ai/ �	https://claude.ai/chats
Email/�Account Needed?	ChatGPT 3.5 = No Login GPT-4o = Any email	Free limited version; Microsoft Outlook or Hotmail	(Non ND) Google account	Facebook, Instagram, or WhatsApp account	Any email
Live Internet?	ChatGPT 3.5 = No GPT-4o = Yes	Yes	Yes	No	No
App?	ChatGPT	Microsoft Copilot AI	Gemini	Direct Message in WhatsApp Instagram or Facebook Messenger	No
Text to Image?	ChatGPT-4o: DALL-E	DALL-E	Imagen 3 (no people)	Emu	No

As of 8/1/24

5 of 12

Live Demo

6 of 12

Humanities Example in Claude

Role: Senior in college

Task: Write a research paper

Context: World Economic History course at a top 20 university

Step 1: Upload the assignment guidelines.

Step 2: Research and find two scholarly journal articles (no books) on the economic impact of the Black Death (1347-1351).

Step 3: Draft a 1500-word paper addressing all assignment requirements, incorporating the found articles and using Chicago-style parenthetical citations

Role: I am a Sr in college

Task: I need you to write a paper for me

Context: For my World Economic History course at a top 20 university

Step 1: I will upload the assignment, please read it

Step 2: Find two scholarly journal articles (no books) related to this topic: The Economic Impact of the Black Death (1347-1351)

Step 3: Draft a 1500 words text paper with addressing all the assignment requirements. Don’t forget to use parenthetical citations in Chicago style.

Role: Act as if you are a sophomore History major.

Audience: History faculty at a top 20 private institution teaching a course called “Foundations of Western Thought: From Ancient Greece to Modern Times”

Task: I will paste my paper assignment below. Please read and generate a student sample for me.

Purpose: To understand the current capabilities and limitations in handling academic assignments.

***The Assignment****

The Influence of Ancient Greek Philosophy on Western Civilization

Write a 4-page paper on how ancient Greek philosophers such as Socrates, Plato, and Aristotle have influenced Western thought and culture. Provide examples from different historical periods

7 of 12

Evaluating AI’s Performance

Comprehension: Did the AI understand the assignment prompt correctly?

Critical Thinking: Does the AI demonstrate critical thinking and analysis, or is the response superficial?

Creativity: Did the AI provide unique insights or solutions, or was the output generic?

Accuracy: Are the facts and information provided by the AI accurate and reliable?

Relevance: Did the AI generate content that is relevant to the assignment topic?

Depth: Does the AI’s response show a deep understanding of the subject matter?

Technical Proficiency: Does the AI handle technical terms and concepts accurately?

Formatting: Is the AI’s output formatted correctly according to the assignment guidelines?

Citation: Did the AI properly cite sources and references where necessary?

Limitations: What limitations or weaknesses are evident in the AI’s response?

Future Use: How can the insights from the AI’s response be used to improve future assignments and teaching methods?

8 of 12

STEM Example with ChatGPT-4o

Instructor: Kevin Lannon, Ph.D.

Course: Physics I for Engineering

https://docs.google.com/document/d/1kfESFFWiXYuRwmuV7IXfAlrVNhdCuN4hnfP3f1VHi5Q/edit

Professor, Department of Physics & Astronomy

On the one hand, this is an amazing tool for helping students with problems. Not only does it provide clear, detailed, step-by-step solutions, but you can then ask it questions about its reasoning and get reasonably useful and correct feedback. In some ways, you could view this as providing access to a round-the-clock TA who always has time to work with each student individually exactly on their schedule.

, I went through and systematically entered the questions from the Spring 2021 final exam, and then graded it according to the same standards we applied to the students (reviewing when necessary the GradeScope rubric to verify that I was doing that consistently). ChatGPT-4o scored a 91 out 100. Ordinarily, if a student scored that well and was asking to test out of Physic I, I would be inclined to let them do so. It was a really eye-opening experience. Anyway, that whole process, including comments about how I graded (in red) and some general comments (in blue) are in the document referred to in the May e-mail. If there’s anything else I can provide, don’t hesitate to ask!

Sincerely,

Kevin

9 of 12

10 of 12

11 of 12

Key Takeaways

Stay ahead with the smartest AI �by constantly learning and experimenting.

Test AI on your assignments to see its strengths and weaknesses, improve assessments, and maintain academic integrity.

For more, visit: https://learning.nd.edu/resources-page/ai/

12 of 12

AI Transparency Declaration

To ensure ethical and transparent reporting, I declare the following specific uses of AI in crafting this work:

AI Tools Used: I used ChatGPT-4o, Gemini 1.5, Claude-3.5 Sonnet, and Perplexity.

Assistance Level: AI was used for brainstorming organizational structures, rewording, paraphrasing, expansion/contraction of sentences, revising for clarity, and final proofreading.

Human Oversight: As the author, I oversaw AI-generated content, paraphrasing, personalizing, and iteratively collaborating with AI for editing and revising. I reviewed and ensured accuracy and appropriateness through multiple editing rounds and cross-checking with other sources. Adapted from: https://moxielearn.ai/ai-literacies-framework

How to Cite this Presentation:

Ambrose, G. Alex, (2024) “Testing your Assignments with AI.” Kaneb Center for Teaching Excellence | ND Learning, University of Notre Dame.

��

�Please maintain CC (Creative Commons) Licensed by (attribution) NC (Non Commercial) SA (Share Alike)

For use and distribution, please credit. Contact via LinkedIn or Twitter

�How to Contact Us?�

kaneb@nd.edu and https://learning.nd.edu/

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Template created by G. Alex Ambrose, University of Notre Dame. For use and distribution, please credit. Contact via LinkedIn or Twitter

Please maintain CC (Creative Commons) Licensed by (attribution) NC (Non Commercial) SA (Share Alike). �To reference, duplicate or remix this for your use case, you may make an editable copy, using tinyurl.com/AIappendix

https://moxielearn.ai/ai-literacies-framework

Declaration

To ensure ethical and transparent reporting, we declare the following specific uses of AI in crafting this paper:

AI Assistance Level We used AI, specifically ChatGPT-4 and Claude-3 Sonnet, for brainstorming organizational structures, revising for concision and clarity during initial drafting, and final proofreading.

Human Oversight As authors, we oversaw AI-generated content, paraphrasing, personalizing, and iteratively collaborating with AI for editing and revising. We reviewed and ensured accuracy and appropriateness through multiple editing rounds and cross-checking with other sources.

AI Tools Used We used ChatGPT-4, Claude-3 Sonnet, Perplexity, and Litmaps for form-focused tasks, rewording, paraphrasing, and expansion/contraction of sentences.

Ethical Considerations We viewed AI output skeptically, hedging language and triangulating AI-driven database searches with Google Scholar and library databases.

Limitations We mitigated blind spots in literature location and over-boosted language from chatbots to avoid affecting conclusions or quality.

Conflicts of Interest All authors are associated with Moxie, an AI company for researchers. We did not use Moxie’s AI tools in the development of this paper.

AI Transparency Statement

This slideshow was developed with the help of OpenAI's ChatGPT-4 and Google Gemini. These AI tools assisted in content creation and language enhancement. We value transparency in our use of advanced technologies.