Assessment Design Framework for the Age of AI

A Framework for Enhancing Assessment

Presenting: Andrew Larner

Collaborators: Jack Sutherst, Sarah Underwood and Carmen-Elena Dorobat

Key Questions:

• How vulnerable are our assessments to AI collusion?
• What assessment parameters correlate with these vulnerabilities?
• How can we design AI-conversant assessments by turning these vulnerabilities into opportunities for AI collaboration?

ZONE OF DESIRED AI USE

AI Exclusion

The deliberate structuring of assessments that renders the use of Gen AI impractical or impossible.

For example, unseen exams or professional discussions, where the immediate application of AI tools for generating or accessing information is restricted or irrelevant.

AI Collaboration

A partnership between students and Gen AI tools to enhance the

creative and cognitive processes involved in academic work.

AI may be used to aid in idea generation, research, and refinement of projects while maintaining the students' responsibility for critical decision-making and intellectual contributions.

AI is a supplementary resource within the bounds of academic integrity.

AI Collusion

All AI

The surreptitious use of Gen AI tools to produce academic work without

transparently acknowledging the AI's involvement.

AI is employed as a substitute for genuine learning efforts.

This constitutes a form of academic dishonesty and undermines the principles of integrity, fairness, and originality in scholarly pursuits.

No AI

ZONE OF DESIRED AI USE

AI Exclusion

Operationally difficult or time intensive.

Often not reflective of real-world expectations.

AI Collaboration

Desired zone

AI Collusion

Academically undesirable

A form of academic malpractice No detection system (at MMU)

No AI

All AI

PROJECT OBJECTIVES

How can we design assessments that preclude AI-collusion, i.e. AI use that substitutes learning?

How can we design assessments that purposefully include AI-collaboration, i.e. AI use that stimulates and increases learning?

Passed the Test

Key issues:

• Passed all units with user level 1 or 2 with only 2 hours work and no prior knowledge of the subject area.
• Achieved sometimes better results than actual

students on that module.

• No misconduct evidence could be gathered, nor could marking distinguish student contribution from AI output.

• With some prompts AI can tackle (rather successfully) assessments that previously appeared AI-proof, e.g. generic reflections, and provide excellent scaffolding for oral assessments such as professional dialogues.

Stage 1 – Information Gathering

Stage 2 – Working with AI

2hrs

FINDING #1

Assessment Design encourages different AI user levels

USER LEVEL 1: text generator - no or negligible editing, little or no learning necessary

USER LEVEL 2: text editor -significant editing, including some peripheral learning necessary

​

USER LEVEL 3: concept curator - finding sources and explaining concepts, some text generation possible but not suitable without significant user curation (core learning required)

Considered AI-Collusion

Considered AI-Collaboration

CHECK YOUR ASSESSMENT’S AI USER LEVEL

Fill in the questionnaire to calculate the AI user level score for one of your modules.

​

Assessment AI Score

ASSESSMENT PARAMETERS

SIZE

(small – medium –

large)

TURNAROUND

(hours – days – months)

FORMAT

(written – artifact –

oral)

FOUNDATION

(theoretical – mixed-

practical)

CONTINUITY

(yes or no)

GROUPWORK

(yes or no)

CRITICALITY &

REFLECTIVITY

(analytical – evaluative-

reflective)

STRUCTURE

(open – scaffolded-

closed)

CONTEXT

(broad – focused-

embedded)

FINDING #2 continued:

Assessment parameters relevance to AI user level

• Format and foundation are most relevant, followed closely by student effort hours.
• Context, criticality and reflectivity continuity, and groupwork have a moderate impact.
• Duration or structure have a low impact.

AI Exclusion

*Desired Zone*

AI Collaboration

AI Collusion

Hours

Weeks

Months

Oral Assessment

Artifact or Recording

Written Assessment

Practice & Experimentation

Theory & Application

Theory & Research

Closed Structure

Scaffolded Structure

Open Structure

Groupwork

No Groupwork

With Pre-Requisites

Without Pre-Requisites

Reflective

Evaluative

Analytical

Embedded Context

Focused Context

Broad or No Context

Over 60 SEH (5,000 W)

30-40 SEH (2,500 – 3,500 W) *Student Effort Hours (SEH), Approximate Word Count Equivalent (W)*

15-20 SEH ( 2,000 W)

Turnaround

Structure

Criticality & Reflectivity

Group Work (Yes/No)

Continuity

Context

Format

Size

Foundation

FINDING #3: Multi-dimensional assessments encourage AI collaboration

• Each individual parameter is not sufficiently impactful on their own.
• Combinations of parameters (3 or more) are more potent in eliminating collusion and encouraging collaboration.
• Combining parameters means designing assessments as multifaceted

challenges, blending diverse cognitive processes and skills.

• Each component or element of assessment should require students to demonstrate proficiency across multiple dimensions of knowledge and application.
• Multi-dimensional assessments require designing tasks that exercise diverse, specific skills and touch on combined learning outcomes, and are authentic.
• Functional over Declarative Knowledge/Process over Outcome.

VARY THE PARAMETERS

WRITE A STRATEGIC PLAN

Version 1

• 2,500 words strategic plan
• A company of your choice or from a case study
• Written for your line manager
• Containing an explanation and analysis of the factors and theories underpinning the plan (detailed structure provided)

Version 2

• Write your own case study for a public company in the North West. Use data

database

from a company like Fame or

strategy plan

Mintel.

• Create a for this enterprise

• Justify the plan to your line manager get their feedback and redo the plan
• Include case study (500 words), both plans (1,500 words), the plan justification

(500 words), and a reflection

on how you

implemented the manager’s feedback

(500)

BEFORE: 1 parameter in the desired zone

AI Exclusion

AI Collaboration

Desired zone

hours

oral assessment

artifact or recording

written assessment

practice & experimentation

theory & application

theory & research

closed structure

AI Collusion

​

weeks months

​

scaffolded structure open structure

embedded context

focused context

b

road or no co

ntext

groupwork

no groupwork

with pre-requisites

no pre-requisites

reflective

evaluative

analytical

over 60 student effort h (5,000 w)

30-40 student effort h (2,500 – 3,500 w)

15-20 student effort h (under 2,000 w)

AFTER: 4 parameters in the desired zone

AI Exclusion

AI Collaboration

Desired zone

hours

oral assessment

artifact or recording

written assessment

practice & experimentation

theory & application

theory & research

closed structure

AI Collusion

​

weeks months

​

scaffolded structure open structure

embedded context

focused context

broad or no context

groupwork

no groupwork

with pre-requisites

no pre-requisites

reflective

evaluative

analytical

over 60 student effort h (5,000 w)

30-40 student effort h (2,500 – 3,500 w)

15-20 student effort h (under 2,000 w)

SUMMING UP

• Balancing academic integrity with AI proficiency will help students stand out amidst technological progress.
• Multi-dimensional assessments preclude AI collusion and foster AI collaboration.
• These require designing assessment tasks that exercise diverse, specific skills and touch on combined learning outcomes.
• Varying specific parameters is a way to systematically include AI in assessments without compromising academic integrity or assessment authenticity.

Our Open Resource

Generative AI & Assessment in AI Website