1 of 27

MISSOURI ONLINE

AI Plagiarism & Cheating

Welcome, my name is Guy Wilson. I am an instructional technologist with Missouri Online.
We are going to be looking at the issue of AI plagiarism & cheating, and discuss some technical and non-technical ways to approach it.
We will be focusing first on text, but it is vital to note that AI can generate any kind of content, and those capabilities are rapidly developing and being rolled out to the public.
We'll start by looking over recent changes to the Student Code of Conduct regarding use of AI.
We will ask just what AI plagiarism & cheating are.
Then we will a look a bit at how generative AIs produce new text based on the probabilities of words occurring in a given context before looking at how this is used by AI detectors.
Next, we'll talk a little bit AI detectors and their limitations, both at present and in the future. We'll look at some specific ways they can be beaten.
We will follow that up with some ways of talking to students about this, both in general and in the case of students who have come under suspicion, before looking quickly at all the many different areas in which cheating may take place.
I may be jumping back and forth between these slides and the web as we go along.
Please put your questions in Chat. If we run short on time, we will follow up by email later.
Information in this presentation is current as of November 6, 2023. Given the rate of change in AI and AI detection, some of this information may not be current when viewed.
I'll drop some links to our resources and my contact information into chat.

Drop these links in chat:

Missouri Online AI Documentation

https://teachingtools.umsystem.edu/support/solutions/articles/11000122321-artificial-intelligence-ai-

Calendar of Missouri Online Events

https://teaching.missouri.edu/events

Recent Blog Post: Teaching in the time of AI

https://teaching.missouri.edu/blog/teaching-time-ai

Academic Integrity Sprint Signup

https://umsystem-teaching.catalog.instructure.com/courses/umsys-noncredit-academicintegrity

Contact me:

Guy Wilson

wilsong@umsystem.edu

Get help and assistance:

-Email: teachingtools@umsystem.edu

-Phone: 855-675-0755

-Chat: https://teachingtools.umsystem.edu/ (Live Chat button at bottom right of page)

2 of 27

The term unauthorized use of artificially generated content, includes, but is not limited to (i) use of artificial intelligence tools or other tools that generate artificial content in taking quizzes, tests, examinations, or other assessments without permission from the instructor; (ii) submitting work for evaluation as ones own that was produced in material or substantial part through use of artificial intelligence tools or other tools that generate artificial content without permission from the instructor; (iii) using artificial intelligence tools or other tools that generate artificial content in a manner contrary to instructions from the instructor; or (iv) using artificial intelligence tools or other tools that generate artificial content in a manner that violates any other provision of these rules concerning academic dishonesty. Use of commonly available tools such as spelling or grammar checking software or features of software that propose anticipated words or phrases while text is being written will not be considered unauthorized use of artificially generated content unless such use is contrary to instructions from the instructor.

200.010 STANDARD OF CONDUCT

§C.1.C (EFFECTIVE 8/1/23)

On June 29, the Board of Curators approved an addition to the Student Standard of Conduct regarding AI.

I have had some conversations with Academic Integrity officers on some of the campuses but do not know if a definitive interpretation has been reached yet. Here is how I understand them at this point - this is just my interpretation.

(i) and (ii) prohibit use of AI on assessments and assignments without permission from the instructor.
(iii) gives the instructor additional discretion to prohibit other uses in the class.
(iv) coordinates with other university policies
These are followed by an unnumbered provision that specifically allows use of grammar checking, autocorrect and autocomplete, and similar features of software such as Grammarly or even Word, unless the instructor specifically forbids it.

There is a slight change of wording to another part of the policy that emphasizes that instructors do have authority over what students can use and that these policies cover both cheating and plagiarism.

This will have implications for your syllabi and class policies.

3 of 27

Syllabus/Instructions

Specify exactly what students are allowed to do with AI.
If you think a use of AI is not covered by the policy, and wish to prohibit, be explicit.
If you do not want students to use tools like Grammarly, or some parts of them, be specific.

Also:

Remember that AI can be used for almost any part of any assignment.
Explain any technologies or techniques you will use to deter the use of AI.
Discuss with students.

Image generated with Bing AI

It is clear that instructors need to be explicit in their syllabi and/or test and assignment instructions about what students are allowed to do with AI, because the new policy prohibits AI use without permission.

You may also wish to clarify specific things that are prohibited if you think they are not sufficiently clear. For instance, you might want to prohibit students using an AI for brainstorming ideas for a paper or presentation.

Be aware that tools like Grammarly, Word, and Google Docs are going well beyond current ideas about grammar checking, spell checking, and autocomplete. They are incorporating more and more generative AI capabilities that will make them able to complete or generate student work in the same way that ChatGPT, Bard, or Claude can now.

Whatever you specify, it is imperative that you communicate and discuss the policies with your students. This allows you to make sure that they understand, but also to give them a chance to express their anxieties and concerns about the technologies they are using, and also over any technologies you plan to use in enforcement before problems occur.

4 of 27

Situation - General

Constant, rapid, chaotic change
Synergies

AI + Search (Bing)
AI + Office Apps (MS Office/G Suite)
AI + OS (Windows 11 Copilot)
AI + plugins (ChatGPT Plugins)
AI + mobile
AI + everything

Rapid rise of multi-modal AI
Embodied AI

Image generated with Bing AI

5 of 27

GenAI Capabilities

Create:

Text
Images
Video
Audio
Animation
3D-Models
Code
Deepfake A/V

Services

Data Analysis
Translation
Search
Summarize
Research
Interrogate Images/Video

Autonomous Action
Universal Interface

Image generated with Bing AI

6 of 27

Scope

Not just plagiarism, potential to affect everything
Not just an issue of writing, it cuts across media
Need to think beyond the present
Need to think about all phases of assignments/assessments

Image generated with Bing AI

We are not just talking about plagiarism, and we may well end up having to redefine plagiarism in the next few years. GenAI can potentially affect every aspect of your coursework, from the way you create content to the way students complete their homework, their tests, and their projects. It cuts across all media, including ones we rarely consider, such as 3-D printed objects. Because of the speed of change, we need to think not just about where we are now, or at the end of the semester, or at the end of the academic year. We cannot future proof our classes, but if we can become aware of what is on the horizon and stay abreast of developments, we may be able to keep up.

We need to think about all aspects of assignments and assessments. We will have to consider what we can realistically prohibit and police, but also what skills students may be able to acquire by being allowed to use AI. It is a fine balancing act of course, as we work through issues of creativity, accuracy, integrity with new workplace and life skills.

7 of 27

The State of Play

40-50% use GenAI for homework
~20% admit using it in unauthorized ways
50% who use it for papers do extensive rewriting
Tii reports 10.3% >20% AI, 4.2% >80%
No foolproof way of determining AI plagiarism
Existing reporting tools do not provide solid evidence

Image generated with Bing AI

Surveys of students are showing a generally similar pattern.

The 40-50% using it for homework are not all cheating. There are many instructors who are allowing some use of GenAI. About 20% of those who use it are cheating or plagiarizing. That works about to 8-10% of students surveyed.

Students who are using it for papers may not be using it for entire papers. They may be using it for some parts that are giving them trouble. Other may use it for some early drafts, then do extensive rewriting. Note that when done maliciously, it will often blur the results from detectors, fooling them into thinking that the paper was written entirely or almost entirely by a human.

There are no foolproof ways of determining whether a paper was written by an AI or not. There are indications, detector scores are one, but nothing certain. Regular plagiarism detection tools are not going to catch AI cheating. Even using the same prompt (instructions) with the same AI, it will produce differently worded versions each time with little or no overlap.

8 of 27

What is AI Plagiarism?

Research?
Brainstorming?
Outlining?
Abstracts?

Notes & Bibliography?
Drafts?
Editing & Proofreading?
Polishing?

Creating content with an AI without attribution

But what about having an AI help with:

Image generated with Bing AI

So what is AI plagiarism?

At its simplest, we are talking about creating & using content without attribution. That differs a little from conventional plagiarism, which is simply using someone else's content without any creativity at all. The complexity of the prompt (i.e., the questions, and instructions given to an AI) will often determine the quality of the output. We'll talk about prompts shortly, but notice that prompt-writing is now a skill that students need to develop for their future employment.

How much use of AI is a problem? I've listed eight general areas here. If I, as a student, use AI for the items on the left, is that cheating? Is that plagiarism? What about the ones on the right? Would you consider all of those plagiarism if left unacknowledged in my submission?

Of course, generative AI is better at some of these than others, but that is changing. Right now bibliographies and citations are not going to pass muster if generated by an AI, but GPT-4-based products are good at outlining & summarizing articles, so they might be useful in creating the annotations in an annotated bibliography.

9 of 27

GLOSSARY

Token = a word, syllable, word fragment, letter, or punctuation mark.
Perplexity = entropy/lack of randomness in the text
Burstiness = variation of perplexity across longer blocks of text

Image generated with Bing AI

These are three important terms to understand in working with text AIs.

The token is the smallest unit the AI or the detector works with. As we'll see, in many cases the text is processed syllable by syllable. GPT-3 remembers the last 4000 tokens. GPT-4 remembers the last 8000. The tokens it remembers provide the context for creating the next piece(s) of text.

Perplexity is a measure of the probability that the token will be selected by the model. AIs tend to select the highest probability tokens.

Burstiness is a measure of variability across larger blocks of text. I've seen at least three different explanations of this. The block of text might range from a sentence to a document. Most detectors seem to use sentences as the unit of analysis. While lower scores are more likely to be from an AI, a large difference in the burstiness of sentences could indicate that some of the text was written or polished by a human.

10 of 27

Human content above.

ChatGPT-3 to the right.

Screenshots from GLTR: http://gltr.io/dist/

11 of 27

GPT picks tokens (words or fragments) that make sense in context.

The color shows how likely that word was. (Green most, Red least likely.)

The numbers show the position in the list. This varies by context.

Detectors work in reverse.

They look at the probability of the words being picked by the model.

The more high probability words, the higher the likelihood it is written by an AI.

Colossus was an electronic digital computer...

GPT picks tokens (words or fragments) that make sense in context.

The color shows how likely that word was. (Green most, Red least likely.)

The numbers show the position in the list.

This varies by context. (GPT-3 uses the last 4000 tokens for context, GPT-4 uses the last 8000.)

Colossus was part of the prompt for this paper, which was written by ChatGPT-3 in February.

Detectors work in reverse.

They look at the probability of the words being picked by the model.

The more high probability words, the higher the likelihood it is written by an AI.

They also look at variations between sentences.

If the vast majority of the sentences have a large proportion of high-probability words, then it is flagged as AI or likely-AI written.

If the vast majority of the sentences have a low proportion of high-probability words, then it is flagged as Human or likely Human-written.

If there is a great deal of difference between sentences, then it is usually marked as uncertain or partly written by an AI.

This is based on the MIT-Harvard GLTR project demo.

12 of 27

Violet boxes indicate purely human content - scores should be 0%

Orange boxes indicate mixed human/AI content - scores should be be ~40-90%

All other is AI content - scores should be 100%

Overlapping bullets may hide others

🟪

🟧

Comparison of Four AI Detectors

This chart compares four AI detectors evaluated 22 different writing samples. As you can see, there is considerable variation. AI-generated content was created with Bard, Claude, and various configurations of ChatGPT.
There are a number of free detectors available of varying quality.
The university has not approved any of them for use, so if you do use them, it is at your own risk (issues include FERPA, security, accessibility, and student copyright).
Many of these have premium versions available.
Some may require you to create an account and login.
Never, ever use ChatGPT or another GenAI chatbot to tell you whether a paper was written by a human or AI. It will often tell you that everything is its own work. In an infamous case at Texas A&M, a professor failed an entire class of seniors because ChatGPT stated that all of the papers he gave it were AI generated.

13 of 27

A Bit About Turnitin

Why aren't we rolling out the AI Indicator yet?

Unable to test it before rolling out.
Always on
Students unable to see results
Uncertain about reliability
Timing
Licensing issues

Image generated with Bing AI

14 of 27

Limitations

Detectors lag behind generative AI
Skilled prompt writing
False positives
Edited or rephrased content

Image generated with Bing AI

By definition, an AI detector lags behind the AIs it is intended to catch. AIs continuously evolve, but, from time to time, they undergo major upgrades. These may result in geometric, rather than linear, progress in capabilities. This may happen more than once a year at this point. They may catch up from time to time. When reading papers that examine the efficiency of AI detection, it is important to note when the study was done, with what tools, and against what AIs. It is likewise important to pay close attention to what they are studying and if the way they the experiments were constructed reflect the way the tools would be used in a real class.
A student who is skilled at writing prompts, which is something we should increasingly expect, as this is already becoming a job skill, may be able to get the AI to vary the word choice and syntax sufficiently to fool the detector. The prompt may even ask the AI to misspell a specific number of words at random.
Some people write in a very stilted way that may look like an AI to the detector.
A paper released in arXiv in April 2023 shows that non-native speakers of English are at a significant disadvantage with AI detectors. Other studies have broadly confirmed this. They demonstrated that all of the detectors had a higher rate of false positives on those papers than those from native speakers. By having ChatGPT polish the non-native papers, the percentage detected as AI dropped dramatically. This has recently been challenged by GPT Zero and Turnitin, who argue that improvements they have made to their software have solved the bias problem.
The detector can only work on what is submitted. If a student does a minimal amount of editing or runs it through a rewriting tool to create more variety of usage, it may not register as having been written by an AI. (Presently the AI rephrases, Quillbot, even its free version, is consistently beating a lot of detectors.)
Demo Quillbot (https://quillbot.com/) if time permits with an AI detector to show what can be accomplished with paraphrasing. Also show what a prompt is like.

15 of 27

Other Issues

Privacy
Security
Student Copyright
Accessibility
Equity

Image generated with Bing AI

Using unapproved 3rd-party tools can pose both privacy (certainly FERPA, and HIPPA in some cases) and security risks for students and faculty. If student writing has not been de-identified before being sent to an unapproved detector that could have consequences. Many of these tools will require you to have an account, free or otherwise, so the data could at least be tracked back to you and your classes.
None of these tools have been evaluated for accessibility at this time. Not all of their features may be accessible to all faculty at this time. Likewise, if you plan to share the report with students, accessibility issues may also apply.
Student copyright is a real thing. By sharing a student's work with an unauthorized vendor's products, and if the student wrote the work, issues of student copyright might emerge. I'm no attorney, but it is a possibility.
There are at least two equity issues to consider with detectors:

Some student writing styles may trigger false positives. This may be because of specific ways students were taught to write and might involve neurodiversity, first language, or other aspects of a student's language environment. Also, if they are quoting or badly paraphrasing other sources, this may be an issue. In two notorious cases, a popular detector stated that parts of the US Constitution and the Book of Genesis were written by an AI. (We can only hope the latter is not an act of AI hubris.)
The other is that the detectors do not make it easy to share the reports with students (or with others). With some of them, you may need to take screen captures. Many feel that students should be made aware that their content is being run through a detector beforehand and that students should have access to the same data as the instructor.

16 of 27

Good Prompts

Good prompts should:

Simulate a persona ("Act as _____")
Specify the task clearly, including the goal
Include specific directions (for instance formatting output, requiring certain steps or certain items to be included)
List any constraints or provide context for the AI
May need to be iterative to fine-tune it.

Image generated with Bing AI

17 of 27

Act as a senior in college

Write an approximately 1000-word essay on early computers of the 1940s

Include at least these four computers: The Atanasoff-Berry Computer (ABC)

Colossus I & Il, the Zuse Z3, and ENIAC.

Discuss the purpose and main features of each of those four.

Format as an essay. Do not use bulleted lists.

Avoid first person and mentions of yourself.

Be more fluent. Do not write like an AI.

Vary your vocabulary more than usual and use more synonyms.

18 of 27

Some scholars have been thinking about this for a while. Sarah Eaton at the University of Calgary recently suggested that AI will (or should) change both what we think about writing and the way we think about plagiarism. She breaks that down into six tenets:

Hybrid Human-Al Writing Will Become Normal
Human Creativity is Enhanced
Language Barriers Disappear
Humans can Relinquish Control, but not Responsibility
Attribution Remains Important
Historical Definitions of Plagiarism No Longer Apply

These are matters of exploring AI tools, learning to use them creatively and responsibly, allowing them to take hold and deal with issues we might have because of different languages (or even barriers to expression we may have due to background, neural divergence, etc.), and transcending our current ideas of authorship and plagiarism. At the same time, scholars like Maha Bali at the American University in Cairo, while substantially agreeing with Eaton's position, note ethical and practical issues around some of these, such as this allowing the continued dominance of English, or serious issues with the quality of machine translation and consequent loss or distortion of meaning.

19 of 27

20 of 27

Students

Before the assignment:

Set clear policies/common vocabulary
Explain why assignments can/cannot use AI
If using a detector, explain why
Discuss student concerns

Image generated with Bing AI

Be explicit in your syllabus about what is and is not allowed. Define any terms (e.g., plagiarism, rephrasing) you think might be open to different interpretations between you or your students.
Be clear about your reasons for including or excluding AI in all or part of the assignment. They may not have thought through what using the AI is taking away from them in the long run.
Detectors may be seen by students as unreliable (they are certainly of limited reliability) and a sign of distrust. Make sure students understand under what conditions you will use them, what you are looking for, and how you will use the data.
Talk over their concerns. These could be about a variety of matters. Perhaps give them an opportunity to open up about what they have done or seen others do with AI.

They may have valid privacy or security concerns, particularly given a recent data breach by OpenAI that exposed some names, email addresses, and partial credit card information. Perhaps ask them to do extra reflection in a non-written format if the detectors are a major issue.
Some may raise issues of equity: easier access to AI for economically better-off students, racial issues, accessibility issues, etc.
They may have other ethical issues they want to discuss. Encourage them to do so.
For anyone with serious concerns, give them a chance to do an alternative assignment or to opt out of parts that require them to use an AI.

21 of 27

Students

Evaluating the submissions:

Detectors give probabilities not evidence
What other clues do you have?
Consider your course of action

Image generated with Bing AI

This may be the most important takeaway from this hour: "Detectors give probabilities not evidence." Having spent more than a decade supporting anti-plagiarism software (SafeAssign and then Turnitin), I cannot stress what a big change this is. Before, we were given evidence - this piece of text is the same as this one found here, and here, and here. Occasionally we might be presented with ways the writer has monkeyed with the text to throw off the detectors by inserting invisible characters or swapping Cyrillic characters for Latin characters they resemble. Instructors had evidence, they had to judge the extent of malicious intent on the part of the student, but they had proof that the letter of the policy or the law had been violated.

Everything about generative AI is probabilistic and so is everything about its detection. Text it creates does not get flagged by the similarity detectors because it is not copied from something already out there on the web or in the detector's database - unless, of course, someone copied someone else's AI-generated text. It likely will not match other text from the same AI, even created with the same prompt. Because of its probabilistic approach to writing, with each iteration, the AI should spit out text sufficiently different from previous versions so that it won't trigger significant matches of different students using the same AI and prompt.
We now have to rely on pattern differences that are common to AI and human writing, but which may or may not occur in either. The best we can get is a suggestion, often expressed as a percentage, based on the way the detector scores the probabilities that a given word or syllable will follow another, and the variation in those scores from sentence to sentence. That is it. You can view this as evidence of a sort, but of a very different sort.
Think about it legalistically. Could you get a conviction with it or an acquittal? To my way of thinking, without a confession, or without other supporting, circumstantial evidence, a great attorney might get a conviction, but more likely the accused would be acquitted. In some common law traditions, you might manage a scotch verdict.

You may have other clues.

I've mentioned the present problems with bibliographies. Spot-check a couple of articles to see if they exist or contain a mix of incorrect and correct information. If there are DOI links, see if they are valid. We don't know how long this vulnerability will last, but it is a powerful indicator at this time.
Look for grammatical and spelling errors. While it is possible to instruct an AI to randomly make x-number of mistakes of a certain type, they usually do not make mistakes. When they do respond to prompts telling them to, for instance, misspell three words at random, the errors may be very odd, not something a native speaker of English with a high-school education would do. My guess is that AIs currently do not take phonics into account.
Are there strange factual errors?

These may be about specific facts, anachronisms, misattributions of actions to a particular person or group, etc.
They may reflect a lack of understanding about the way things work in the mundane world in terms of physical laws or time.
If there are mathematical or statistical elements, examine these closely for odd errors. Currently, GPT-4 is bad at math, but this may not last long, as it is gaining access to Wolfram Alpha.

If you find something consider your course of action.

What can you prove?
What are you inferring?
What are your departmental or campus guidelines?
How serious is the infraction?
What might mitigate it?

22 of 27

Students

After evaluating:

Be open and respectful
Explain academic dishonesty
Go over the reasons for suspicion
Discuss student concerns

Image generated with Bing AI

Do not be an inquisitor. Let the student know you are going to treat them fairly within the guidelines the university and your department have established.
Make sure they understand academic dishonesty, why it is important, and how it applies in the circumstances.

Some students may feel that AI has changed what is and is not acceptable.
Others may see it as a way to level the playing field. In one of the first cases where a student admitted it, he pointed out that there was no policy excluding its use (the university only specified that students could not copy work from other people), and claimed he understood the material but was a poor writer.

Explain why you are suspicious, what evidence you have, etc.
Discuss their concerns and next steps.

If you feel there is a genuine chance they may have misunderstood the rules, perhaps give them a second chance.
If they claim that this is just the way they write, after all, some people do write in a very stilted and predictable manner, ask them to write something out while you are with them for comparison.

23 of 27

Other Media

Threats

Computer Code
Art Work
Music
Video clips

"Fixes"

Version/Change Tracking
Document/Reflect on Process
Work in Controlled Environment
Realtime, Live Demonstration of Skills

Image generated with Bing AI

So far, we've only looked at plagiarism of text. Other areas are vulnerable to plagiarism from AI. LLMs are getting increasingly good at producing code in a number of programming languages. Art work from text-to-image programs is improving rapidly.. Music can now be generated, so can video, though the quality lags about a year behind images, and clips are short.

There are already AI-image detectors, but they face the same sorts of problems as the other AI-plagiarism tools. For other types of content, and, indeed for images, requiring a record o different versions, reflection on the process, working in a controlled (surveilled) environment, or face-to-face demonstrations of skills are about all we have to prevent plagiarism in any of these areas.

This is going to be true of most of the other kinds of cheating we will encounter. This is why redesigning assignments is so important. We face the choice between increased surveillance and changes in attitudes and assignments.

24 of 27

Tests

Threats

AI Browser Plugins
AI for Answers
AI Plagiarism
AI for Math
AI for Translation

"Fixes"

In-Seat Tests
Viva Voce testing
Lockdown Browser
Online Proctoring
Different Kinds of Assessments or Questions

Image generated with Bing AI

25 of 27

Homework

Threats

Fact Checking
Research
General Assistance
Solving Math Problems
Analyzing Data
Visualizations
Translation

"Fixes"

Version/Change Tracking
Document/Reflect on Process
Work in Controlled Environment
Realtime, Live Demonstration of Skills

Image generated with Bing AI

26 of 27

Presentations

Threats

Create Slide Decks

Text
Images
Video & Audio
Graphs & Charts
Entire Decks

Synthetic Voice
Synthetic Video

"Fixes"

Version/Change Tracking
Document/Reflect on Process
Work in Controlled Environment
Live, F2F Presentations

Image generated with Bing AI

When we come to presentations, we are moving into more speculative territory. The the three major headings I have under threats, only only the creation of slide decks is really feasible now, except perhaps for very advanced users of GenAI tools. The ability to create all of different elements for a slide deck and combine them into an entire presentation is well within the capabilities of most users after a little experimentation. The exception might still be audio and video clips, but that will change over the next few months.

I will admit that creating an online presentation with deep fake audio and video avatars is not something you need to worry much about yet, but the tools are there, getting better, and will be able to generate longer clips from scripts (whether created by a human or AI) soon. I wouldn't worry about them yet but be aware that these are possibilities that may be feasible within a year or so.

27 of 27

THE DILEMMA

Prohibition/Permission
Surveillance/Exploration
Plagiarism/Hybrid Creativity
Cheating/Changing Skillsets

Image generated with Bing AI

Coming out of all the changes COVID wrought, we may have hoped for a period of less change and less challenge in higher education. Technology and the economy are not allowing that. I dislike binaries, but one the hand we are left with trying to prohibit use of AI in assignments and increasing the use of surveillance and detection to try to enforce it. On the other other we begin to question the very nature of creativity, of skills, even of what it means to be human. Finding the balance between them is now the task. Different answers are required for different disciplines and different points in a student's career. Some tests will still be required. They will require more vigilance. There will be times when the writing should come entirely from the student. There will be times when hybrid creativity should be encouraged. For most of the instances of AI plagiarism and cheating, technical fixes will prove of only limited utility. Changes to assignments and tests, changes of attitude, changes of orientation are inevitable.