1 of 16

Challenges and Opportunities of Moderating Usage of Large Language Models in Education

Lars Krupp, Steffen Steinert, Maximilian Kiefer-Emmanouilidis, Karina E. Avila, Paul Lukowicz, Jochen Kuhn, Stefan Küchemann, Jakob Karolus

2 of 16

LLMs exist, and students might use them

2

1: Krupp, Lars, et al. "Unreflected Acceptance-Investigating the Negative Consequences of ChatGPT-Assisted Problem Solving in Physics Education." arXiv preprint arXiv:2309.03087 (2023).

Students critically miss reflection when interacting with LLMs to help them answer physics questions [1]

Top French university bans use of ChatGPT to prevent plagiarism – Reuters, January 27, 2023

Despite Cheating Fears, Schools Repeal ChatGPT Bans - The New York Times, Aug. 24, 2023

OpenAI, Arizona State University partner to expand AI in academia – Reuters, January 19, 2024

Moderate LLM behavior to mitigate the loss of reflection

3 of 16

Study - Conditions

3

Baseline Conditions

Moderation Conditions

Classic Bot

Search Engine

Hint Bot

Multi-Response Bot

4 of 16

Hint Bot

4

Hint Bot

To calculate the length of the hypotenuse of a triangle without a right angle, you can use the Law of Cosines. The formula is c²=a²+b²-2ab*cos(C) […]

How do I calculate the length of the hypothenuse of a triangle without a right angle?

5 of 16

How do I calculate the length of the hypothenuse of a triangle without a right angle?

5

You cannot calculate the length of the hypotenuse without a right angle.

Multi-response Bot

To calculate the length of the hypotenuse without a right angle, you can use the Law of Cosines.

To calculate the length of the hypotenuse without a right angle, you can use the Pythagorean theorem or trigonometric functions.

Multi-Response Bot

6 of 16

Selected Results: Student-Tool Interaction Strategies

6

7 of 16

Takeaways

7

Moderation can positively influence interaction behavior

Moderation must set the right expectation

The search engine is still the best option

8 of 16

Lars Krupp�lars.krupp@dfki.de

We must find a way to make students want to use these moderated LLMs for their studies

9 of 16

Results: Correctness, Quality and Usability

9

10 of 16

LLMs exist, and students might use them

10

1: Krupp, Lars, et al. "Unreflected Acceptance-Investigating the Negative Consequences of ChatGPT-Assisted Problem Solving in Physics Education." arXiv preprint arXiv:2309.03087 (2023).

students critically miss reflection when interacting with LLMs to help them answer physics questions [1]

Moderate LLM behavior to mitigate the loss of reflection

11 of 16

Results: Scores

11

	n	mean	sd
Search Engine	12	2.5	1.38
Multi-response Bot	10	2.3	2.21
Hint Bot	10	1.1	0.57
Classic Bot	8	1.6	1.77

12 of 16

Study design

12

40 participants
Between-subject design
Physics domain
4 conditions (classic bot, search engine, hint bot, multi-response bot)

13 of 16

Conditions

13

14 of 16

How do I calculate the length of the hypothenuse of a triangle without a right angle?

14

Hint Bot

Calculate c if a = 5, b = 7 and C = 60°

To calculate the length of the hypotenuse of a triangle without a right angle, you can use the Law of Cosines. The formula is c²=a²+b²-2ab*cos(C) […]

To calculate the length of the hypotenuse (c) using the law of cosines, you can use the formula: […]

15 of 16

Hint Bot Example

15

16 of 16

Multi-Response Bot example

16