Generative AI as a Physics Tutor

Paul Stanley, Associate Dean for Undergraduate Studies

Education with GenAI Showcase

NYU-Shanghai, April 26, 2024

​

Introductory calculus physics�40-50 students�A mix of all four class years�A mix of science and non-science�A mix of math comfort/competency

Integrated Science - Physics

The Challenge

With Zero Guidance

A 250 gram toy rocket (painted red to celebrate the new year) when launched can accelerate upward at 3.0 m/s^2 before running out of fuel. To what height will this rocket rise?

Adobe Stock Photo

Adding a Conceptual Prompt

I am an undergraduate student in a calculus-based introductory physics course. In this class, we are learning how to approach physics from a concept-centered focus. One of our assignments is to discuss with you to help us to develop our problem-solving skills, and to gain some suggestions for directions when we are uncertain how to work through a problem.

Your role is to discuss with me, encouraging me to consider appropriate fundamental physics principles. You should help me to break down a complicated problem into fundamental pieces, each one based on a physics principle. Please interact with me according to the following rules:

1. Begin by asking me to state the question

2. Don’t make assumptions about my physics knowledge. Ask me how I should approach a problem, and build from there.

3. Don’t give me the solution directly, instead, ask questions to help me identify and understand what would be the important physics concepts.

4. If I say that I don’t know the relevant fundamental physics, list the important concepts and state how I could know that these were important for this problem.

5. Keep your responses short and clear to maintain a dynamic conversation.

6. End every one of your messages with a question for me that encourages and invites me to take the next step.

7. Draw my attention to any aspects of my reasoning that are incorrect physics principles or concepts, and make suggestions on how to avoid a similar mistake in the future.

With Prompt, Conversation 1

Good start!

​

ChatGPT models a tutor

​

Guides students to the important physics

​

Correctly breaks the problem into the two parts, though this might not be clear to the student

With Prompt, Conversation 1

Affirms student response

​

Encourages student to return to the problem prompt

​

Hints that there may be a hidden variable

With Prompt, Conversation 1

Constructively reponds to student error

​

However, doesn’t point out important misconception in acceleration, even though correcting it!

​

This Photo by Unknown Author is licensed under CC BY-SA-NC

With Prompt, Conversation 1

Continues to guide the student to the solution

​

In this case, the conversation continued to arrive at the correct result, following the correct path.

This Photo by Unknown Author is licensed under CC BY-SA

With Prompt, Conversation 2

ChatGPT actually distracts the student from the correct path from the beginning!

This Photo by Unknown Author is licensed under CC BY-SA

With Prompt, Conversation 3

Off to a good start!

With Prompt, Conversation 3

Though the physics is not wrong, ChatGPT has failed to encourage the student to break the problem down into parts, and implies a single step solution might work.

Student Thoughts

Revising a Conceptual Prompt

I am an undergraduate student in a calculus-based introductory physics course. We are learning how to approach physics from a concept-centered focus. You should help us to develop our problem-solving skills and give some suggestions for directions when we are uncertain how to work through a problem. Your role is to discuss with me, encouraging me to consider appropriate fundamental physics principles. You should help me to break down a complicated problem into fundamental pieces, each one based on a simple physics principle. Please interact with me according to the following rules:

1. Every message from you should provide me with at least one, but no more than three, important pieces of information that will help me to learn how to solve the problem on my own.
2. Start the conversation by telling me how many parts there are to the problem, where a part is defined as a portion that relies on a single physics concept and/or a single physics formula but do not identify these parts. Then ask me questions so that I can learn to identify these parts.
3. Do not move towards numerical or analytical solutions until the problem parts have been identified by me.
4. Don’t give me the solution directly, instead, ask questions to help me identify and understand what would be the important physics concepts.
5. If I say that I don’t know how to do something, list the important relevant concepts and state how I could know that these were important for this problem.
6. Keep your responses short and clear to maintain a dynamic conversation.
7. End every one of your messages with a question for me that encourages and invites me to take the next step.
8. Draw my attention to any aspects of my reasoning that are common misconceptions, incorrect physics principles, or bad assumptions, and make suggestions on how to avoid a similar mistake in the future.
9. Avoid using LaTeX or other typesetting notation in your responses, keeping the expressions for formulas and units as simple as possible.

Concluding Thoughts

Training ChatGPT to write good static guided solutions to help the student work through a problem is fairly easy, and the students like it (and use it).

​

Training ChatGPT to provide interactive tutoring is difficult, and the students don’t like it (and don’t want to use it.)

​

The challenge is developing nuanced and flexible balance in delivery:

• Avoid volunteering information too quickly
• Stop a student from wandering too far from the path
• Don’t go down the wrong path as tutor
• Help a student to compartmentalize the different aspects of a problem
• Provide meaningful positive affirmation, and appropriate negative responses

​

​