HARNESSING GENERATIVE AI IN HIGHER EDUCATION: USES AND CONCERNS

JAMES PITARRESI

VICE PROVOST AND DISTINGUISHED TEACHING PROFESSOR

EXECUTIVE DIRECTOR, CENTER FOR LEARNING AND TEACHING

BINGHAMTON UNIVERSITY

JMP@BINGHMATON.EDU

CLT WORKSHOP NOVEMBER 22, 2024

SOMETHING CRAZY HAPPENED

November 2022 ChatGPT 3.5

The Homework Apocalypse

(Ethan Mollick July 2023)

THEN SOMETHING REALLY CRAZY HAPPENED

March 2023 ChatGPT 4

The Llama 3 Herd of Models, Meta, July 2024

Generation 2 LLMs

The Llama 3 Herd of Models, Meta, July 2024

WHERE DO LLMS GET THEIR DATA?

Wikipedia

Reddit

“We are living in the experiment.”

-Dave Copps

“The key to surviving this new educational landscape is to embrace AI’s potential while being mindful of its limitations and ethical challenges.”

- Ethan Mollick

LLM CHATBOTS

• Natural Language input (“prompt”)
• Next word (token) predictor (3 words ≈ 4 tokens)
• Context is very important
• LLMs do not have a database of every single word written
• LLMs are very good at recognizing and predicting patterns in words
• The next token predictor is surprisingly accurate
• Responses seem plausible and realistic but could be a hallucination

​

LLM CHATBOTS

Remember, it is a language model – talk to it!

​

CHALLENGES WITH CHATBOTS IN HIGHER ED

Impact Research and the Walton Family Foundation Survey. (2024, May). AI Chatbots in Schools: Findings from a Poll of K-12 Teachers, Students, Parents, and College Undergraduates

CHALLENGES WITH CHATBOTS IN HIGHER ED

• Generative AI Chatbots provide information
• Reading the information is not necessarily learning
• Reading + processing/analyzing/synthesizing information => learning
• Illusion of learning; consuming information “feels” like learning

​

Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, O., & Mariman, R. (2024). Generative AI Can Harm Learning. Available at SSRN 4895486.

• No ChatGPT
• ChatGPT
• ChatGPT Tutor

“Learning Loss”

CHALLENGES WITH CHATBOTS IN HIGHER ED

• LLM Chatbot writing detectors are biased against non-native English speakers and they are easily fooled; they're not reliable.
• Bias in training data
• Privacy concerns
• Environmental concerns
• Banning LLM Chatbot use for your class is OK, but how to enforce?
• Laissez-faire approach “use it but you’re responsible for errors” may lead to learning loss
• Thoughtfully incorporate LLM Chatbots in teaching and learning

HOW CAN WE USE LLM CHATBOTS?

WHERE DO WE SPEND TIME AS INSTRUCTORS?

• Teaching tasks (syllabus, content, assessments)
• Course tasks (grading, office hours, help sessions)
• Administrative tasks (advising, service, committees)
• Research (mentoring, proposals, publications, reports)

​

LET US SUPPOSE…

You are assigned to teach (or revamp) a course.

• Develop the course scope, a customized list of topics.
• Identify key learning objectives for the course.
• Transform topics to a schedule for a 15-week semester.
• Tie sections of the book to the topics for each class meeting.
• Review/revise course description.

About how long would this take?

SIMPLE PROMPT CRAFTING: �TALK TO IT!

https://alchemy.works/

DEMO

​

As a university researcher, please give me a short summary of this article and include a bulleted list of key findings.

As an expert in teaching mechanical engineering to undergraduate students, can you list the typical topics covered in a Strength of Materials course?

But wait, there’s more! When starting a project, I often insert an additional important prompt:

​

“Before we start, what clarifying questions do you have for me?”

“Acting as a prompt engineer, review the prompt and optimize it; ask me any questions about it.”

BUILDING A COURSE WITH ChatGPT 4o

Wonderful! I plan to use Beer, et al., Mechanics of Materials, latest edition. I'd like a general list of topics that are covered in a 15-week, 3-credit course. What do you suggest for the main topics?

Exclude Shear Force and Bending Moments as well as Energy Methods. My class meets three times per week, every Monday, Wednesday, and Friday, for one hour at each meeting. Can you draft a 15-week schedule for me? And put it into a table.

Thank you. Now, I have a tricky task for you. Can you match the topics we put in our table to chapters and sections of the textbook I mentioned?

With the correct table of contents in these photos, can you match the topics we put in our table to chapters and sections of the textbook? Put it in a MS Word format I can download.

BTW, I can also pull in ABET criteria.

FREQUENT FORMATIVE ASSESSMENT

I want to prepare some practice problems for my students. As an expert teacher in Mechanics of Materials, can you give me 10 problems based on Ch. 2: Stress and Strain - Axial Loading in Beer, et al, Mechanics of Materials 8th edition. Before we start, any questions for me?

1) A mix of basic to medium difficulty would be best.2) Basic calculation problems, some real-world problems if possible. No derivations.3) Students can use the book 4) A mix of both 5) A mix from those sections is fine.

Hello, as an expert in mechanical engineering, I would like you to solve the following problem.

​

A turbine engine produces 7,000 kW at 5,500 RPM. A solid circular shaft of diameter D1 connects the output to a 10:1 reduction gear set. From there, a second solid shaft of diameter D2 connects the electrical generator. Note 10:1 signifies that shaft D2 is rotating at 10 times less than shaft D1 (i.e., its angular velocity (ω) is less).

​

If the allowable shear stress in the shaft material is 100MPa, what are the diameters of the two solid circular shafts?

Frequent Formative Assessment

This should be 85.2 mm

Your approach is correct. However, you did not calculate the cubed root for both D₁ and D₂ correctly. Please review your work, take your time, and check your math!

ChatGPT4o1 got it correct on the first try.

• This is a great opportunity to talk with students about their reliance on the output from generative AI.
• Consuming information is not learning.
• It is also a good exercise for students to check for errors!
• Note: There were a few errors in the practice problems, too!

​

In week 10 of my course, I plan to cover pressure vessels. Are there any noteworthy failures of pressure vessels that I could use in my course as a quick active learning exercise?

Can you expand on Case Study 4? I would like you to gather information on the Apollo 13 explosion, use the Internet if needed.

​

I would like you to gather background information as well as technical information, specifically regarding the failed pressure vessel's design.

​

I'd like you to outline a slide deck of four to six slides based on what you find.

AI Generated Slide Deck via Gamma

(gamma.app)

As an expert in higher education pedagogy, please review the course table you made and develop a course description including reference to ABET 2024.

By-the-way, I’ve spent less than 15 minutes on this task.

I HAVE A QUESTION…

WHO IS THIS “EXPERT” IN HIGHER EDUCATION PEDAGOGY?

Let’s explore a more intentional prompting approach.

What if I could leverage the knowledge an expert, like Ken Bain of the Best Teacher Institute, and get feedback on my course description?

Using the internet, can you identify up to 8 sources of information on him?

Acting through the “lens” of Ken Bain’s work, can you review the course description and learning objectives above and give me feedback?

How might Ken Bain revise the course description and learning objectives?

I CAN NOW BUILD TEAMS OF EXPERTS

CHEATING

“Students will cheat with AI.

​

But they also will begin to integrate AI into everything they do, raising new questions for educators. Students will want to understand why they are doing assignments that seem obsolete thanks to AI. They will want to use AI as a learning companion, a co-author, or a teammate.

​

They will want to accomplish more than they did before, and also want answers about what AI means for their future learning paths.”

​

-Ethan Mollick The Homework Apocalypse, July 2023

Cheating has been around for a long time, accelerated by the Internet, smartphones, organized “cheating” providers, and now generative AI.

The Unpleasantness of Thinking: A Meta-Analytic Review of the

Association Between Mental Effort and Negative Affect

David, et al. https://doi.org/10.1037/bul0000443

People are good at finding ways of avoiding something they don’t want to do, like mental effort.

ENCOURAGE THINKING, NOT REPLACE IT WITH AI

• Generative AI detectors for essays don’t work (unreliable witness).
• Blindly using LLM’s for “answer bots” leads to high homework grades and low exam scores; learning loss.
• Use LLM’s as tutors – guiding students, not giving them answers.
• Khanmigo and CognifyTutors; Home-built tutors
• LLM’s are best used as a co-intelligence tool for humans to do better thinking
• We need to rethink assessments.

Other Fun Tools: Google’s Notebook LM

ENT 280A: Innovation Scholars Program Study Guide - Fall 2023

Short Answer Quiz

Instructions: Answer the following questions in 2-3 sentences each.

1. What are the key differences between invention, entrepreneurship, and innovation?
2. What are the four criteria a successful innovation must meet? Briefly explain each.
3. What are three benefits of conducting user interviews for innovation purposes?
4. Describe the "Jobs-to-be-Done" (JTBD) theory and provide an example.
5. What is the purpose of reframing a problem in the context of innovation?
6. Explain the “Double Diamond Approach” and how it applies to problem-solving.
7. Why is it important to “fall in love with the problem, not the solution” in the innovation process?
8. What is the primary focus of low-resolution prototyping and experiments?
9. Why is it important to incorporate “transactions” into prototype testing? Provide examples of transactions.
10. What are three advantages of using storyboards for prototyping?

Short Answer Quiz Answer Key

1. Invention focuses on creating something entirely new. Entrepreneurship involves forming a new business and assuming the risks and rewards. Innovation identifies and solves a problem, adding value for the user.
2. A successful innovation must be valuable (offering real benefits), usable (easy to use and understand), viable (financially sustainable), and feasible (technically possible).
3. User interviews allow for deep understanding of user behaviors and motivations, uncover hidden needs and pain points, and provide insights for developing targeted solutions.
4. JTBD theory focuses on understanding the progress a person desires to make in a specific circumstance. The “Job” is that desired progress. For example, a student’s “Job” might be to effectively learn course material and achieve a good grade.
5. Reframing challenges initial assumptions and allows exploration of alternative perspectives, potentially leading to more effective and innovative solutions.
6. The Double Diamond Approach is a design process model with two diamond-shaped phases: problem definition and solution development. Each diamond represents a divergent phase (expanding possibilities) followed by a convergent phase (narrowing down to the best options).
7. Focusing on the problem ensures efforts are directed towards addressing a real need rather than getting attached to a specific solution that may not be optimal.
8. Low-resolution prototypes aim to quickly and cheaply test the desirability of an idea. They prioritize gathering user feedback and iterating based on that feedback.
9. Transactions represent concrete user actions that indicate commitment or interest, offering valuable insights into actual user behavior. Examples include clicking a button, signing up, making a purchase, or downloading.
10. Storyboards help visualize user experience, identify potential pain points, and facilitate communication and feedback among team members and potential users.

Essay Questions

1. Discuss the importance of user research in the innovation process. How can techniques like interviewing and applying the Jobs-to-be-Done framework contribute to developing successful innovations?
2. Explain the role of brainstorming and idea generation in fostering innovation. What strategies can teams employ to maximize their creative output during brainstorming sessions?
3. Describe the concept of low-resolution prototyping. Why is it beneficial to prioritize desirability over feasibility in early prototyping stages? Discuss various methods of low-resolution prototyping and their respective advantages.
4. Analyze the statement, "Innovation is not about finding the right answer, but about asking the right questions." How does this statement relate to the process of reframing problems and developing user-centered solutions?
5. Discuss the importance of iteration and feedback in the innovation process. How can teams effectively gather and incorporate user feedback to refine and improve their prototypes and experiments?

Glossary of Key Terms

• Creative Confidence: The belief in one's ability to generate innovative ideas and solutions.
• Human-Centered Design: A design process that prioritizes understanding the needs and experiences of the intended users.
• Jobs-to-be-Done (JTBD): A theory that focuses on understanding the progress a person desires to make in a specific circumstance.
• Low-Resolution Prototype: A basic, inexpensive representation of an idea used to test its desirability and gather feedback.
• Minimum Viable Product (MVP): A version of a product with just enough features to attract early-adopter customers and validate a product idea early in the product development cycle.
• Reframing: Challenging initial assumptions and exploring alternative perspectives to gain a new understanding of a problem.
• Storyboard: A visual representation of a user's experience with a product or service, typically using a series of illustrations or images.
• Transaction: A concrete user action that indicates commitment or interest in a product or service (e.g., clicking, signing up, purchasing).
• User Feedback: Information and insights gathered from users about their experiences and opinions, used to improve a product or service.
• Value Proposition: A statement that clearly articulates the benefits a product or service offers to customers.
• Double Diamond Approach: A design process model with divergent and convergent phases for both problem definition and solution development.

Google’s Illuminate

https://illuminate.google.com/

arXiv is a free distribution service and an open-access archive for nearly 2.4 million scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics.

ADMINISTRATIVE TASK: �STUDENT EVALUATION OF TEACHING

Goal: develop questions that avoid bias due to gender, ethnicity, and physical attractiveness in evaluations of instructors.

Based on the search results, there are several key researchers and experts who have contributed to developing questions and methods to avoid bias due to gender, ethnicity, and physical attractiveness in evaluations of instructors:

​

Philip B. Stark - Professor of Statistics at UC Berkeley who has done extensive research on issues with student evaluations of teaching and developing more valid methods.

​

Anne Boring - Assistant Professor at Erasmus University Rotterdam who has studied gender bias in student evaluations.

​

Justin Esarey - Associate Professor of Politics at Wake Forest University, who has studied statistical issues with student evaluations.

​

Bob Uttl - Professor of Psychology at Mount Royal University, who has conducted meta-analyses on the validity of student evaluations.

​

Now, I would like you to evaluate the questions I provided you, but this time, I want your feedback through the "lens" of the four experts mentioned above. Take your time and give me high-quality feedback and suggested changes to the questions.

USING VIDEO AND PODCAST TRANSCRIPTS

• Load transcript into ChatGPT
• Request summary
• List highlights, key points
• Ask for quiz questions
• Slide prep for class

AI ASSISTED GRADING

• The promise:
• Quick turn-around and feedback on assignments
• Opens time for high-value interactions with students
• Encourage the use of frequent formative assessments
• The nightmare:
• May lack nuances and judgement
• Bias and fairness
• Superficial or formulaic feedback

https://mitsloanedtech.mit.edu/2024/05/09/ai-assisted-grading-a-magic-wand-or-a-pandoras-box/

AI TUTORING

www.cognifytutors.com

www.khanmigo.ai

ETHICAL AND OTHER CONSIDERATIONS

• Equity and Access: the Digital Divide
• Access to AI tools should not exacerbate educational disparities
• Challenges and Advocacy
• Open-source alternatives
• Advocacy for Equitable Access
• Research on Impact and Needs
• Challenges facing society due to generative AI

Key Strategic Priorities

BU INITIATIVES

• Generative AI Advisory Committee to help with initial investments at the CLT
• Conversations with OpenAI
• Institute on AI and Society

​