Production LLMs aligned to be helpful, harmless, and honest

• Helpful to whom? Demands of a typical user (left) vs a student (right) differ!
• Necessary to develop critical thinking and aim is to promote learning in students

Help me with Q: What is the importance of cell?

Let’s break it into subproblems. First, do you know the fundamental components of a cell?

Of course. Think about the outer boundary and control center that support a cell's functions.

Cell membrane and nucleus

I don’t know. Give me a hint

Help me with Q: What is the importance of cell?

Cells are the fundamental units of life, serving as the building blocks of all living organisms. They play a crucial role in various functions, including metabolism, reproduction, and the maintenance of genetic information.

Let’s align LLMs to be helpful for students

Approaches to elicit desired behavior�1. Prompt engineering�2. SFT (supervised fine-tuning)� - this paper�3. RLHF (PPO, DPO, etc)� - follow up paper: Pedagogical alignment of LLMs (https://arxiv.org/abs/2402.05000)

Help me with Q: What is the importance of cell?

Let’s break it into subproblems. First, do you know the fundamental components of a cell?

Of course. Think about the outer boundary and control center that support a cell's functions.

Cell membrane and nucleus

I don’t know. Give me a hint

CLASS:�A Design Framework for building LLM based Intelligent Tutoring Systems grounded on Learning Science Principles

Shashank Sonkar, Naiming Liu,�Debshila Basu Mallick, Richard G. Baraniuk

CLASS

• Conversational Learning with Analytical Step-by-Step Strategies
• GOAL
• Build a LLM to help students with questions
• Can’t we use any out-of-the-box models like LLaMA, Mistral, GPT?

Production LLMs aligned to be helpful, harmless, and honest

• Helpful to whom? Demands of a typical user (left) vs a student (right) differ!
• Necessary to develop critical thinking and aim is to promote learning in students

Help me with Q: What is the importance of cell?

Let’s break it into subproblems. First, do you know the fundamental components of a cell?

Of course. Think about the outer boundary and control center that support a cell's functions.

Cell membrane and nucleus

I don’t know. Give me a hint

Help me with Q: What is the importance of cell?

Cells are the fundamental units of life, serving as the building blocks of all living organisms. They play a crucial role in various functions, including metabolism, reproduction, and the maintenance of genetic information.

Let’s align LLMs to be helpful for students

Let’s define “helpful” to the students�Scaffolding: Learning science to the rescue�1. Break the problem into smaller manageable subproblems�2. Walk students through these subproblems� - Goal: solve on his own� - Nudge him in right direction� - Regular hints� - Fill knowledge gaps� - Promotes critical thinking

Help me with Q: What is the importance of cell?

Let’s break it into subproblems. First, do you know the fundamental components of a cell?

Of course. Think about the outer boundary and control center that support a cell's functions.

Cell membrane and nucleus

I don’t know. Give me a hint

CLASS: Let’s align LLMs to be helpful for students

Approaches to elicit desired behavior�1. Prompt engineering�2. SFT (supervised fine-tuning)� - create datasets using GPT-4� - then finetune�3. RLHF (PPO, DPO, etc)

Help me with Q: What is the importance of cell?

Let’s break it into subproblems. First, do you know the fundamental components of a cell?

Of course. Think about the outer boundary and control center that support a cell's functions.

Cell membrane and nucleus

I don’t know. Give me a hint

CLASS Framework

Two Datasets�- Scaffolding�- Conversational

GPT generated datasets�- followed by Subject Matter Expert (SME) validated�

Subproblem-Augmented Dual Retrieval Technique

1. Hallucinations
2. Retrieval to counteract hallucinations
3. Learning science theory of scaffolding can also improve retrieval
4. CLASS implements a dual-layered retrieval process
1. Main problem, as well as
2. Subproblems
5. MMLU benchmark (biology MCQ subsets)
• Main problems: 3% increase LLaMA's accuracy
• With subproblems: 6.5% increase LLaMA's accuracy

Evaluation Metrics (Subject Matter Expert Protocol)

1. Factual Correctness
1. Are generated hints factually true or false?
2. Are generated final answers provided factually true or false?
2. Relevancy
• Are generated hints relevant/ helpful when student is stuck (provided the hints are factually correct) ?
3. Completeness
• Are all aspects of an answer completed before the next question is asked?

Thank you !�Email Shashank Sonkar (ss164@rice.edu) for any information !