Scott Crossley

Personalization in Intelligent Textbooks Using Large Language Models

Intelligent Textbooks for Enhanced Lifelong Learning (iTELL)

• iTELL is a framework
• Not domain specific
• Input is a markdown file

​

Markdown for Chapter 1, Section of Think Python textbook

The iTELL framework

Our Tech Stack

​

• Rebuilt and redesigned using Next.js
• Containerized API backend for feedback systems
• 3 main components can be hosting using any cloud provider:
• Next.js webapp
• SQL database
• Summary feedback API (Python)

The iTELL framework

• Personalization
• AI integration

The iTELL framework

• Personalization
• Keystroke logging

The iTELL framework

• Personalization
• Reading focus

Personalization: AI integration

• Summaries
• Help students consolidate knowledge by reconstructing information from the text (Nelson & King 2022)
• Assess reading comprehension (Graham & Harris, 2015)

Personalization: AI integration

• Summary scoring models
• Built on Longformer pretrained model from AllenAI
• Predict human evaluations in two dimensions of competence
• Trained on ~5,000 summaries from ~100 source texts

Main Idea

Details

Cohesion

How well did the summary capture the main idea of the source?

How accurately did the summary capture the details from the source?

How well did the summary transition from one idea to the next?

Voice

Paraphrase

Language

Was the summary written using objective language?

Is the summary appropriately paraphrased?

How well did the summary use lexis and syntax?

Content Model

Wording Model

Personalization: AI integration

• Summary scoring model performance

Personalization: AI integration

• Keyphrases are extracted automatically from the markdown file provided by the content creator.
• iTELL does not require that authors define keyphrases
• Provide specific feedback to users about missing ideas in summary writing
• Bloomberg’s KeyBART provided the highest quality key phrases.
• Key phrase overlap between the summary and the source correlated with Content scores.

Correlation between Score and Key Phrase Overlap

Personalization: AI integration

Summary feedback

• Feedback provided on
• Wording
• Fine-tuned large-language model
• Objective language, paraphrasing, borrowing
• Content
• Fine-tuned large-language model
• Main point, details, cohesion
• Topic borrowing
• Containment score
• Topic similarity
• doc2vec

Personalization: Click stream data

• Keystroke logging
• Provides insights into the cognitive processes at work during the summary writing task

​

• Allows detection of cheating through LLMs

​

Personalization: Reading focus

• Tracking reading
• Time on screen is recorded for each subsection of the text
• Using data to personalize user experience
• If a learner fails to write a passing summary, iTELL will suggest that they “reread” skipped subsections.
• Short question/answer generation
• Pinpointed natural questions based on reading patterns
• Using generative AI
• assess comprehension
• Increase interactivity
• Question types
• recall questions
• summary questions
• inference questions

​

​

​

Personalization: Automatic Question Generation

• iTell uses chatGPT to generate short-answer questions and answers for each subsection during preprocessing

• Answers are scored automatically by comparing student answers to automatically generated correct answers using one of two strategies:
• Using similarity measures derived from Rouge, Bleu, and Bleurt
• Training an MPnet classifier

• Development
• Short question/answers generation for videos
• Guide on the side
• Integrate AskJill
• Integrate concepts maps from SMART
• Enhanced visualization with assistance from AI-ALOE vis team
• Teacher dashboards
• Useability and efficacy
• Field testing in Economics classes at GaTech in fall 2023
• User documentation

​

​

​

LEAR Lab

Hongyu Dai

Qiushi Yan

Kai Wen

Shu Yang

Yu Ning

Lydia Liu

Scott Crossley

Wesley Morris

Langdon Holmes