Kapotaksha Das, Amber Wood, Lance Sloan, Kenny Moore, and Zhen Qian
Insights into Peer Reviews:
A Text Analysis Pipeline Built on Google Cloud
Introductions
Presentation Outline
Overview
Connecting the Data
Deep Learning & RoBERTa
Review to Classification Pipeline
Visualization and Usage
Wrapping up
What is MWrite?
Helping students critique better
Student drafts an essay
Reviewed by Peers
Peers leave comments based on criteria
Feedback is incorporated to improve work
Peer Review Comments
A treasure trove of valuable insights
Over 500 comments are made in a single assignment
Identifying trends in feedback against criteria
Help understand the intent within feedback
Helping students give better feedback
But this cannot be done manually
Classifying Comments
A system to break comments down effectively
The author has identified the issue correctly but has not presented their argument in a clear manner.
Taking a written peer review:
Problem Identified
Verification
Praise
Writing or Formatting Issues
Incorrect Content
Missing Content
Tier 1: Type of Comment
Tier 2: Type of Problem
To actionable classification:
The Deep Learning Architecture
Classification Results
Vertex AI Notebook Manager
PyTorch
Hugging Face
Peer Review Comments
Ro
a
Model for Tier 1
Ro a
Model for Tier 2
RoBERTa is built atop
BigQuery Database
Using RoBERTa for Classification
Model for Tier 1 Tri-label, single-class classifier
Identifies the type of comment
Model for Tier 2 Tri-label, multi-class classifier
Identifies the type of problem
Verification
Praise
Problem Identified
Peer Review Comments
Writing or Formatting Issues
Incorrect Content
Writing or
Formatting Issues
Missing Content
Only comments identified as ʻProblem Identifiedʼ
Unlocking More Potential
Using Deep Learning to gain insight
Pushed to Google Cloud Platform
Used by Tableau
MWrite Database
Prediction on Google Cloud Platform
Visualization in Tableau
From DB To GCP
Canvas MWrite Database
GCP BigQuery Database
Openshift container executes the transfer job
Jenkins schedules and triggers the job daily
Controlling Data
Ensuring Privacy and Security
Only Comment ID is sent to GCP
User provides authentication and credentials
Authenticated ID used to retrieve and show pertinent information to User
Only data user should see is allowed by Tableau
Vertex AI Notebook Manager
BigQuery Database
Preventing Data Leaks
Instructors & Fellows
Students
2 distinct views to separate data
Authenticate User on Login
Only serves data where student is allowed
Only serves data where instructor is allowed
Answering through Visualizing
Making classification accessible
Instructors & Fellows
Students
Macro View
Micro View
How is the class doing?
How am I doing?
Are the criteria encouraging better feedback?
Am I able to understand the criteria?
Instructor View
Instructor View
Student View
Key Challenges
Limiting access, maintaining privacy
Keeping databases connected and up to date
Building views specific to courses and needs
Working with specific limitations of Vertex AI for model deployment
Solutions Designed
Develop a daily schedule to retrieve course data requiring classification dynamically
Fine-tune PeerBERT to perform two tiers of classifications on peer reviews
Use Google Cloud to handle large data and computational demands
Work with instructors and fellows to build dashboards as per their needs
Thank you for joining!
We will now take any questions.
Instructor View
Student View