Reading group: Knowledge Distillation

Model Compression by Caruana et. al

https://www.cs.cornell.edu/~caruana/compression.kdd06.pdf

Goal

• Explore techniques for compressing complex machine learning models into simpler ones without significant loss in performance
• Existing model: Ensemble of multiple different models
• Proposed final model: single NN that is much smaller and faster

​

​

​

kachow!

What is model compression?

• Compressing large models (e.g., ensembles, deep neural networks) into smaller models (e.g., shallow neural networks, decision trees)
• Benefits: Reduced memory footprint, increased speed, lower computational requirements, easier deployment on resource-constrained devices

​

Main Technique - Distillation

​

​

• Train a smaller model (student) to mimic the behavior of a larger model (teacher) or ensemble
• Use the teacher's output (soft targets) to train the student
• Caruana et. al: Synthetically generate unlabeled data (MUNGE), label the data using the ensemble, train a NN on the generated data

Experimental Setup

​

​

• Tested on 8 datasets: Train 4000 points, Val 1000 points.
• Teacher models: ensembles trained using Ensemble Selection
• Student models: Neural Networks
• Data generation: Random, Naive Bayes, MUNGE

Image credits: gwern

Results

​

​

• Student models achieved performance close to their respective teacher models
• Compression successful across various datasets and model types

Takeaways

​

​

• “Compression works by labeling a large unlabeled data set with the target model, and then training a neural net using the newly labeled data.”
• MUNGE can be used when no unlabeled data is available

Image credits: gwern

Knowledge Distillation by Hinton et al.

• Builds upon the ideas in Model Compression
• Focuses on transferring knowledge from deep neural networks to shallow ones
• Introduces temperature scaling to soften probability distributions, providing more detailed information to student models
• Demonstrated successful compression of deep neural networks for image classification tasks

Image credits: gwern

Knowledge Distillation by Hinton et al.

• Set T > 1, produce a softer probability distribution
• Lot of helpful information in soft targets than in a hard target
• Use same T when training, set T=1 during eval

Image credits: gwern

Feedback

• Reach out to us if you’d like to present at the reading group!
• We’d love to hear your thoughts! 💙

One more thing…

​

​

💬 Please subscribe to the @ml-efficiency channel so that you get notifications

❕Allows everyone to get fewer @everyone notifications!

Thank you!

@viraat, @bhavnicksm