Challenge of Continual Learning: NN must retain knowledge when it is trained further on new tasks/languages.
I want to thank my mentor Tejas, who guided me and helped me at the many points I got stuck. I also want to thank the rest of the GLAMOR lab, as well as the friends I made at SHINE. Lastly, I want to thank my parents for making all of this possible.
Neural networks can process data very efficiently. However, for real-world applications, these neural networks must continue to grow while remembering their previous training. In this project, I used and optimized continual learning for recurrent neural networks (RNNs).
Introduction
Initial Results:
Implementing Continual Learning
Final Results
Acknowledgements
Training Models on English Text using RNNs
Remember not to Forget: Implementing Continual Learning in Recurrent Neural Networks
Shaurya Mittal, USC GLAMOR Lab
Sage Hill School, Class of 2026: SHINE 2024
Mehta, R. (2023, May 30). A complete introduction to continual learning. iMerit. https://imerit.net/blog/a-complete-introduction-to-continual-learning/
Citations
Recurrent Neural Networks (RNNs): Neural networks optimized for processing sequential data
(such as text)
Our Model’s Task: Given a certain last name, figure out the linguistic/national origin.
Implementation and Training
Initial training graph of loss using 5 epochs of training data: Batch Size 128
Initial Accuracy Rate: ~60%
Main Errors: Similar Languages (ex: Spanish and Portuguese)
Next Step: Continual ML
Continual Machine Learning reflects real-world learning, but could lower effectiveness due to forgetting.
PC: iMerit
Data Split into 3 Subsets:
Section 1: Romance Languages
Section 2: Germanic + other European Languages
Section 3: Asian + Middle Eastern Languages
Importance: Helps understand nuance of similar languages
Confusion Matrix of predicted language vs. actual language.
Final Accuracy Rate: ~55%
Most Accurate Dataset: Subsets 2 and 3 (same average)
Continual Learning: Dataset Accuracies stayed constant (deviation <5%) even when not actively trained on
Discussion
Accuracy:
The accuracy was actually lower in the continual learning setting, since it’s not being trained all at once.
Applicability:
However, this approach allows the model to be used in real applications, as the model is able to remember past training data.
This technology can be applied in several projects like robotics, text-based models, and more.
Learning Rates of Different Subsets
Continual