2 of 14

Overview

Continual Learning (CL): train a NN on a sequence of tasks without significant performance degradation to previous tasks.

Constraint: no access to the original datasets in previous tasks.

Solutions:

Rehearsing past experiences
Weight regularization
Parameter isolation

Non-convex Optimization Surface

3 of 14

Overview

Our contributions:

Propose Batch Model Consolidation (BMC) and a distributed learning framework to support CL for training multiple expert models on a single task stream composed of tasks from diverse domains.
Propose a stability loss as regularization to expert models and a batched distillation loss combines multiple expert models to update a single base model in a single incremental step.
We introduce Stream dataset of 71 image classification tasks and show that BMC is robust against large domain-shifts and for a large number of tasks.

Previous approaches in Continual Learning suffer significant performance degradation and unacceptable cost when faced with a large number of diverse tasks [1, 2].

4 of 14

BMC - The Intuition

Base Model : the NN expected to perform on all tasks in the CL task sequence.

Expert Model: a NN specializing on one task.

BMC: consolidate the knowledge (parameters) from Expert Models into the Base Model.

5 of 14

BMC - The Intuition

The expert model is regularized for not deviating from the base model by a large margin.

We train the base model incrementally by regularized experts to get the new base model .

Batched consolidation reduces gradient noise from distant tasks, and regularization improves the stability of base model.

6 of 14

BMC - Regularization Phase

A single incremental step of BMC

Interim. Feature Knowledge Distillation

Applied between experts and base model

Stability Loss - Regularizing experts

Used on the expert device and

between the base model

After experts training: sample consolidation artifacts as Buffers

7 of 14

BMC - Consolidation Phase

A single incremental step of BMC

Interim. Feature Knowledge Distillation

Applied between experts and base model

Batched Distillation Loss

Apply distillation between multiple experts and a single base model on data

Task Loss and Batched Distillation Loss is applied on

8 of 14

Distributed CL Training Framework

Experts are trained individually on remote devices.
Each remote device passes the Buffer data to central device once after expert training.
The central device uses the Buffer and Memory data to update the Base Model.

9 of 14

Experiments - Stream Benchmark

Stream dataset: 71 image classification datasets concatenated, with 6,770,722 training images, 743,977 validation images, and 2866 classes.

BMC achieved 70.4% final mean accuracy compared to the second best Experience Replay (ER) 41.4%, a 70% improvement.

10 of 14

Experiments - Cost Analysis

Total Cost (TC)

= Communication Cost (buffer size)

Memory Cost (memory size)

The Pareto front of our method shows a trade-off between the Total Cost of a memory and a buffer with Mean Accuracy.

11 of 14

Experiments - Cost Analysis

Cost Accuracy (blue)

The ratio between final mean accuracy and Total Cost, representing the performance gained per unit cost. BMC has the highest Cost Accuracy 6.27.

Relative Time Performance (orange)

The relative time of optimizing w.r.t. training sequentially without CL (SGD/fine-tuning). BMC has the highest time efficiency of 78% and is the only one faster than fine-tuning (100%, red dotted line).

12 of 14

Conclusions

BMC is a combined approach of expert model regularization, rehearsal by experience replay, and parameter-isolation by training then consolidating disjoint experts.
BMC allows distributed training where each expert reside on a different device and specialize in a given task.
BMC is the only method that can maintains performance for our long sequence of 71 tasks, while being more efficient than sequential fine-tuning.
A more sophisticated baseline such as DER++ [3] does not outperform Experience Replay in a more realistic dataset like Stream, calls for more research.

So in conclusion, our method BMC is a robust Continual Learning approach with state-of-the-art performance on

Large task sequences and great time efficiency.

Our method can be seen as the combination of weight regularization, which is the regularization phase;

And Parameter isolation; which in the regularization phase, BMC isolates the task parameters in separate expert models;

As well as replaying, by using a memory holding the experiences for rehearsal in the consolidation phase.

Our model consolidation framework not only can improve time efficiency by distributed training, but also

Is the only method that can maintain performance on the Stream dataset.

According to our observation, the recent baselines do not outperform a simple experience replay method

In a complicated real-world scenario like Stream.

So we call for more research into the realistic evaluation of Continual Learning,

Where the task sequence should be long and the tasks should be in various domains to

Evaluate the learning algorithm for its robustness.

13 of 14

References

[1] Raia Hadsell, Dushyant Rao, Andrei A. Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences, 24(12):1028–1040, 2020.

[2] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory G. Slabaugh, and Tinne Tuytelaars. Continual learning: A comparative study on how to defy forgetting in classification tasks. CoRR, abs/1909.08383, 2019.

[3] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and SIMONE CALDERARA. Dark experience for general continual learning: a strong, simple baseline. Advances in Neural Information Processing Systems, volume 33, pages 15920–15930. Curran Associates, Inc., 2020.

1 of 14