1 of 13

ON THE SOFT-SUBNETWORK FOR�FEW-SHOT CLASS INCREMENTAL LEARNING

Presented by: Lucas Wu

2 of 13

Few-Shot Class Incremental Learning

Incrementally learn a sequence of sessions

each containing only a small number of training examples for each class, while retaining previously learned knowledge.

Due to the small amount of training data for new tasks, the model tends to severely overfit to new classes and quickly forget old classes, deteriorating the model performance.

3 of 13

Challenges

Due to the small amount of training data for new tasks, the model tends to severely overfit to new classes and quickly forget old classes, deteriorating the model performance.

Catastrophic forgetting

Overfitting

4 of 13

Current work

1. OCS (Yoon et al., 2022) Focusing on class imbalance problems

- cons: do not consider the overfitting issues caused by training a sequence of few-shot tasks

2. FSLL (Mazumder et al., 2021) tackles overfitting for few-shot CIL by partially-splitting model parameters for different sessions through multiple sub steps of iterative reidentification and weight selection.

- cons: led to computationally inefficient.

5 of 13

Motivation

Regularized Lottery Ticket Hypothesis (RLTH).

A randomly-initialized dense neural network contains a regularized subnetwork that can retain the prior class knowledge while providing room to learn the new class knowledge through isolated training of the subnetwork

6 of 13

Sketch of solution

softNet uses two main sub-networks: a primary sub-network and a secondary sub-network.

The primary subnetwork is used to remember previously learned knowledge, while the secondary subnetwork is used to learn new knowledge.

This way, when learning new categories, previous knowledge is not forgotten.

7 of 13

Sketch of solution

softNet uses two main sub-networks: a primary sub-network and a secondary sub-network.

The primary subnetwork is used to remember previously learned knowledge, while the secondary subnetwork is used to learn new knowledge.

This way, when learning new categories, previous knowledge is not forgotten.

SoftNet starts with a randomly initialized dense neural network.
It then jointly learns the model weights and a soft mask during a base training session. This soft mask consists of a major subnetwork (which retains knowledge of previously learned sessions) and a minor subnetwork (which learns new sessions without overfitting).
After the base training session, the major part of the pre-trained subnetwork weights is frozen to maintain prior class knowledge. Only the minor part of the weights is updated for the novel class knowledge. [Page 2]
The SoftNet approach divides the dense neural network into two parts: a major subnetwork and a minor subnetwork. The major subnetwork retains the base task knowledge, while the minor subnetwork acquires new task knowledge. During training, SoftNet updates the model weights using a combination of these two subnetworks, effectively regularizing the weights for incremental learning

8 of 13

Overall Algorightm

Basic class training (t = 1):��First, the model weights θ and fractional weights s are randomly initialized.��For each training cycle:��Get the soft mask msoft, which consists of a major mmajor and a slave mminor, both of which are randomly selected.��For each data batch bt:��Calculate the base loss Lbase using Equation 3.��Update model weights θ.��Update the score weight s.��Incremental learning (t ≥ 2):��Combine the current training data Dt with the examples saved from the previous session.��For each training cycle:��For each data batch bt:��Calculate the loss Lm using Equation 5.��Update model weights θ.��Output: model parameters θ, s, and m.��To put it simply: the algorithm is first trained on the base class and then fine-tuned in an incremental learning phase. During the base training phase, it uses a soft mask to update the model and score weights. During the incremental learning phase, it only updates the model weights.

9 of 13

Base Training (t = 1):

Find the model weight
Score S

θ^ and m^_t**: These are the optimal weights and subnetworks we want to find.��L_t: This is a function that measures the performance of the model on the given data D_t.��θ \odot m_t: This represents the combination of model weight θ and subnetwork m_t. Here, we combine the two with a single operation (\odot, which stands for element-wise multiplication).��D_t: This is the training data for the current stage.

10 of 13

Obtain soft subnet by supplementing winning tickets

11 of 13

Result

12 of 13

Result with SOTA

13 of 13

Conclusion

Inspired by the "Regularized Lottery Ticket Hypothesis (RLTH)", which proposes the existence of smooth subnetworks in dense networks.

The author proposed Soft-SubNetworks (SoftNet) as an incremental learning strategy that aims to retain learned class knowledge and learn new knowledge.

SoftNet simultaneously learns model weights and adaptive soft masks to minimize catastrophic forgetting.