1 of 8

Collaborative Learning for Deep Neural Networks

UEF Reading Club, Spring 2021

​

Xuechen Liu

2 of 8

Previous Learning methods

  • Auxiliary training
  • Multi-task learning
  • Knowledge Distillation/Teacher-Student learning
  • Other methods for improvement: General label smoothing, Two-way distillation, born-again DNN, etc...

3 of 8

Major Contributions

  • Improve system accuracy without extra inference cost and with minimal tuning effort
  • Training with intermediate-level representation (ILR) leads to better training efficiency and generalization
  • Improve robustness to label noise “by nature”
  • It looks like it’s not a “single” method, but a collection of “single” methods on multiple modules of the training pipeline

4 of 8

Generation of training graph

  • Instead of multiple instances for parallel training/model averaging, all sub-heads share same ILRs

​

  • Very similar to multi-tasking learning, except that there is no ‘multi-task’

​

  • Multiple patterns can be designed, including a binary-tree like one

5 of 8

Sharing objectives

  • Making shared ‘soft’ values across different ILR heads

​

  • Combines multiple views on the same data

​

  • The objective function is between averaged prediction of each head and data

​

  • “Contains additional information than ground truth label”?

6 of 8

Optimization via group Back-Prop

  • Alternative SGD experiences slower training speed and lower convergence efficiency

​

  • Simultaneous SGD thus is brought back

​

  • Back-prop scaling is proposed to take over the stability issue of gradient

7 of 8

Experiments and Results

8 of 8

My Personal Thoughts

  • Instead of having multiple instances and do model averaging, multi-head ILR can be another option?
    • Especially based on some constraints/practices on training DNN ASV models….
  • Alternative SGD may also be useful based on this work - the work cited is not that persuasive and for more complicated tasks, we cannot avoid it
  • Using multi-head ILR approach can be useful for handling label noise
    • and this can be very useful for speech processing tasks, I think
  • Hierarchical architecture can be useful for faster convergence
    • Is there any further work to prove this? I believe there are….