�Finding Sparse Subnetworks�for Self-Supervised ASR and Speech Synthesis
Cheng-I Jeff Lai 1.2
with Yang Zhang 2*, Alexander H. Liu 1*, Shiyu Chang 2*, Erica Cooper 3*
Yi-Lun Liao 1, Yung-Sung Chuang 1, Kaizhi Qian 2, Sameer Khurana 1,
Junichi Yamagishi 3, David Cox 2, James Glass 1
1MIT CSAIL 2MIT-IBM Watson AI Lab 3National Institute of Informatics
PARP
https://people.csail.mit.edu/clai24/parp
TTS-Pruning
https://people.csail.mit.edu/clai24/prune-tts
Self-Supervised Learning Framework in Speech
2
Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (Baevski et al. NeurIPS 2020)
Unsupervised Cross-lingual Representation Learning for Speech Recognition (Conneau et al. Interspeech 2021)
HuBERT: How much can a bad teacher benefit ASR pre-training? (Hsu et al. ICASSP 2021)
Self-Supervised Learning Framework in Speech
3
Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (Baevski et al. NeurIPS 2020)
Unsupervised Cross-lingual Representation Learning for Speech Recognition (Conneau et al. Interspeech 2021)
HuBERT: How much can a bad teacher benefit ASR pre-training? (Hsu et al. ICASSP 2021)
Let’s Wind the Clock back 2 years
4
Year 2019: Speech SSL only as a proof of concept
5
Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-Sequence Autoencoder (Chung et al. Interspeech 2016)
Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddings (Liu et al. Interspeech 2018)
Completely Unsupervised Phoneme Recognition by a Generative Adversarial Network Harmonized with Iteratively Refined Hidden Markov Models (Chen et al. Interspeech 2019)
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech (Chung et al. Interspeech 2018)
An Unsupervised Autoregressive Model for Speech Representation Learning (Chung et al. Interspeech 2019)
Year 2020: Break-Through
6
Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (Baevski et al. NeurIPS 2020)
Unsupervised Cross-lingual Representation Learning for Speech Recognition (Conneau et al. Interspeech 2021)
HuBERT: How much can a bad teacher benefit ASR pre-training? (Hsu et al. ICASSP 2021)
Year 2021: the Takeaway from SUPERB
7
SUPERB: Speech processing Universal PERformance Benchmark (Yang et al. Interspeech 2021)
Year 2021: the Takeaway from SUPERB
8
SUPERB: Speech processing Universal PERformance Benchmark (Yang et al. Interspeech 2021)
Scaling up representation module is necessary
regardless of the SSL objectives
Self-Supervised Learning Framework in Speech
9
Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (Baevski et al. NeurIPS 2020)
Unsupervised Cross-lingual Representation Learning for Speech Recognition (Conneau et al. Interspeech 2021)
HuBERT: How much can a bad teacher benefit ASR pre-training? (Hsu et al. ICASSP 2021)
Subnetwork Discovery for Speech SSL
10
Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (Baevski et al. NeurIPS 2020)
Unsupervised Cross-lingual Representation Learning for Speech Recognition (Conneau et al. Interspeech 2021)
HuBERT: How much can a bad teacher benefit ASR pre-training? (Hsu et al. ICASSP 2021)
Subnetwork Discovery for Speech SSL
11
Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (Baevski et al. NeurIPS 2020)
Unsupervised Cross-lingual Representation Learning for Speech Recognition (Conneau et al. Interspeech 2021)
HuBERT: How much can a bad teacher benefit ASR pre-training? (Hsu et al. ICASSP 2021)
Low-resource scenarios
e.g. 10min/1h/10h paired data
Overarching Goal & Broader Impact
Develop an orthogonal approach to existing speech SSL studies that
12
Overarching Goal & Broader Impact
Develop an orthogonal approach to existing speech SSL studies that
Broader impact in
13
Two Types of Subnetwork Discovery
14
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Frankle et al. ICLR 2019)
The Lottery Ticket Hypothesis for Pre-trained BERT Network (Chen et al. NeurIPS 2021)
Two Types of Subnetwork Discovery
15
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Frankle et al. ICLR 2019)
The Lottery Ticket Hypothesis for Pre-trained BERT Network (Chen et al. NeurIPS 2021)
Two Types of Subnetwork Discovery
16
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Frankle et al. ICLR 2019)
The Lottery Ticket Hypothesis for Pre-trained BERT Network (Chen et al. NeurIPS 2021)
Two Types of Subnetwork Discovery
17
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Frankle et al. ICLR 2019)
The Lottery Ticket Hypothesis for Pre-trained BERT Network (Chen et al. NeurIPS 2021)
Two Types of Subnetwork Discovery
18
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Frankle et al. ICLR 2019)
The Lottery Ticket Hypothesis for Pre-trained BERT Network (Chen et al. NeurIPS 2021)
Two Types of Subnetwork Discovery
19
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Frankle et al. ICLR 2019)
The Lottery Ticket Hypothesis for Pre-trained BERT Network (Chen et al. NeurIPS 2021)
One-Shot Magnitude Pruning (OMP)
Two Types of Subnetwork Discovery
20
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Frankle et al. ICLR 2019)
The Lottery Ticket Hypothesis for Pre-trained BERT Network (Chen et al. NeurIPS 2021)
multiple iterations:
Iterative Magnitude Pruning (IMP)
Two Types of Subnetwork Discovery
21
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Frankle et al. ICLR 2019)
The Lottery Ticket Hypothesis for Pre-trained BERT Network (Chen et al. NeurIPS 2021)
multiple iterations:
Iterative Magnitude Pruning (IMP)
Issue!!
each downstream finetuning requires 8/24 GPUs 🡪 computationally infeasible for many
Two Types of Subnetwork Discovery
22
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Frankle et al. ICLR 2019)
The Lottery Ticket Hypothesis for Pre-trained BERT Network (Chen et al. NeurIPS 2021)
Two Types of Subnetwork Discovery
23
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Frankle et al. ICLR 2019)
The Lottery Ticket Hypothesis for Pre-trained BERT Network (Chen et al. NeurIPS 2021)
Pruning mask obtained without target task finetuning
Two Types of Subnetwork Discovery
24
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Frankle et al. ICLR 2019)
The Lottery Ticket Hypothesis for Pre-trained BERT Network (Chen et al. NeurIPS 2021)
Two Types of Subnetwork Discovery
25
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Frankle et al. ICLR 2019)
The Lottery Ticket Hypothesis for Pre-trained BERT Network (Chen et al. NeurIPS 2021)
Two Types of Subnetwork Discovery
26
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Frankle et al. ICLR 2019)
The Lottery Ticket Hypothesis for Pre-trained BERT Network (Chen et al. NeurIPS 2021)
Magnitude Pruning at Initialization (MPI)
or Cross-lingual Mask Transfer
Two Types of Subnetwork Discovery
27
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Frankle et al. ICLR 2019)
The Lottery Ticket Hypothesis for Pre-trained BERT Network (Chen et al. NeurIPS 2021)
Computationally cheap
Two Types of Subnetwork Discovery
28
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Frankle et al. ICLR 2019)
The Lottery Ticket Hypothesis for Pre-trained BERT Network (Chen et al. NeurIPS 2021)
Issue!!
Worse performance than (Type I)
task-aware subnetwork discovery
Can we discover subnetworks at pretrained initialization with similar sparsity as (Type I) task-aware subnetwork discovery,
Can we discover subnetworks at pretrained initialization with similar sparsity as (Type I) task-aware subnetwork discovery,
while inducing the same computational cost as (Type II) task-agnostic subnetwork discovery?
Prune-Adjust-Re-Prune (PARP)
31
same as Task-Agnostic Subnetwork Discovery:
Pruning mask obtained without target task finetuning
Prune-Adjust-Re-Prune (PARP)
32
same as Task-Agnostic Subnetwork Discovery
Prune-Adjust-Re-Prune (PARP)
33
Dynamic Network Surgery for Efficient DNNs (Guo et al. NIPS 2016)
Prune-Adjust-Re-Prune (PARP)
34
Dynamic Network Surgery for Efficient DNNs (Guo et al. NIPS 2016)
Prune-Adjust-Re-Prune (PARP)
35
Dynamic Network Surgery for Efficient DNNs (Guo et al. NIPS 2016)
Pruned-out weights made updatable was already proposed in the past
Prune-Adjust-Re-Prune (PARP)
36
Prune-Adjust-Re-Prune (PARP)
37
final subnetwork is different from
the initial subnetwork
Prune-Adjust-Re-Prune (PARP)
38
Computationally cheap
Same/better sparsity as
Task-Aware Subnetwork Discovery
Why though?
39
A Surprising Observation
40
A Surprising Observation
41
Unsupervised Pretraining Transfers Well Across Languages (Riviere et al. ICASSP 2020)
Unsupervised Cross-lingual Representation Learning for Speech Recognition (Conneau et al. Interspeech 2021)
A Surprising Observation
42
Unsupervised Pretraining Transfers Well Across Languages (Riviere et al. ICASSP 2020)
Unsupervised Cross-lingual Representation Learning for Speech Recognition (Conneau et al. Interspeech 2021)
A Surprising Observation
43
Non-trivial, high IOUs
across spoken languages at any model scale / amount of supervision (10min, 1h, 10h) / sparsity level (10-90%)
Unsupervised Pretraining Transfers Well Across Languages (Riviere et al. ICASSP 2020)
Unsupervised Cross-lingual Representation Learning for Speech Recognition (Conneau et al. Interspeech 2021)
A Surprising Observation
44
Non-trivial, high IOUs
w/ SSL pretrained initialization
Unsupervised Pretraining Transfers Well Across Languages (Riviere et al. ICASSP 2020)
Unsupervised Cross-lingual Representation Learning for Speech Recognition (Conneau et al. Interspeech 2021)
Any (Type II) task-agnostic subnetwork is a sufficiently good initial subnetwork for PARP Step 1 due to the high IOUs.
Any (Type II) task-agnostic subnetwork is a sufficiently good initial subnetwork for PARP Step 1 due to the high IOUs.
Furthermore, PARP Step 2 merely makes minimal adjustment
to the initial subnetwork.
Low-Resource English ASR
47
Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (Baevski et al. NeurIPS 2020)
PARP (black line) attains lower
WER than IMP/OMP/MPI
Low-Resource English ASR
48
Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (Baevski et al. NeurIPS 2020)
w/o LM decoding,
PARP offers 10% lower WER than full wav2vec 2.0 base/large, whereas IMP/OMP/MPI don’t
Low-Resource English ASR
49
Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (Baevski et al. NeurIPS 2020)
Computational Cost in # of SSL Finetuning
50
Low-Resource Multi-Lingual ASR
51
Unsupervised Pretraining Transfers Well Across Languages (Riviere et al. ICASSP 2020)
Unsupervised Cross-lingual Representation Learning for Speech Recognition (Conneau et al. Interspeech 2021)
52
53
54
55
Cross-Lingual Mask Transferability
56
Cross-Lingual Mask Transferability
57
Mask Transfer with Regular Finetuning
58
Mask Transfer with Regular Finetuning
59
mismatches are PER increase over the
same pair transfer
Mask Transfer with Regular Finetuning
60
Are similar spoken language more transferrable?
Spanish, French, Italian, Kyrgyz, Dutch are less transferrable to Russian
Swedish, Turkish, Tatar, Mandarin are more transferrable to Russian
Mask Transfer with PARP Step 2
61
Mask Transfer with PARP Step 2
62
PER degradation is over the same pair transfer is gone!
Mask Transfer with PARP Step 2
63
Random Pruning does not improve – PARP Step 2 indeed makes minimal adjustments
(see also Section 4.2 for more verifications)
Observation on Pre-Trained BERT/XLNet
64
Non-trivial, high IOUs
across GLUE tasks
Observation on Pre-Trained BERT/XLNet
65
Non-trivial, high IOUs
across GLUE tasks
Mask Transfer with Regular Finetuning/PARP
66
Mask Transfer with Regular Finetuning/PARP
67
performance degradation over the same pair transfer is much smaller
Mask Transfer with Regular Finetuning/PARP
68
LM-Decoding Sparse wav2vec 2.0
69
https://github.com/facebook/Ax
“Middle Contextualized Representations are more Valuable for Downstream ASR”
70
HuBERT: How much can a bad teacher benefit ASR pre-training? (Hsu et al. ICASSP 2021)
SUPERB: Speech processing Universal PERformance Benchmark (Yang et al. Interspeech 2021)
Unsupervised Speech Recognition (Baevski et al. NeurIPS 2021)
“Middle Contextualized Representations are more Valuable for Downstream ASR”
71
HuBERT: How much can a bad teacher benefit ASR pre-training? (Hsu et al. ICASSP 2021)
SUPERB: Speech processing Universal PERformance Benchmark (Yang et al. Interspeech 2021)
Unsupervised Speech Recognition (Baevski et al. NeurIPS 2021)
Pruned Weights Localization Across Layers
72
Consistent across 10 spoken languages and sparsities, See Appendix 18.
Pruned Weights Localization Across Layers
73
Pruning as an alternative means for probing!
as less important weights are discarded.
Low-Complexity Probing via Finding Subnetworks (Cao et al. NAACL 2021)
Cross-Task Mask Transfer on SUPERB
74
SUPERB: Speech processing Universal PERformance Benchmark (Yang et al. Interspeech 2021)
Cross-Task Mask Transfer on SUPERB
75
SUPERB: Speech processing Universal PERformance Benchmark (Yang et al. Interspeech 2021)
Cross-Task Mask Transfer on SUPERB
76
SUPERB: Speech processing Universal PERformance Benchmark (Yang et al. Interspeech 2021)
Cross-Task Mask Transfer on SUPERB
77
SUPERB: Speech processing Universal PERformance Benchmark (Yang et al. Interspeech 2021)
A Good Pruning Algorithm avoids pruning out the Important Weights in Pre-Trained Initializations
78
Dynamic Network Surgery for Efficient DNNs (Guo et al. NIPS 2016)
Identifying and Controlling Important Neurons in Neural Machine Translation (Bau et al. ICLR 2018)
SNIP: Single-Shot Network Pruning based on Connection Sensitivity (Lee et al. ICLR 2019)
Importance Estimation for Neural Network Pruning (Molchanov et al. CVPR 2019)
Analyzing Redundancy in Pretrained Transformer Models (Dalvi et al. EMNLP 2020)
Low-Complexity Probing via Finding Subnetworks (Cao et al. NAACL 2021)
A Good Pruning Algorithm avoids pruning out the Important Weights in Pre-Trained Initializations
79
A Good Pruning Algorithm avoids pruning out the Important Weights in Pre-Trained Initializations
80
A Good Pruning Algorithm avoids pruning out the Important Weights in Pre-Trained Initializations
81
��On the Interplay between �Sparsity, Naturalness, Intelligibility, and Prosody
On the Interplay Between Sparsity, Naturalness, Intelligibility, and Prosody in Speech Synthesis (Lai et al. arxiv 2021)
Out of 500 Neural TTS papers,
a few papers on Efficient TTS
83
A Survey on Neural Speech Synthesis (Tan et al. arxiv 2021)
Out of 500 Neural TTS papers,
a few papers on Efficient TTS,
0 papers on TTS pruning*
84
*Efficient Neural Audio Synthesis (Kalchbrenner et al. ICML 2018)
A Survey on Neural Speech Synthesis (Tan et al. arxiv 2021)
“We point out future research direction on neural TTS, mainly in two categories…
Efficient speech synthesis: how to reduce the cost of speech synthesis including the cost of collecting and labeling data, training and serving TTS models, etc.”
85
A Survey on Neural Speech Synthesis (Tan et al. arxiv 2021)
“We point out future research direction on neural TTS, mainly in two categories…
Efficient speech synthesis: how to reduce the cost of speech synthesis including the cost of collecting and labeling data, training and serving TTS models, etc.”
86
A Survey on Neural Speech Synthesis (Tan et al. arxiv 2021)
87
A Survey on Neural Speech Synthesis (Tan et al. arxiv 2021)
88
Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions (Shen et al. ICASSP 2018)
Neural Speech Synthesis with Transformer Network (Li et al. AAAI 2019)
Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram (Yamamoto et al. ICASSP 2020)
89
Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions (Shen et al. ICASSP 2018)
Neural Speech Synthesis with Transformer Network (Li et al. AAAI 2019)
Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram (Yamamoto et al. ICASSP 2020)
Module 1: Encoder
90
Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions (Shen et al. ICASSP 2018)
Neural Speech Synthesis with Transformer Network (Li et al. AAAI 2019)
Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram (Yamamoto et al. ICASSP 2020)
Module 2: Decoder
91
Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions (Shen et al. ICASSP 2018)
Neural Speech Synthesis with Transformer Network (Li et al. AAAI 2019)
Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram (Yamamoto et al. ICASSP 2020)
Module 3: Pre-Net
92
Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions (Shen et al. ICASSP 2018)
Neural Speech Synthesis with Transformer Network (Li et al. AAAI 2019)
Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram (Yamamoto et al. ICASSP 2020)
Module 4: Post-Net
93
Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions (Shen et al. ICASSP 2018)
Neural Speech Synthesis with Transformer Network (Li et al. AAAI 2019)
Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram (Yamamoto et al. ICASSP 2020)
Module 4: Post-Net
Module: Generator
Model Setup
94
Pruning Algorithms & Training
95
Pruning Algorithms & Training
96
Pruning Algorithms & Training
97
98
Pruned Model Evaluation
99
Naturalness
5-point scale MOS tests, 100 HITs/test
Intelligibility
synthetic WER via Google ASR API
Prosody
mean/std F0 estimation (Hz) and averaged utterance duration (sec.)
Demo
100
have made it difficult for the Treasury to maintain close and continuing supervision.
Demo
101
have made it difficult for the Treasury to maintain close and continuing supervision.
Unpruned
30% sparse
70% sparse
90% sparse
Demo
102
no sizable organization can achieve efficiency without the careful analysis and demarcation of responsibility
Demo
103
no sizable organization can achieve efficiency without the careful analysis and demarcation of responsibility
Unpruned
30% sparse
90% sparse
90% + 88% sparse
Does Sparsity Improve Naturalness?
104
Does Sparsity Improve Naturalness?
105
Does Sparsity Improve Naturalness?
106
No statistical difference up to 90% sparsity
Does Sparsity Improve Naturalness?
107
Hypothesis I: pruned model could train better
Does Sparsity Improve Naturalness?
108
No statistical difference up to 88% sparsity
Does Sparsity Improve Naturalness?
109
Hypothesis II: Neural TTS are over-parameterized in terms of Naturalness
Does Sparsity Improve Intelligibility?
110
Does Sparsity Improve Intelligibility?
111
Does Sparsity Improve Intelligibility?
112
1) Transformer-TTS: WER reduces prior to 90% 2) Tacotron2: WER maintains up to 40%
Does Sparsity Improve Intelligibility?
113
1) Transformer-TTS: WER reduces prior to 90% 2) Tacotron2: WER maintains up to 40%
3) P-WaveGAN: no WER change at all
Does Sparsity Improve Intelligibility?
114
Hypothesis III: Neural TTS are over-parameterized in terms of Intelligibility
Does Sparsity Change Prosody?
115
Does Sparsity Change Prosody?
116
117
1) Transformer-TTS: minimal duration fluctuation
2) Tacotron2: linear decrease (speedup)
118
1) Transformer-TTS: minimal duration fluctuation
2) Tacotron2: linear decrease (speedup)
3) P-WaveGAN: no duration change at all
119
Instability of Tacotron2 (RNN) pruning
c.f. Transformer-TTS F0 variation
120
Hypothesis IV: vocoder is not
responsible for prosody generation
121
Minimal F0 std variation c.f. ground truth 53 Hz F0 std
122
Minimal F0 over-smoothing
A Survey on Neural Speech Synthesis (Tan et al. arxiv 2021)
123
Hypothesis V: pruning does not hurt prosodic expressivity
Does more finetuning data improve Sparsity?
124
Does more finetuning data improve Sparsity?
125
30% of finetuning data (~7h) is enough
TTS-Augmentation for Unspoken Text
126
TTS-Augmentation for Unspoken Text
127
100h more data,
without domain mismatch
Does more finetuning data improve Sparsity?
128
TTS-Augmentation has minimal effects
129
130
Neural TTS models are
over-parameterized and highly prunable
Pruned models produce speech with
equal naturalness & intelligibility, with
similar prosody.
Pruning as an alternative tool for understanding deeper how
modern speech models are trained
Cheng-I Jeff Lai
Yang Zhang *
Alexander H. Liu *
Shiyu Chang *
Yi-Lun Liao
Yung-Sung Chuang
Kaizhi Qian
Sameer Khurana
David D. Cox
James R. Glass
Erica Cooper *
Junichi Yamagishi
Thank you!
PARP
https://people.csail.mit.edu/clai24/parp
TTS-Pruning
https://people.csail.mit.edu/clai24/prune-tts
Thank you!
PARP
https://people.csail.mit.edu/clai24/parp
TTS-Pruning
https://people.csail.mit.edu/clai24/prune-tts
Suggestions:
137