Representation Selective Self-distillation

and wav2vec 2.0 Feature Exploration

for Spoof-aware Speaker Verification

Jin Woo Lee¹, Eungbeom Kim², Junghyun Koo¹, Kyogu Lee^1,2,3

jinwlee@snu.ac.kr

¹ Department of Intelligence and Information, Seoul National University

² Interdisciplinary Program in Artificial Intelligence, Seoul National University

³ AI Institute, Seoul National University

eb.kim@snu.ac.kr

dg22302@snu.ac.kr

kglee@snu.ac.kr

• Motivation
• Contributions
• Representation Learning for CM
• Front-end feature exploration
• Back-end model selection
• RSSD: Training Strategy for SASV
• Representation-Selective Self-Distillation
• Evaluation using the proposed front-/back-end combinations
• Conclusions

Outline

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

Motivation

• Spoofing-aware Automatic Speaker Verification (SASV) Challenge
• Automatic Speaker Verification (ASV) systems should reject both of:
• Voice uttered by different speaker (ASV non-target)
• Spoofed (synthesised or converted) utterances (CM non-target)
• How to integrate two distinct systems (ASV + CM) ^[1]
• Ensemble solutions based upon separate ASV and CM systems
• Integrated single system solutions
• ASVspoof 2019 Logical Access (LA) database ^[2]
• Collection of labeled speech data (target, non-target, spoofed)

[1] Jung, J. W., Tak, H., Shim, H. J., Heo, H. S., Lee, B. J., Chung, S. W., ... & Kinnunen, T. (2022). SASV challenge 2022: A spoofing aware speaker verification challenge evaluation plan. arXiv preprint arXiv:2201.10283.

[2] Wang, X., Yamagishi, J., Todisco, M., Delgado, H., Nautsch, A., Evans, N., ... & Ling, Z. H. (2020). ASVspoof 2019: A large-scale public database of synthesized, converted and replayed speech. Computer Speech & Language, 64, 101114.

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

Motivation

• ASV systems should additionally reject spoofed utterances
• Text-to-speech and voice conversion studies are constantly improving
• Difference between the target and the non-target for the CM becomes more indistinguishable
• Improve not only the SASV system, but also those for the CM
• Not many anti-spoofing studies analyzed to find clues about the attributes that are important for finding artifacts
• Answer to what is an effective representation for finding artifacts remains veiled

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Representation learning for spoofing CM
• Effective feature extractor for the CM
• Explore Transformer layer outputs from wav2vec 2.0 model
• Analyze feature spaces that effectively distinguish spoofed utterances
• Effective CM encoder for the feature extractor
• Study encoder structure that effectively distinguish spoofed utterances
• Compare EERs for each combination of feature extractor and encoder
• Representation Selective Self-Distillation (RSSD)
• Inspired by representation distillation^[1] and self-distillation^[2].

Contributions

[1] Tian, Y., Krishnan, D., & Isola, P. (2019, September). Contrastive Representation Distillation. In International Conference on Learning Representations.

[2] Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., & Ma, K. (2019). Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 3713-3722).

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

Contributions

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• ASV Network: ECAPA-TDNN ^[1]
• CM Network: wav2vec 2.0 ^[2,3] + Attentive statistics pooling ^[1]
• SASV Network: Representation-Selective Self-Distillation (RSSD)

[1] Desplanques, B., Thienpondt, J., & Demuynck, K. (2020). Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification. arXiv preprint arXiv:2005.07143.

[2] Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449-12460.

[3] Conneau, A., Baevski, A., Collobert, R., Mohamed, A., & Auli, M. (2020). Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006.13979.

Contributions

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Experimental setup: front-end

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• XLSR-53 Feature Exploration

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Experimental setup: back-end

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Spoofing CM using ASP

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• CM EER Comparison

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Using ASP as back-end model outperforms the others�for most front-ends
• Using 5^th layer output as front-end outperforms the others for most back-ends

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Using ASP as back-end model outperforms the others�for most front-ends
• Using 5^th layer output as front-end outperforms the others for most back-ends

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Using ASP as back-end model outperforms the others�for most front-ends
• Using 5^th layer output as front-end outperforms the others for most back-ends

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Using ASP as back-end model outperforms the others�for most front-ends
• Using 5^th layer output as front-end outperforms the others for most back-ends

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Using ASP as back-end model outperforms the others�for most front-ends
• Using 5^th layer output as front-end outperforms the others for most back-ends

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Spoofing countermeasure feature exploration
• XLSR-53 n^th layer + ASP

Representation Learning for CM

TTS

VC

VC

TTS

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• t-SNE Plot of Evaluated CM embeddings

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• t-SNE Plot of Evaluated CM embeddings

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Spoofing countermeasure EER breakdown

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Spoofing countermeasure EER breakdown

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Spoofing countermeasure EER breakdown

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Spoofing countermeasure EER breakdown

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Spoofing countermeasure EER breakdown

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Spoofing countermeasure EER breakdown

Representation Learning for CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

RSSD: Training Strategy for SASV

• Representation Selective Self-Distillation (RSSD)

ASV

CM

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

RSSD: Training Strategy for SASV

• Representation Selective Self-Distillation (RSSD)

ASV

CM

Distort the Test Spk Emb

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

RSSD: Training Strategy for SASV

• Representation Selective Self-Distillation (RSSD)

ASV

CM

Keep the Test Spk Emb

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Training with RSSD

RSSD: Training Strategy for SASV

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Training with RSSD

RSSD: Training Strategy for SASV

minimize

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Training with RSSD

RSSD: Training Strategy for SASV

minimize

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

RSSD: Training Strategy for SASV

• Representation Selective Self-Distillation (RSSD)

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Evaluation using the proposed front-/back-end combinations

RSSD: Training Strategy for SASV

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU

• Investigate from which layer of XLSR-53 the features obtained are advantageous for defending spoof attacks.
• Output from the 5^th Transformer layer shows the lowest CM EER.
• CM encoder with a simple attentive statistics pooling layer outperformed AASIST and MLP.
• Propose RSSD for SASV
• Selective training with representation self-distillation helps utilizing the learned representations of each ASV and CM task.
• The proposed CM model advantages both CM and SASV task.

Conclusion

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification

-

Jin Woo Lee et al.

MARG@SNU