1 of 19

1

Whispering LLaMA: A Cross-Modal Generative Error Correction �Framework for Speech Recognition

Srijith Radhakrishnan, Huck Yang, Sumeer Ahmad Khan, Rohit�Kumar, Narsis A. Kiani, David Gomez-Cabrero, Jesper N. Tegner

2 of 19

Could You Recognize this Speech?

[Context] at a noisy party

[Context] referring to someone who has not been uninvited or excluded from an event or social circle. �

2

“He has not been dropped.”

Slide sourced from Dr. Huck Yang on �ASRU 23 Tutorial on LLM for ASR

3 of 19

Open Questions During this Talk

Do we need better ears or a better brain to recognize speech?

Could audio and text work as a single cross-modal correction framework

3

Slide sourced from Dr. Huck Yang on �ASRU 23 Tutorial on LLM for ASR

4 of 19

LLMs are Generative Error Correctors

4

5 of 19

Whispering-LLaMA: Prompt-Template

5

Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting, �ASRU 2023, Yang et al.

Related Zero-Shot Correction Ref.

6 of 19

Whispering-LLaMA (1/2): Model-Level

6

7 of 19

Whispering-LLaMA (2/2): Layer-Level

7

We have a more detailed diagram of our layer_wise adapter mechanism here. The left side illustrates the LLAMA adapter mechanism employing a transformer backbone to finetune the model.

The right side represents the fusion adapter. In the fusion adapter, we extract audio features from the Whisper encoder and the key and value linear layers from the Whisper decoder model. Passing the audio features through these linear layers mirrors the key and value pairs produced within the whisper decoder model. Subsequently, we implement an adapter with an autoencoder backbone to align the audio features with the language model. As the shapes of the Whisper tensors and LLaMA tensors differ, we incorporate a padding mechanism to ensure that the latent structure in the audio features are not destroyed

8 of 19

N-best Hypotheses Dataset

8

9 of 19

Model Performance

9

We report the results of our framework in this slide.

We futher coduct abaltaion experimients in row 4 to 8.

During our experimients we discovered that masking the hypotheses in the prompt in the loss function significantly boosts performance. In row 5 we remove this masking mechanism and the performance drops as shown in the table

In row 6 we replace the acoustic embedding with random normal distribution to check if the framework actually makes use of the audio features. And doing so make the model unable to converge leading to high wer

In row 7 we replace our initialization mechanism with random normal distribution to check if the mechanism helps in preserving the latent structure of the whisper audio features. And doing so makes the model unable to converge again leading to the high wer.

These two experiments show that the audio features and the padding mechanism play a significant role in the design of the framework

In rows 8 and 9 we report the performance after removing the fusion adapter. Representing standard fine tuning performance.

10 of 19

Whispering-LLama Conclusion (1/2)

We propose a multimodal decoder based layer-level fusion to combine Whisper decoder and LLaMA decoder to improve speech transcripts.

10

11 of 19

Whispering-LLama Conclusion (2/2)

We demonstrate that LLM integration is a promising direction

Utilizing audio information and quality transcripts further boosts performance

Whisper Tiny (70M) + LLM (7B) = Whisper Large (1.5B)

11

12 of 19

More References

Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting, ASRU 2023�
HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models, NeurIPS 2023, Dataset and Benchmark�
Generative error correction for code-switching speech recognition using large language models, Arxiv 2023�
Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition, ASRU 2023

12

13 of 19

Resource & Thank you all!

13

https://github.com/Srijith-rkr/Whispering-LLaMA/tree/main

100+ stars or GitHub 😊

srijithrkr@gmail.com; hucky@nvidia.com

14 of 19

14

Link to our slides

15 of 19

Appendix - Fine-tuning the LLM with adapters

15

16 of 19

Appendix - Where to extract the audio information from?

16

17 of 19

Appendix - How to learn cross-modal representations?

17

18 of 19

Appendix - Dealing with shape mismatch?

18

We have a more detailed diagram of our layer_wise adapter mechanism here. The left side illustrates the LLAMA adapter mechanism employing a transformer backbone to finetune the model.

The right side represents the fusion adapter. In the fusion adapter, we extract audio features from the Whisper encoder and the key and value linear layers from the Whisper decoder model. Passing the audio features through these linear layers mirrors the key and value pairs produced within the whisper decoder model. Subsequently, we implement an adapter with an autoencoder backbone to align the audio features with the language model. As the shapes of the Whisper tensors and LLaMA tensors differ, we incorporate a padding mechanism to ensure that the latent structure in the audio features are not destroyed

19 of 19

Appendix - Whisper

Trained on to 680,000 hours of multilingual (99 languages) and multi task audio data

�

19