Lecture 9

1

Transformers for TTS

People before 2017

People after 2017

Pre-requisites

• Transformer, difference between encoder-only, encoder-decoder and decoder-only models.
• VectorQuantisation. What it does, but not how it works
• Adversarial learning like in GANs
• LoRA
• Architecture and Losses for HFG [*Add question]

​

2

TTS tasks

3

• TTS - Text-to-Speech. Speaker from training dataset.
• Continuation - you have 3s of audio, continue it. No text prompt.
• Voice Cloning - Audio synthesis from 3-10 seconds audio of unknown speaker and text prompt.
• (speech/text)-to-(speech/text) - map speech and text to one latent space, than generate response in text or audio.

​

Motivation: more data

4

We want:

• Voice cloning
• Multispeaker models
• Big noisy datasets
• Better domain adaptation capabilities (pretrain-finetune, or few-shot)

Data: sources

• We want 10kh+ hours of audiodata
• We want pairs of speech and text which match
• Recording professional speakers is costly

5

LibriVox link

Main sources:

• YouTube
• Very diverse: speakers, topics
• No GT texts -> ASR markup -> mistakes + poor punctuation
• Poor audio quality
• Background music + noises
• Not everything is licensed
• Podcasts
• No GT texts
• Better audio quality
• Not everything is licensed
• AudioBooks
• Fewer open-source sources
• Poor speaker diversity
• GT texts with punctuations
• Good pronunciation and rich intonations
• LibriVox project

Data: pipeline

• VAD - audio segmenting
• ASR - labeling
• Noise detection / background music detection - filtering
• [optional] Text aligning (for audiobooks)
• [optional] Audio enhancement (optional) - audio postprocessing
• [optional] Speaker biometry - if speaker labels are needed

6

Conclusion: As we apply a lot of networks to our data, we will have a lot of errors. That’s why we need datasets to be big.

Data: datasets

Previous TTS datasets:

• LJ speech: monospeaker, 24h
• LibriTTS : ~2k speakers , 585h, clean split of LibriSpeech

Current datasets:

• GigaSpeech: 24kh, ASR labeled datasets from YouTube, podcasts and books
• LibriLight: 60kh, ASR labeled, from LibriVox project
• LibriHeavy - Lirilight with aligned texts

7

Questions ?

8

Tacotron2 disadvantages

• Works for average (>10h) high-quality datasets
• Vulnerable to bad examples in dataset (text-audio mismatch/mispronunciations/noise)
• Poor domain adaptation: emotion/intonation/whisper
• GST – is a very complicated tool for prosody control

​

9

Tacotron is not suitable here…

Post Tacotron2 models

10

Flow based models

Transformer based models

Diffusion based models

- VITS, VITS2, so-VITS

- Grad-TTS

- ResGrad

- GuidedTTS 1,2

- StyleTTS 1,2

​

- Encodec, Descript, Soundstorm

- Vall-E, MQTTS

- AudioLM, Spear-TTS

- SpeechGPT, NaturalSpeech 1,2

Good for prosody-sensitive applications

Good for applications like conversion

Good for big noisy datasets

Motivation

11

Tacotron 2

Our LLM bros

• Train on comparably small amount of high quality data, which is expensive to collect

• Train on huge amounts of very noisy data, scraped from internet
• Then finetune or use few-shot techniques for the dedicated task

• If you want to adopt new domain (emotion/intonation/whisper), than you have to find corresponding data.

​

• If you want to adopt new domain (code/writing style/emotion), you need to find the corresponding prompt

​

• Work with discrete tokens. And in generative tasks use techniques like beam-search/top-k sampling

​

• Work with continuous target (mel-spectrogram). Which is hard to generate.

​

Recap: encoder-only transformer (like BERT)

12

Encoder-only transformer

Recap: decoder-only transformer (like GPT)

13

Decoder-only transformer

Recap: decoder-only, teacher-forcing

14

Decoder-only transformer

Previously on speech_course …

15

Codecs sequence

Consider they are almost identical

• Long sequence
• Float samples

• Shorter sequence
• Codecs
• Each token consists of several tokens

Recap: namings

16

Time

Index

Questions ?

17

Pipeline

18

Seq2Seq

Encoder

Decoder

text

Discrete tokens

1. Encoder-decoder unsupervised training

2. Seq2seq (phonemes to tokens) supervised training

Previous lecture

Difference with NLP

• We have to merge different modalities in one latent space
• Instead of 1-dimensional token-sequence, we have N tokens per codec
• We have to take information about speaker somewhere

19

Vall-E (microsoft, Jan 2023)

• Voice cloning
• Tokenized text + reference codec -> continuation codecs
• 60kh of audiobooks from LibriLight
• Decoder-only autoregressive transformer (GPT approach to TTS)
• Uses Encodec

20

Vall-e: encodec

• Hierarchical codecs
• Lowest row is the most important one

21

Vall-e: encodec

• Hierarchical codecs
• Lowest row is the most important one
• Further layers of tokens improve restoration quality

22

Vall-e: encodec

• Hierarchical codecs
• Lowest row is the most important one
• Further layers of tokens improve restoration quality
• Authors chose 8 rows

23

Vall-E: Train AR stage

24

Decoder-only autoregressive model (like GPT)

Training

Phonemized text

<EOS>

Audio codes

<EOS>

AR = autoregressive

Vall-E: Train AR stage

25

Decoder-only autoregressive model (like GPT)

Training

Phonemized text

<EOS>

Vall-E: Train AR stage

26

Decoder-only autoregressive model (like GPT)

Training

Phonemized text

<EOS>

Vall-E: Train AR stage

27

Decoder-only autoregressive model (like GPT)

Training

Phonemized text

<EOS>

Vall-E: Train AR stage

28

Decoder-only autoregressive model (like GPT)

Training

Phonemized text

<EOS>

Vall-E: Train AR stage

29

Decoder-only autoregressive model (like GPT)

Training

Phonemized text

<EOS>

Vall-E: Train NAR stage

30

Encoder-only model (like BERT)

Training

Phonemized text

Reference audio

NAR = non autoregressive

Vall-E: AR inference

31

Vall-E: inference AR & NAR stages

32

Time

Codes

AR (autoregressive)

NAR (non-autoregressive)

Questions ?

33

MQTTS (ByteDance, Feb 2023)

• Have their own codec model
• Voice cloning
• ~1kh of audio from podcasts and youtube
• Encoder-decoder (aka original transformer) architecture with tuned cross-attention mechanism

​

34

MQTTS: Architecture

35

Encoder

Decoder

Single attention

h

ɛ

l

ʊ

…

13

47

52

60

32

01

68

89

53

47

42

86

29

01

75

89

…

13

47

52

60

32

01

68

89

29

01

75

89

…

53

47

42

86

eos

eos

eos

eos

bos

bos

bos

bos

MQTTS: transducer

36

Big decoder

13

47

52

60

32

01

68

89

53

47

42

86

29

01

75

89

…

13

47

52

60

32

01

68

89

29

01

75

89

…

53

47

42

86

eos

eos

eos

eos

Sub-decoder

e0

e1

e2

e3

en

• Decoder-only transformer
• Condition of embeddings from decoder
• Makes 4 steps for each code

MQTTS: Inference

37

Decoder

Cross-Attention and softmax

13

47

52

60

Embeddings from encoder

Training

MQTTS: Inference

38

Decoder

Cross-Attention and softmax

13

47

52

60

32

01

68

89

Embeddings from encoder

Training

MQTTS: Inference

39

Decoder

Cross-Attention and softmax

13

47

52

60

32

01

68

89

29

01

75

89

Embeddings from encoder

Training

MQTTS: Inference

40

Decoder

Cross-Attention and softmax

13

47

52

60

32

01

68

89

29

01

75

89

53

47

42

86

Embeddings from encoder

…

Training

MQTTS: Inference

41

Decoder

Cross-Attention and softmax

13

47

52

60

Embeddings from encoder

Inference

• We want alignment to be more monotonic
• We restrict attentions span
• Here span size N = 5

MQTTS: Inference

42

Decoder

Cross-Attention and softmax

13

47

52

60

32

01

68

89

Embeddings from encoder

Inference

When we spot that attention for first element is smaller than 1/N, we move the span

MQTTS: Inference

43

Decoder

Cross-Attention and softmax

13

47

52

60

32

01

68

89

29

01

75

89

Embeddings from encoder

Inference

MQTTS: Inference

44

Decoder

Cross-Attention and softmax

13

47

52

60

32

01

68

89

29

01

75

89

53

47

42

86

Embeddings from encoder

Inference

MQTTS: Inference

45

Decoder

Cross-Attention and softmax

13

47

52

60

32

01

68

89

29

01

75

89

53

47

42

86

Embeddings from encoder

…

Inference

Questions ?

46

Coffee break

47

AudioLM (Google, Sep 2022)

• Continuation task (Language modeling for audio)
• Uses 2 pre-trained models: W2V-BERT and Soundstream
• Does not require texts, uses audio-only data!!
• Data: 60kh from LibriLight

​

​

48

W2V-BERT: recap

49

https://arxiv.org/abs/2108.06209

Latent Features Encoder

(Convolutional network with striding)

z₀

z₁

z₂

z₃

z_T-1

z_T

…

Context Network

(Transformer encoder)

c₀

c₁

c₂

c₃

c_T-1

c_T

…

Time masking

(BERT like)

Quantization module

(Gumbel softmax trick)

Contrastive loss

(distinguish current masked positions label among others)

Waveform

Masking N% of latent vectors

(paper proposed 50%)

Masked latent

feature vector

Quantized target vectors

Quantized (audio) token IDs

Masked Language Model

loss

MLM Context Network

(Transformer encoder)

Architecture overview

Stolen slide from ASR II presentation

AudioLM: semantic and acoustic tokens

50

AudioLM: pipeline

51

• Semantic tokens from W2V-BERT + kMeans
• Coarse acoustic tokens from bottom 4 codes in a codec
• Fine acoustic tokens are the remaining 12 codes
• Flatten everything for sequence modeling

…

…

4

12

Coarse acoustic tokens

Fine acoustic tokens

AudioLM: demo

52

Demo

Questions ?

53

AudioLM(5/4)

• Continuation
• Two types of tokens:
• Semantic tokens
• Used to preserve long-term structural coherence
• Uses w2v-BERT, a self-supervised representation model for audio representation. Loss = MLM + contrastive
• kMeans clustering, k=1024
• 1s (16kHz) -> 1 x 25 ints
• Acoustic tokens
• Used to decode into wav
• 1s (16kHz) -> 16 x 50 ints
• As we use RVQ, we define top 4 tokens – coarse tokens and other 12 – fine tokens
• Training
• Nothing –> semantinc tokens
• Semantic tokens -> coarse acoustic tokens
• Coarse acoustic tokens -> fine acoustic tokens

​

​

​

​

54

Thoughts

- It is weird, that we can predict fine tokens only with coarse ones. If so, why does decoder learn this

- Continuations are logically reasonable !

AudioLM(5/4)

• Continuation
• Two types of tokens:
• Semantic tokens
• Used to preserve long-term structural coherence
• Uses w2v-BERT, a self-supervised representation model for audio representation. Loss = MLM + contrastive
• kMeans clustering, k=1024
• 1s (16kHz) -> 1 x 25 ints
• Acoustic tokens
• Used to decode into wav
• 1s (16kHz) -> 16 x 50 ints
• As we use RVQ, we define top 4 tokens – coarse tokens and other 12 – fine tokens
• Training
• Nothing –> semantinc tokens
• Semantic tokens -> coarse acoustic tokens
• Coarse acoustic tokens -> fine acoustic tokens

​

​

​

​

55

Thoughts

- It is weird, that we can predict fine tokens only with coarse ones. If so, why does decoder learn this

- Continuations are logically reasonable !

W2V-BERT

• Big audio-only self-supervised pre-train for ASR
• Self-supervised losses: Contrastive loss and masked language modeling loss
• Pre-training: Libri-Light unlab-60k subset, 60kh of unannotated speech
• Used with Kmeans for tokenization of speech
• Produces semantic token, which are good at encoding phoneme information, with limited info about speaker and prosody
• 1s (16kHz) -> 25 vectors -> 25 tokens

​

​

​

​

​

​

56

Spear-TTS (Google, Feb 2023)

• TTS, low-source TTS, cloning
• Inherits ideas from AudioLM
• SPEAR stands for “speak, read and prompt”
• Uses a few parallel data and a lot of unsupervised data

57

Spear-TTS: pipeline

58

Audio 60kh

SoundStream

W2V-BERT

Audio 60kh

Problem: Few parallel data

Problem: 1-to-many mapping

G2P

Spear-TTS: reading

59

Encoder

Decoder

Cross-attention

Phonemes from G2P

Semantic tokens from W2V-BERT

Spear-TTS: reading

• Text -> Semantic tokens
• Seq2seq model
• Pretraining + Backtranslation
• We want to get a big chunk of synthesized data
• Then train seq2seq on it

60

SPEAR-TTS: speaking

• Semantic tokens -> Acoustic tokens
• We can use audio only dataset
• We split utterance into two parts: prompt and target. Then we generate semantic and acoustic tokens for both prompt and target
• Now we use acoustic-prompt, semantic-prompt and semantic-target to generate acoustic-target
• Libri-light 60k is used here

61

Speaker and prosody are inherited from the prompt

Speaker and prosody can very because of sampling

Spear-TTS: what for…

• Uses small amount of parallel data, as small as 15 minutes (low-resource TTS)
• Good domain adaptation. Model can do different speakers and emotions even-though there were none in training data

62

Demo

SPEAR-TTS(5/4+): part1

• Same story with semantic tokens
• S1 – reading. Text -> sematic tokens
• S2 – speaking. Semantic -> acoustic
• Distinguish task into several representations to ease mappings between them and target different problems separately
• Uses very small source of parallel data (monospeaker lj-speech 24h)

63

Audio 60kh

SoundStream

W2V-BERT

Audio 60kh

• S1 – text to semantic
1. Training a seq2seq model P on big audio-only dataset on a denoising task
2. Training and ASR model B (Backtranslation) on a small parallel text-speech dataset
3. Using this ASR model B to transcribe audio only dataset
4. Fine-tune model B on this synthetic dataset to become S1

​

Problem: Few parallel data

Problem: 1-to-many mapping

Questions ?

64

SpeechGPT (Fudan university, may 2023)

• (speech/text)-to-(speech/text)
• Exploits LLM knowledge from LLaMa
• Uses TTS and ASR datasets

65

SpeechGPT

• Architecture
• HuBERT + kMeans for audio tokenization (predecessor of W2V-BERT)
• HiFi-GAN on tokens for token-to-audio transformation
• Pre-trained LLaMa GPT
• Data
• GigaSpeech (ASR)
• CommonVoice
• LibriLight

66

SpeechGPT: training

• We have Llama, which haven’t seen speech yet
• We expand the embedding matrix of Llama by random vectors, which correspond to audio tokens
• We train the model on tokens from audio-only data

67

1. Modality adaptation

​

Decoder-only pretrained LLama

SpeechGPT: training

2. Cross-model instruction finetuning

68

• We take ASR and TTS datasets with pairs of text-speech pairs
• We take ChatGPT to create 100 instructions for ASR and TTS task
• We assemble this into instruction-text-speech or instruction-speech-text triplets
• We train model on NTP task

SpeechGPT: training

3. Chain-of-Modality Instruction Fine-Tuning

69

• Create a model, which generates HuBERT tokens from text. Encoder-Decoder transformer, trained on LibriSpeech
• Take text dialogue dataset
• Using the above model generate quadruplets for each sample in a dialogue dataset: (TextI, TextR) -> (SpeechI, TextI, TextR, SpeechR)
• Sample from dataset a pair
• Fine tune model with LoRA method

SpeechGPT: result

70

Demo

SpeechGPT(3+/5)

• (speech/text)-to-(speech/text)
• Exploits LLM knowledge from LLaMa
• TTS and ASR datasets
• Architecture
• HuBERT + kMeans audio tokenization
• Pre-trained LLaMa GPT with
• Adopted HiFiGAN vocoding

71

Three steps training

1. Modality-adaptation pre-training

- Expand the vocabulary of LLM

- Take speech only dataset

- Train on NTP for audio units only2 .

2. Cross-modal Instruction Fine-Tuning

- We take ASR and TTS datasets with speech-text pairs

- We take GPT-4 and create 100 prompt for ASR and TTS part

- We compose

3. Chain-of-Modality Instruction Fine-Tuning

- Efficient fine tuning using LoRA method

- Take dialogue dataset, using TTS and ASR generate quadruplets (SpeechI, TextI, TextR, SpeechR)

- Sample from dataset a pair and train on it

​

Questions ?

72