1 of 15

2023.04.07

제네시스랩 AI연구팀

신재영

https://valle-demo.github.io/

2 of 15

Motivation

Introduction

Because the training data is relatively small, current TTS systems still suffer from poor generalization

To tackle the zero-shot TTS problem, existing work leverages speaker adaptation methods, requiring additional fine-tuning, complex pre-designed features, or heavy structure engineering.

3 of 15

Aim

Introduction

To have a flexible generation pipeline to perform various acoustic speech generation based on a single acoustic input

Transformer-style learning

To generate high quality speech

Good quality representation via Encodec�

4 of 15

Main Idea

Introduction

Large pre-training data

Pre-training stage, scale up the text-to-speech synthesis (TTS) training data to 60K hours of English speech which is hundreds of times larger than existing systems

Discrete Pipeline

Traditional: phoneme → mel-spectrogram → waveform
New: phoneme → discrete code → waveform �

5 of 15

Normal TTS system VS VALL-E

Introduction

6 of 15

Background

Speech Quantization

Encodec (used in VALL-E)

High Fidelity Neural Audio Compression, Défossez et. Al. 2022

7 of 15

Architecture

Overall Structure

8 of 15

Architecture

Neural Audio Codel model

Residual Vector Quantization (RVQ)

9 of 15

Architecture

Decoder

Auto-Regressive (AR)

used for first quantizer
Attended only to previous tokens
Generated sequence length is unknown
O(T) time complexity�

Non-Auto-Regressive (NAR)

used for 2^nd to 8^th quantizer
Generated sequence length is known
O(1) time complexity
Can generate all sequences in parallel

Separate sinusoidal position embeddings for prompts and output

x : phoneme sequence, 즉 text

c : 어느 시점에서 나온 값일지는 몰라도, quantizer layer에서 나온 output (sequence predict에서 이전 output -> 현재 input 정도로 이해하는 그 값)

틸다 C : 음성 prompt (3초 넣는다는 그 음성)

참고 - C가 대문자면 전체를, 소문자면 특정 구간을 의미

autoregressive language model은 첫번째 quantizer에서 토큰을 생성한다. 그것은 phoneme embedding W_x 와 acoustic embedding X_a, transformer decoder 그리고 prediction layer로 구성되어있다.

The reason for training 2 models is because of the way EnCodec model generates acoustic tokens.

It works in a hierarchical fashion where the current layer is trained on the residual from the previous layer i.e very first layer after input will contain the most important information about the speaker identity while the later layers will contain some other fine acoustic details in a diminishing return fashion.

Therefore, the very first layer of acoustic tokens are the most important tokens, hence different training for first layer of acoustic codes and the remaining acoustic codes to get the best possible reconstruction of speaker identity.

10 of 15

Method

Loss Function

Token embeddings e are summed up from all 8 layers
AR Loss is autoregressive loss of predicting next token for layer 1
NAR Loss is summed loss of predicting all tokens for a chosen layer between 2-8

Which layer to predict is randomly sampled

11 of 15

Program based Evaluation

Results

• Word Error Rate evaluated by using an Automatic Speech Recognition program

• Speaker Similarity predicted by WavLM-TDNN (Chen et. al, 2022)

**Previous SOTA (YourTTS, GSLM, AudioLM) not trained on VALL-E’s large dataset
Overall, VALL-E performs the best

12 of 15

Human Evaluation

Results

CMOS (comparative mean opinion score) :

Indicator of speech naturalness (evaluated by 6–12 native speakers manually)
Ranked from -3 to 3�

SMOS (similar mean opinion score) :

Indicator of speaker similarity (evaluated by 6–12 native speakers manually)
Ranked from 1 to 5

13 of 15

Speaker Similarity

Results

14 of 15

Limitation

Results

Unable to perform well on accented speakers, maybe they’re not covered in the training dataset.

Word level errors in generated audio i.e missing, duplicated or unclear utterances of word.

Fall short of training a single universal model since it utilises two language models to be trained as well as Encodec.

15 of 15

THANK YOU FOR LISTENING