2023.07.07

제네시스랩 AI연구팀

신재영

PMLR 2022

https://edresson.github.io/YourTTS/

Motivation

Introduction

• There were several approaches for zero-shot multi-speaker TTS (ZS-TTS)
• i.e. Attentron, ZSM-SS, SC-GlowTTS

​

• Despite these advances, the similarity gap between observed and unobserved speakers during training is still an open research question.

​

• In parallel with the ZS-TTS, multilingual TTS has also evolved aiming at learning models for multiple languages at the same time.

​

• In this paper, we propose YourTTS with several novel ideas focused on zero-shot multi-speaker and multilingual training.

Background

Introduction

VITS

JaeHyeon Kim, et al. “Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech”, PMLR 2021

Background

Introduction

Speaker Consistency Loss

Detai Xin, et al. “Cross-lingual Speaker Adaptation using Domain Adaptation and Speaker Consistency Loss for Text-To-Speech Synthesis”, INTERSPEECH 2021

Architecture

Overall Structure

Architecture

Speaker Encoder

Hee Soo Heo, et al. “Clova Baseline System for the VoxCeleb Speaker Recognition Challenge 2020”, Arxiv 2020

• As speaker encoder, we use the H/ASP model publicly available, that was trained with the Angular Prototypical plus Softmax loss functions in the VoxCeleb 2 dataset.
• This model was chosen for achieving state-of-the-art results in VoxCeleb 1 test subset.
• This model reached an average Equal Error Rate (EER) of 1.967 while the speaker encoder used in the SC-GlowTTS paper reached an EER of 5.244.

H/ASP model configuration

Method

Objective Function

​

• Speaker Consistency Loss (SCL) in the final loss

​

​

�

• φ(.) : a function outputting the embedding of a speaker
• α : positive real number that controls the influence of the SCL in the final loss
• n : the batch size
• g and h : represent, respectively, the ground truth and the generated speaker audio

Experiment

Audio datasets

​

• English: VCTK dataset, which contains 44 hours of speech and 109 speakers, sampled at 48KHz.

​

• Portuguese: TTS-Portuguese Corpus, a single speaker dataset of the Brazilian Portuguese language with around 10 hours of speech, sampled at 48KHz.

​

• French: fr FR set of the M-AILABS dataset, which is based on LibriVox6 . It consists of 2 female (104h) and 3 male speakers (71h) sampled at 16KHz

​

�

Experiment

Setup

​

• Start from a model trained 1M steps on LJSpeech and continue the training for 200K steps with the VCTK dataset
• For each of the experiments a fine-tuning was performed using Speaker Consistency Loss (α = 9)
• Training is done by continuing from the previous experiment for approximately 140k steps, learning one language at a time

​

• Experiment 1: using VCTK dataset (monolingual)
• Experiment 2: using both VCTK and TTS-Portuguese datasets (bilingual)
• Experiment 3: using VCTK, TTS-Portuguese and MAILABS french datasets (trilingual)
• Experiment 4: starting with the model obtained in experiment 3 we continue training with 1151 additional English speakers from both LibriTTS partitions trainclean-100 and train-clean-360.

​

�

Experiment

Setup

Detai Xin, et al. “Cross-lingual Speaker Adaptation using Domain Adaptation and Speaker Consistency Loss for Text-To-Speech Synthesis”, INTERSPEECH 2021

Zero-shot Multi-Speaker TTS

Results

• Speaker Encoder Cosine Similarity (SECS) :

between the speaker embeddings of two audios extracted from the speaker encoder. �It ranges from -1 to 1, and a larger value indicates a stronger similarity

Contribution

Results

• State-of-the-art results in the English Language

​

• The first work proposing a multilingual approach in the zero-shot multi-speaker TTS scope

​

• Ability to do zero-shot multi-speaker TTS and zero-shot Voice Conversion with promising quality and similarity in a target language using only one speaker in the target language during model training

​

• Require less than 1 minute of speech to fine-tune the model for speakers who have voice/recording characteristics very different from those seen in model training, and still achieve good similarity and quality results

Limitation

Results

• It seems that using SCL slightly decreases the quality of generated audio.

​

• We believe this is because with the use of SCL, our model learns to generate recording characteristics present in the reference audio, producing more distortion and noise.

​

• However, it should be noted that in our tests with high-quality reference samples, the model is able to generate high-quality speech.

THANK YOU FOR LISTENING