2023.01.27

제네시스랩 AI연구팀

신재영

NeurIPS 2021

Goals of Modern TTS

Motivation

• Fast : fast inference speed
• Lightweight : small size and low runtime memory footprint
• High-quality : capture the details in natural speech
• Expressive : generate expressive speech with powerful prosody modeling method

We propose PortaSpeech, which combines the advantages of VAE and normalizing flows.

VAE and Flow in NAR-TTS

Introduction

• VAE
• VAE is good at capturing the low frequency parts while discarding high-frequency parts.
• Thus we can intuitively infer that VAE is good at modeling the prosody while not good at modeling the details in speech.

​

• Flow-based model
• Flow-based models can generate high-quality images at the cost of very large model size and we may infer that flow-based models can model the details in speech �well with large model size.

Preliminary Analyses on VAE and Flow

Introduction

• MOS-P
• The prosody quality of VAE-based model only drops slightly when reducing the model capacities
• Inspiration: Apply VAE-based mel-spectrogram decoder to lightweight TTS model

​

• MOS-Q
• VAE-based model has poorer audio quality upper bound than Flow-based model
• Inspiration: introduce a flow-based post-net to refine the mel-spectrograms generated �by VAE

Architecture

PortaSpeech

Architecture

Linguistic Encoder with Mixture Alignment

• Phoneme-level hard alignment has several issues
• Challenging to obtain very accurate phoneme-level boundaries
• Affect the training of duration predictor and hurts the prosody

​

• Introduce mixture alignment
• Soft alignment in phoneme level
• Keeps hard alignment in word level

​

• Encourage the attention to be close to the diagonal in �text-to-spectrogram alignment
• Add a word-level relative positional encoding embedding

Architecture

Variational Generator

c (outputs of linguistic encoder)

Architecture

Post-Net

• The architecture adopts Glow
• simple loss-based (L1 or MSE-based) or VAE-based methods that often generate blurry outputs
• flow-based models can overcome the over-smoothing problem and generate more realistic outputs

​

• Conditioned on the outputs of:
• The variational generator (prosody information)
• The linguistic encoder (text information)

​

• Post-Net only focuses on modeling the details in �mel-spectrograms, greatly reducing requirements for model capacity

Objective Function

Training Loss Term

• Duration prediction loss : MSE between the predicted and the ground-truth word-level duration in log scale

​

• Reconstruction loss of variational generator : MAE between the ground-truth mel-spectrogram and that generated by the variational generator

​

• The KL-divergence of variational generator

​

• The negative log-likelihood of the post-net

Experimental Setup

Experiments

• Datasets
• LJSpeech
• 13,100 English audio clips (about 24 hours)
• 12,228 samples for training, 349 samples for validation and 523 samples for testing

​

• Data Preprocess
• Converting the text sequence into the phoneme sequence
• Transform the raw waveform into mel-spectrograms

Performance

Experiments

• Audio quality : PortaSpeech (normal) outperforms previous TTS models in audio quality and prosody and only has slight performance degradation when reducing the model size.
• Model size and memory footprint : PortaSpeech (small) has the smallest model size and memory footprint.
• Inference speed: PortaSpeech (small) achieves similar RTF with other NAR-TTS models.

​

Visualizations (Quality)

Experiments

PortaSpeech can generate mel-spectrograms with rich details in frequency bins �which results in natural sounds.

Performance on Multi-Speaker

Experiments

• Similar conclusions as that on LJSpeech that PortaSpeech can achieve good prosody and audio quality in terms of MOS-P and MOS-Q, even in more complicated (multi-speaker) scenarios.

​

Ablation Studies

Experiments

• Enhanced normalizing flow-based Prior
• compare with simple Gaussian prior of the original VAE
• enhanced prior can improve the prosody

​

• Flow-based Post-Net
• can improve the audio quality of the generated mel-spectrograms

​

• Mixture Alignment
• can improve the prosody, which may benefit from more accurate duration extraction and prediction

THANK YOU FOR LISTENING