1 of 147

ISCA Odyssey 2020

Tutorial 4

Neural statistical parametric

text-to-speech synthesis

Xin WANG

National Institute of Informatics, Japan

Contact: wangxin@nii.ac.jp

/ʃɪn/

/wʌn/

1

2 of 147

Slides by Xin Wang

National Institute of Informatics

This work is licensed under the Creative Commons Attribution 3.0 license.

See http://creativecommons.org/ for details.

Note: Natural Japanese speech data belonging to ATR Ximera corpus are deleted in this public available version

2

3 of 147

Self - introduction

https://researchmap.jp/wangxin

http://tonywangx.github.io/

Basic info

Post-doc at NII, Yamagishi Lab
Ph.D (2015 – 2018) at NII

Research topics

Text-to-speech synthesis (for Ph.D)
Speech & music audio modeling

Activities

ASVspoof 2019 & voice privacy challenge

3

4 of 147

Note

About the abstract

For waveform models, pleae check reading list

In this tutorial, ... recent methods neural-network-based acoustic models (e.g., Tacotron and its variants) and waveform generators ...

We also explain some of the classical methods such as the hidden-Markov-model-based ones.

... we make an excursion to voice conversion ...

Apologize for leaving these topics out

4

5 of 147

Note

Goal

Recent approaches (Tacotron, ...)
How do they work and what are the differences

Based on my own reading list

It cannot be unbiased
Notes on each slide may be useful
Appendix & reading list may be useful

Feedback is welcome!

5

6 of 147

Contents

Introduction
TTS overview
Recent sequence-to-seqeuence TTS

Soft attention
Hard attention
Hybrid approaches

Summary

6

7 of 147

Introduction

7

8 of 147

Intro – TTS?

Kewley-Port, D. & M. Nearey, T. Speech synthesizer produced voices for disabled, including Stephen Hawking. J. Acoust. Soc. Am. 148, R1--R2 (2020)

Input text

8

9 of 147

Intro – TTS?

See more in Taylor, P. Text-to-Speech Synthesis. (Cambridge University Press, 2009).

Input text

Formant synthesis

Hidden Markov models (HMM)

Waveform concatenation

LSTM-RNN

...

Unit-selection

Tacotron

WaveNet

9

10 of 147

Intro – Rapid advance of TTS

Wu, Z. et al. ASVspoof: the automatic speaker verification spoofing and countermeasures challenge. IEEE J. Sel. Top. Signal Process. 11, 588–604 (2017)

Unit-selection

(S10)

HMM-based TTS

(S3, S4)

Fig2. from (Wu 2017)

ASVspoof 2015

i-vector distance between spoofed & bona fide speech

10

11 of 147

Intro – Rapid advance of TTS

Wu, Z. et al. ASVspoof: the automatic speaker verification spoofing and countermeasures challenge. IEEE J. Sel. Top. Signal Process. 11, 588–604 (2017)

Wang, X. et al. ASVspoof 2019: a large-scale public database of synthesized, converted and replayed speech. Comput. Speech Lang. (2020)

Fig2. from (Wu 2017)

ASVspoof 2015

Fig3. from (Wang 2020)

ASVspoof 2019

11

12 of 147

Intro – Rapid advance of TTS

Wu, Z. et al. ASVspoof: the automatic speaker verification spoofing and countermeasures challenge. IEEE J. Sel. Top. Signal Process. 11, 588–604 (2017)

Wang, X. et al. ASVspoof 2019: a large-scale public database of synthesized, converted and replayed speech. Comput. Speech Lang. (2020)

Fig2. from (Wu 2017)

ASVspoof 2015

Fig3. from (Wang 2020)

ASVspoof 2019

Unit-section

(S10), (A16, A04)

Sequence-to-sequence (seq-to-seq) TTS

(A10, A11)

WaveNet

(A12)

HMM-DNN

(A08)

12

13 of 147

Intro – Rapid advance of TTS

Wang, X. et al. ASVspoof 2019: a large-scale public database of synthesized, converted and replayed speech. Comput. Speech Lang. (2020)

ASVspoof 2019: human listening test

Speaker similarity

Speech quality

Non-target

Target

A10

Non-target

Target bona fide

A10

A16

A08

HMM-DNN

Unit-selection

Seq-to-seq

13

14 of 147

Intro – Rapid advance of TTS

More samples: https://nii-yamagishilab.github.io/samples-xin/main-asvspoof2019

Database: https://datashare.is.ed.ac.uk/handle/10283/3336

See reference for methods before unit-selection

Unit-selection A16	HMM-DNN A08	Latest Seq2seq A10	Target bona fide (natural)

Waveform concantenation

Formant synthesis

~2016

~2013

~2000

HMM-based TTS

HMM-DNN

Seq-to-seq

Unit-selection

~1996

Now

14

15 of 147

Intro – Rapid advance of TTS

More samples: https://nii-yamagishilab.github.io/samples-xin/main-asvspoof2019

Database: https://datashare.is.ed.ac.uk/handle/10283/3336

See reference for methods before unit-selection

Unit-selection A16	HMM-DNN A08	Latest Seq2seq A10	Target bona fide (natural)

Waveform concantenation

Formant synthesis

~2016

~2013

~2000

HMM-based TTS

HMM-DNN

Seq-to-seq

Unit-selection

~1996

Now

15

16 of 147

Intro – Rapid advance of TTS

More samples: https://nii-yamagishilab.github.io/samples-xin/main-asvspoof2019

Database: https://datashare.is.ed.ac.uk/handle/10283/3336

See reference for methods before unit-selection

Unit-selection A16	HMM-DNN A08	Latest Seq2seq A10	Target bona fide (natural)

Waveform concantenation

Formant synthesis

~2016

~2013

~2000

HMM-based TTS

HMM-DNN

Seq-to-seq

Unit-selection

~1996

Now

16

17 of 147

TTS Overview

17

18 of 147

Overview – TTS easy?

Sentence from: Beckman, M. E. & Ayers, G. Guidelines for ToBI labelling. OSU Res. Found. 3, (1997)

Marianna made the marmalade

Discrete

Continuous

Alignment

M

a

m

e

a

…

m

…

d

a

l

d

e

r

Ambiguity

Speaker identity,

Prosody …

18

19 of 147

Overview – Classical TTS

Sentence from: Beckman, M. E. & Ayers, G. Guidelines for ToBI labelling. OSU Res. Found. 3, (1997)

LOGIOS Lexicon tool: http://www.speech.cs.cmu.edu/tools/lextool.html

H*, L-L%: ToBI labels Beckman, M. E. & Ayers, G. Guidelines for ToBI labelling. OSU Res. Found. 3, (1997)

Marianna made the marmalade

M AA R IY AA N AH M EY D DH AH M AA R M AH L EY D

To phone

Waveform

generation

M

a

m

e

a

…

m

…

d

a

l

d

e

r

Normali-zation

Acoustic

realization

+Prosody

20 of 147

Overview – Classical TTS

Taylor, P. Text-to-Speech Synthesis. (Cambridge University Press, 2009).

Dutoit, T. An Introduction to Text-to-speech Synthesis. (Kluwer Academic Publishers, 1997).

Marianna made the marmalade

M AA R IY AA N AH M EY D DH AH M AA R M AH L EY D

To phone

+prosody

Acoustic

realization

Waveform

generation

M

a

m

e

a

…

m

…

d

a

l

d

e

r

Normali-zation

H*

L-L%

M AA R IY AA N AH M EY D DH_AHM AA R M AH L EY _D

Back-end

Main topic of this talk

How can we do alignment?

Front-end

Pitch accent, intonation, phrasing

Syllabification, lexical stress

Grapheme-to-phoneme

Text-normalization

Taylor 2009

Dutoit 1997

20

21 of 147

Overview – Classical TTS

Formant synthesis: Klatt, D. H. Review of text-to-speech conversion for English. J. Acoust. Soc. Am. 82, 737–793 (1987)

Concatenative & signal processing: See Dutoit An Introduction to Text-to-speech Synthesis. (1997)

Unit-selection: Hunt, A. J. & Black, A. W. Unit selection in a concatenative speech synthesis system using a large speech database. ICASSP (1996).

See history of unit-selection in Chapter 16.7.3, Paul Taylor, Text-to-speech Synthesis (2009)

Acoustic

realization

Waveform

generation

H*

L-L%

M AA R IY AA N AH M EY D DH_AHM AA R M AH L EY _D

Back-end

Formant

synthesis

Concatenative &

Signal processing

Unit-selection

Statistical

parametric

sd

Speech signal

analysis &

re-synthesis

Statistical model

Multi-speaker

modeling

F0 modeling

Zero-shot

Robustness

Multi-lingual

Emotion

...

Vocoders

Spectral modeling

Alignment & duration

Adaptation

21

22 of 147

Overview – Unit-selection

Hunt, A. J. & Black, A. W. Unit selection in a concatenative speech synthesis system using a large speech database. in Proc. ICASSP 373–376 (1996).

Black, A. W. & Taylor, P. A. Automatically clustering similar units for unit selection in speech synthesis. (1997)

...

Marianna made the marmalade

H*

L-L%

M AA R IY AA N AH [ ] M EY D DH_AH[ ]M AA R M AH L EY _D

Speech unit database

^{(Sec. 2.1, Hunt 1996)}

Find best units

No explicit modeling of alignment

22

23 of 147

Overview – Unit-selection

Hunt, A. J. & Black, A. W. Unit selection in a concatenative speech synthesis system using a large speech database. in Proc. ICASSP 373–376 (1996).

Black, A. W. & Taylor, P. A. Automatically clustering similar units for unit selection in speech synthesis. (1997)

...

Marianna made the marmalade

H*

L-L%

M AA R IY AA N AH [ ] M EY D DH_AH[ ]M AA R M AH L EY _D

Speech unit database

Find best units

No explicit modeling of alignment

23

24 of 147

Overview – HTS (HMM-based speech synthesis system)

MLSA: Imai, S., Sumita, K. & Furuichi, C. Mel log spectrum approximation (MLSA) filter for speech synthesis. Electron. Commun. Japan (Part I Commun. 66, 10–18 (1983)

STRAIGHT: Kawahara, H., Masuda-Katsuse, I. & Cheveigne, A. de. Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds. Speech Commun. 27, 187–207 (1999)

WORLD: Morise, M., Yokomori, F. & Ozawa, K. WORLD: A vocoder-based high-quality speech synthesis system for real-time applications. IEICE Trans. (2016)

Marianna made the marmalade

H*

L-L%

M AA R IY AA N AH [ ] M EY D DH_AH[ ]M AA R M AH L EY _D

…

Acoustic feature vector

Mel-cepstrum coefficients

F0

Other features…

Vocoders

STRAIGHT

WORLD

Glottal

Sinusoidal

24

25 of 147

Overview – HTS (HMM-based speech synthesis system)

Zen H. An example of context-dependent label format for HMM-based speech synthesis in English https://wiki.inf.ed.ac.uk/twiki/pub/CSTR/F0parametrisation/hts_lab_format.pdf (2006)

Tokuda, K., Zen, H. & Black, A. W. An {HMM}-based speech synthesis system applied to English. in Proc. SSW 227–230 (2002).

…

Marianna made the marmalade

H*

L-L%

…

M AA R IY AA N AH [ ] M EY D DH_AH[ ]M AA R M AH L EY _D

Context-dependent labels (or linguistic feature vector)

Previous previous phone: R

Previous phone: IY

Current phone: AA

Next phone: N

Next Next phone: AH

Stressed syllable: True

Word position: 1

…

25

26 of 147

Overview – HTS (HMM-based speech synthesis system)

Note: only one state is plotted for simplicity

DNN: deep neural networks

…

Marianna made the marmalade

H*

L-L%

…

M AA R IY AA N AH [ ] M EY D DH_AH[ ]M AA R M AH L EY _D

Input sequence is short,

Output sequence is long!?

Statistical parametric speech synthesis (SPSS)

Parametric: speech parameterized as acoustic features vectors

Statistical: decision trees, HMM, and DNN for input-to-output mapping

26

27 of 147

Overview – HTS (HMM-based speech synthesis system)

Note: only one state is plotted for simplicity

DNN: deep neural networks

…

Marianna made the marmalade

H*

L-L%

…

M AA R IY AA N AH [ ] M EY D DH_AH[ ]M AA R M AH L EY _D

Input sequence is short,

Output sequence is long!?

27

28 of 147

Overview – HTS (HMM-based speech synthesis system)

Note: only one state is plotted for simplicity

…

Marianna made the marmalade

H*

L-L%

…

M AA R IY AA N AH [ ] M EY D DH_AH[ ]M AA R M AH L EY _D

Vowel?

Phone is AA?

HMM model

(state)

N

Y

N

Y

Decision tree(s)

28

29 of 147

Overview – HTS (HMM-based speech synthesis system)

Better methods: Zen, H., Tokuda, K., Masuko, T., Kobayasih, T. & Kitamura, T. A hidden semi-Markov model-based speech synthesis system. IEICE Trans. Inf. Syst. 90, 825–834 (2007)

For training: forward-backward algorithm & statistics accumulation

…

Marianna made the marmalade

H*

L-L%

…

M AA R IY AA N AH [ ] M EY D DH_AH[ ]M AA R M AH L EY _D

…

Regression is much simpler

Duration prediction

(explicit way)

29

30 of 147

Overview – HTS (HMM-based speech synthesis system)

Tokuda, K., Yoshimura, T., Masuko, T., Kobayashi, T. & Kitamura, T. Speech parameter generation algorithms for HMM-based speech synthesis. in Proc. ICASSP 936–939 (2000).

…

Marianna made the marmalade

H*

L-L%

…

M AA R IY AA N AH [ ] M EY D DH_AH[ ]M AA R M AH L EY _D

…

Maximum likelihood parameter generation

30

31 of 147

Overview – HTS (HMM-based speech synthesis system)

Tokuda, K., Yoshimura, T., Masuko, T., Kobayashi, T. & Kitamura, T. Speech parameter generation algorithms for HMM-based speech synthesis. in Proc. ICASSP 936–939 (2000).

Training requires forward-backward algorithm of HMM. See Rabiner, L. R. A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77, 257–286 (1989)

…

H*

L-L%

…

M AA R IY AA N AH [ ] M EY D DH_AH[ ]M AA R M AH L EY _D

…

Marianna made the marmalade

Waveform generation (vocoder)

Explicit duration prediction�(alignment for generation)

Linguistic feature extraction

Decision tree searching

Maximum likelihood parameter generation

Aligned!

31

32 of 147

Overview – HTS & DNN

Zen, H., Senior, A. & Schuster, M. Statistical parametric speech synthesis using deep neural networks. in Proc. ICASSP 7962–7966 (2013).

Fan, Y., Qian, Y., Xie, F. & Soong, F. K. TTS Synthesis with Bidirectional LSTM Based Recurrent Neural Networks. in Proc. Interspeech 1964–1968 (2014).

…

H*

L-L%

…

M AA R IY AA N AH [ ] M EY D DH_AH[ ]M AA R M AH L EY _D

Marianna made the marmalade

Waveform generation (vocoder)

Linguistic feature extraction

…

Deep neural networks (Feedforward, RNN …)

Explicit duration prediction

(alignment for generation)

Aligned!

32

33 of 147

Overview – HTS & DNN

Graves, A. Sequence Transduction with Recurrent Neural Networks. in Proc. ICML (2012).

…

H*

L-L%

…

M AA R IY AA N AH [ ] M EY D DH_AH[ ]M AA R M AH L EY _D

Marianna made the marmalade

…

Aligned!

RNNs are usually restricted to problems where the alignment between the input and output sequence is known in advance ^{(Graves 2012)}

HMM is needed for alignment learning

33

34 of 147

Overview – Seq-to-seq models

…

H*

L-L%

…

M AA R IY AA N AH [ ] M EY D DH_AH[ ]M AA R M AH L EY _D

Marianna made the marmalade

Joint alignment & regression

Single neural network for

Joint alignment & regression & linguistic analysis

34

35 of 147

Recent Seq-to-seq TTS��How do they work?

35

36 of 147

Seq-to-seq model

Task

M

a

i

r

a

n

a

n

…

Seq-to-seq model

Derive linguistic features from input
Learn & generate alignment
Generate output sequence

Attention

mechanism

36

37 of 147

Seq-to-seq model

Task

M

a

i

r

a

n

a

n

…

Seq-to-seq model

Proposed by machine translation community

Wide application in speech recognition

37

38 of 147

Seq-to-seq model

Task

38

39 of 147

Seq-to-seq model

Sutskever, I., Vinyals, O. & Le, Q. V. Sequence to sequence learning with neural networks. in Proc. NIPS 3104–3112 (2014).

Kalchbrenner, N. & Blunsom, P. Recurrent Continuous Translation Models. in Proc. EMNLP 1700–1709 (Association for Computational Linguistics, 2013).

Encoding-decoding

Simple & effective for short sequence
Single code c
Long sequence?

a scalar or vector

Encoder

Decoder

39

40 of 147

Seq-to-seq model

Note: decodor is simplified for presentation. See appendix – Bahdanau model for details

Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. in Proc. ICLR (2015).

Attention

Softmax

40

41 of 147

Seq-to-seq model

Note: decodor is simplified for presentation. See appendix – Bahdanau model for details

Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. in Proc. ICLR (2015).

Attention

Softmax

41

42 of 147

Seq-to-seq model

Note: decodor is simplified for presentation. See appendix – Bahdanau model for details

Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. in Proc. ICLR (2015).

Attention

Softmax

42

43 of 147

Seq-to-seq model

Note: decodor is simplified for presentation. See appendix – Bahdanau model for details

Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. in Proc. ICLR (2015).

Attention

Softmax

Alignment matrix

43

44 of 147

Seq-to-seq model

Attention

Softmax

Encoder

Alignment!

44

45 of 147

Recent Seq-to-seq TTS��Attention mechanism

45

46 of 147

Attention

Soft attention

Hard attention

Location-based attention

Content-based attention

Forward attention

Self-attention

Dot attention

Additive attention

Global attention

Local attention

Posterior attention

Monotonic attention

46

47 of 147

Attention

What features to compute

Content-based

Location-based

Location-aware

How to compute

Scaled-dot

Additive

Dot

Constraints on

Local

Monotonic / forward

Global

Self-attention

(content from x)

Soft-attention

(Deterministic attention)

47

48 of 147

Attention

Deterministic & stochastic attention: Xu, K. et al. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. in Proc. ICML (eds. Bach, F. & Blei, D.) vol. 37 2048–2057 (PMLR, 2015).

What features to compute

Content-based

Location-based

Location-aware

Scaled-dot

How to compute

Additive

Dot

Constraints on

Local

Global

Monotonic / forward

Soft-attention

(Deterministic attention)

Hard-attention

(Stochastic attention)

Alignment as latent

48

49 of 147

Soft attention

Dot, scaled-dot, additive

Luong, T., Pham, H. & Manning, C. D. Effective Approaches to Attention-based Neural Machine Translation. in Proc. EMNLP 1412–1421 (2015).

Vaswani, A. et al. Attention is all you need. in Proc. NIPS 5998–6008 (2017).

Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. in Proc. ICLR (2015).

Softmax

How to compute?

Dot ^{(Luong 2015, Eq(8))}:

Additive ^{(Bahdanau 2015, A2.2)}:

Scaled dot ^{(Vaswani 2017, Eq.(1))}:

49

50 of 147

Soft attention

Luong, T., Pham, H. & Manning, C. D. Effective Approaches to Attention-based Neural Machine Translation. in Proc. EMNLP 1412–1421 (2015).

Vaswani, A. et al. Attention is all you need. in Proc. NIPS 5998–6008 (2017).

Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. in Proc. ICLR (2015).

Dot, scaled-dot, additive

Softmax

Dot ^{(Luong 2015, Eq(8))}:

Additive ^{(Bahdanau 2015, A2.2)}:

Scaled dot ^{(Vaswani 2017, Eq.(1))}:

How to compute?

50

51 of 147

Soft attention

Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. in Proc. ICLR (2015).

Content, location, …

Softmax

What features?

Content-based^{(Bahdanau 2015, Eq(6))}

Dot, scaled-dot, or additive

51

52 of 147

Soft attention

Chorowski, J. K., Bahdanau, D., Serdyuk, D., Cho, K. & Bengio, Y. Attention-based models for speech recognition. in Proc. NIPS 577–585 (2015).

Content, location, …

Softmax

Content-based^{(Bahdanau 2015, Eq(6))}

Location-aware^{(Chorowski 2015, Eq.(9)}

52

53 of 147

Soft attention

Graves, A. Generating sequences with recurrent neural networks. arXiv Prepr. arXiv1308.0850 (2013)

Graves, A., Wayne, G. & Danihelka, I. Neural turing machines. arXiv Prepr. arXiv1410.5401 (2014)

Content, location, …

Softmax

Content-based^{(Bahdanau 2015, Eq(6))}

Location-aware^{(Chorowski 2015, Eq.(9)}

Location-based^{(Graves 2013, Eq(46-52)}

53

54 of 147

Soft attention

Graves, A. Generating sequences with recurrent neural networks. arXiv Prepr. arXiv1308.0850 (2013)

Graves, A., Wayne, G. & Danihelka, I. Neural turing machines. arXiv Prepr. arXiv1410.5401 (2014)

Content / location / …

Softmax

Content-based^{(Bahdanau 2015, Eq(6))}

Location-aware^{(Chorowski 2015, Eq.(9)}

Location-based^{(Graves 2013, Eq(46-52)}

Attention parameterized as GMM

54

55 of 147

Soft attention

Graves, A. Generating sequences with recurrent neural networks. arXiv Prepr. arXiv1308.0850 (2013)

Graves, A., Wayne, G. & Danihelka, I. Neural turing machines. arXiv Prepr. arXiv1410.5401 (2014)

Content, location, …

Softmax

Content-based^{(Bahdanau 2015, Eq(6))}

Location-aware^{(Chorowski 2015, Eq.(9)}

Location-based^{(Graves 2013, Eq(46-52)}

Location-based 2 ^{(Luong 2015, Eq(9))}:

55

56 of 147

Soft attention

Graves, A. Generating sequences with recurrent neural networks. arXiv Prepr. arXiv1308.0850 (2013)

Graves, A., Wayne, G. & Danihelka, I. Neural turing machines. arXiv Prepr. arXiv1410.5401 (2014)

Content, location, …

Softmax

Content-based^{(Bahdanau 2015, Eq(6))}

Location-aware^{(Chorowski 2015, Eq.(9)}

Location-based^{(Graves 2013, Eq(46-52)}

Location-based 2 ^{(Luong 2015, Eq(9))}:

Compute e_n,m for every m

56

57 of 147

Soft attention

Luong, T., Pham, H. & Manning, C. D. Effective Approaches to Attention-based Neural Machine Translation. in Proc. EMNLP 1412–1421 (Association for Computational Linguistics, 2015). doi:10.18653/v1/D15-1166

Chorowski, J. K., Bahdanau, D., Serdyuk, D., Cho, K. & Bengio, Y. Attention-based models for speech recognition. in Proc. NIPS 577–585 (2015).

Global / local / …

Softmax

Global ^{(Luong 2015)}:

The idea of a global attentional model is to consider all the hidden states of the encoder

Local ^{(Luong 2015, Chorowski 2015 Sec2.3)}:

a small window of context

for some m

57

58 of 147

Soft attention

What features to compute

Content-based

Location-based

Location-aware

Scaled-dot

How to compute

Additive

Dot

Constraints on

Local

Monotonic/forward

Global

Self-attention

(content from x)

Self-attention

58

59 of 147

Soft attention

Vaswani, A. et al. Attention is all you need. in Proc. NIPS 5998–6008 (2017).

Self-attention

Softmax

Global, scaled-dot

Content-based

Self-input N = M

59

60 of 147

Soft attention

Vaswani, A. et al. Attention is all you need. in Proc. NIPS 5998–6008 (2017).

Self-attention

Softmax

Everything can be parallel�In matrix form�

Query, Key, Value

60

61 of 147

Soft attention

Vaswani, A. et al. Attention is all you need. in Proc. NIPS 5998–6008 (2017).

“Self-attention” for alignment

Softmax

Query, Key, Value

Decoder pre-net

61

62 of 147

Soft attention

Battenberg, E. et al. Location-relative attention mechanisms for robust long-form speech synthesis. in Proc. ICASSP 6194–6198 (2020).

What features to compute

Content-based

Location-based

Location-aware

Scaled-dot

How to compute

Additive

Dot

Constraints on

Local

Monotonic/forward

Global

Self-attention

(content from x)

More variants are explored in

(Battenberg 2020)

62

63 of 147

Soft attention

TTS systems

Softmax

Decoder (autogressive)

Attention

Encoder

Phoneme / characters

Acoustic feature vectors

63

64 of 147

Soft attention

TTS systems

Softmax

Decoder (autogressive)

Attention

Encoder

Free to add more layers

Phoneme / characters

Acoustic feature vectors

64

65 of 147

Soft attention

Wang, Y. et al. Tacotron: Towards End-to-End Speech Synthesis. in Proc. Interspeech 4006–4010 (2017).

TTS systems

Softmax

Free to add more layers

Phoneme / characters

Acoustic feature vectors

Decoder (autogressive)

Attention

Encoder

Tacotron

65

66 of 147

Soft attention

Wang, W., Xu, S., Xu, B. & others. First step towards end-to-end parametric TTS synthesis: Generating spectral parameters with neural attention. in Proc. Interspeech 2243–2247 (2016).

Sotelo, J. et al. Char2wav: End-to-end Speech Synthesis. in Proc. ICLR (Workshop Track) (2017).

Ping, W. et al. Deep voice 3: Scaling text-to-speech with convolutional sequence learning. in Proc. ICLR (2018).

Tachibana, H., Uenoyama, K. & Aihara, S. Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention. in Proc. ICASSP 4784–4788 (2018).

Shen, J. et al. Natural TTS synthesis by conditioning WaveNet on Mel spectrogram predictions. in Proc. ICASSP 4779–4783 (2018).

Li, N., Liu, S., Liu, Y., Zhao, S. & Liu, M. Neural speech synthesis with transformer network. in Proceedings of the AAAI Conference on Artificial Intelligence vol. 33 6706–6713 (2019).

Taigman, Y., Wolf, L., Polyak, A. & Nachmani, E. Voiceloop: Voice fitting and synthesis via a phonological loop. in Proc. ICLR (2018).

TTS systems – A few examples

Before Tacotron:

Alex Graves’ work in 2015 ^{(https://youtu.be/-yX1SYeDHbg?t=2331)}
Wenfu Wang’s Mandarin TTS in 2016 ^{(Wang 2016)}

Similar encoder-attention-decoder structure

Char2wav ^{(Sotelo 2017)}
Tacotron 2 ^{(Shen 2018)}
DCTTS ^{(Tachibana 2018)}
Deep voice 3 ^{(Ping 2018)}
Transformer TTS ^{(Li 2019)}

Memory buffer + attention: voiceloop ^{(Taigman 2018)} . See appendix for illustration

66

67 of 147

Soft attention

Similar table in page 40: https://www.slideshare.net/jyamagis/tutorial-on-endtoend-texttospeech-synthesis-part-2-tactron-and-related-endtoend-systems

TTS systems – A few examples

	Tacotron	Tacotron2	Char2wav	DCTTS	DeepVoice 3	Transformer	Voiceloop
Speaker	Single	Single	Multiple?	Single	Multiple	Single	Multiple
Waveform Gen.	Griffin-Lim	WaveNet	Sample RNN	Griffin-Lim	Multiple	WaveNet	WORLD
Acous. feat. y_1:N	Mel.spec	Mel.spec	Mel.cep, F0, BAP	Mel.spec	Multiple	Mel.spec	Mel.cep, F0, BAP
Post-net Core Pre-Net	CBHG	CNN	-	dilatedCNN	CNN	CNN	Memory buffer + Full-con.
	RNNs	LSTMs	RNN	dilatedCNN	CNN	Self.att.
	Full-con.	Full-con.	-	dilatedCNN	Full-con.	Self.att.
Attention	Additive, content	Additive, location-aw.	Location	Scale-dot, content	Scale-dot, content / monotonic	Scale-dot, content (self-att.)	Additive Location
Encoder	CBHG Full-con.	Bi-LSTM CNN	Bi-RNN	Highway dilatedCNN	CNN	Full-con. Self-att.	Full-con.
Input x_1:M	Character	Character	Character / phoneme	Character	Character / phoneme	Phoneme	Phoneme

Decoder

67

68 of 147

Soft attention

TTS systems – A few examples

Samples from official websites, public domain audio

Char2wav: http://josesotelo.com/speechsynthesis/, DCTTS: https://tachi-hi.github.io/tts_samples/,

DeepVoice3: http://research.baidu.com/Blog/index-view?id=91, Voiceloop https://ytaigman.github.io/loop/

	Tacotron	Tacotron2	Char2wav	DCTTS	DeepVoice 3	Transformer	Voiceloop
Speaker	Single	Single	Multiple?	Single	Multiple	Single	Multiple
VCTK	https://google.github.io/tacotron/ Astonishingly good!					https://neuraltts.github.io/transformertts/ Also good!

LJ-speech

Others

68

69 of 147

Soft attention

TTS systems – A few examples

Samples from official websites, public domain audio

Char2wav: http://josesotelo.com/speechsynthesis/, DCTTS: https://tachi-hi.github.io/tts_samples/,

DeepVoice3: http://research.baidu.com/Blog/index-view?id=91, Voiceloop https://ytaigman.github.io/loop/

	Tacotron	Tacotron2	Char2wav	DCTTS	DeepVoice 3	Transformer	Voiceloop
Speaker	Single	Single	Multiple?	Single	Multiple	Single	Multiple
VCTK	https://google.github.io/tacotron/ Astonishingly good!					https://neuraltts.github.io/transformertts/ Also good!

LJ-speech

Others

69

70 of 147

Soft attention

TTS systems – A few examples

Samples from official websites, public domain audio

Char2wav: http://josesotelo.com/speechsynthesis/, DCTTS: https://tachi-hi.github.io/tts_samples/,

DeepVoice3: http://research.baidu.com/Blog/index-view?id=91, Voiceloop https://ytaigman.github.io/loop/

	Tacotron	Tacotron2	Char2wav	DCTTS	DeepVoice 3	Transformer	Voiceloop
Speaker	Single	Single	Multiple?	Single	Multiple	Single	Multiple
VCTK	https://google.github.io/tacotron/ Astonishingly good!					https://neuraltts.github.io/transformertts/ Also good!

LJ-speech

Others

70

71 of 147

Soft attention

What features to compute

Content-based

Location-based

Location-aware

Scaled-dot

How to compute

Additive

Dot

Constraints on

Local

Monotonic/forward

Global

Self-attention

(content from x)

Other techniques

71

72 of 147

Soft attention with constraints

Global Alignment

Hard to learn, sometimes fail to work!

72

73 of 147

Soft attention with constraints

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

https://mutiann.github.io/papers/interspeech2019/

Samples & audios from (He 2019)

Crashes \ ReadAVs \ 00000000 \ foo , doc WriteAVs \ 11111111 \ foo 2 , doc: That makes post-processing a little painful since the fuzzer reports crashes in the hierarchical structure mentioned above.

73

74 of 147

Soft attention with constraints

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

https://mutiann.github.io/papers/interspeech2019/

Samples & audios from (He 2019)

Crashes \ ReadAVs \ 00000000 \ foo , doc WriteAVs \ 11111111 \ foo 2 , doc: That makes post-processing a little painful since the fuzzer reports crashes in the hierarchical structure mentioned above.

74

75 of 147

Soft attention with constraints

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

https://mutiann.github.io/papers/interspeech2019/

Samples & audios from (He 2019)

Crashes \ ReadAVs \ 00000000 \ foo , doc WriteAVs \ 11111111 \ foo 2 , doc: That makes post-processing a little painful since the fuzzer reports crashes in the hierarchical structure mentioned above.

75

76 of 147

Soft attention with constraints

More suitable for speech-related tasks?

Constrained alignment

76

77 of 147

Soft attention with constraints

Zhang, J.-X., Ling, Z.-H. & Dai, L.-R. Forward attention in sequence-to-sequence acoustic modeling for speech synthesis. in Proc. ICASSP 4789–4793 (2018).

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

77

78 of 147

Soft attention with constraints

Note: original paper uses y rather than \alpha to denote alignment

Zhang, J.-X., Ling, Z.-H. & Dai, L.-R. Forward attention in sequence-to-sequence acoustic modeling for speech synthesis. in Proc. ICASSP 4789–4793 (2018).

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

Forward & “monotonic”

Example on forward attention ^{(Zhang 2018, Algorithm 1)}

78

79 of 147

Soft attention with constraints

Zhang, J.-X., Ling, Z.-H. & Dai, L.-R. Forward attention in sequence-to-sequence acoustic modeling for speech synthesis. in Proc. ICASSP 4789–4793 (2018).

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

Forward & “monotonic”

Example on forward attention ^{(Zhang 2018, Algorithm 1)}

1

0

79

80 of 147

Soft attention with constraints

Zhang, J.-X., Ling, Z.-H. & Dai, L.-R. Forward attention in sequence-to-sequence acoustic modeling for speech synthesis. in Proc. ICASSP 4789–4793 (2018).

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

Forward & “monotonic”

Example on forward attention ^{(Zhang 2018, Algorithm 1)}

1

0

0.1

0.4

0.5

0.0 = (0 + 0) * 0.1

0.4 = (0 + 1) * 0.4

0.5 = (0 + 1) * 0.5

Alternatively ^{(Zheng 2018, algorithm 2)}

A tiny example

80

81 of 147

Soft attention with constraints

Zhang, J.-X., Ling, Z.-H. & Dai, L.-R. Forward attention in sequence-to-sequence acoustic modeling for speech synthesis. in Proc. ICASSP 4789–4793 (2018).

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

Forward & “monotonic”

Example on forward attention ^{(Zhang 2018, Algorithm 1)}

1

0

81

82 of 147

Soft attention with constraints

Zhang, J.-X., Ling, Z.-H. & Dai, L.-R. Forward attention in sequence-to-sequence acoustic modeling for speech synthesis. in Proc. ICASSP 4789–4793 (2018).

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

Forward & “monotonic”

Example on forward attention ^{(Zhang 2018)}

1

0

82

83 of 147

Soft attention with constraints

Note: (He 2019) can use strictly monotonic hard attention during inference.

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

Tjandra, A., Sakti, S. & Nakamura, S. Local Monotonic Attention Mechanism for End-to-End Speech And Language Processing. in Proc. IJCNLP 431–440 (2017).

Forward & “monotonic”

Similar idea but a more rigid parametric form ^{(He 2019)}
“Monotonic” + local attention ^{(Tjandar 2017)}
Alignment is NOT guaranteed to be monotonic

Still soft attention during inference

… using soft attention at inference as well … at the cost of strict guarantee of locality and monotonicity ^{(He 2019)}

83

84 of 147

Soft attention with constraints

Tachibana, H., Uenoyama, K. & Aihara, S. Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention. in Proc. ICASSP 4784–4788 (2018).

Chen, M. et al. MultiSpeech: Multi-Speaker Text to Speech with Transformer. arXiv Prepr. arXiv2006.04664 (2020)

Park, J., Han, K., Jeong, Y. & Lee, S. W. Phonemic-level duration control using attention alignment for natural speech synthesis. in Proc. ICASSP 5896–5900 (2019).

Battenberg, E. et al. Location-relative attention mechanisms for robust long-form speech synthesis. in Proc. ICASSP 6194–6198 (2020).

Other techniques

Diagonal “constraints” in training loss ^{(Tachibana 2018, Chen 2020)}

Alignment loss based on external alignment ^{(Park 2019)}
Prior bias to alignment ^{(Battenberg 2020)}

Figure 2 from (Chen 2020)

84

85 of 147

Soft attention with constraints

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

https://mutiann.github.io/papers/interspeech2019/

Good enough? – Sample 1

Crashes \ ReadAVs \ 00000000 \ foo , doc WriteAVs \ 11111111 \ foo 2 , doc: That makes post-processing a little painful since the fuzzer reports crashes in the hierarchical structure mentioned above.

Figure 4 & samples from (He 2019)

Forward attention

Baseline soft-attention

Sample in page 73

85

86 of 147

Soft attention with constraints

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

https://mutiann.github.io/papers/interspeech2019/

Good enough? – Sample 1

Crashes \ ReadAVs \ 00000000 \ foo , doc WriteAVs \ 11111111 \ foo 2 , doc: That makes post-processing a little painful since the fuzzer reports crashes in the hierarchical structure mentioned above.

Forward attention

Baseline soft-attention

Sample in page 73

Figure 4 & samples from (He 2019)

86

87 of 147

Soft attention with constraints

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

https://mutiann.github.io/papers/interspeech2019/

Good enough? – Sample 1

Figure 4 & samples from (He 2019)

Forward attention

Sample in page 73

Baseline soft-attention

Crashes \ ReadAVs \ 00000000 \ foo , doc WriteAVs \ 11111111 \ foo 2 , doc: That makes post-processing a little painful since the fuzzer reports crashes in the hierarchical structure mentioned above.

87

88 of 147

Soft attention with constraints

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

https://mutiann.github.io/papers/interspeech2019/

Good enough? – Sample 2

The preliminary ruling by Judge Lucy in the U.S. District Court for the Northern District of California said that Qualcomm must license some patents involved in making so-called modem chips, which help smart phones connect to wireless data networks, to rival chip firms. The ruling is a setback for Qualcomm because the chip company and the FTC had jointly asked the judge last month to delay ruling on the issue for up to 30 days while they pursued settlement talks.

Figure 4 & samples from (He 2019)

Forward attention

Baseline soft-attention

88

89 of 147

Soft attention with constraints

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

https://mutiann.github.io/papers/interspeech2019/

Good enough? – Sample 2

Soft-attention doesn’t guarantee monotonicity

The preliminary ruling by Judge Lucy in the U.S. District Court for the Northern District of California said that Qualcomm must license some patents involved in making so-called modem chips, which help smart phones connect to wireless data networks, to rival chip firms. The ruling is a setback for Qualcomm because the chip company and the FTC had jointly asked the judge last month to delay ruling on the issue for up to 30 days while they pursued settlement talks.

Figure 4 & samples from (He 2019)

Forward attention

Baseline soft-attention

89

90 of 147

Soft attention with constraints

Zhang, J.-X., Ling, Z.-H. & Dai, L.-R. Forward attention in sequence-to-sequence acoustic modeling for speech synthesis. in Proc. ICASSP 4789–4793 (2018).

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

Chen, M. et al. MultiSpeech: Multi-Speaker Text to Speech with Transformer. arXiv Prepr. arXiv2006.04664 (2020) https://speechresearch.github.io/multispeech/

Battenberg, E. et al. Location-relative attention mechanisms for robust long-form speech synthesis. in Proc. ICASSP 6194–6198 (2020). https://google.github.io/tacotron/publications/location_relative_attention/

TTS systems – a few examples

	Zhang 2018	He 2019	Chen 2020	Battenberg 2020
Base model	Tacotron	Tacotron 2	Transformer Multi-speaker	Tacotron
Attention	Forward attention	Stepwise “monotonic”	Diagonal constraints	Dynamic convoluion attention

Samples (VCTK, Lessac, LJ-speech)	Mandarin TTS https://jxzhanggg.github.io/ForwardAttention/	English TTS https://mutiann.github.io/papers/interspeech2019/

90

91 of 147

Recent Seq-to-seq TTS��Hard attention

91

92 of 147

From soft to hard attention

Alignment 1

Alignment 2

Alignment 3

92

93 of 147

From soft to hard attention

Alignment 1

Alignment 2

Alignment 3

Dynamic progamming, forward-backward, search ...

93

94 of 147

Hard vs Soft – Generation

…

Sampling (select)

Softmax

sum

Soft-attention

Hard-attention

94

95 of 147

Hard vs Soft – training

…

Dynamic programming

Softmax

sum

Soft-attention

Hard-attention

95

96 of 147

Hard attention

What features for

Content-based

Location-based

Location-aware

Scaled-dot

How to compute

Additive

Dot

Constraints on

Global

Local

Monotonic / forward

Soft-attention

Hard-attention

Probabilistic assumption

Joint output & alignment

Separate output & alignment

…

Sampling

Marginalizing

Training

Inference

Greedy

Beam

96

97 of 147

Hard attention

See slides later for more through explanation

Alignment 1

Alignment 2

Alignment 3

y_naligns with x_m

Separate output & align.

Probabilistic assumption

Joint output & alignment

Separate output & alignment

…

Sampling

Marginalizing

Training

Inference

Greedy

Beam

97

98 of 147

Hard attention

Separate output & align.

Probabilistic assumption

Joint output & alignment

Separate output & alignment

…

Sampling

Marginalizing

Training

Inference

Greedy

Beam

Depends on the current state (like HMM)

For simplicity,

x_1:M is ignored

98

99 of 147

Hard attention

For simplicity, x_1:M is ignored in all equations

Wu, S., Shapiro, P. & Cotterell, R. Hard Non-Monotonic Attention for Character-Level Transduction. in Proc. EMNLP 4425–4438 (Association for Computational Linguistics, 2018)

Wu, S. & Cotterell, R. Exact Hard Monotonic Attention for Character-Level Transduction. in Proc. ACL 1530–1537 (Association for Computational Linguistics, 2019).

Yu, L., Buys, J. & Blunsom, P. Online Segment to Segment Neural Transduction. in Proc. EMNLP 1307–1316 (Association for Computational Linguistics, 2016).

Separate output & align.

Probabilistic assumption

Joint output & alignment

Separate output & alignment

…

Sampling

Marginalizing

Training

Inference

Greedy

Beam

1-th order

^{(wu 2019, Yu2016)}

0-th order

^{(wu 2018, wu 2019)}

99

100 of 147

Hard attention

Yu, L., Buys, J. & Blunsom, P. Online Segment to Segment Neural Transduction. in Proc. EMNLP 1307–1316 (Association for Computational Linguistics, 2016).

Emission probability in original paper is slightly more complicated

Separate output & align.

1th order Markovian

1st way to parameterize transition ^{(Yu 2016, Eq.(9-10))}

Probabilistic assumption

Joint output & alignment

Separate output & alignment

…

Sampling

Marginalizing

Training

Inference

Greedy

Beam

x₃

x₂

x₁

y₁

y₂

y₃

x_m

y_n

Skip

Emit

100

101 of 147

Hard attention

Wu, S. & Cotterell, R. Exact Hard Monotonic Attention for Character-Level Transduction. in Proc. ACL 1530–1537 (Association for Computational Linguistics, 2019).

Separate output & align.

1th order Markovian

2nd way to parameterize transition ^{(Wu 2019, pp-1533)}

Probabilistic assumption

Joint output & alignment

Separate output & alignment

…

Sampling

Marginalizing

Training

Inference

Greedy

Beam

x₃

x₂

x₁

y₁

y₂

101

102 of 147

Hard attention

Yasuda, Y., Wang, X. & Yamagishi, J. Effect of choice of probability distribution, randomness, and search methods for alignment modeling in sequence-to-sequence text-to-speech synthesis using hard alignment. in Proc. ICASSP 6724–6728 (2020).

Separate output & align.

1th order Markovian

3nd way to parameterize transition ^{(Yasuda 2020, Eq(3))}

Probabilistic assumption

Joint output & alignment

Separate output & alignment

…

Sampling

Marginalizing

Training

Inference

Greedy

Beam

x₃

x₂

x₁

y₁

y₂

102

103 of 147

Hard attention

Separate output & align.

1th order Markovian

All monotonic, but what’s the difference?

Probabilistic assumption

Joint output & alignment

Separate output & alignment

…

Sampling

Marginalizing

Training

Inference

Greedy

Beam

x₃

x₂

x₁

y₁

y₂

y₃

x₃

x₂

x₁

y₁

y₂

x₂

x₁

y₁

y₂

Not normalized

Normalized

y₁, y₂, y₃

x₁, x₃, x₃

Allows skip

y₁, y₂

x₁, x₃

Allows skip

Most stricted

103

104 of 147

Hard attention

Lawrence R Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings ofthe IEEE, 77(2):257–286.

Separate output & align.

1th order Markovian

Training by marginalization & back-prop.

Probabilistic assumption

Joint output & alignment

Separate output & alignment

…

Sampling

Marginalizing

Training

Inference

Greedy

Beam

x₃

x₂

x₁

y₁

y₂

y₃

Forward-algorithm ^{(Rabiner 1989)}

104

105 of 147

Hard attention

Xu, K. et al. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. in Proc. ICML (eds. Bach, F. & Blei, D.) vol. 37 2048–2057 (PMLR, 2015).

Separate output & align.

1th order Markovian

Training by marginalization & back-prop.
Training by sampling ^{(Xu 2015, Sec 4.1)}

Probabilistic assumption

Joint output & alignment

Separate output & alignment

…

Sampling

Marginalizing

Training

Inference

Greedy

Beam

Not differentiable

Requires REINFORCE

x₂

x₁

y₁

y₂

y₃

105

106 of 147

Hard attention

Yasuda, Y., Wang, X. & Yamagishi, J. Initial investigation of encoder-decoder end-to-end TTS using marginalization of monotonic hard alignments. in Proc. SSW 211–216 (2019). doi:10.21437/SSW.2019-38

Separate output & align.

Inference (decoding)

Probabilistic assumption

Joint output & alignment

Separate output & alignment

…

Sampling

Marginalizing

Training

Inference

Greedy

Beam

x₃

x₂

x₁

y₁

y₂

y₃

Stop

…

^{(Yasuda 2019, Sec3.2)}

106

107 of 147

Hard attention

Joint output & alignemnt: CTC and RNN transducer

Probabilistic assumption

Joint output & alignment

Separate output & alignment

…

Sampling

Marginalizing

Training

Inference

Greedy

Beam

How to sum efficiently?

How to generate efficiently?

107

108 of 147

Hard attention

Yasuda, Y., Wang, X. & Yamagishi, J. Effect of choice of probability distribution, randomness, and search methods for alignment modeling in sequence-to-sequence text-to-speech synthesis using hard alignment. in Proc. ICASSP 6724–6728 (2020).

Yasuda, Y., Wang, X. & Yamagishi, J. Initial investigation of encoder-decoder end-to-end TTS using marginalization of monotonic hard alignments. in Proc. SSW 211–216 (2019). doi:10.21437/SSW.2019-38

TTS with hard attention ^{(Yasuda 2019, Yasuda 2020)}

Training: marginalization
Inference: search & tricks ^{(Yasuda 2020)}

x₃

x₂

x₁

y₁

y₂

y₃

Samples are available

(for Japanese TTS)

https://nii-yamagishilab.github.io/sample-ssnt-sampling-methods/

108

109 of 147

Recent Seq-to-seq TTS��Hybrid approaches

109

110 of 147

Hybrid approaches

He, M., Deng, Y. & He, L. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. in Proc. Interspeech 1293–1297 (2019). doi:10.21437/Interspeech.2019-1972

Yasuda, Y., Wang, X. & Yamagishi, J. Effect of choice of probability distribution, randomness, and search methods for alignment modeling in sequence-to-sequence text-to-speech synthesis using hard alignment. in Proc. ICASSP 6724–6728 (2020).

Why not only attention

Soft attention may fail to deal with long utterances

Hard attention may not be accurate enough

Figure 4 & samples from (He 2019)

Figure 4 from (Yasuda 2019)

110

111 of 147

Hybrid approaches

Why not only attention

Larget search space

Short

111

112 of 147

Hybrid approaches

Ren, Y. et al. Fastspeech: Fast, robust and controllable text to speech. in Proc. NIPS 3171–3180 (2019).

Yu, C. et al. Durian: Duration informed attention network for multimodal synthesis. arXiv Prepr (2019)

Zeng, Z., Wang, J., Cheng, N., Xia, T. & Xiao, J. AlignTTS: Efficient feed-forward text-to-speech system without explicit alignment. in Proc. ICASSP 6714–6718 (2020).

Training

Duration

Alignment matrix

Duration model

…

Waveform gen.

Decoder

Attention

Encoder

d₁= 1

d₂= 3

d₃= 2

d₄= 4

d₅= 2

d₆= 2

112

113 of 147

Hybrid approaches

Ren, Y. et al. Fastspeech: Fast, robust and controllable text to speech. in Proc. NIPS 3171–3180 (2019).

Yu, C. et al. Durian: Duration informed attention network for multimodal synthesis. arXiv Prepr (2019)

Zeng, Z., Wang, J., Cheng, N., Xia, T. & Xiao, J. AlignTTS: Efficient feed-forward text-to-speech system without explicit alignment. in Proc. ICASSP 6714–6718 (2020).

Generation: similar to DNN-HMM approach

…

Waveform gen.

Decoder

Attention

Encoder

…

Waveform gen.

Decoder

Encoder

…

Aligned!

“Up-sample”

113

114 of 147

Hybrid approaches

Ren, Y. et al. Fastspeech: Fast, robust and controllable text to speech. in Proc. NIPS 3171–3180 (2019). https://speechresearch.github.io/fastspeech/

Yu, C. et al. Durian: Duration informed attention network for multimodal synthesis. arXiv Prepr (2019)

Zeng, Z., Wang, J., Cheng, N., Xia, T. & Xiao, J. AlignTTS: Efficient feed-forward text-to-speech system without explicit alignment. in Proc. ICASSP 6714–6718 (2020). https://tencent-ailab.github.io/durian/

Ren, Y. et al. FastSpeech 2: Fast and High-Quality End-to-End Text-to-Speech. arXiv Prepr. arXiv2006.04558 (2020)

TTS systems – A few examples

FastSpeech^{(Ren 2019)}: soft-attention (Transformer TTS)
AlignTTS^{(Zeng 2020)} : hard-attention, 0^th order Markovian
DurIAN^{(Yu 2019)} & FastSpeech2^{(Ren 2020)}: HMM-aligner

	FastSpeech	AlignTTS	DurIAN	FastSpeech2
For alignment	Extracted from Transformer TTS	Learned by hard-attention	HMM-alignment?	HMM-alignment

Samples (open domain data)		No samples? (even trained on LJ-speech)	Mandarin https://tencent-ailab.github.io/durian/

114

115 of 147

Hybrid approaches

Ren, Y. et al. Fastspeech: Fast, robust and controllable text to speech. in Proc. NIPS 3171–3180 (2019). https://speechresearch.github.io/fastspeech/

Yu, C. et al. Durian: Duration informed attention network for multimodal synthesis. arXiv Prepr (2019)

Zeng, Z., Wang, J., Cheng, N., Xia, T. & Xiao, J. AlignTTS: Efficient feed-forward text-to-speech system without explicit alignment. in Proc. ICASSP 6714–6718 (2020). https://tencent-ailab.github.io/durian/

Ren, Y. et al. FastSpeech 2: Fast and High-Quality End-to-End Text-to-Speech. arXiv Prepr. arXiv2006.04558 (2020)

TTS systems – A few examples

FastSpeech^{(Ren 2019)}: soft-attention (Transformer TTS)
AlignTTS^{(Zeng 2020)} : hard-attention, 0^th order Markovian
DurIAN^{(Yu 2019)} & FastSpeech2^{(Ren 2020)}: HMM-aligner

	FastSpeech	AlignTTS	DurIAN	FastSpeech2
For alignment	Extracted from Transformer TTS	Learned by hard-attention	HMM-alignment?	HMM-alignment

Samples (open domain data)		No samples? (even trained on LJ-speech)	Mandarin https://tencent-ailab.github.io/durian/

115

116 of 147

Hybrid approaches

Ren, Y. et al. Fastspeech: Fast, robust and controllable text to speech. in Proc. NIPS 3171–3180 (2019). https://speechresearch.github.io/fastspeech/

Yu, C. et al. Durian: Duration informed attention network for multimodal synthesis. arXiv Prepr (2019)

Zeng, Z., Wang, J., Cheng, N., Xia, T. & Xiao, J. AlignTTS: Efficient feed-forward text-to-speech system without explicit alignment. in Proc. ICASSP 6714–6718 (2020). https://tencent-ailab.github.io/durian/

Ren, Y. et al. FastSpeech 2: Fast and High-Quality End-to-End Text-to-Speech. arXiv Prepr. arXiv2006.04558 (2020)

TTS systems – A few examples

FastSpeech^{(Ren 2019)}: soft-attention (Transformer TTS)
AlignTTS^{(Zeng 2020)} : hard-attention, 0^th order Markovian
DurIAN^{(Yu 2019)} & FastSpeech2^{(Ren 2020)}: HMM-aligner

	FastSpeech	AlignTTS	DurIAN	FastSpeech2
For alignment	Extracted from Transformer TTS	Learned by hard-attention	HMM-alignment?	HMM-alignment

Samples (open domain data)		No samples? (even trained on LJ-speech)	Mandarin https://tencent-ailab.github.io/durian/

116

117 of 147

Summary

117

118 of 147

Summary on Seq-to-seq TTS

van den Oord, A. et al. WaveNet: A generative model for raw audio. arXiv Prepr. arXiv1609.03499 (2016)

…

Acoustic model

Waveform gen.

M AA R IY AA (H*) …D

Front-end

…

Duration model

M

a

e

…

d

a

l

…

Waveform gen.

Seq-to-seq model

WaveNet

(by Deepmind)

M AA R IY AA (H*) …D

Front-end

…

Duration model

118

119 of 147

Summary on Seq-to-seq TTS

About autoregressive models, see:

Shannon, M., Zen, H. & Byrne, W. Autoregressive models for statistical parametric speech synthesis. IEEE Trans. Audio, Speech, Lang. Process. 21, 587–597 (2013)

Wang, X., Takaki, S. & Yamagishi, J. Autoregressive Neural F0 Model for Statistical Parametric Speech Synthesis. IEEE/ACM Trans. Audio, Speech, Lang. Process. 26, 1406–1419 (2018)

From pipeline to seq-to-seq

M

a

e

…

d

a

l

…

Acoustic model

Waveform gen.

M AA R IY AA (H*) …D

Front-end

…

Duration model

…

Waveform gen.

Seq-to-seq model

Trainable implicit

front-end

Duration through

attention & alignment

Autoregressive decoding

(not explained)

Neural waveform models

119

120 of 147

Summary on Seq-to-seq TTS

About autoregressive models, see:

Shannon, M., Zen, H. & Byrne, W. Autoregressive models for statistical parametric speech synthesis. IEEE Trans. Audio, Speech, Lang. Process. 21, 587–597 (2013)

Wang, X., Takaki, S. & Yamagishi, J. Autoregressive Neural F0 Model for Statistical Parametric Speech Synthesis. IEEE/ACM Trans. Audio, Speech, Lang. Process. 26, 1406–1419 (2018)

Attention mechanism in seq-to-seq models

…

Acoustic model

Waveform gen.

M AA R IY AA (H*) …D

Front-end

…

Duration model

M

a

e

…

d

a

l

…

Waveform gen.

Seq-to-seq model

Trainable implicit

front-end

Duration through

attention & alignment

Autoregressive decoding

(not explained)

Neural waveform models

Attention-based seq-to-seq model

What features for

Content-based

Location-based

Location-aware

Scaled-dot

How to compute

Additive

Dot

Constraints on

Global

Local

Monotonic / forward

Soft-attention

Probabilistic assumption

Joint output & alignment

Separate output & alignment

…

Sampling

Marginalizing

Training

Inference

Greedy

Beam

Hard-attention

Hybrid approach

less or no attention

For robustness

120

121 of 147

Summary on Seq-to-seq TTS

Henter, G. E., Merritt, T., Shannon, M., Mayo, C. & King, S. Measuring the perceptual effects of modelling assumptions in speech synthesis using stimuli constructed from repeated natural speech. in Proc. Interspeech 1504–1508 (2014).

Watts, O., Henter, G. E., Merritt, T., Wu, Z. & King, S. From HMMs to DNNs: where do the improvements come from? in Proc. ICASSP 5505–5509 (2016).

Watts, O., Henter, G. E., Fong, J. & Valentini-Botinhao, C. Where do the improvements come from in sequence-to-sequence neural TTS? in Proc. SSW vol. 10 217–222 (2019).

Attention mechanism in seq-to-seq models

M

a

e

…

d

a

l

…

Acoustic model

Waveform gen.

M AA R IY AA (H*) …D

Front-end

…

Duration model

…

Waveform gen.

Seq-to-seq model

Trainable implicit

front-end

Duration through

attention & alignment

Autoregressive decoding

(not explained)

Neural waveform models

Where do the improvements come from ^{(Watts 2019)}?

121

122 of 147

Other topics in TTS��based on seq-to-seq model

122

123 of 147

Other topics in TTS

Neural waveform generator

WaveNet “vocoder”, WaveRNN, etc.
LPCNet, GELP, etc.
☞ readling list

Speaker, style, emotion, etc.

Style token, etc.
Speaker embedding / vector, etc.

Prosody

M

a

e

…

d

a

l

…

Waveform gen.

Seq-to-seq model

123

124 of 147

Other topics in TTS

Lorenzo-Trueba, J. et al. Towards Achieving Robust Universal Neural Vocoding. in Proc. Interspeech 2019 181–185 (2019). doi:10.21437/Interspeech.2019-1424

Liu, L.-J., Ling, Z.-H., Jiang, Y., Zhou, M. & Dai, L.-R. WaveNet Vocoder with Limited Training Data for Voice Conversion. in Proc. Interspeech 2018 1983–1987 (2018). doi:10.21437/Interspeech.2018-1190

Jia, Y. et al. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. in Proc. NIPS 4480–4490 (2018).

Taigman, Y., Wolf, L., Polyak, A. & Nachmani, E. Voiceloop: Voice fitting and synthesis via a phonological loop. in Proc. ICLR (2018).

Cooper, E. et al. Zero-shot multi-speaker text-to-speech with state-of-the-art neural speaker embeddings. in Proc. ICASSP 6184–6188 (2020).

Cooper, Erica, et al. "Can Speaker Augmentation Improve Multi-Speaker End-to-End TTS?." arXiv preprint arXiv:2005.01245 (2020).

Gibiansky, A. et al. Deep voice 2: Multi-speaker neural text-to-speech. in Proc. NIPS 2962–2970 (2017).

Chen, M. et al. MultiSpeech: Multi-Speaker Text to Speech with Transformer. arXiv Prepr. arXiv2006.04664 (2020)

Multi-speaker (with limited data)

M

a

e

…

d

a

l

…

Waveform gen.

Seq-to-seq model

Speaker-independent ^{(Lorenzo-Trueba 2019)} or adapted to target speaker ^{(Liu 2018)}

Jointly trained speaker vectors

Voiceloop ^{(Tagiman 2018)}

Deep voice 2 ⁽^{Gibiansky 2017}⁾

And others ^{(Chen 2020, )}

Zero-shot (with separate speaker model)

Tacotron2 + ASV model ^{(Jia 2018)}

Tacotron + LDE ^{(Cooper 2020, Cooper 2020)}

Most methods use speaker vector

124

125 of 147

Other topics in TTS

Course 2.3: Veilleux, N., Shattuck-Hufnagel, S. & Brugos, A. 6.911 Transcribing Prosodic Structure of Spoken Utterances with ToBI. MIT Open Course Ware https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-911-transcribing-prosodic-structure-of-spoken-utterances-with-tobi-january-iap-2006/# (2006)

Veilleux, N., Shattuck-Hufnagel, S. & Brugos, A. 6.911 Transcribing Prosodic Structure of Spoken Utterances with ToBI. MIT Open Course Ware https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-911-transcribing-prosodic-structure-of-spoken-utterances-with-tobi-january-iap-2006/# (2006)

“Prosody”

Marianna made the marmalade

Who made the marmalade?

TTS

125

126 of 147

Other topics in TTS

Zhu, J. Probing the phonetic and phonological knowledge of tones in Mandarin TTS models. in Proc. 10th International Conference on Speech Prosody 2020 930–934 (2020). doi:10.21437/SpeechProsody.2020-190

Fujimoto, T., Hashimoto, K., Oura, K., Nankaku, Y. & Tokuda, K. Impacts of input linguistic feature representation on Japanese end-to-end speech synthesis. in Proc. SSW10 166–171 (2019).

Yasuda, Y., Wang, X., Takaki, S. & Yamagishi, J. Investigation of enhanced Tacotron text-to-speech synthesis systems with self-attention for pitch accent language. in Proc. ICASSP 6905–6909 (2019).

Skerry-Ryan, R. J. et al. Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron. in (eds. Dy, J. & Krause, A.) vol. 80 4693–4702 (PMLR, 2018).

Lee, Y. & Kim, T. Robust and fine-grained prosody control of end-to-end speech synthesis. in Proc. ICASSP 5911–5915 (2019).

Kenter, T., Wan, V., Chan, C.-A., Clark, R. & Vit, J. {CH}i{VE}: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network. in (eds. Chaudhuri, K. & Salakhutdinov, R.) vol. 97 3331–3340 (PMLR, 2019). Sun, G. et al. Generating diverse and natural text-to-speech samples using a quantized fine-grained vae and autoregressive prosody prior. in Proc. ICASSP 6699–6703 (2020).

“Prosody”

Lexical tone & pitch accent

Mandarin ^{(Zhu 2020)}
Japanese ^{(Yasuda 2019, Fujimoto 2019)}

Supra-segmental variation

Tacotron + Prosody embedding ^{(Skerry-Ryan 2018, Lee 2019)}
Variational auto-encoder ^{(Wan 2020, Sun 2020)}

126

127 of 147

Other topics in TTS

Entertainment

Challenging for traditional TTS

Kato, S. et al. Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences. IEEE Access 8, 138149–138161 (2020)

This photo is transformed from “DP3M2471” by Akira Kawamura licensed under CC BY 2.0.

https://nii-yamagishilab.github.io/samples-rakugo/201910_IEEE_access

Samples in Japanese

127

128 of 147

Thank your for listening

You can find this slide on

http://tonywangx.github.io/slide.html

It is recommeded to check

unexplained slides & reading list & appendix

128

129 of 147

Readling list

129

130 of 147

(My) Readling list

Books on TTS

Many topics on front-end & classical speech synthesis methods (e.g., formant synthesis, concatenative & signal processing)

Dutoit, T. An Introduction to Text-to-speech Synthesis. (Kluwer Academic Publishers, 1997).
Taylor, P. Text-to-Speech Synthesis. (Cambridge University Press, 2009).

Others:

Chapter 16, Huang, X., Acero, A., Hon, H.-W. & Reddy, R. Spoken Language Processing: A Guide to Theory, Algorithm, and System Development. (Prentice Hall PTR, 2001).
Chapter 8, Jurafsky, D. & Martin, J. H. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. (Prentice Hall PTR, 2000).

130

131 of 147

(My) Readling list

Tutorial & talk

On DNN-based statistical parametric speech synthesis

King, S., Watts, O., Ronanki, S., Espic, F., Wu, Z. Deep Learning for Text-to-Speech Synthesis, using the Merlin toolkit. Tutorial at Interspeech 2017 (Many details explained) http://speech.zone/courses/one-off/merlin-interspeech2017/
Qian, Y, Soong, F. Deep learning for speech generation and synthesis. Tutorial at ISCSLP 2014 https://www.superlectures.com/iscslp2014/tutorial-4-deep-learning-for-speech-generation-and-synthesis
Dr. Heigz Zen’s talk on DNN or HMM-based statistical parametric speech synthesis http://rtthss2015.talp.cat/ and more on http://www.sp.nitech.ac.jp/~zen/english/index.php?Publications%2FTalks

131

132 of 147

(My) Readling list

Tutorial & talk

More recent topics in statistical parametric speech synthesis:

Zen, H. Generative Model-Based Text-to-Speech Synthesis, CBMM Workshop https://cbmm.mit.edu/video/generative-model-based-text-speech-synthesis

On neural attention

Alex Graves: Attention & Memory, EEML 2020, https://www.youtube.com/watch?v=POtAGvwnB5s
Smola, A. Attention in deep learning, ICML 2019, http://alex.smola.org/talks/ICML19-attention.pdf
Alex Graves: Hallucination with RNNs https://www.youtube.com/watch?v=-yX1SYeDHbg

132

133 of 147

(My) Readling list

Papers on TTS front-end topics

Text normalization (classical, DNN):

Sproat, R. et al. Normalization of non-standard words. Comput. Speech Lang. 15, 287–333 (2001)
Sproat, R. & Jaitly, N. RNN approaches to text normalization: A challenge. arXiv Prepr. arXiv1611.00068 (2016)

Graphme-to-phoneme, letter-to-sound (DT, statistical, DNN):

Black, A. W., Lenzo, K. & Pagel, V. Issues in building general letter to sound rules. in Proc. SSW3 77–80 (1998).
Bisani, M. & Ney, H. Joint-sequence models for grapheme-to-phoneme conversion. Speech Commun. 50, 434–451 (2008)
Yao, K. & Zweig, G. Sequence-to-Sequence Neural Net Models for Grapheme-to-Phoneme Conversion. in Proc. Interspeech 3330–3334 (2015).

133

134 of 147

(My) Readling list

Papers on TTS front-end topics

Prosody: I found this book and paper useful as introdutction.

Gussenhoven, C. The phonology of tone and intonation. (Cambridge University Press, 2004).

Prosody labelling (e.g., ToBI):

Beckman, M. E. & Ayers, G. Guidelines for ToBI labelling. OSU Res. Found. 3, (1997)
Silverman, K. E. A. et al. ToBI: a standard for labeling English prosody. in Proc. ICSLP 867–870 (1992).

Predicton prosodic labels from text:

Hirschberg, J. Pitch accent in context predicting intonational prominence from text. Artif. Intell. 63, 305–340 (1993)

Unsupervised front-end:

Watts, O. S. Unsupervised learning for text-to-speech synthesis. (University of Edinburgh, 2013).

134

135 of 147

(My) Readling list

Overview papers on statistical parametric speech synthesis

HMM & decision tree

Tokuda, K. et al. Speech synthesis based on hidden Markov models. Proc. IEEE 101, 1234–1252 (2013)
Zen, H., Tokuda, K. & Black, A. W. Statistical parametric speech synthesis. Speech Commun. 51, 1039–1064 (2009)

DNN - HMM

Ling, Z. H. et al. Deep learning for acoustic modeling in parametric speech generation: A systematic review of existing techniques and future trends. IEEE Signal Process. Mag. 32, 35–52 (2015)

135

136 of 147

(My) Readling list

Neural waveform models for TTS

One talk

Wang, X. Neural waveform models for text-to-speech synthesis. https://tonywangx.github.io/slide.html

One of the latest Ph.D thesis; you may find many papers in references

Juvela, L. Neural waveform generation for source-filter vocoding in speech synthesis. Aalto Universiry (2020)

136

137 of 147

(My) Readling list

Neural waveform models for TTS

They are many papers besides WaveNet

SampleRNN: Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, Yoshua Bengio, “SampleRNN: An Unconditional End-to-End Neural Audio Generation Model”, ICLR 2017

WaveRNN: Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, Koray Kavukcuoglu, “Efficient Neural Audio Synthesis”, arXiv:1802.08435, 2018

FFTNet: Zeyu Jin, Adam Finkelstein, Gautham J. Mysore, Jingwan Lu, “FFTnet: A Real-Time Speaker-Dependent Neural Vocoder”, ICASSP 2018

Universal vocoder: Jaime Lorenzo-Trueba, Thomas Drugman, Javier Latorre, Thomas Merritt, Bartosz Putrycz, Roberto Barra-Chicote, Alexis Moinet, Vatsal Aggarwal, “Towards achieving robust universal neural vocoding,” Interspeech 2019

FloWaveNet Sungwon Kim, Sang-gil Lee, Jongyoon Song, and Sungroh Yoon. FloWaveNet: “A Generative Flow for Raw Audio” arXiv:1811.02155, 2018.

LPCNet: Jean-Marc Valin, Jan Skoglund, “LPCNet: Improving Neural Speech Synthesis Through Linear Prediction”, ICASSP 201

LP-Wavenet: Min-Jae Hwang, Frank Soong, Eunwoo Song, Xi Wang, Hyeonjoo Kang, Hong-Goo Kang, “LP-WaveNet: Linear Prediction-based WaveNet Speech Synthesis,” arXiv:1811.11913, 2018

ExcitNet: Eunwoo Song, Kyungguen Byun, Hong-Goo Kang, “ExcitNet vocoder: A neural excitation model for parametric speech synthesis systems”, EUSIPCO 2019

137

138 of 147

(My) Readling list

Neural waveform models for TTS

They are many papers besides WaveNet

GELP: Juvela, L., Bollepalli, B., Yamagishi, J. & Alku, P. GELP: GAN-excited linear prediction for speech synthesis from Mel-spectrogram. in Proc. Interspeech 694–699 (2019).

Parallel WaveGAN: Yamamoto, R., Song, E. & Kim, J.-M. Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. in Proc. ICASSP 6199–6203 (2020).

MelGAN: Kumar, K. et al. MelGAN: Generative adversarial networks for conditional waveform synthesis. in Proc. NIPS 14910–14921 (2019).

Hi-Net: Ai, Y. Ling, Z.H. A Neural Vocoder with Hierarchical Generation of Amplitude and Phase Spectra for Statistical Parametric Speech Synthesis, IEEE TASLP (2020)

ClariNet: Ping, W., Peng, K. & Chen, J. ClariNet: Parallel wave generation in end-to-end text-to-speech. in arXiv preprint arXiv:1807.07281 (2019).

Parallel WaveNet: van den Oord, A. et al. Parallel WaveNet: Fast high-fidelity speech synthesis. in Proc. ICML 3918–3926 (2018).

WaveGlow: Prenger, R., Valle, R. & Catanzaro, B. WaveGlow: A Flow-based Generative Network for Speech Synthesis. in Proc. ICASSP 3617–3621 (2019).

GlotNet: Prenger, R., Valle, R. & Catanzaro, B. WaveGlow: A Flow-based Generative Network for Speech Synthesis. in Proc. ICASSP 3617–3621 (2019).

138

139 of 147

(My) Readling list

Tools

List by ISCA-Synsig https://www.synsig.org/index.php/Software

Just search end-to-end TTS on github

139

140 of 147

Appendix

140

141 of 147

Appendix

Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. in Proc. ICLR (2015).

Soft-attention: Bahdanau’s model

Softmax

141

142 of 147

Appendix

Voiceloop (use notation in the paper)

d

k

N_a

Input

Encoding

LUT_p

Output

Spk. ID

LUT_s

F_o

F_u

Attention

Step1. compute attention

142

143 of 147

Appendix

Voiceloop (use notation in the paper)

d

k

N_u

N_a

Input

Encoding

LUT_p

Output

Spk. ID

LUT_s

F_o

F_u

Feedback

Step2. update memory

143

144 of 147

Appendix

Voiceloop (use notation in the paper)

d

k

N_u

N_a

Input

Encoding

LUT_p

Output

Spk. ID

LUT_s

N_o

F_o

+

F_u

Step3. output

144

145 of 147

Appendix

Yu, L., Buys, J. & Blunsom, P. Online Segment to Segment Neural Transduction. in Proc. EMNLP 1307–1316 (Association for Computational Linguistics, 2016). doi:10.18653/v1/D16-1138

Raffel, C., Luong, M.-T., Liu, P. J., Weiss, R. J. & Eck, D. Online and Linear-Time Attention by Enforcing Monotonic Alignments. in (eds. Precup, D. & Teh, Y. W.) vol. 70 2837–2846 (PMLR, 2017).

Hard alignment

Un-normalized transition probability

x₃

x₂

x₁

y₁

y₂

y₃

Eq (9) in (Yu 2016)

145

146 of 147

Appendix

Graves, A. Sequence Transduction with Recurrent Neural Networks. in Proc. ICML (2012).

Graves, A., Fernández, S., Gomez, F. & Schmidhuber, J. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. in Proc. ICML 369–376 (2006).

Hard alignment - joint output & alignment

CTC ^{(Graves 2006)}
RNN transducer ^{(Graves 2012)}

Fig1. RNN transducer ^{(Graves 2012)}

Emit labels

Probabilistic assumption

Joint output & alignment

Separate output & alignment

…

Sampling

Marginalizing

Training

Inference

Greedy

Beam

146

147 of 147

Appendix

Shankar, S. & Sarawagi, S. Posterior attention models for sequence to sequence learning. in Proc. ICLR (2018).

Hard alignment - Posterior attention model

Marginalize

Unfold

State assumption

Probabilistic assumption

Joint output & alignment

Separate output & alignment

Posterior attention

Sampling

Marginalizing

Training

Inference

Greedy

Beam

147