1 of 50

1

Using Cyclic Noise as Source Signal for Neural Source-Filter-based Speech Waveform Model

Xin WANG, Junichi YAMAGISHI

National Institute of Informatics, Japan

INTERSPEECH 2020

Paper 1018, INTERSPEECH 2020

Shanghai, China

Contact: wangxin@nii.ac.jp

2 of 50

Slides by Xin Wang

National Institute of Informatics

This work is licensed under the Creative Commons Attribution 3.0 license.

See http://creativecommons.org/ for details.

3 of 50

3

Introduction

4 of 50

4

Introduction

Task

Existing models

WaveNet, WaveRNN, WaveGlow, Parallel WaveNet / WaveGAN…
LPCNet, GELP, GlotNet, ExcitNet …

…

1

2

3

4

…

Waveform

values

Neural waveform models

Mel-spectrogram, F0, etc.

Reference in Appendix

5 of 50

5

Introduction

Neural source-filter waveform model (NSF)

Xin Wang, and Junichi Yamagishi. 2019. Neural Harmonic-plus-Noise Waveform Model with Trainable Maximum Voice Frequency for Text-to-Speech Synthesis. In Proc. SSW, 1–6. ISCA: ISCA.
Xin Wang, Shinji Takaki, and Junichi Yamagishi. 2020. Neural Source-Filter Waveform Models for Statistical Parametric Speech Synthesis. IEEE/ACM TASLP 28: 402–415. doi:10.1109/TASLP.2019.2956145.

1

2

3

4

…

T

…

Neural

filter

Source

Generated

waveform

6 of 50

6

Introduction

Neural source-filter waveform model (NSF)

Xin Wang, and Junichi Yamagishi. 2019. Neural Harmonic-plus-Noise Waveform Model with Trainable Maximum Voice Frequency for Text-to-Speech Synthesis. In Proc. SSW, 1–6. ISCA: ISCA.
Xin Wang, Shinji Takaki, and Junichi Yamagishi. 2020. Neural Source-Filter Waveform Models for Statistical Parametric Speech Synthesis. IEEE/ACM TASLP 28: 402–415. doi:10.1109/TASLP.2019.2956145.

1

2

3

4

…

T

…

Neural

filter

Source

Natural

waveform

1

2

3

4

…

T

Back-propagation

Multi-resolution spectral distance

Generated

waveform

7 of 50

7

Introduction

Question: sine-based source is the best?

Thomas Drugman, Paavo Alku, Abeer Alwan, and Bayya Yegnanarayana. "Glottal source processing: From analysis to applications.” Computer Speech & Language, 28(5):1117–1138, 2014.

(a) Speech waveform (b) LP residual

8 of 50

8

Proposed methods

9 of 50

9

Harmonic-plus-noise NSF with trainable sinc-filters

Spectral features & F0

Condition module

Source module

Frequency-domain distance

Generated

waveform

F0

Generated waveform

Gradients

Spectral & F0

Harmonic branch

HP

LP

+

MVF

Baseline

Noise branch

Xin Wang, and Junichi Yamagishi. 2019. Neural Harmonic-plus-Noise Waveform Model with Trainable Maximum Voice Frequency for Text-to-Speech Synthesis. In Proc. SSW, 1–6. ISCA: ISCA.
Details: https://www.dropbox.com/sh/gf3zp00qvdp3row/AABEVzUUqnJ4QbkxiQcjOhM5a/web/2019-ssw.pdf?raw=1

MVF: Maximum voiced frequency
HP & LP: high-pass, low-pass finite impulse response filter

Natural

waveform

10 of 50

10

Spectral features & F0

Condition module

Frequency-domain distance

Natural

waveform

Generated

waveform

Generated waveform

Gradients

Spectral & F0

Harmonic branch

Noise branch

HP

LP

+

MVF

Sine-based source signal

Baseline

Noise

FF

Sine

generator

harmonics

FF: fully-connected feedforward layer

Fundamental component

Random Initial phase

Sampling rate

Additive noise

11 of 50

11

Spectral features & F0

Condition module

Frequency-domain distance

Natural

waveform

Generated

waveform

Generated waveform

Gradients

Spectral & F0

Harmonic branch

Noise branch

HP

LP

+

MVF

Source module

F0

Motivation

Proposed methods (1/2)

Sine
Stable pitch Strong assumption on source signal

Random noise
No assumption on source signal Unstable pitch (see experiments)

12 of 50

12

Spectral features & F0

Condition module

Frequency-domain distance

Natural

waveform

Generated

waveform

Generated waveform

Gradients

Spectral & F0

Harmonic branch

Noise branch

HP

LP

+

MVF

Cyclic-noise-based

source signal

F0

Motivation

Proposed methods (1/2)

Sine
Stable pitch Strong assumption on source signal

Random noise
No assumption on source signal Unstable pitch (see experiments)

Periodicity

Randomness

13 of 50

13

Proposed methods (1/2)

Cyclic noise source module

F0

Impulse train

generation

Convolution

Gaussian noise

generation

Exponential decay

FF

Source module

Decayed noise

14 of 50

14

F0

Impulse train

generation

Convolution

Gaussian noise

generation

Exponential decay

FF

Source module

Proposed methods (1/2)

Cyclic noise source module

15 of 50

15

F0

Impulse train

generation

Convolution

Gaussian noise

generation

Exponential decay

FF

Source module

Proposed methods (1/2)

Cyclic noise source module

16 of 50

16

F0

Impulse train

generation

Convolution

Gaussian noise

generation

Exponential decay

FF

Source module

Cyclic noise source module

Proposed methods (1/2)

17 of 50

17

F0

Impulse train

generation

Convolution

Gaussian noise

generation

Exponential decay

FF

Source module

100 Hz

Cyclic noise source module

Proposed methods (1/2)

18 of 50

18

F0

Impulse train

generation

Convolution

Gaussian noise

generation

Exponential decay

FF

Source module

Cyclic noise source module

Proposed methods (1/2)

19 of 50

19

F0

Impulse train

generation

Convolution

Gaussian noise

generation

Exponential decay

FF

Source module

Cyclic noise source module

Proposed methods (1/2)

20 of 50

20

F0

Impulse train

generation

Convolution

Gaussian noise

generation

Exponential decay

FF

Source module

Cyclic noise source module

Proposed methods (1/2)

21 of 50

21

Cyclic noise source module

Noise is decayed by before the next impulse

Proposed methods (1/2)

22 of 50

22

Cyclic noise source module

Proposed methods (1/2)

23 of 50

23

Spectral features & F0

Condition module

Source module

Harmonic branch

HP

LP

+

MVF

Noise branch

Hidden features

Frequency-domain distance

Natural

waveform

Generated waveform

Proposed methods (1/2)

24 of 50

24

Frequency-domain distance

Source module

Natural

waveform

Hidden features

Harmonic branch

HP

LP

+

MVF

Noise branch

Up-samp.

F0

Mel-spectrogram

Bi-LSTM & CONV

Concat. & up-samp.

Up-samp. & merge

F0 to UV

Generated waveform

Differentiable !

Proposed methods (1/2)

25 of 50

25

Frequency-domain distance

Source module

Natural

waveform

Hidden features

Harmonic branch

HP

LP

+

MVF

Noise branch

Up-samp.

F0

Mel-spectrogram

Bi-LSTM & CONV

Concat. & up-samp.

Up-samp. & merge

F0 to UV

Generated waveform

Additional masked spectral loss (see paper)

Masked spectral loss

Sines

mask

Proposed methods (2/2)

26 of 50

26

Experiments

27 of 50

27

Experiments

Data & features

CMU-ARCTIC: SLT, BDL, RMS, CLB
Input Features: Mel-spectrogram (80 dim), F0

Models

Speaker independent training, w/o speaker vector

28 of 50

28

Results

Copy-synthesis
Crowd-sourcing, 710 evaluators did 792 test sets
Each system received 1584 MOSs

Average results over all four speakers

https://nii-yamagishilab.github.io/samples-nsf/nsf-v4.html

* p > 0.01

*

Experiments

Baseline source

Cyclic noise source

29 of 50

29

https://nii-yamagishilab.github.io/samples-nsf/nsf-v4.html

BDL (male)

RMS (male)

CLB (female)

SLT (female)

Experiments

30 of 50

30

https://nii-yamagishilab.github.io/samples-nsf/nsf-v4.html

BDL (male)

RMS (male)

CLB (female)

SLT (female)

Impulse train noise

Experiments

31 of 50

31

BDL

Natural

Sin

He was manifestly distressed by my coming.

Experiments

32 of 50

32

Summary

33 of 50

33

Summary

Previously:

Sine source

Proposed:

Cyclic noise

source

Period by F0

Noise

Source

module

Neural filter module

34 of 50

34

Samples (CMU & VCTK)

https://nii-yamagishilab.github.io/samples-nsf/index.html

Code, scripts & Jupyter notebook tutorial

https://github.com/nii-yamagishilab/project-NN-Pytorch-scripts

This and related slides

https://tonywangx.github.io/slide.html

Thank you for your attention

35 of 50

35

Appendix

36 of 50

36

Cyclic noise-based source signals

Additional training criterion

Masked spectral loss

Spectral features & F0

Condition module

Source module

Frequency-domain distance

Natural

waveform

Generated waveform

HP

LP

+

MVF

Noise branch

Hidden features

Masked spectral loss

Sines

mask

b1

b2

b3

b4

b5

37 of 50

37

Cyclic noise-based source signals

Additional training criterion

Masked spectral loss

Spectral features & F0

Condition module

Source module

Natural

waveform

HP

LP

+

MVF

Noise branch

Hidden features

Masked spectral loss

b1

b2

b3

b4

b5

Sine mask

Sines

mask

38 of 50

38

Source signals (BDL_0474)

upsampled F0

Sine + noise

Pulse + noise

bdl_0474

39 of 50

39

Source signals (BDL_0474)

upsampled F0

cyclic noise \beta = 0.21

cyclic noise \beta = 0.43

40 of 50

40

Source signals (BDL_0474)

upsampled F0

cyclic noise \beta = 0.86

cyclic noise \beta = 1.74

41 of 50

41

Source signals (SLT_0474)

upsampled F0

Sine + noise

Pulse-train + noise

42 of 50

+

Noise branch

b1

b2

b3

b4

b5