2023.04.28

제네시스랩 AI연구팀

신재영

Interspeech 2022

https://daisyqk.github.io/Automatic-Prosody-Annotation_w/

Motivation

Introduction

• In expressive text-to-speech synthesis (TTS), prosody modeling plays an important role in synthesizing high naturalness and intelligibility speech.

​

• However, the acquisition of prosodic boundary labels relies on manual annotation, which is costly and time-consuming.

​

• In this paper, we propose to automatically extract prosodic boundary labels from text-audio data.

What Is Prosody?

Background

​

• Prosodic Features of Speech
• Pause – used to separate grammatical construction (ex. ‘.’, ‘,’)
• Pitch – different pitch level or intonation changes meaning (ex. ‘?’, ‘!’)
• Stress – degree of emphasis given to a speech to provide context of meaning in specific language
• Volume – apart from the slight increase in loudnesss to indicate stress, volume is used to convey emotions
• Tempo – the speed of voice

�

https://www.youtube.com/watch?v=spemPRr3IYM

https://www.youtube.com/watch?v=LHLbvdt9Xyg

Prosodic boundaries of Mandarin

Introduction

​

• CC (Character): smallest unit of Chinese (ex. 사)
• LW (Lexicon Word) : syntactic boundary between words (ex. 사과)
• PW : Prosodic Word (ex. 사|과)
• PPH : Prosodic Phrase (ex. 사과를)
• IPH : Intonational Phrase (ex. 사과를 먹는다.)
• Prosodic Word (PW), Prosodic Phrase (PPH) and Intonational Phrase (IPH) correspond to three different lengths of pause in speech from short to long

The hierarchical prosody annotation adopted in this work categorizes the prosodic boundaries of Mandarin speech into five levels, which from low to high are Character (CC), Lexicon Word (LW), Prosodic Word (PW), Prosodic Phrase (PPH) and Intonational Phrase (IPH). Prosodic Word (PW), Prosodic Phrase (PPH) and Intonational Phrase (IPH) correspond to three different lengths of pause in speech from short to long. Lexicon Word (LW) indicates syntactic boundary between words, and Chinese Character (CC) is the smallest unit of Chinese.

Architecture

Overall Structure

Architecture

Audio Encoder

​

• PPG (Phonetic posteriogram) extractor
• The PPG extractor transforms acoustic features to the frame based posterior probabilities of phonetic classes (phonemes)
• PPGs can retain acoustic information while excluding speaker identities
• We believe that PPG can represent the duration and transition information of each phoneme in the audio, which is important to prosodic boundary estimation

​

​

�

An example posteriorgram for the utterance “I had to do that.”

Mark Dredze, et al. “NLP on Spoken Documents without ASR”, EMNLP 2020

Architecture

Audio Encoder

​

• PPG (Phonetic posteriogram) extractor
• A conformer-based model consisting of 2 convolutional layers and 12 conformer blocks are adopted to build the pre-trained PPG-based encoder.
• The phone alignment generated from a GMM-HMM model is used as the training target.
• The PPG extractor is trained to classify frame-based MFCCs to the corresponding phoneme classes by minimizing cross-entropy loss.

�

Shengkui Zhao, et al. “TOWARDS NATURAL AND CONTROLLABLE CROSS-LINGUAL VOICE CONVERSION BASED ON NEURAL TTS MODEL AND PHONETIC POSTERIORGRAM”, EMNLP 2020

Architecture

Audio Encoder

​

• Limitation of PPG (Phonetic posteriogram) extractor
• However, as prosodic boundaries are usually related to the word and character language information
• The phoneme-based PPG model misses the characterlevel language context information
• For example, “大 学生物，必修课” and “大学生, 务必修课” have the same phone sequence, however, their prosodic boundaries are different. (“아버지 가방에 들어가신다. / 아버지가 방에 들어가신다.)

�

Mark Dredze, et al. “NLP on Spoken Documents without ASR”, EMNLP 2020

Architecture

Audio Encoder

​

• Character-based encoder
• The character-based model transforms each input time frame to posterior probabilities of character classes, which can better keep the character-level information
• Two character-based ASR models based on CNN and conformer architectures are investigated.
• CNN-based model focuses on the local information, while the conformer-based model considers the whole utterance information

​

​

�

Architecture

Audio Encoder

​

• Character-based encoder
• The CNN Character-based encoder only contains two 2D convolution layers and a linear output layer.
• The conformer-based encoder uses the same model architecture as the PPG encoder, except the final output layer
• The character based audio encoder is optimized using CTC cost function in an End-to-End manner

​

​

�

Architecture

Multi-Modal Fusion Decoder

​

• With the audio and text hidden representations, a multi-modal fusion decoder is needed to fuse these two representations and then estimate the prosodic boundary
• Since the length of output prosodic boundary estimation is the same as the length of the input text sequence,
• in cross attention, the text modality is used as query, the audio modality is used as key and value

​

​

​

�

​

• Such a cross attention layer is designed to allow the model to automatically learn the alignment of the audio and text hidden inputs

Method

Training Objective

​

• The fused hidden vector H is then fed into an output linear layer with softmax function to obtain the probability distribution of prosodic boundaries
• Cross Entropy (CE) criterion is adopted as the training objective of the proposed model:�

Automatic Evaluation

Results

Human Evaluation

Results

Contribution

Results

• The proposed text-speech prosody annotation framework significantly outperforms text-only baselines

​

• The quality of automatic prosodic boundary annotations is comparable to human annotations

​

• TTS systems trained with model-annotated boundaries are slightly better than systems that use manual ones

THANK YOU FOR LISTENING