1 of 17

2023.06.09

제네시스랩 AI연구팀

신재영

ICASSP 2022

2 of 17

What Is Grapheme-to-Phoneme?

Background

predicting the pronunciation of words from their spellings

Considering that a pronunciation dictionary can never cover all possible words in a language, G2P conversion is essential for any applications that depend on the mapping relationship between the spoken and written forms of a language

https://ko.wikipedia.org/wiki/%EC%9C%84%ED%82%A4%EB%B0%B1%EA%B3%BC:IPA

https://github.com/sigmorphon/2021-task1

https://github.com/Kyubyong/g2pK

3 of 17

Limitations of Previous Studies

Introduction

Nevertheless, building neural G2P models usually relies on a large, language-specific pronunciation dictionary, which may not be available for a lot of languages

Previous approaches to address this issue are mainly focused on utilizing the data resources of other languages of other modals to improve the performance of limited-resource G2P conversion

4 of 17

Motivation

Introduction

This paper proposes a pre-trained grapheme model named grapheme BERT (GBERT) to improve the Transformer-based G2P model.

Inspired by the success of the pre-trained BERT, which provides contextual word representations, this paper proposes a pretrained grapheme model called grapheme BERT (GBERT)

The training of GBERT requires only easily accessible word lists.

5 of 17

GBERT

Introduction

GBERT is designed to capture the contextual relationship among the graphemes in a word, which is essential for the G2P task since the same grapheme may have different pronunciations due to different contexts.

The pre-training task of GBERT is a masked grapheme prediction task, i.e, predicting the masked graphemes from the seen graphemes in a word.

Furthermore, two approaches are developed to improve the Transformer-based G2P model with GBERT. They are fine-tuning GBERT and fusing GBERT into the Transformer model by attention.

6 of 17

Introduction

GBERT vs BERT

Following BERT, the model architecture of GBERT is a multilayer bidirectional Transformer encoder where each input token can see all input tokens.

GBERT differs from BERT in that the input to GBERT is the grapheme sequence in a single word while the input to BERT is the wordpiece sequence in a single sentence or two sentences.

7 of 17

Method

Pre-training GBERT for G2P Conversion

GBERT is pre-trained using a masked grapheme prediction task
mask token(80%), random grapheme(10%), original grapheme(10%)

8 of 17

Method

Fine-tuning GBERT for G2P Conversion

Fine-tuning a pre-trained language model is a typical way to apply pre-trained representations to downstream tasks.

Since GBERT is a pre-trained Transformer encoder while the vanilla Transformer includes an encoder and a decoder, fine-tuning GBERT means that we replace the encoder in the vanilla Transformer with GBERT and train the new model in an end-to-end manner.

9 of 17

Method

BERT-fused model

Jinhua Zhu, et al. “Incorporating BERT into Neural Machine Translation”, ICLR 2020

In addition to fine-tuning, another approach to integrate pre-trained language models is utilizing them as feature extractors

One example is the BERT-fused model, which controls how each encoder and decoder layer of the vanilla Transformer model interacts with the output features of BERT

In each encoder layer, an additional GBERT Enc attention module is added to the original Transformer encoder layer
In each decoder layer, there is a similar GBERT-Dec attention module for a similar purpose.

10 of 17

Method

BERT-fused model

Jinhua Zhu, et al. “Incorporating BERT into Neural Machine Translation”, ICLR 2020

https://arxiv.org/pdf/2002.06823.pdf

Let X and Y denote the source language domain and target language domain respectively, which are the collections of sentences with the corresponding languages. For any sentence x ∈ X and y ∈ Y, let lx and ly denote the number of units (e.g., words or sub-words) in x and y.

The i-th unit in x/y is denoted as xi /yi

�Any input x ∈ X is progressively processed by the BERT, encoder and decoder.

Step-1:

임의의 입력 x ∈ X가 주어지면, BERT는 먼저 HB = BERT(x)로 인코딩�HB는 BERT의 마지막 레이어의 출력�hB,i∈HB는 x에서 i번째 wordpiece 나타냄

Step-2: Let Hl E denote the hidden representation of l-th layer in the encoder, Denote the i-th element in Hl E as h l i for any i ∈ [lx].

In the l-th layer, l ∈ [L], 수식(1).

attnS, attB는 서로 다른 파라미터를 가진 attention model
그런 다음 h~il는 FFN에 의해 추가로 처리되고 우린 l번째 레이어의 출력을 얻음

Step-3 : S<tl가 time step t 이전의 디코더에서 l번째 레이어의 hidden state를 나타냄

S<tl=(s1l,...,st−1l)�s10은 시퀀스의 시작을 나타내는 특수 토큰이고 st0는 time step t-1에서 예측 단어의 임베딩
attnS, attnB, attnE는 각각 self-attention model,BERT-decoder attention model,encoder-decoder attention model을 나타냄
식2는 레이어를 반복하고 결국 stL을 얻을 수 있음
마지막으로 stL는 선형 변환과 softmax를 통해 매핑되어 t번째 예측 단어 y~t를 얻음
디코딩 프로세스는 문장 종료 토큰을 충족할 때 까지 계속

이는 tokenization 방식에 관계없이 사전 학습된 모델을 활용하는 일반적인 방법
우리의 프레임워크에서 BERT의 출력은 외부 시퀀스 표현의 역할을 하며, NMT mode에 통합하기 위해 attention module 사용

우리의 프레임워크에서 BERT의 출력은 외부 시퀀스 표현의 역할을 하며, NMT mode에 통합하기 위해 attention module 사용

또한 drop net은 network 학습을 정규화하는데 사용되며, 학습 중에 두 입력 또는 평균 결과 중 하나를 출력하고 추론 중에 평균 결과를 출력

11 of 17

Method

Fusing GBERT into Transformer-based G2P Model

12 of 17

Datasets

Experiments

The four most difficult languages (with the highest word error rates) among �all ten languages in the medium-resource (8000 pronunciation records for training) subtask of the SIGMORPHON 2021 task was chosen

Korean letters were decomposed into single-sound letters with hangul-jamo , �e.g., 가감 → ㄱ ㅏ ㄱ ㅏ ㅁ

13 of 17

WER and PER results

Results

※Low-resource : randomly sampled 1000 records from the original training set

WER is the percentage of words whose predicted phoneme sequences were not identical to the gold references.

PER is the sum of the Levenshtein distance between the predicted and the reference phoneme sequences, divided by the sum of the reference lengths on the test set.

The lower the WER and PER, the better the performance.

IL (Imitation Learning) :

다른 강화학습이 기존 경험을 통해 파라미터를 수정해가는 방식을 사용한다면,

Imitation Learning의 역할은 전문가가 겪은 경험을 통해 학습하는 것이다.

몬테주마처럼 보상을 받는 텀이 너무 길거나 모호할 때,

또는 최적의 행동지침을 바로 세우기가 어려울 때 사용하면 좋다.

https://mech-literacy.tistory.com/24

----

왜 low-resource 에서는 finetuning 이 fusing 보다 더 좋지??

-> 이러한 결과는 GBERT를 Transformer에 fusing하는 방법이 모델 구조가 복잡하기 때문에 GBERT를 미세 조정하는 방법보다 학습 데이터의 양에 더 민감

한국어는 왜 Transformer가 더좋지?

-> The contextual relationship among graphemes in Korean was weaker than that in the other three languages.

Limitation:

The Transformer baseline implemented by us outperformed the IL baseline on the medium-resource task of Dutch, but was not as good as IL on the other three languages

For the GBERT w/o fine-tuning model, its performance was much worse than the Transformer baseline in almost all experiments, especially for Korean. This indicates that the grapheme representations derived from the pre-trained GBERT can not provide all the necessary information for G2P conversion.

14 of 17

WER and PER results

Results

Fine-tuning a pre-trained language model is a typical way to apply pre-trained representations to downstream tasks.

These results show that the GBERT-fused model may be more sensitive to the amount of training data than the method of fine-tuning GBERT because the former has a more complex model structure

The reason that our proposed method did not achieve satisfactory performance for Korean on the low-resource G2P task may be that the contextual relationship among graphemes in Korean was weaker than that in the other three languages.

----

왜 low-resource 에서는 finetuning 이 fusing 보다 더 좋지??

-> 이러한 결과는 GBERT를 Transformer에 fusing하는 방법이 모델 구조가 복잡하기 때문에 GBERT를 미세 조정하는 방법보다 학습 데이터의 양에 더 민감

한국어는 왜 Transformer가 더좋지?

-> The contextual relationship among graphemes in Korean was weaker than that in the other three languages.

Limitation:

The Transformer baseline implemented by us outperformed the IL baseline on the medium-resource task of Dutch, but was not as good as IL on the other three languages

For the GBERT w/o fine-tuning model, its performance was much worse than the Transformer baseline in almost all experiments, especially for Korean. This indicates that the grapheme representations derived from the pre-trained GBERT can not provide all the necessary information for G2P conversion.

15 of 17

Conclusion

Results

The training of GBERT requires only easily accessible word lists.

Experiments on the Dutch, Serbo-Croatian, Bulgarian and Korean datasets of the SIGMORPHON 2021 G2P task show that the method of fusing GBERT can reduce the WER and PER of all languages on the medium-resource G2P task

The method of fine-tuning GBERT can improve the G2P performance of most languages on the low resource G2P task. (except Korean)

16 of 17

THANK YOU FOR LISTENING

17 of 17

Limitations of Previous Studies

Introduction

The hierarchical prosody annotation adopted in this work categorizes the prosodic boundaries of Mandarin speech into five levels, which from low to high are Character (CC), Lexicon Word (LW), Prosodic Word (PW), Prosodic Phrase (PPH) and Intonational Phrase (IPH). Prosodic Word (PW), Prosodic Phrase (PPH) and Intonational Phrase (IPH) correspond to three different lengths of pause in speech from short to long. Lexicon Word (LW) indicates syntactic boundary between words, and Chinese Character (CC) is the smallest unit of Chinese.

Nevertheless, building neural G2P models usually relies on a large, language-specific pronunciation dictionary, which may not be available for a lot of languages

multilingual neural networks

Models of high-resource languages showed better ability

utilizing multimodal data

Additional audio supervision help model to learn a more optimal�intermediate representation of graphemes

An early work is a wFST-based system [11]. Subsequently, multilingual neural networks [12] and pre-trained G2P models of high-resource languages [13] showed better cross-lingual G2P modeling ability.

Another approach is utilizing multimodal data. Route et al. [14] found that additional audio supervision can help the G2P model to learn a more optimal intermediate representation of graphemes.

However, these studies mainly focused on utilizing the data resources of other languages or other modals to improve the performance of limited resource G2P conversion, which do not explore better grapheme representations for G2P in an unsupervised way.

WFST(Weighted Finite-State Transducers) : ASR(autiomatic speech recognition)에서 사용되는 HMM phonetic model, Context-dependent phones, Pronunciation lexicons and grammars를 transducer 형태로 표현하고 각각의 transducer를 결합하여 하나의 transducer로 만들어, 그것을 이용하여 sound를 decoding하는 일련의 구조.