Does Fine-tuning by Reinforcement Learning Improve Generalization in Binary Speech Deepfake Detection?

Xin Wang, Wanying Ge, Junichi Yamagishi

National Institute of Informatics, Japan

Interspeech 2026

Area 4 - Oral 9 - 3

wangxin@nii.ac.jp

1

Does Fine-tuning by Reinforcement Learning Improve Generalization in Binary Speech Deepfake Detection?

Xin Wang, Wanying Ge, Junichi Yamagishi

National Institute of Informatics, Japan

Interspeech 2026

Area 4 - Oral 9 - 3

wangxin@nii.ac.jp

mitigate catastrophic forgetting

similarly to

2

Slides by Xin Wang

National Institute of Informatics

​

​

​

© 2026, Xin Wang. All rights reserved.

​

​

​

​

This work is licensed under the Creative Commons Attribution 3.0 license.

See http://creativecommons.org/ for details.

​

​

​

​

Note: Speech data is from ASVspoof 2019 database https://datashare.ed.ac.uk/handle/10283/3336

3

One-minute summary

• Context of our experiments

​

​

​

​

• Main findings:
• Fine-tuning by supervised learning (i.e., cross-entropy) leads to catastrophic forgetting
• Fine-tuning by Group Relative Policy Optimization (GRPO)-like criterion mitigates catastrophic forgetting

Pre-training

(self-supervised learned front-end)

Post (or middle) training

(large-scale data, supervised)

Fine-tuning

(small-scale)

4

Introduction

5

Introduction

• Task: binary classification

​

​

​

​

​

​

​

​

Detector

Deepfake generator

REAL

FAKE

Transmission

VoIP

mp3

or …

6

Introduction

• Goal: generalization on unseen data

Detector

7

Introduction

Juan M. Martin-Donas and Aitor Alvarez. 2022. The Vicomtech Audio Deepfake Detection System Based on Wav2vec2 for the 2022 ADD Challenge. In Proc. ICASSP, May 23, 2022. 9241–9245.

Xin Wang and Junichi Yamagishi. 2022. Investigating Self-Supervised Front Ends for Speech Spoofing Countermeasures. In Proc. Odyssey, 2022. 100–106.

Hemlata Tak, Massimiliano Todisco, Xin Wang, Jee-weon Jung, Junichi Yamagishi, and Nicholas Evans. 2022. Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation. In Proc. Odyssey, 2022. 112–119.

Wanying Ge, Xin Wang, Xuechen Liu, and Junichi Yamagishi. 2025. Post-training for Deepfake Speech Detection. In Proc. ASRU, December 06, 2025.

Yi Zhu, Brahmi Dwivedi, Jayaram Raghuram, and Surya Koppisetti. 2026. Alethia: A Foundational Encoder for Voice Deepfakes. In Proc. ICML, 2026.

• Promising solution: large-scale training

​

​

​

​

​

​

​

​

• self-supervised learning (SSL) front-end ^{(Martin-Donas 2022, Wang 2022, Tak 2022)}
• (large-scale) supervised training ^{(Ge 2025, Zhu 2026)}

Detector

Unseen deepfakes, speakers, languages, channels …

8

Introduction – Base method

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Proc. NIPS, 2020. 12449–12460.

​

Detector

SSL-based

front end

Classifier

Input wav.

(REAL)

(FAKE)

9

Introduction – Base method

SSL-based

front end

Classifier

Cross-entropy

Training set

~500k hours

Real data only

SSL-based

front end

Contrastive loss

Pseudo

labels

download

& plug in

pre-training

supervised fine-tuning

~0.1k

Labels

10

Introduction – Base method II

SSL-based

front end

Classifier

Cross-entropy

Training set

~500k hours

Real data only

SSL-based

front end

Contrastive loss

Pseudo

labels

pre-training

supervised fine-tuning

A middle stage to fill the gap between pre-training and fine-tuning?

Labels

11

Introduction – Base method II

Wanying Ge, Xin Wang, Xuechen Liu, and Junichi Yamagishi. 2025. Post-training for Deepfake Speech Detection. In Proc. ASRU, 2025.

Labels

SSL-based

front end

Classifier

Cross-entropy

~500k hours

Real data only

SSL-based

front end

Contrastive loss

Pseudo

labels

pre-training

post-training

Real, vocoded, TTS/VC data

~75k

SSL-based

front end

Classifier

Cross-entropy

supervised fine-tuning

Target domain real & fake data

0.1 k

Labels

12

Introduction – Base method II

Wanying Ge, Xin Wang, Xuechen Liu, and Junichi Yamagishi. 2025. Post-training for Deepfake Speech Detection. In Proc. ASRU, 2025.

Labels

Cross-entropy

SSL-based

front end

Classifier

Cross-entropy

~500k hours

Real data only

SSL-based

front end

Contrastive loss

Pseudo

labels

pre-training

post-training

Real, vocoded, TTS/VC data

~75k

SSL-based

front end

Classifier

supervised fine-tuning

Target domain real & fake data

0.1 k

Labels

Large-scale, many domains

Fine-tuning, a target challenging domain

13

Introduction – Issue of catastrophic forgetting

Wanying Ge, Xin Wang, Xuechen Liu, and Junichi Yamagishi. 2025. Post-training for Deepfake Speech Detection. In Proc. ASRU, 2025.

Challenging target domain

Other “easy” domains

Equal error rate

5%

50%

How to keep this error rate low after supervised fine-tuning?

14

Method

15

Method

• The same pre-training / post-training pipeline

SSL-based

front end

Classifier

Cross-entropy

Target domain real & fake data

0.1 k

supervised fine-tuning

16

Method

Daya Guo, Dejian Yang, et al. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645, 8081 (September 2025), 633–638.

​

​

• The same pre-training / post-training pipeline

SSL-based

front end

Classifier

Target domain real & fake data

0.1 k

• Replace the fine-tuning criterion
• cross-entropy supervised fine-tuning (SFT)
• similar to Group Relative Policy Optimization (GRPO) ^{(Guo 2025)}

​

GRPO

GRPO-like

P(REAL)

P(FAKE)

17

Daya Guo, Dejian Yang, et al. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645, 8081 (September 2025), 633–638.

​

SSL-based

front end

Classifier

Target domain real & fake data

0.1 k

GRPO

Reinforcement learning-like

P(REAL)

P(FAKE)

18

Method – why GRPO-like fine-tuning?

Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. 2025. RL’s Razor: Why Online Reinforcement Learning Forgets Less. In NeurIPS 2025 Workshop on Continual and Compatible Foundation Model, September 04, 2025.

​

​

General domains

Reinforcement learning

(e.g., GRPO)

Supervised training

by cross entropy

19

Method – implementation

SFT

Detector

​

Cross-entropy

s

Label

Wav

Logit

REAL (R)

Prob.

Detector

​

s

Sampled predictions

REAL (R)

Reward

​

Evaluate only using ground-truth label

20

Method – implementation

Label

Wav

Prob.

Detector

​

REAL (R)

Reward

​

​

Detector

​

Sampled

predictions

Checkpoint (previous epoch)

gradients

Prediction is correct?

21

Method – implementation

Wav

Prob.

Detector

​

REAL (R)

Sampled

predictions

Reward

​

​

Detector

​

Checkpoint (previous epoch)

Label

gradients

Prediction is correct?

See paper (appendix on arxiv) & code

1. Reward clipping, KLD, …
2. Gradients of GRPO are different from those of SFT

22

Experiment

23

Experiment setup

Wanying Ge, Xin Wang, Xuechen Liu, and Junichi Yamagishi. 2025. Post-training for Deepfake Speech Detection. In Proc. ASRU, 2025.

Nuria Alina Chandra, Ryan Murtfeldt, Lin Qiu, Arnab Karmakar, Hannah Lee, Emmanuel Tanumihardja, Kevin Farhat, Ben Caffee, Sejin Paik, Changyeon Lee, Jongwook Choi, Aerin Kim, and Oren Etzioni. 2025. Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024.

​

• Research question
• Can GRPO-like fine-tuning mitigate catastrophic forgetting?

Real, vocoded, & fake data ^{(Ge 2025)}

~75k hours

Deepfake-Eval-2024 train set, real & fake data ^{(Chandra 2025)}

~50 hours

Real data

e.g., XLS-R

~500k hours

​

Evaluation

In-domain

Out-of-domain

Deepfake-Eval-2024 test set

• ADD2023
• FoR
• DeepVoice
• In-the-wild

Challenging target domain

• Data protocol ^{(Ge 2025)}

We use post-trained checkpoints

24

Experiment setup

• Research question
• Can GRPO-like fine-tuning mitigate catastrophic forgetting?

SFT (cross-entropy)

Pre-training

Post training

Fine-tuning

GRPO

Pre-training

Post training

No fine-tuning

• Experimental systems

We use post-trained checkpoints

25

Experiment results

• Highlighted results
• Model architecture: XLS-R-2B + global average pooling+linear output

Pre-training

Post training

Fine-tuning

v

v

x

v

v

SFT

v

v

GRPO

Average EER (%)

Challenging

target domain

Other easier non-target domains

22.64

2.69

10.26

6.28

Out-of-domain performance degraded after SFT

26

Experiment results

• Highlighted results
• Model architecture: XLS-R-2B + global average pooling+linear output

Pre-training

Post training

Fine-tuning

v

v

x

v

v

SFT

v

v

GRPO

Average EER (%)

22.64

2.69

10.26

6.28

GRPO mitigates catastrophic forgetting

9.23

2.69

Challenging

target domain

Other easier non-target domains

27

Experiment results

• Highlighted results
• Model architecture: XLS-R-2B + global average pooling + linear output

Pre-training

Post training

Fine-tuning

v

v

x

v

v

SFT

v

v

GRPO

Detailed EER (%)

Other easier non-target domains

​

22.64

10.26

9.23

ADD

FoR

DeepVoice

ItW

4.67

2.61

2.23

1.24

6.09

3.92

8.75

6.35

5.34

0.47

2.76

2.19

GRPO mitigates catastrophic forgetting

Challenging

target domain

28

Experiment results

• Highlighted results
• Model architecture: XLS-R-2B + global average pooling + linear output

Pre-training

Post training

Fine-tuning

v

v

x

v

v

SFT

v

v

GRPO

Average EER (%)

In-domain

Out-of-domain

22.64

2.69

10.26

6.28

9.23

2.69

w/o sampling

11.41

3.41

Sampling incorrect labels is necessary to GRPO

29

Experiment results

Equal error rate (EER)

Average EER of 3 runs

SSL

Target domain EERs

Non-target domain

Training pipelines

30

Experiment results

• Other results in the paper
• Similar findings when using other multi-lingual SSL front-ends
• MMS-1B, MMS-300M

​

• Other pipelines were sub-optimal

​

​

​

• Ablation study on GRPO
• KLD regularization is NOT a must

​

​

31

Summary

32

Summary

• Does GRPO-like fine-tuning mitigate catastrophic forgetting?
• YES, at least in our experiments

​

​

​

• Limitations
• Not all factors are ablated
• Size of sampling
• Time to update old checkpoint
• Test on other target domains

Pre-training

Post training

Fine-tuning

GRPO

• Future work
• GRPO for post training?
• Reward beyond simple label comparison?
• …

33

Thank you

Code

github/nii-yamagishilab/AntiDeepfake

34

Appendix

35

Implementation details

(appendix https://arxiv.org/abs/2603.02914)

​

0 If equal, else 1

36

Compare gradients

(appendix https://arxiv.org/abs/2603.02914)

​

​

without group-mean / var

without clipping, KLD …

GRPO-like

Cross entropy

37

Compare gradients

(appendix https://arxiv.org/abs/2603.02914)

​

​

without group-mean / var

without clipping, KLD …

GRPO-like

Mixup

38

Introduction – Base method

Xin Wang and Junichi Yamagishi. 2023. Spoofed Training Data for Speech Spoofing Countermeasure Can Be Efficiently Created Using Neural Vocoders. In Proc. ICASSP, 2023.

Xin Wang and Junichi Yamagishi. 2024. Can Large-Scale Vocoded Spoofed Data Improve Speech Spoofing Countermeasure with a Self-Supervised Front End? In Proc. ICASSP, 2024. 10311–10315.

SSL-based

front end

Classifier

Cross-entropy

Training set

~500k hours

Real data only

SSL-based

front end

Contrastive loss

Pseudo

labels

pre-training

supervised fine-tuning

SSL-based

front end

Contrastive loss

Pseudo

labels

40k hours

Vocoded data

pre-training II

distillation

39

Introduction – Base method

Xin Wang and Junichi Yamagishi. 2023. Spoofed Training Data for Speech Spoofing Countermeasure Can Be Efficiently Created Using Neural Vocoders. In Proc. ICASSP, 2023.

Xin Wang and Junichi Yamagishi. 2024. Can Large-Scale Vocoded Spoofed Data Improve Speech Spoofing Countermeasure with a Self-Supervised Front End? In Proc. ICASSP, 2024. 10311–10315.

SSL-based

front end

Classifier

Cross-entropy

Training set

~500k hours

Real data only

SSL-based

front end

Contrastive loss

Pseudo

labels

pre-training

supervised fine-tuning

SSL-based

front end

Contrastive loss

Pseudo

labels

40k hours

Vocoded data

pre-training II

Slightly lower EERs than pre-training + fine-tuning ^{(Wang 2024)}

More complicated: distillation & additional loss needed

distillation

40