1 of 10

MoE-GRPO: Optimizing Mixture-of-Experts via

Reinforcement Learning in Vision-Language Models

Dohwan Ko¹ Jinyoung Park² Seoung Choi² Sanghyeok Lee² Seohyun Lee² Hyunwoo J. Kim²

¹Korea University ²KAIST

Korea University

MLV Lab

CVPR 2026

2 of 10

BACKGROUND

Deterministic Top-K Routing in MoE

One MoE layer, one token

token h_t

gating network g(h_t) = softmax(·)

select top-K (K = 2) — always the arg-max

output = gate-weighted sum of the K experts

greedy · identical choice for similar tokens, every layer, every step

No exploration

Alternative expert combinations are never even tried during fine-tuning.

Better routings overlooked

The greedy arg-max may miss expert selections that would earn higher reward.

Overfits to few experts

Traffic collapses onto a small subset of experts; the rest are rarely trained.

Prior fixes (e.g. Gaussian-noise routing) inject randomness — but never optimize the selection policy.

Korea University

MLV Lab

CVPR 2026

3 of 10

KEY IDEA

Expert selection as sequential decisions, optimized with RL

Deterministic Top-K

token 1 → E2 + E6

token 2 → E2 + E6

token 3 → E2 + E6

E1

E2

E3

E4

E5

E6

E7

E8

gating scores fixed by SFT → same arg-max for every similar token

6 of 8 experts almost never selected

no exploration → overfits to a few experts

MoE-GRPO (ours)

1 · Explore

2 · Reward

3 · Update

4 · Exploit

E1

E2

E3

E4

E5

E6

E7

E8

rollout 1

E1

E5

→

✓

Â > 0

rollout 2

E3

E6

→

✗

Â < 0

rollout 3

E7

E8

→

✓

Â > 0

rollout 4

E1

E7

→

✓

Â > 0

group-relative advantage Âⁱ pushes gate probability toward rewarded experts

explore → reward → exploit: a learned routing policy

Korea University

MLV Lab

CVPR 2026

[Key idea] video 48.2s – 102.9s (55s)

Our key idea is to treat expert selection as a sequential decision-making problem, and optimize the routing policy with reinforcement learning. Let's compare the two behaviors. ▶ On the left, deterministic top-K: token after token, the router picks exactly the same highest-scoring experts. Six of the eight experts are almost never selected. ▶ On the right, MoE-GRPO. During training, we sample multiple expert routings — these are our rollouts. Each rollout generates an answer and receives a verifiable reward. ▶ Rollouts that chose good expert combinations get positive group-relative advantages, and the gating network shifts probability toward them. ▶ Early on, the policy explores widely; as rewards accumulate, it exploits the combinations that actually work. The router learns a policy, instead of following a fixed rule.

4 of 10

METHOD

MoE-GRPO framework

G routing rollouts

g_old samples G expert-selection sequences across all layers & tokens

Verifiable reward → group advantage

answer correct = 1, else 0; advantages normalized within the group

Token-GRPO + Gate-GRPO

dense supervision at every layer & token · gates trained from scratch → no reference model, no KL

Korea University

MLV Lab

CVPR 2026

5 of 10

METHOD

Modality-aware router guidance

Routing a vision token

modality-awareness score ŝ_v(e) — how often expert e serves vision tokens

E1

E2

E3

E4

E5

E6

E7

E8

72%

18%

55%

12%

66%

38%

61%

47%

✗

deactivate bottom 25% per modality (gate score → −∞)

sample K experts from the remainder (multinomial)

exploration stays modality-consistent, yet diverse

Why?

unrestricted search over sequential expert choices is slow & unstable — guidance prunes irrelevant directions

At inference

plain deterministic top-K — zero extra cost at test time

Korea University

MLV Lab

CVPR 2026

6 of 10

EXPERIMENTS

Main results — 9 image & video benchmarks

InternVL3.5-1B → MoE (2 of 8 experts active, 1.3B / 2.9B params) · 100K visual-instruction samples · G = 8 rollouts

Fine-tuning

MME

MMBench

MMStar

MMT

AI2D

VideoMME

MLVU

LongVB

MVBench

Avg.

Det-FT (top-K)

1,660

75.8

45.6

51.8

62.7

45.6

48.6

45.3

56.7

54.0

Stoch-FT-Multi

1,458

73.9

43.3

51.2

61.8

45.9

50.3

47.0

56.5

53.7

Stoch-FT-Noise

1,684

76.3

46.1

52.0

62.4

45.1

51.1

45.3

55.8

54.3

MoE-GRPO (ours)

1,693

77.5

45.7

54.8

65.8

46.6

53.1

46.5

58.4

56.0

+2.0 vs deterministic top-K

+2.3 vs multinomial sampling

+1.7 vs Gaussian noise

Randomness alone is not enough — stochastic variants don't consistently beat Det-FT. Explicitly optimizing the routing policy is what helps (best on 7 / 9 benchmarks).

Korea University

MLV Lab

CVPR 2026

7 of 10

EXPERIMENTS

The learned routing generalizes

Cross-dataset transfer · CLIP-MoE, ImageNet → 10 datasets

70.8

CLIP-MoE

68.8

+ Det-FT

71.9

+ MoE-GRPO

Domain generalization · ImageNet → V2 / S / A / R

63.4

CLIP-MoE

66.0

+ Det-FT

67.5

+ MoE-GRPO

Det-FT degrades the base model — overfitting

diverse expert use → robust out-of-domain

+3.1 over Det-FT · best on 9 / 10 datasets

+4.1 over CLIP-MoE · +1.5 over Det-FT

Korea University

MLV Lab

CVPR 2026

8 of 10

EXPERIMENTS

Ablations — both objectives & the guidance matter

Training objective (avg. of 4 benchmarks)

Token-GRPO only

53.9

Gate-GRPO only

50.9

Token-GRPO + Gate-GRPO

55.7

Exploration strategy

modality-agnostic · Gaussian noise

54.2

modality-agnostic · multinomial

54.8

modality-aware guidance (ours)

55.7

faster reward growth

lower in-group variance

Modality-aware guidance avoids wasted exploration → more stable policy learning

Korea University

MLV Lab

CVPR 2026

9 of 10

ANALYSIS

What routing policy is learned?

Token-level expert utilization

entropy 1.05 → 1.82

Det-FT

MoE-GRPO

47.0%

43.7%

30.9%

16.5%

15.9%

18.5%

Task-level specialization

JSD 0.06 → 0.20

Qualitative example

balanced per token · specialized per task → correct answer

Korea University

MLV Lab

CVPR 2026

10 of 10

MoE-GRPO

RL-optimized expert routing for MoE vision-language models

Routing as sequential decision-making — Token-GRPO + Gate-GRPO turn one binary reward into dense, layer-wise routing supervision (no reference model, no KL).

Modality-aware guidance keeps exploration efficient and stable.

Consistent gains over deterministic & stochastic routing — plus better generalization and task-level expert specialization.

arXiv 2603.24984

Thank you!

Korea University

MLV Lab

CVPR 2026