1 of 16

RL based LLM post-training (part 1)

Soo Kyung Kim

​

Department of AI�Ewha Womans University

2 of 16

Agenda

  • Reinforcement Learning based LLM-post training (part1)� - PPO, DPO, GRPO

​

​

  • Reinforcement Learning based LLM-post training (part2)� - DAPO, GSPO, TR-GRPO (On-going)

�

  • Multi-Turn Reinforcement Learning for LLM Agents (part3)��

​

3 of 16

How to train LLM?

GPT-3 base

GPT-3 �instruction-tuned

4 of 16

Online RL vs Offline RL

DPO

GRPO

PPO

On-policy RL

Imitation learning

Off-policy�(Hybrid btw �on and off policy)

(c) Imitation learning

5 of 16

RLHF – Reward Model

Bradley–Terry model:

6 of 16

Reward Model

Bradley–Terry model:

7 of 16

RLHF

 

LLM specific KL term

�In RLHF they used PPO

8 of 16

PPO (Proximal Policy Optimization)

  • Advantage: Expected reward if I take the action a at state s

​

Solve high-variance: �We don’t want

my policy becomes too different with old policy��🡪 Conservative update

9 of 16

PPO (Proximal Policy Optimization)

  • Policy gradient: Stable training
  • Reward model and Value function are required 🡪 Reward model might not robust
  • Limited exploration
  • Preference data, and roll-out is required 🡪 Slow
  • High Computing cost: Policy (LLM trained), Ref Policy (LLM frozen), Reward Model, Value Function.

​

+ Pro, - Con

10 of 16

DPO (Deterministic Preference Optimization)

  • maximizes the log-probability difference between preferred and less preferred responses

11 of 16

DPO (Deterministic Preference Optimization)

  • Fast: Do not require value function update, no reward model
  • Low Computing cost: No value function + reward model, only require policy and reference policy
  • Limited exploration
  • Supervised Setting: Prone to overfitting, No outside of distribution, as trajectory getting longer more prone to get error

​

+ Pro, - Con

12 of 16

GRPO (Group Relative Policy Optimization)

Solve high-variance:�1. We don’t want

my policy becomes too different with old policy�🡪 Conservative update��2. Average-out in the group with same query �🡪 Unstable Trust-Region

LLM specific KL term

13 of 16

General RL-based post-training

  • Sparse reward

​

�🡪 Conservative update

�🡪 Value-over estimation�🡪 Low Entropy of Policy�(Deterministic Policy Collapse, Reward Hacking)

�🡪 Unstable TR

14 of 16

GRPO (Group Relative Policy Optimization)

  • No Value function: Faster than PPO
  • Better reasoning: You sample k candidate solutions per prompt � 🡪 Group-relative update
  • Advantage function is not prompt based
  • Clipping 🡪 conservative update 🡪 lack of exploration

​

+ Pro, - Con

15 of 16

Problems in RL-based LLM post-training

​

PPO (Online)

GRPO (Hybrid)

DPO (Imitation Learning)

​

�����

​

​

Cons

Stable Training

- Robust to reward-model calibration�- More sample-efficient

- Very sample-efficient

Pros

- Slow, large computing cost

- Conservative policy update�- Not robust to reward score (both preference or scalar)

- Conservative policy update, Limited exploration

- Low entropy of policy (Deterministic policy collapse)

- Require pre-sampled data�Limited exploration�- Low entropy of policy�- Difficult to explore�- Value(Q) over-estimation�

 

 

 

DAPO, GSPO,TR-GRPO

GoodLiar

(DA-ILQL)

16 of 16

DPO : Appendix - derivation