# Reinforcement learning for LLM post-training Abbreviations: reinforcement learning (RL), reinforcement learning from human feedback (RLHF), Proximal Policy Optimization (PPO), Generalized Advantage Estimation (GAE), Direct Preference Optimization (DPO), Group Relative Policy Optimization (GRPO), Group Sequence Policy Optimization (GSPO), Kullback-Leibler divergence (KL), supervised fine-tuning (SFT), mixture of experts (MoE). After pre-training and SFT, LLMs are aligned or taught to reason by optimizing a policy $\pi_\theta$ (the LLM) against a reward signal. This page summarizes the main methods, from classic RLHF with PPO to the critic-free group methods used for reasoning models. See [LLM Taxonomy](LLM%20Taxonomy.md) for where these methods sit in model development, and [Unsloth training](../Machine%20Learning/Deep%20Learning/Unsloth%20PEFT/Training.md) for practical trainer setup. ## Objective and loss RL maximizes an objective $J(\theta)$, typically the expected reward; optimizers minimize a loss $\mathcal{L}(\theta)$. The two usually differ only by sign: $\mathcal{L}(\theta) = -J(\theta)$. $\nabla_\theta J(\theta)$ is the direction in parameter space that increases the objective the most. ## Reward functions Rewards for LLMs come from one or a combination of: - A learned reward model, such as one trained on human preference data (see [Human preference reward model](#Human%20preference%20reward%20model)). - Rule-based rewards, such as format or length checks. - Verifiable rewards, such as unit tests passing or a math answer matching the reference. Learned rewards are usually combined with a KL penalty that discourages the policy from drifting too far from a frozen reference policy (typically the SFT model). This regularizes the reward, and therefore the advantage. ## Human preference reward model For each prompt $x$, humans pick a preferred response $y_A$ over a rejected response $y_B$. In RLHF, the reward model usually starts from a pretrained LM whose LM head is replaced by a scalar reward head mapping the final token's hidden state to one number: $r = w^\top h_{\text{final}} + b$. The same model scores both responses independently, $r_A = R_\phi(x, y_A)$ and $r_B = R_\phi(x, y_B)$, and is trained with the logistic Bradley–Terry preference loss: $\mathcal{L}_{RM} = -\log \sigma(r_A - r_B)$ Training is pairwise, but inference is per response: after training, $R_\phi(x, y)$ returns an unbounded scalar score for any single response. ## RLHF with PPO The classic InstructGPT recipe optimizes the policy with PPO against the human preference reward model. Each generated token gets a sampled per-token KL estimate against the reference policy: $r_t^{KL} = \log \pi_\theta(a_t \mid s_t) - \log \pi_{\mathrm{ref}}(a_t \mid s_t)$ where $a_t$ is the token chosen given the preceding text $s_t$. The total reward for response $y$ to prompt $x$ is the reward-model score minus the scaled KL penalty: $R_{\mathrm{total}} = R_{\mathrm{RM}}(x, y) - \beta \sum_{t=1}^{T} r_t^{KL}$ PPO then needs a per-token advantage $A_t$: how much better or worse the outcome after token $t$ was than expected. In simplified form, $A_t = R_t - V(s_t)$, where $R_t$ is the realized (KL-shaped) future return from step $t$ onward and $V(s_t)$ is the return predicted by a learned critic (value function). In practice, PPO estimates $A_t$ with GAE. If $A_t > 0$, the update increases the probability of token $t$ relative to the old policy. PPO's clipped surrogate objective takes $\min\big(\rho_t A_t,\ \operatorname{clip}(\rho_t, 1-\epsilon, 1+\epsilon) A_t\big)$, where $\rho_t$ is the probability ratio between the current and the old policy. Once the ratio leaves $[1-\epsilon, 1+\epsilon]$ in the direction favored by the advantage, the clipped term wins the $\min$ and its gradient is zero, removing the incentive to move further. Clipping discourages excessively large policy changes, but does not guarantee a bound: the actual ratio can still end up outside the clip range. The cost: four models in memory (policy, reference, reward model, and critic). ## DPO DPO skips the explicit reward model and the RL loop and optimizes directly on preference pairs, using a frozen reference model. For a preferred response $y_w$ and a rejected response $y_l$, with $\log \pi(y \mid x) = \sum_t \log \pi(y_t \mid x, y_{<t})$: $r_w = \log \pi_\theta(y_w \mid x) - \log \pi_{\mathrm{ref}}(y_w \mid x), \quad r_l = \log \pi_\theta(y_l \mid x) - \log \pi_{\mathrm{ref}}(y_l \mid x)$ $\mathcal{L}_{\mathrm{DPO}} = -\log \sigma\big(\beta (r_w - r_l)\big)$ $\beta$ is the same KL-regularization coefficient as in [RLHF with PPO](#RLHF%20with%20PPO). DPO is derived from the same KL-constrained reward-maximization objective: $\max_{\pi_\theta}\ \mathbb{E}_{x,\, y \sim \pi_\theta}\big[r(x, y)\big] - \beta\, \mathrm{KL}\big(\pi_\theta(\cdot \mid x)\ \|\ \pi_{\mathrm{ref}}(\cdot \mid x)\big)$ Its optimal policy is $\pi^*(y \mid x) \propto \pi_{\mathrm{ref}}(y \mid x) \exp\big(r(x, y) / \beta\big)$. Solving for the reward gives $r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)} + \beta \log Z(x)$, where the partition function $Z(x)$ depends only on the prompt. Substituting this into the Bradley–Terry loss of the [human preference reward model](#Human%20preference%20reward%20model) cancels $Z(x)$ and yields the DPO loss above. So $\beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)}$ acts as an implicit reward, and $\beta$ both scales it and controls how far the policy may drift from the reference: a smaller $\beta$ allows larger deviations. DPO lowers the loss by making the preferred response gain more probability relative to the reference model than the rejected one. ## GRPO GRPO, introduced with DeepSeekMath, replaces PPO's learned critic and token-level advantages with group-relative, sequence-level advantages. For each prompt $q$, sample a group of $G$ outputs $o_1, \ldots, o_G$, score them, and use each output's z-score within the group as its advantage: $\hat A_i = \frac{R_i - \operatorname{mean}(R_1, \ldots, R_G)}{\operatorname{std}(R_1, \ldots, R_G)}$ Every token of output $i$ gets the same $\hat A_i$. The per-token probability ratio between the current and the old policy enters the same clipped surrogate as in PPO: $r_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t} \mid q, o_{i,<t})}$ $\mathcal{L}_{\mathrm{GRPO}}(\theta) = -\frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \min\Big(r_{i,t}(\theta) \hat A_i,\ \operatorname{clip}\big(r_{i,t}(\theta), 1-\epsilon, 1+\epsilon\big) \hat A_i\Big)$ The original formulation also adds a KL penalty against the reference model. Groups are processed in batches: a batch averages the gradient over many noisy groups and uses the GPUs more efficiently than optimizing one group at a time. Known biases, pointed out by Dr. GRPO: - The per-response length normalization $\frac{1}{|o_i|}$ favors long responses when the model is wrong and short responses when it is right. Dr. GRPO removes it. - Dividing by the group's standard deviation gives extra weight to prompts that are too easy or too hard (rewards nearly all 1 or all 0, so little variance). Dr. GRPO drops the standard deviation; normalizing across the whole batch instead of per group is another common fix. ## GSPO GSPO, introduced with Qwen3, moves the importance ratio and clipping from the token to the sequence level, to match the sequence-level reward. The sequence ratio is the geometric mean of the token ratios: $s_i(\theta) = \exp\left(\frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \log \frac{\pi_\theta(y_{i,t} \mid x, y_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(y_{i,t} \mid x, y_{i,<t})}\right)$ $\mathcal{L}_{\mathrm{GSPO}}(\theta) = -\mathbb{E}\left[\frac{1}{G} \sum_{i=1}^{G} \min\Big(s_i(\theta) \hat A_i,\ \operatorname{clip}\big(s_i(\theta), 1-\epsilon, 1+\epsilon\big) \hat A_i\Big)\right]$ The advantage $\hat A_i$ is the group z-score of the reward, as in GRPO. Pipeline comparison: - GRPO: token ratio → per token, $\min$(unclipped ratio × sequence advantage, clipped ratio × sequence advantage) → average token-level objective. - GSPO: token ratios → geometric mean per sequence → per sequence, $\min$(unclipped ratio × sequence advantage, clipped ratio × sequence advantage) → sequence-level objective. One ratio per sequence reduces noise from individual token ratios and improves training stability, especially for long outputs and MoE models. GSPO is not universally better, but it showed more stable RL training than GRPO. In TRL (and Unsloth, which uses TRL's trainers), `importance_sampling_level="sequence"` in the `GRPOConfig` only selects the sequence-level ratio. It does not alone reproduce the objective above: TRL's default `loss_type="dapo"` aggregates over tokens, while the equation weights each sequence equally, which corresponds to `loss_type="grpo"`. TRL's paper index gives this configuration to reproduce the GSPO paper: ```python from trl import GRPOConfig training_args = GRPOConfig( importance_sampling_level="sequence", loss_type="grpo", beta=0.0, # no KL penalty epsilon=3e-4, epsilon_high=4e-4, gradient_accumulation_steps=1, steps_per_generation=4, # must be 4x gradient_accumulation_steps ) ``` The sequence-level ratio only has an effect when training is slightly off-policy, for example with `steps_per_generation > gradient_accumulation_steps` or `num_iterations > 1`; otherwise the ratio is 1. ## Comparison | Method | Reward source | Critic | Ratio and clipping | Notes | | --- | --- | --- | --- | --- | | PPO (RLHF) | Human preference reward model + KL penalty | Yes (GAE) | Per token | Four models in memory; complex to tune | | DPO | Implicit, from preference pairs | No | None (no sampling) | Offline; simple and cheap | | GRPO | Any (often verifiable) | No (group z-score) | Per token | Standard for reasoning models; length and difficulty biases | | GSPO | Any (often verifiable) | No (group z-score) | Per sequence | More stable for long outputs and MoE | ## Sources - Ouyang et al. (2022), [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155) (InstructGPT) - Schulman et al. (2017), [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347) - Schulman et al. (2015), [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://arxiv.org/abs/1506.02438) - Rafailov et al. (2023), [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/abs/2305.18290) - Shao et al. (2024), [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://arxiv.org/abs/2402.03300) (GRPO) - Liu et al. (2025), [Understanding R1-Zero-Like Training: A Critical Perspective](https://arxiv.org/abs/2503.20783) (Dr. GRPO) - Zheng et al. (2025), [Group Sequence Policy Optimization](https://arxiv.org/abs/2507.18071) - Hugging Face TRL, [Paper index: Group Sequence Policy Optimization](https://huggingface.co/docs/trl/main/en/paper_index#group-sequence-policy-optimization)