TL;DR
- Modern reinforcement learning (RL) for LLM reasoning relies on importance sampling (IS) for sample efficiency, but IS creates an exploration-stability dilemma: pure IS explodes, and the clipping used to tame it strictly caps how much the policy is allowed to move.
- We formalize the Probability Capacity (Cap), the update budget a token has before its gradient is truncated, and show that conservative clipping ties this budget to \(\pi_{\text{old}}\), structurally starving correct but low-confidence reasoning paths.
- Unbounded Positive Asymmetric Optimization (UP) replaces the IS ratio with a stop-gradient self-anchored ratio for positive advantages, recovering a stable, unclipped REINFORCE gradient, while keeping standard clipping for negative advantages as a safeguard.
- UP is a plug-and-play objective that instantiates as UP-DAPO, UP-GRPO, and UP-GSPO, and improves accuracy and exploration across DAPO, GSPO, and GRPO, across dense, MoE, and vision-language models, and across language and multimodal data.
Try It: How Does Clipping Throttle Exploration?
Figure 1 below is an interactive map of the update budget a token receives before its gradient is clipped, drawn over the plane of the current policy probability \(\pi_\theta\) (x axis) and the behavior probability \(\pi_{\text{old}}\) (y axis). Switch Method between DAPO and UP-DAPO for positive advantages (\(\hat{A} > 0\)), inspect the shared non-positive regime (\(\hat{A} \le 0\)), drag \(\epsilon_{\text{high}}\), \(\epsilon_{\text{low}}\), and \(c\), and hover anywhere to read the exact capacity. The key contrast: DAPO ties the budget to \(\pi_{\text{old}}\) through a thin band, while UP-DAPO frees it to the full \(1 - \pi_\theta\), independent of \(\pi_{\text{old}}\).
Hover over the map to read Cap at any (πθ, πold).
Figure 1. Interactive Probability Capacity map (this paper's Figure 1). The color encodes the update budget Cap over the plane of the current policy probability πθ (x axis) and the behavior probability πold (y axis). For positive advantages, DAPO confines Cap to a thin band bounded by πθ = (1 + εhigh)πold, while UP-DAPO frees it to the full 1 − πθ independent of πold; for non-positive advantages both methods share the clipped wedge.
Background
From REINFORCE to GRPO, DAPO, and GSPO
RL has become the standard paradigm for eliciting complex reasoning from LLMs. The vanilla REINFORCE objective is strictly on-policy and yields a mathematically stable gradient:
\[\nabla_\theta J_{\text{REINFORCE}}(\theta) = \mathbb{E}_{q \sim Q,\, o \sim \pi_\theta}\!\left[ \sum_{t=1}^{|o|} \hat{A}\, \nabla_\theta \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) \right].\]REINFORCE is stable but sample inefficient, since each trajectory is discarded after a single step. To reuse sampled data across multiple steps, GRPO and DAPO introduce importance sampling (IS), estimating the current policy \(\pi_\theta\) from samples drawn by a historical policy \(\pi_{\text{old}}\) through the ratio
\[r_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\text{old}}(o_{i,t} \mid q, o_{i,<t})}.\]To stop this ratio from exploding, GRPO bounds it inside a trust region \([1 - \epsilon, 1 + \epsilon]\) via clipping. DAPO decouples the bounds into \((\epsilon_{\text{low}}, \epsilon_{\text{high}})\) and drops the KL penalty, so it relies exclusively on clipping to trade off stability against exploration. GSPO instead moves to a sequence-level ratio, using the length-normalized likelihood of the whole response:
\[s_i(\theta) = \left( \frac{\pi_\theta(o_i \mid q)}{\pi_{\text{old}}(o_i \mid q)} \right)^{\frac{1}{|o_i|}} = \exp\!\left( \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \big( \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) - \log \pi_{\text{old}}(o_{i,t} \mid q, o_{i,<t}) \big) \right).\]Every one of these methods keeps \(\pi_{\text{old}}\) in the denominator, and every one leans on clipping. That shared structure is exactly what creates the dilemma below.
Probability Capacity (Cap)
To make the cost of clipping precise, we define the Probability Capacity (Cap) as the maximum allowable increase in \(\pi_\theta\) for a token with positive advantage (\(\hat{A} > 0\)), or the maximum allowable decrease for a token with non-positive advantage (\(\hat{A} \le 0\)), before the optimization gradient is truncated. Cap is the effective budget a token has for policy exploration relative to \(\pi_{\text{old}}\). Figure 1 plots this budget directly, and the rest of this post is a story about how UP enlarges it.
Motivation
IS-based RL is caught between two failure modes. Pushing updates too aggressively (pure IS) triggers instability; constraining them too conservatively (clipping) stifles exploration. We examine each in turn.
Dilemma 1: Importance Sampling Induces Instability
The unclipped IS objective maximizes the advantage weighted by the ratio, \(J_{\text{IS}}(\theta) = \mathbb{E}[\hat{A}\, r_{i,t}(\theta)]\), whose gradient is
\[\nabla_\theta J_{\text{IS}}(\theta) = \mathbb{E}_{q \sim Q,\, o \sim \pi_{\text{old}}}\!\left[ \hat{A}\, r_{i,t}(\theta)\, \nabla_\theta \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) \right].\]Comparing this to the REINFORCE gradient, the flaw is explicit: the IS gradient is scaled by \(r_{i,t}(\theta)\). For a rare, high-reward reasoning path where the behavior probability \(\pi_{\text{old}}\) is infinitesimally small, this ratio explodes as the active policy marginally improves. The pathological scaling destroys stable representations and drives training to diverge, which is precisely why clipping is treated as indispensable.
Dilemma 2: Clipping Stifles Exploration
For positive advantages, the clipped objective is \(J_{\text{IS+CLIP}}(\theta) = \min\!\big[r_{i,t}(\theta), 1 + \epsilon\big]\hat{A}\), and its gradient vanishes the moment the ratio exceeds the trust-region boundary. Mapping this to absolute probability space, the policy is upper-bounded by \(\pi_\theta \le (1 + \epsilon_{\text{high}})\pi_{\text{old}}\), so the remaining budget follows a piecewise Cap:
\[\mathrm{Cap}(\pi_\theta, \pi_{\text{old}}) = \begin{cases} \min\!\big(1,\, (1 + \epsilon_{\text{high}})\pi_{\text{old}}\big) - \pi_\theta & \text{if } \pi_\theta < (1 + \epsilon_{\text{high}})\pi_{\text{old}}, \\[4pt] 0 & \text{if } \pi_\theta \ge (1 + \epsilon_{\text{high}})\pi_{\text{old}}. \end{cases}\]This exposes a structural bottleneck: the budget is linearly dependent on \(\pi_{\text{old}}\). For a low-likelihood reasoning token with \(\pi_{\text{old}} = 0.01\), a standard \(\epsilon_{\text{high}} = 0.28\) caps the absolute probability increase at a mere \(0.0028\). Once \(\pi_\theta\) reaches \(0.0128\), the capacity drops to zero, the token enters the Upper Clip region, and its gradient is nullified even when the advantage is exceptionally high. Set the method to DAPO in Figure 1 and read along the bottom of the map, where \(\pi_{\text{old}}\) is small, to see this sliver of budget for yourself. Structurally, this stifles exploration and leaves promising long-tail trajectories undiscovered.
The question. How can we maximize a model’s exploration while preventing training instability in RL? Aggressive updates destabilize training; conservative clipping starves exploration. We need a formulation that escapes both.
Method
UP dismantles the dilemma with two mathematical moves: replace \(\pi_{\text{old}}\) with the current policy \(\pi_\theta\) via a stop-gradient, and route the gradient asymmetrically by the sign of the advantage.
An Unbounded Objective for Positive Advantages
For correct rollouts (\(\hat{A} > 0\)), we replace the IS ratio with a self-anchored ratio \(\tilde{r}_{i,t}(\theta)\) built from the stop-gradient operator \(\mathrm{sg}(\cdot)\) (Eq. 12):
\[J^{+}_{\text{UP}}(\theta) = \mathbb{E}_{q \sim Q,\, o \sim \pi_{\text{old}}}\!\left[ \sum_{t=1}^{|o|} \hat{A}\, \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\mathrm{sg}\!\big(\pi_\theta(o_{i,t} \mid q, o_{i,<t})\big)} \right], \qquad \hat{A} > 0.\]During backpropagation \(\mathrm{sg}(\pi_\theta)\) is a constant scalar, yet its forward value equals \(\pi_\theta\). Applying the log-derivative trick \(\nabla x / x = \nabla \log x\) collapses the objective to (Eq. 13):
\[\nabla_\theta J^{+}_{\text{UP}}(\theta) = \mathbb{E}_{q \sim Q,\, o \sim \pi_{\text{old}}}\!\left[ \sum_{t=1}^{|o|} \hat{A}\, \nabla_\theta \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) \right].\]This is a profound simplification: optimizing the self-anchored ratio is mathematically equivalent to maximizing the REINFORCE objective. By anchoring the policy to \(\pi_\theta\) instead of \(\pi_{\text{old}}\), UP eradicates the root cause of IS instability from Dilemma 1, and its Cap becomes fully unconstrained by history, \(\mathrm{Cap} = 1 - \pi_\theta\). Switch the method to UP-DAPO with \(\hat{A} > 0\) in Figure 1: the thin clipped band expands into a full budget that depends only on \(\pi_\theta\).
Why Asymmetric: Keep Clipping for Negative Advantages
The unbounded mechanism is deliberately restricted to correct rollouts. For wrong rollouts the advantage is negative, which flips the gradient direction; applying an unbounded update there performs aggressive gradient ascent and destroys the original representation. UP therefore retains standard clipping for \(\hat{A} \le 0\), where DAPO and UP-DAPO share an identical Cap governed by the lower-clip and dual-clip boundaries (see the \(\hat{A} \le 0\) view in Figure 1):
\[\mathrm{Cap}(\pi_\theta, \pi_{\text{old}}) = \begin{cases} \pi_\theta - (1 - \epsilon_{\text{low}})\pi_{\text{old}} & \text{if } (1 - \epsilon_{\text{low}})\pi_{\text{old}} \le \pi_\theta \le c\,\pi_{\text{old}}, \\[4pt] 0 & \text{if } \pi_\theta < (1 - \epsilon_{\text{low}})\pi_{\text{old}} \quad (\text{Lower Clip}), \\[4pt] 0 & \text{if } \pi_\theta > c\,\pi_{\text{old}} \quad (\text{Dual Clip}). \end{cases}\]This is the crux of UP: exploration for correct rollouts, safety for wrong ones.
UP-GxPO: One Plugin, Three Instantiations
Because it only rewrites the positive branch, UP plugs into any group-based policy optimizer. The token-level UP-DAPO combines the unbounded positive objective with the DAPO clipped objective for negatives (Eq. 14):
\[J_{\text{UP-DAPO}}(\theta) = \mathbb{E}\!\left[ \frac{1}{\sum_i |o_i|} \sum_{i}\sum_{t} \begin{cases} \hat{A}_{i,t}\, \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) & \text{if } \hat{A}_{i,t} > 0, \\[4pt] \min\!\big( r_{i,t}(\theta)\hat{A}_{i,t},\, \mathrm{clip}(r_{i,t}(\theta), 1 - \epsilon_{\text{low}}, 1 + \epsilon_{\text{high}})\hat{A}_{i,t} \big) & \text{if } \hat{A}_{i,t} \le 0. \end{cases} \right].\]The same principle gives UP-GRPO, which keeps GRPO’s symmetric clipping and KL penalty on the negative branch (Eq. 15):
\[J_{\text{UP-GRPO}}(\theta) = \mathbb{E}\!\left[ \frac{1}{G} \sum_{i} \frac{1}{|o_i|} \sum_{t} \begin{cases} \hat{A}_i\, \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) - \beta D_{\text{KL}}(\pi_\theta \,\|\, \pi_{\text{ref}}) & \text{if } \hat{A}_i > 0, \\[4pt] \min\!\big( r_{i,t}(\theta)\hat{A}_i,\, \mathrm{clip}(r_{i,t}(\theta), 1 - \epsilon, 1 + \epsilon)\hat{A}_i \big) - \beta D_{\text{KL}}(\pi_\theta \,\|\, \pi_{\text{ref}}) & \text{if } \hat{A}_i \le 0. \end{cases} \right].\]Finally, UP extends to the sequence level as UP-GSPO, whose positive branch provably reduces to a length-normalized REINFORCE gradient (Eq. 16):
\[J_{\text{UP-GSPO}}(\theta) = \mathbb{E}\!\left[ \frac{1}{G} \sum_{i} \begin{cases} \hat{A}_i \left( \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) \right) & \text{if } \hat{A}_i > 0, \\[4pt] \min\!\big( s_i(\theta)\hat{A}_i,\, \mathrm{clip}(s_i(\theta), 1 - \epsilon, 1 + \epsilon)\hat{A}_i \big) & \text{if } \hat{A}_i \le 0. \end{cases} \right].\]Takeaway. UP is a drop-in objective. It swaps the positive branch of any GxPO algorithm for a stop-gradient, unclipped, REINFORCE-equivalent update, and leaves the negative branch untouched as a safeguard.
Experiments
We evaluate UP on dense LLMs (Qwen3-14B-Base, Qwen3-8B), an MoE model (Qwen3-30B-A3B-Base), and a vision-language model (Qwen3-VL-8B-Instruct), using the verl framework. Across every setting UP raises accuracy and exploration while keeping training stable.
Performance, Exploration, and Stability Analysis of UP-DAPO
On Qwen3-14B-Base trained on DAPO-17K-MATH and evaluated on AIME24, UP-DAPO comprehensively outperforms DAPO. As shown in Figure 2, it lifts peak Avg@32 from \(47.71\) to \(51.15\) and Maj@32 from \(58.36\) to \(60.88\). Figure 3 traces the source of these gains: a consistently higher generation entropy alongside a higher Best@32 (from \(80.49\) to \(81.79\)) confirms that removing the \(\pi_{\text{old}}\) anchor genuinely enlarges exploration capacity. Crucially, Figure 4 shows the gradient norm and the KL divergence of UP-DAPO stay comparable to, or slightly below, DAPO’s, so this added exploration does not cost stability.
Figure 2. Performance comparison of DAPO and UP-DAPO on Qwen3-14B-Base during training: (a) Avg@32 and (b) Maj@32 accuracy on AIME24. UP-DAPO reaches a peak Avg@32 of 51.15 against DAPO's 47.71, and a peak Maj@32 of 60.88 against 58.36.
Figure 3. Exploration capacity of UP-DAPO: (a) entropy of generation probabilities on DAPO-17K-MATH and (b) Best@32 accuracy on AIME24. UP-DAPO sustains a higher entropy and lifts peak Best@32 to 81.79 against DAPO's 80.49.
Figure 4. Training stability of UP-DAPO: (a) gradient norm and (b) KL divergence between the active policy and the reference model on DAPO-17K-MATH. Both quantities track DAPO closely, so the extra exploration does not destabilize training.
Ablation Studies of UP-DAPO
Two ablations pin down why both design choices are necessary. As Figure 5 shows, simply relaxing DAPO’s upper clip to infinity while keeping the \(\pi_{\text{old}}\) anchor stays stable at first but explodes beyond 80 steps, with the gradient norm surging to \(10^{13}\); this shows the self-anchored ratio, not merely a loosened bound, is what enables safe unbounded updates. Applying the unbounded update symmetrically to both positive and negative advantages collapses within 25 steps, confirming that clipping on the negative branch is an indispensable safeguard.
Figure 5. Training stability of DAPO, DAPO with \(\epsilon_{\text{high}} = \infty\), REINFORCE (off-policy), and UP-DAPO on Qwen3-14B-Base: (a) gradient norm and (b) KL divergence, both on a log scale. DAPO with \(\epsilon_{\text{high}} = \infty\) explodes to \(10^{13}\) after about 80 steps, and the symmetric unbounded variant collapses within 25 steps, while UP-DAPO stays stable.
Comparison with Other RL Baselines
Under the unified protocol of Liu et al., we train Qwen3-8B on MATH (Levels 3 to 5) and evaluate Pass@1 across five reasoning benchmarks. As reported in Table 1, UP-GRPO attains the best average accuracy of \(61.31\%\), surpassing the strongest prior baseline GSPO (\(60.15\%\)) by an absolute \(1.16\%\). UP-GRPO ranks first or ties for first on four of the five individual benchmarks, and remains highly competitive on OlympiadBench, second only to ASPO. Consistent gains across benchmarks of varying difficulty show that the unbounded positive update is a broadly effective, rather than benchmark-specific, improvement over the GxPO family. Figure 6 explains these downstream gains: UP-GRPO’s policy entropy keeps rising throughout training while most baselines suffer entropy collapse, and its KL divergence stays among the lowest of the twelve methods.
| Method | AIME24 | AMC23 | MATH500 | Minerva | OlympiadBench | Average |
|---|---|---|---|---|---|---|
| GRPO | 35.73 | 75.00 | 86.00 | 30.88 | 51.34 | 55.79 |
| Dr. GRPO | 33.33 | 85.00 | 85.80 | 30.15 | 51.19 | 57.09 |
| CISPO | 38.02 | 87.50 | 86.60 | 29.04 | 55.65 | 59.36 |
| DPPO | 40.10 | 87.50 | 86.20 | 30.51 | 53.27 | 59.52 |
| GMPO | 37.50 | 87.50 | 87.00 | 31.25 | 55.06 | 59.66 |
| GSPO | 40.52 | 85.00 | 88.20 | 31.25 | 55.80 | 60.15 |
| SAPO | 39.90 | 82.50 | 87.20 | 30.88 | 55.65 | 59.23 |
| REINFORCE++ | 20.52 | 62.50 | 78.80 | 29.78 | 42.26 | 46.77 |
| RLOO | 31.67 | 80.00 | 85.20 | 28.68 | 50.60 | 55.23 |
| W-REINFORCE | 35.52 | 80.00 | 85.80 | 30.15 | 53.27 | 56.95 |
| ASPO | 37.50 | 85.00 | 87.60 | 29.78 | 58.48 | 59.67 |
| UP-GRPO (Ours) | 41.04 | 87.50 | 88.40 | 31.25 | 58.33 | 61.31 |
Table 1. UP-GRPO against eleven RL baselines on AIME24, AMC23, MATH500, Minerva, and OlympiadBench (Pass@1, %), trained on MATH (Levels 3 to 5) with Qwen3-8B. Best per column in bold.
Figure 6. Exploration capacity and training stability of UP-GRPO against eleven RL baselines on Qwen3-8B during training: (a) entropy and (b) KL divergence between the active policy and the reference model on MATH (Levels 3 to 5). Most baselines suffer entropy collapse, while UP-GRPO keeps a rising entropy and holds its KL divergence among the lowest.
Universality of UP across Algorithms, Architectures, and Modalities
UP is a universal plugin, not a token-level trick. On the MoE model Qwen3-30B-A3B-Base, Figure 7 shows UP-GSPO raises peak Avg@32 on AIME24 from GSPO’s \(52.71\%\) to \(55.73\%\), an absolute \(3.02\%\), while keeping KL divergence nearly identical to the baseline. On the vision-language model Qwen3-VL-8B-Instruct trained on Geometry3K, Figure 8 shows UP-GRPO lifts accuracy from GRPO’s \(59.30\%\) to \(62.60\%\), an absolute \(3.30\%\), again without any added instability. Together these results show UP resolves the exploration-stability dilemma regardless of optimization granularity (token-level to sequence-level), model architecture (dense to MoE), or modality (language to multimodal).
Figure 7. Performance and stability comparison of GSPO and UP-GSPO on Qwen3-30B-A3B-Base (MoE): (a) Avg@32 accuracy on AIME24 and (b) KL divergence on DAPO-17K-MATH. UP-GSPO reaches a peak Avg@32 of 55.73 against GSPO's 52.71, an absolute gain of 3.02.
Figure 8. Performance and stability comparison of GRPO and UP-GRPO on Qwen3-VL-8B-Instruct: (a) accuracy on the Geometry3K test set and (b) KL divergence. UP-GRPO reaches 62.60 against GRPO's 59.30, an absolute gain of 3.30.
BibTeX
@misc{fan2026upunboundedpositiveasymmetric,
title={UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma},
author={Chongyu Fan and Pengfei Liu and Jingjia Huang and Sijia Liu and Yi Lin},
year={2026},
eprint={2607.06987},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2607.06987},
}
TL;DR
- 现代面向 LLM 推理的强化学习 (RL) 依赖重要性采样 (IS) 来提升样本效率，但 IS 带来了探索与稳定的两难：纯 IS 会梯度爆炸，而用于抑制爆炸的裁剪又会严格限制策略的可变动幅度。
- 我们把一个 token 在梯度被截断前所拥有的更新预算形式化为 概率容量 (Probability Capacity, Cap)，并指出保守的裁剪把该预算与 \(\pi_{\text{old}}\) 线性绑定，从结构上饿死了那些正确但低置信的推理路径。
- Unbounded Positive Asymmetric Optimization (UP) 对正优势用 stop-gradient 自锚定比值 替换 IS 比值，从而还原出稳定、无裁剪的 REINFORCE 梯度；对负优势则保留标准裁剪作为安全阀。
- UP 是一个即插即用的目标，可实例化为 UP-DAPO、UP-GRPO 与 UP-GSPO，并在 DAPO、GSPO、GRPO，以及 dense / MoE / 视觉语言模型、语言与多模态数据上全面提升准确率与探索能力。
试一试：裁剪如何抑制探索？
下方的 图 1 是一张交互式地图，绘制了一个 token 在梯度被裁剪之前所能获得的更新预算，横轴为当前策略概率 \(\pi_\theta\)，纵轴为行为策略概率 \(\pi_{\text{old}}\)。在正优势 (\(\hat{A} > 0\)) 下把 方法 在 DAPO 与 UP-DAPO 之间切换，也可以查看非正优势区间 (\(\hat{A} \le 0\))，拖动 \(\epsilon_{\text{high}}\)、\(\epsilon_{\text{low}}\)、\(c\)，并把鼠标移到任意位置读取精确的容量值。核心对比在于：DAPO 通过一个很窄的带把预算与 \(\pi_{\text{old}}\) 绑定，而 UP-DAPO 把预算放大为完整的 \(1 - \pi_\theta\)，与 \(\pi_{\text{old}}\) 无关。
将鼠标移到图上，即可读取任意 (πθ, πold) 处的 Cap。
图 1. 交互式概率容量图（即本文的 Figure 1）。颜色编码更新预算 Cap，横轴为当前策略概率 πθ，纵轴为行为策略概率 πold。对正优势，DAPO 把 Cap 限制在由 πθ = (1 + εhigh)πold 界定的狭窄带内，而 UP-DAPO 把它放大为完整的 1 − πθ，与 πold 无关；对非正优势，两种方法共享同一个被裁剪的楔形区。
背景
从 REINFORCE 到 GRPO、DAPO 与 GSPO
RL 已成为激发 LLM 复杂推理能力的标准范式。原始的 REINFORCE 目标严格 on-policy，其梯度在数学上是稳定的：
\[\nabla_\theta J_{\text{REINFORCE}}(\theta) = \mathbb{E}_{q \sim Q,\, o \sim \pi_\theta}\!\left[ \sum_{t=1}^{|o|} \hat{A}\, \nabla_\theta \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) \right].\]REINFORCE 稳定但样本效率低，因为每条轨迹在一步更新后即被丢弃。为了在同一批采样数据上进行多步优化，GRPO 与 DAPO 引入 重要性采样 (IS)，用历史策略 \(\pi_{\text{old}}\) 采样并通过如下比值来估计当前策略 \(\pi_\theta\)：
\[r_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\text{old}}(o_{i,t} \mid q, o_{i,<t})}.\]为防止该比值爆炸，GRPO 通过裁剪把它约束在信任域 \([1 - \epsilon, 1 + \epsilon]\) 内。DAPO 进一步将上下界解耦为 \((\epsilon_{\text{low}}, \epsilon_{\text{high}})\) 并去掉 KL 惩罚，因此完全依赖裁剪来权衡稳定与探索。GSPO 则转向 序列级 比值，采用整条回复的长度归一化似然：
\[s_i(\theta) = \left( \frac{\pi_\theta(o_i \mid q)}{\pi_{\text{old}}(o_i \mid q)} \right)^{\frac{1}{|o_i|}} = \exp\!\left( \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \big( \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) - \log \pi_{\text{old}}(o_{i,t} \mid q, o_{i,<t}) \big) \right).\]这些方法都把 \(\pi_{\text{old}}\) 留在分母中，也都依赖裁剪。正是这一共同结构催生了下面的两难。
概率容量 (Cap)
为精确刻画裁剪的代价，我们把 概率容量 (Probability Capacity, Cap) 定义为：对正优势 token (\(\hat{A} > 0\))，在梯度被截断前 \(\pi_\theta\) 所能增大的最大幅度；对非正优势 token (\(\hat{A} \le 0\))，则为所能减小的最大幅度。Cap 就是一个 token 相对 \(\pi_{\text{old}}\) 用于探索的有效 预算。图 1 直接绘制了这一预算，而本文其余部分讲的正是 UP 如何把它放大。
动机
IS-based RL 夹在两种失效模式之间：更新过猛（纯 IS）会触发不稳定；约束过严（裁剪）会抑制探索。我们逐一分析。
困境一：重要性采样导致训练不稳定
无裁剪的 IS 目标用比值加权优势，\(J_{\text{IS}}(\theta) = \mathbb{E}[\hat{A}\, r_{i,t}(\theta)]\)，其梯度为：
\[\nabla_\theta J_{\text{IS}}(\theta) = \mathbb{E}_{q \sim Q,\, o \sim \pi_{\text{old}}}\!\left[ \hat{A}\, r_{i,t}(\theta)\, \nabla_\theta \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) \right].\]与 REINFORCE 梯度对比，问题一目了然：IS 梯度被 \(r_{i,t}(\theta)\) 显式缩放。对一条稀有的高回报推理路径，其行为概率 \(\pi_{\text{old}}\) 极小，只要当前策略略有提升，该比值便会爆炸。这种病态缩放会破坏稳定的表征，使训练发散，这正是裁剪被视为不可或缺的原因。
困境二：裁剪抑制探索
对正优势，裁剪后的目标为 \(J_{\text{IS+CLIP}}(\theta) = \min\!\big[r_{i,t}(\theta), 1 + \epsilon\big]\hat{A}\)，一旦比值越过信任域上界，其梯度立即归零。映射到绝对概率空间，策略被上界约束为 \(\pi_\theta \le (1 + \epsilon_{\text{high}})\pi_{\text{old}}\)，于是剩余预算服从如下分段 Cap：
\[\mathrm{Cap}(\pi_\theta, \pi_{\text{old}}) = \begin{cases} \min\!\big(1,\, (1 + \epsilon_{\text{high}})\pi_{\text{old}}\big) - \pi_\theta & \text{if } \pi_\theta < (1 + \epsilon_{\text{high}})\pi_{\text{old}}, \\[4pt] 0 & \text{if } \pi_\theta \ge (1 + \epsilon_{\text{high}})\pi_{\text{old}}. \end{cases}\]这暴露出一个结构性瓶颈：预算 线性依赖于 \(\pi_{\text{old}}\)。对一个低似然的推理 token，\(\pi_{\text{old}} = 0.01\)，标准的 \(\epsilon_{\text{high}} = 0.28\) 只允许绝对概率增大区区 \(0.0028\)。一旦 \(\pi_\theta\) 达到 \(0.0128\)，容量便降为零，token 进入 Upper Clip 区间，即便其优势极高，梯度也被抹除。在 图 1 中把方法设为 DAPO，观察地图底部 \(\pi_{\text{old}}\) 很小的区域，即可亲眼看到这一狭窄的预算。从结构上看，这抑制了探索，也让有潜力的长尾轨迹无从被发现。
核心问题。 如何在 RL 中最大化模型探索的同时防止训练不稳定？更新过猛会破坏稳定，保守裁剪会饿死探索。我们需要一种能同时跳出这两者的形式化方法。
方法
UP 通过两步数学操作来化解两难：用 stop-gradient 把 \(\pi_{\text{old}}\) 替换为当前策略 \(\pi_\theta\)，并按优势的符号非对称地路由梯度。
面向正优势的无界目标
对正确 rollout (\(\hat{A} > 0\))，我们用基于 stop-gradient 算子 \(\mathrm{sg}(\cdot)\) 构造的 自锚定比值 \(\tilde{r}_{i,t}(\theta)\) 替换 IS 比值（式 12）：
\[J^{+}_{\text{UP}}(\theta) = \mathbb{E}_{q \sim Q,\, o \sim \pi_{\text{old}}}\!\left[ \sum_{t=1}^{|o|} \hat{A}\, \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\mathrm{sg}\!\big(\pi_\theta(o_{i,t} \mid q, o_{i,<t})\big)} \right], \qquad \hat{A} > 0.\]反向传播时 \(\mathrm{sg}(\pi_\theta)\) 是常数标量，但其前向取值恰等于 \(\pi_\theta\)。应用对数导数技巧 \(\nabla x / x = \nabla \log x\)，目标坍缩为（式 13）：
\[\nabla_\theta J^{+}_{\text{UP}}(\theta) = \mathbb{E}_{q \sim Q,\, o \sim \pi_{\text{old}}}\!\left[ \sum_{t=1}^{|o|} \hat{A}\, \nabla_\theta \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) \right].\]这是一个深刻的简化：优化自锚定比值在数学上等价于最大化 REINFORCE 目标。通过把策略锚定到 \(\pi_\theta\) 而非 \(\pi_{\text{old}}\)，UP 从根源上消除了困境一中的 IS 不稳定，其 Cap 也不再受历史约束，\(\mathrm{Cap} = 1 - \pi_\theta\)。在 图 1 中把方法切换为 UP-DAPO 并取 \(\hat{A} > 0\)：狭窄的裁剪带会扩展为只依赖 \(\pi_\theta\) 的完整预算。
为何非对称：负优势保留裁剪
无界机制被有意限制在正确 rollout 上。对错误 rollout，优势为负，会反转梯度方向；在此处施加无界更新等于做激进的梯度上升，会破坏原有表征。因此 UP 对 \(\hat{A} \le 0\) 保留标准裁剪，此时 DAPO 与 UP-DAPO 共享一个由下界裁剪与 dual clip 边界决定的相同 Cap（见 图 1 中的 \(\hat{A} \le 0\) 视图）：
\[\mathrm{Cap}(\pi_\theta, \pi_{\text{old}}) = \begin{cases} \pi_\theta - (1 - \epsilon_{\text{low}})\pi_{\text{old}} & \text{if } (1 - \epsilon_{\text{low}})\pi_{\text{old}} \le \pi_\theta \le c\,\pi_{\text{old}}, \\[4pt] 0 & \text{if } \pi_\theta < (1 - \epsilon_{\text{low}})\pi_{\text{old}} \quad (\text{Lower Clip}), \\[4pt] 0 & \text{if } \pi_\theta > c\,\pi_{\text{old}} \quad (\text{Dual Clip}). \end{cases}\]这正是 UP 的关键：对正确 rollout 放开探索，对错误 rollout 守住安全。
UP-GxPO：一个插件，三种实例化
由于只改写正优势分支，UP 可插入任意 group-based 策略优化器。token 级的 UP-DAPO 把无界正目标与 DAPO 对负优势的裁剪目标组合（式 14）：
\[J_{\text{UP-DAPO}}(\theta) = \mathbb{E}\!\left[ \frac{1}{\sum_i |o_i|} \sum_{i}\sum_{t} \begin{cases} \hat{A}_{i,t}\, \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) & \text{if } \hat{A}_{i,t} > 0, \\[4pt] \min\!\big( r_{i,t}(\theta)\hat{A}_{i,t},\, \mathrm{clip}(r_{i,t}(\theta), 1 - \epsilon_{\text{low}}, 1 + \epsilon_{\text{high}})\hat{A}_{i,t} \big) & \text{if } \hat{A}_{i,t} \le 0. \end{cases} \right].\]同一原理给出 UP-GRPO，其负优势分支保留 GRPO 的对称裁剪与 KL 惩罚（式 15）：
\[J_{\text{UP-GRPO}}(\theta) = \mathbb{E}\!\left[ \frac{1}{G} \sum_{i} \frac{1}{|o_i|} \sum_{t} \begin{cases} \hat{A}_i\, \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) - \beta D_{\text{KL}}(\pi_\theta \,\|\, \pi_{\text{ref}}) & \text{if } \hat{A}_i > 0, \\[4pt] \min\!\big( r_{i,t}(\theta)\hat{A}_i,\, \mathrm{clip}(r_{i,t}(\theta), 1 - \epsilon, 1 + \epsilon)\hat{A}_i \big) - \beta D_{\text{KL}}(\pi_\theta \,\|\, \pi_{\text{ref}}) & \text{if } \hat{A}_i \le 0. \end{cases} \right].\]最后，UP 扩展到序列级得到 UP-GSPO，其正优势分支可证明地退化为长度归一化的 REINFORCE 梯度（式 16）：
\[J_{\text{UP-GSPO}}(\theta) = \mathbb{E}\!\left[ \frac{1}{G} \sum_{i} \begin{cases} \hat{A}_i \left( \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) \right) & \text{if } \hat{A}_i > 0, \\[4pt] \min\!\big( s_i(\theta)\hat{A}_i,\, \mathrm{clip}(s_i(\theta), 1 - \epsilon, 1 + \epsilon)\hat{A}_i \big) & \text{if } \hat{A}_i \le 0. \end{cases} \right].\]一句话总结。 UP 是一个即插即用的目标：它把任意 GxPO 算法的正优势分支替换为 stop-gradient、无裁剪、等价于 REINFORCE 的更新，而负优势分支原封不动，充当安全阀。
实验
我们在 dense LLM（Qwen3-14B-Base、Qwen3-8B）、MoE 模型（Qwen3-30B-A3B-Base）以及视觉语言模型（Qwen3-VL-8B-Instruct）上评估 UP，实现基于 verl 框架。在所有 setting 上，UP 都能在保持训练稳定的同时提升准确率与探索能力。
UP-DAPO 的性能、探索与稳定性分析
在 Qwen3-14B-Base 上用 DAPO-17K-MATH 训练并在 AIME24 评估，UP-DAPO 全面优于 DAPO。如 图 2 所示，峰值 Avg@32 从 \(47.71\) 提升至 \(51.15\)，Maj@32 从 \(58.36\) 提升至 \(60.88\)。图 3 揭示了增益来源：始终更高的生成熵，加上更高的 Best@32（从 \(80.49\) 提升至 \(81.79\)），共同证实移除 \(\pi_{\text{old}}\) 锚点确实放大了探索能力。更关键的是，图 4 表明 UP-DAPO 的梯度范数与 KL 散度与 DAPO 相当甚至略低，说明这份额外的探索并未以牺牲稳定为代价。
图 2. DAPO 与 UP-DAPO 在 Qwen3-14B-Base 训练过程中的性能对比：(a) AIME24 上的 Avg@32 与 (b) Maj@32 准确率。UP-DAPO 的峰值 Avg@32 为 51.15，DAPO 为 47.71；峰值 Maj@32 为 60.88，DAPO 为 58.36。
图 3. UP-DAPO 的探索能力：(a) DAPO-17K-MATH 上生成概率的熵，(b) AIME24 上的 Best@32 准确率。UP-DAPO 维持更高的熵，并把峰值 Best@32 提升至 81.79，DAPO 为 80.49。
图 4. UP-DAPO 的训练稳定性：(a) 梯度范数，(b) DAPO-17K-MATH 上活跃策略与参考模型之间的 KL 散度。两者都与 DAPO 高度贴合，说明额外的探索不会破坏训练稳定。
UP-DAPO 的消融研究
两项消融进一步锁定了两个设计各自的必要性。如 图 5 所示，仅仅把 DAPO 的上界裁剪放宽到无穷、同时仍保留 \(\pi_{\text{old}}\) 锚点，前期尚能稳定，但在 80 步后爆炸，梯度范数飙升至 \(10^{13}\)；这说明真正让无界更新变得安全的是自锚定比值，而非单纯放宽边界。若把无界更新对称地施加到正负优势两侧，训练在 25 步内即崩溃，证明负优势分支上的裁剪是不可或缺的安全阀。
图 5. DAPO、DAPO（\(\epsilon_{\text{high}} = \infty\)）、REINFORCE（off-policy）与 UP-DAPO 在 Qwen3-14B-Base 上的训练稳定性：(a) 梯度范数，(b) KL 散度，均为对数坐标。DAPO（\(\epsilon_{\text{high}} = \infty\)）在约 80 步后爆炸至 \(10^{13}\)，对称无界变体在 25 步内崩溃，而 UP-DAPO 始终稳定。
与其他 RL 基线的比较
在 Liu et al. 的统一协议下，我们用 Qwen3-8B 在 MATH（难度 3 至 5）上训练，并在五个推理基准上评估 Pass@1。如 表 1 所示，UP-GRPO 取得最高平均准确率 \(61.31\%\)，比此前最强基线 GSPO（\(60.15\%\)）高出 \(1.16\%\) 的绝对值。UP-GRPO 在五个单项基准中的四个上排名第一或并列第一，在 OlympiadBench 上也极具竞争力，仅次于 ASPO。跨越不同难度基准的稳定增益表明，无界正更新是对 GxPO 家族的普遍改进，而非针对某个基准的特例。图 6 为这些下游增益提供了解释：UP-GRPO 的策略熵在整段训练中持续上升，而多数基线出现熵坍缩；其 KL 散度也维持在十二种方法中的最低之列。
| 方法 | AIME24 | AMC23 | MATH500 | Minerva | OlympiadBench | 平均 |
|---|---|---|---|---|---|---|
| GRPO | 35.73 | 75.00 | 86.00 | 30.88 | 51.34 | 55.79 |
| Dr. GRPO | 33.33 | 85.00 | 85.80 | 30.15 | 51.19 | 57.09 |
| CISPO | 38.02 | 87.50 | 86.60 | 29.04 | 55.65 | 59.36 |
| DPPO | 40.10 | 87.50 | 86.20 | 30.51 | 53.27 | 59.52 |
| GMPO | 37.50 | 87.50 | 87.00 | 31.25 | 55.06 | 59.66 |
| GSPO | 40.52 | 85.00 | 88.20 | 31.25 | 55.80 | 60.15 |
| SAPO | 39.90 | 82.50 | 87.20 | 30.88 | 55.65 | 59.23 |
| REINFORCE++ | 20.52 | 62.50 | 78.80 | 29.78 | 42.26 | 46.77 |
| RLOO | 31.67 | 80.00 | 85.20 | 28.68 | 50.60 | 55.23 |
| W-REINFORCE | 35.52 | 80.00 | 85.80 | 30.15 | 53.27 | 56.95 |
| ASPO | 37.50 | 85.00 | 87.60 | 29.78 | 58.48 | 59.67 |
| UP-GRPO (Ours) | 41.04 | 87.50 | 88.40 | 31.25 | 58.33 | 61.31 |
表 1. UP-GRPO 与十一个 RL 基线在 AIME24、AMC23、MATH500、Minerva、OlympiadBench 上的对比（Pass@1，%），使用 Qwen3-8B 在 MATH（难度 3 至 5）上训练。每列最优用粗体。
图 6. UP-GRPO 与十一个 RL 基线在 Qwen3-8B 训练过程中的探索能力与训练稳定性：(a) 熵，(b) MATH（难度 3 至 5）上活跃策略与参考模型之间的 KL 散度。多数基线出现熵坍缩，而 UP-GRPO 的熵持续上升，且 KL 散度维持在最低之列。
UP 在算法、架构与模态上的通用性
UP 是通用插件，而非 token 级的技巧。在 MoE 模型 Qwen3-30B-A3B-Base 上，图 7 显示 UP-GSPO 把 AIME24 的峰值 Avg@32 从 GSPO 的 \(52.71\%\) 提升至 \(55.73\%\)，绝对提升 \(3.02\%\)，同时 KL 散度与基线几乎一致。在 视觉语言 模型 Qwen3-VL-8B-Instruct 上用 Geometry3K 训练，图 8 显示 UP-GRPO 把准确率从 GRPO 的 \(59.30\%\) 提升至 \(62.60\%\)，绝对提升 \(3.30\%\)，同样没有引入任何额外不稳定。这两个 setting 共同表明，无论优化粒度（token 级到序列级）、模型架构（dense 到 MoE）还是模态（语言到多模态），UP 都能化解探索与稳定的两难。
图 7. GSPO 与 UP-GSPO 在 MoE 模型 Qwen3-30B-A3B-Base 训练过程中的性能与稳定性：(a) AIME24 上的 Avg@32 准确率，(b) DAPO-17K-MATH 上的 KL 散度。UP-GSPO 把峰值 Avg@32 从 52.71 提升至 55.73，且 KL 轨迹几乎一致。
图 8. GRPO 与 UP-GRPO 在视觉语言模型 Qwen3-VL-8B-Instruct（Geometry3K）训练过程中的性能与稳定性：(a) Geometry3K 测试集上的准确率，(b) KL 散度。UP-GRPO 把峰值准确率从 59.30 提升至 62.60，且未引入额外不稳定。
BibTeX
@misc{fan2026upunboundedpositiveasymmetric,
title={UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma},
author={Chongyu Fan and Pengfei Liu and Jingjia Huang and Sijia Liu and Yi Lin},
year={2026},
eprint={2607.06987},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2607.06987},
}
