DC-Jev-RL
Five papers on Kev-4B as a reward judge for PPO and GRPO. Every number in them is computed from logged runs. 中文
Code, logs and data (GitHub) Policy adapters Project page
Papers
Language Judges as Reward Sources for PPO in MuJoCo: Kev-4B versus an Autoregressive Judge of the Same Size
In 165 PPO runs (seven reward sources, five tasks), Kev-4B's gated reward reaches an interquartile mean normalised score of 1.00 [0.81, 1.09], the engineered reward 1.05, the autoregressive judge's 0.35. On Reacher every judge's reward ends below a random policy, and the autoregressive judge's reward teaches Hopper and Walker2d to stand still.
One Backbone for the Policy and the Judge: What Kev-4B Saves and What It Adds to GRPO on MATH
With one shared backbone the trainer needs 48% less GPU memory after loading, and the judge's probabilities stay within 0.015 of fp32. The verifier alone takes MATH-500 from 76.5% to 82.5%. With Kev-4B's score added to the reward, or gating it, one seed per condition ends far ahead and the others near the verifier-only runs, so the gain cannot be attributed to the judge. Kev-4B alone as the reward: 70.7%.
Visible Tests, Held-Out Tests and a Judge: Measuring Reward Hacking in Multi-Turn Code Repair with GRPO
No sign that the visible tests were exploited: the share of problems solved rose (KodCode validation 65.5% to 86.6%) and the share of visible-test passes that are wrong fell by about half on two of the three sets. The judge factor changed nothing beyond the variation between seeds.
When the Verifier Labels Only Some Samples: A Frozen, a Recalibrated and a Scratch Judge Head in GRPO
The frozen Kev-4B head ranks the unlabelled samples well (AUROC 0.91), but its policies learned to repeat themselves; on MATH-500 they end −2.3 points [−4.4, −0.1] against dropping those samples. A head trained from scratch ends +4.1 [1.6, 6.6] against dropping them, with both seeds switched to drafting first.
Can a Judge Choose Where to Branch? Prefix Selection for Tree Rollouts in GRPO on MATH
Kev-4B's choice of roots raised the share of correct leaves (+1.4 points over keeping all four roots), and the final policies are as accurate as with random roots (MATH-500 +0.7 [−0.8, 2.3]). The tree runs' lead over flat GRPO comes mostly from the 2 seeds that switched to drafting first. Trees generate 7%–12% fewer tokens than flat sampling of completions of the same lengths.
Across the GRPO papers
- Drafting first. Of the 26 GRPO runs of Parts II, III-B and III-C, the 6 whose final policies open most solutions with an informal draft in plain text are the 6 best on AIME24 (30.4–34.6%, every other run at most 24.4%). They come from 5 conditions, 2 of them rewarded by the verifier alone. We found the switch after the runs had finished and defined its measure after reading the completions.
- Repetition. Kev-4B pays for solutions that solve the problem again after giving the answer. 6 of the 11 runs with Kev-4B's score in the reward ended repeating themselves, against 2 of the 15 others; repeating did not make a policy more accurate.
Intervals are 95% bootstrap intervals: over runs, stratified by task, in Part I; over problems in the other parts, where they do not include the variation between seeds, which in these runs is often larger than the differences between conditions.
中文
五篇论文研究 Kev-4B 作为 PPO 和 GRPO 的奖励评判模型，其中的每个数字都由记录下来的 run 计算得出。每篇论文都有英文版和中文版，链接见上方。
- 第一部分（MuJoCo 上的 PPO）：在 165 个 PPO run（七种奖励来源、五个任务）中，Kev-4B 带门控的奖励的归一化得分四分位均值为 1.00 [0.81, 1.09]，工程化奖励为 1.05，自回归评判模型为 0.35。在 Reacher 上所有评判奖励最终都不如随机策略；自回归评判模型的奖励让 Hopper 和 Walker2d 学会了站着不动。
- 第二部分（共用骨干，MATH 上的 GRPO）：共用一份骨干时，训练器加载后少用 48% 的显存，评判模型的概率与 fp32 相差不超过 0.015。只用验证器就把 MATH-500 从 76.5% 提高到 82.5%。把 Kev-4B 的分数加入奖励或用它给验证器做门控时，每个条件有一个种子遥遥领先，其余与只用验证器的 run 相近，因此无法把增益归于评判模型。只用 Kev-4B 作奖励：70.7%。
- 第三部分 A（多轮代码修复）：没有看到可见测试被钻空子的迹象：解出题目的比例上升（KodCode 验证集从 65.5% 到 86.6%），通过可见测试的程序中错误者的比例在三个集合中的两个上降低了约一半。评判因子带来的变化没有超出种子之间的差异。
- 第三部分 B（部分标签）：冻结的 Kev-4B 输出头对未标注样本排序良好（AUROC 0.91），但它的策略学会了重复自己；在 MATH-500 上，相对于丢掉这些样本，它们最终为 −2.3 个百分点 [−4.4, −0.1]。从头训练的输出头为 +4.1 [1.6, 6.6]，它的两个种子都改成了先写草稿。
- 第三部分 C（树状 rollout）：Kev-4B 选根让正确叶子的比例比保留全部四个根时提高 1.4 个百分点，最终策略与随机选根一样准确（MATH-500 +0.7 [−0.8, 2.3]）。树状 run 相对扁平 GRPO 的领先主要来自改成先写草稿的 2 个种子。与生成同样长度解答的扁平采样相比，树少生成 7%–12% 的词元。
- 先写草稿：26 个 GRPO run 中，最终策略的多数解答以纯文本非正式草稿开头的 6 个 run，正是 AIME24 上最好的 6 个（30.4–34.6%，其他 run 至多 24.4%）；它们分属 5 个条件，其中 2 个的奖励只有验证器。这一切换是在 run 结束后才发现的，度量也是在读过解答之后定义的。
- 重复：Kev-4B 会为给出答案之后把题目再解一遍的解答付分。奖励中含有 Kev-4B 分数的 11 个 run 中有 6 个最终重复自己，其他 15 个 run 中只有 2 个。