FAULT 预印本:把智能体自诊断错误变成有代价上限的逐步信用信号
阿里巴巴、京都大学、NII 等合作的 FAULT 预印本提出:在强化学习里把「错误发生在哪一步」与「这类错误有多贵」分开,用轨迹引用做诊断,并把有界惩罚预算打到被点名的步骤上,从而在全失败组里仍可能产生相对学习信号。
预印本 FAULT(arXiv:2610.01161,阿里巴巴、京都大学、NII 等)针对一类常见困境:当一组 rollout 全部失败时,GRPO 一类方法往往只能看到「平局」,但「只犯一个可恢复错误」和「一路乱决策」本不该学到同一课。作者把自诊断错误转成逐步信用:诊断必须引用轨迹中的某一步与原句;错误代价由同任务内成功/失败差异拟合,而不是按反思措辞有多严重来拍脑袋;再把有界惩罚预算分到被诊断步骤上,且上限低于二元成败奖励差。
帖文作者强调:这能在「全失败但仍有不同惩罚」的组里制造相对信号;错误价格本身仍需从含成功与失败的组里学,不能单靠全失败数据发明代价。ALFWorld 上作者报告可用信号覆盖约 95% 的组(对照 GRPO 约 41%、GiGPO 约 72%,并共享诊断-SFT 初始化);4B 规模成功率约 91.0%,对照同初始化的 GiGPO 约 80.6%。弱环节仍是诊断质量:确认引文存在 ≠ 确认它真是错误;初始诊断器在 ±2 步容差下主错误定位准确率约 0.48,低于文中 0.60 标准;与结果相关的错误价格也不是因果效应。
对做长程智能体 RL 的人,这篇工作的启发是:与其追更华丽的反思,不如把「声称的错误」绑回结果,并限制糟糕诊断能撬动多少学习。论文:https://arxiv.org/abs/2610.01161 ;来源帖:https://x.com/yanlecheng1/status/2114223284896125201 。
FAULT: step-level credit from self-diagnosed errors
When every rollout fails, GRPO sees a tie. But a run with one recoverable mistake and a run full of bad decisions should not necessarily teach the same lesson.
My FAULT, a preprint from Alibaba, Kyoto University, NII and collaborators, turns self-diagnosed errors into step-level credit. The interesting move is separating two questions: where did the error happen, and how costly is that kind of error?
Diagnoses must cite a step and an exact quote from the trajectory. Error costs are then fitted from success/failure differences within the same task, rather than assigned according to how severe a reflection sounds. A bounded penalty budget is distributed over the diagnosed steps, with its cap kept below the binary success–failure reward gap.
This can create relative learning signals even in all-fail groups, provided their diagnosed penalties differ. The error prices themselves are learned from groups containing both outcomes; the method is not discovering costs from all-fail data alone.
On ALFWorld, the authors report usable signal in 95% of groups versus 41% for GRPO and 72% for GiGPO, with shared diagnosis-SFT initialization. At 4B, success reaches 91.0%, versus 80.6% for GiGPO starting from that same initialization. The cold-start training matters to this comparison.
The weak link is still diagnosis. Checking that a quote exists does not establish that it describes a real error. The initial diagnoser's dominant-error localization accuracy is 0.48 even with a ±2-step tolerance, below the paper's stated 0.60 criterion. Outcome-associated error prices are also not causal effects.
For long-horizon agents, I find this more useful than asking for increasingly elaborate reflections: tie the severity of a claimed mistake back to outcomes, and limit how much an imperfect diagnosis can steer learning.