Cognition 发布 FrontierCode 2.0:用不充分规格测“该不该合并”的判断
Cognition 发布编码基准 FrontierCode 2.0:用更多样、更大范围且刻意不充分的提示,考察模型是否不仅能写满足规格的代码,还具备应否合并 PR 的工程判断;当前榜首约 45%。
编码评测从“能不能写对”进一步问“该不该合”。
Cognition 介绍 FrontierCode 2.0:前作 FrontierCode 1.0 已从“符合规格”推进到“你会不会合并这个 PR”;2.0 任务更多样、范围更大、合并标准更高,并加入刻意不充分的提示,留给模型做工程判断的空间。官方称基准远未饱和,顶尖模型得分仅约 45%。配图榜单中 Claude Opus 5.5 约 45.5%,其后为 GPT-6 Astra、Claude Sonnet 5.5 等。
对研发效能与模型选型团队,这提供了另一把尺子:不只看单测绿,还看在模糊需求下的默认决策质量。引用分数时需注明是 Cognition 自建基准,且模型名以官方榜单为准。
当基准提示故意留白,模型必须补齐工程默认值:错误处理、命名、测试与回滚策略。这更接近真实 PR review 的主观部分。选型团队可把 FrontierCode 与 SWE-bench 等已有榜单交叉看,避免被单一厂商基准带偏;对外沟通时应标明分数来源与评测日期。
flowchart TB
P[不充分提示] --> M[编码模型]
M --> PR[生成 PR]
PR --> J{是否该合并?}
J -->|评测| S[FrontierCode 2.0 得分]
对行业读者而言,关键是核对官方文档中的适用范围、权限模型与发布时间表,避免把预览或试点能力误当成全量通用能力;同时保留人工复核节点,把智能体输出当作待确认草稿而非最终生产指令。
Cognition FrontierCode 2.0 Benchmarks Merge Judgment Beyond Specs
Introducing FrontierCode 2.0: our new coding benchmark that measures a model’s ability to go beyond the spec.
Most coding evals measure a model’s ability to write code that meets a spec. With FrontierCode 1.0, we asked a new question: would you actually merge the PR?
FrontierCode 2.0 is comprised of more diverse tasks, featuring larger scopes and higher mergeability standards. We are now asking: not only would you merge this PR, but should you?
One key feature of FrontierCode 2.0 is underspecified prompts. Rather than including every engineering decision, prompts are deliberately ambiguous, giving the models room to demonstrate their judgment.
The result is a benchmark that is far from saturated, with the top model scoring just 45%.