Epoch AI 复核 15 项基准:仅 4 项完全过关,SWE-bench Verified 等被标有缺陷
Epoch AI 的 Benchmark Reviews 在 15 项常用基准中仅完全核实 4 项;含 SWE-bench Verified、Terminal-Bench 等在内的 9 项被标为有缺陷。
Epoch AI 的 Benchmark Reviews 覆盖 15 项被广泛引用的基准测试,结果是:仅 4 项被完全核实——ExploitBench v0.1、SimpleQA Verified、PostTrainBench v1.1 与 WeirdML v2;另外 9 项被标记为有缺陷,包括 SWE-bench Verified、SWE-Bench Pro、Terminal-Bench 4.0.0、Humanity's Last Exam、HealthBench Professional 等;还有 2 项结论不定。
「有缺陷」的门槛是:在分层抽取的 50 个任务样本中错误率不低于 20%,或存在足以让样本量不再重要的系统性评分问题。问题类型包括无效任务、假阴性与假阳性、激发不足、设置偏差,以及排行榜背后基准版本静默变更。同一综述还引用 FrontierMath v1 约 42% 题目存在错误的发现。
对编程智能体市场尤其敏感的是 SWE-bench Verified 上榜:它几乎是各 CLI 与 IDE Agent 对外引用最多的成绩单。评测从业者 Hamel Husain 批评行业依赖与基准质量之间的落差;帖子同时提到 Braintrust 等公司靠自建评测融资与获客,说明「自己跑失败样例」正在变成刚需。
结论很硬:排行榜数字在有人手动复现失败例之前,更接近营销材料。选 Agent 或写对比文时,应优先看任务集版本、评分器与失败案例,而不是只报一个百分比。
综合来看,读者可先核验官方渠道与发布时间,再结合自身业务边界与合规要求决定是否跟进试点;本文仅整理公开信息,不构成采购、投资或法律建议,关键数字与能力边界请以一手文档为准。
Epoch AI Benchmark Review Flags Many Popular Evals Including SWE-bench
Nine benchmarks, not two or three. Epoch AI ran its Benchmark Reviews program across 15 widely cited tests and could only fully verify four of them: ExploitBench v0.1, SimpleQA Verified, PostTrainBench v1.1, and WeirdML v2. Nine others, including SWE-bench Verified, SWE-Bench Pro, Terminal-Bench 4.0.0, Humanity's Last Exam, and HealthBench Professional, got flagged flawed. Two more came back inconclusive.
The bar for flawed is an error rate of 20% or higher in a stratified sample of 50 tasks, or a grading problem systemic enough that sample size stops mattering. The errors aren't typos. Reviewers found invalid tasks, false negatives, false positives, under-elicitation, setup bias, and silent versioning, where a benchmark quietly changed underneath the leaderboard. A separate citation in the same review found errors in 42% of FrontierMath v1 problems.
SWE-bench Verified making that list matters more than the others because it's the single most quoted number in coding-agent marketing right now, the same leaderboard every CLI and IDE agent cites to prove itself. Hamel Husain, who has spent his career building evals, called the gap between benchmark quality and industry reliance on them astounding, and he's not alone in paying to close it. Braintrust raised a 36 million dollar Series A led by a16z at a 150 million dollar valuation this year, with Notion, Stripe, Airtable, and Zapier among the customers paying between five figures and over 100,000 dollars to run their own evals instead of trusting a published score. Braintrust's own numbers claim accuracy often climbs from under 40% to over 80% within weeks of adopting real evaluation.
A leaderboard number is marketing until someone runs the failed examples by hand. Most teams still skip that step because the benchmark did the work of looking rigorous for them.