读懂 METR 曲线Reading the METR Curve
一个测量结果揭示的自动化边界、价值迁移与法律分界What a single measurement reveals about automation, value, and law
METR 每个模型只发布一个数字。这个数字很奇怪——不是准确率,不是分数,而是一个时长:以人类专家完成所需的时间衡量,模型在给定可靠性下能完成多长的任务。
它是当前 AI 评测领域最有用的数字,而几乎所有人都读错了它。
更值得注意的是它背后的方法论。为了能测出任何东西,METR 必须把任务做成自包含、明确定义、可自动判分的。这三个约束不是偶然的技术细节——它们恰好描述了自动化本身的边界。顺着这个含义往外推,还会落到哪些数字产品保值、以及版权保护什么。
本报告从这台仪器讲起,说清它说了什么,然后把这套框架推到它被设计的领域之外。
METR publishes one number per model. It is a strange number — not an accuracy, not a score, but a duration: how long a task, measured by how long a human expert takes, a model can complete at a given reliability.
It is the most useful number in AI evaluation right now, and almost everyone reads it wrong.
More interestingly, the methodology behind it encodes a general theory of what can be automated. METR had to make their tasks self-contained, well-specified and algorithmically scorable in order to measure anything at all. Those constraints are not incidental — they describe the boundary of automation itself. Follow the implication outward and it lands on which digital goods hold value, and on what copyright protects.
This report starts with the instrument, works through what it says, then pushes the framework past the domain it was built for.
第一部分 — 这台仪器Part I — The instrument
一个数据点是怎么产生的How a single point is produced
这个构造方式与直觉相反。METR 并不是看着模型干活然后掐表,而是先给人类计时,再拟合一条曲线。
- 从 RE-Bench、HCAST 和一组更短的新编软件任务里取题,主要是软件工程、机器学习和网络安全,全部自包含、成功判据可自动检验。
- 雇佣熟练从业者(平均约 5 年经验)完成每个任务,取成功者完成时间的几何平均,记作该任务的人类时长 T。
- 让模型在同样环境下作答,每个任务 6 次独立尝试。
- 以 log T 为自变量,对成功与否做 logistic 回归。
The construction runs backwards from intuition. METR does not watch a model work and time it. They time humans, then fit a curve.
- Assemble tasks from RE-Bench, HCAST and a set of shorter novel software tasks — primarily software engineering, ML and cybersecurity, all self-contained with automatically checkable success criteria.
- Contract skilled professionals (averaging ~5 years of experience) to attempt each task. Take the geometric mean of successful completion times. This is the task's human duration T.
- Run the model in the same environment, six independent attempts per task.
- Fit a logistic regression of success against log T.
q-可靠性时间跨度 H_q 是拟合曲线穿过 q 的那个时长:
The q-reliability time horizon H_q is the duration where the fitted curve crosses q:
由于 logit(0.5) = 0,头条数字简化为 H₅₀ = e^(−a/b)。报告两个阈值:H₅₀ 与 H₈₀。
Since logit(0.5) = 0, the headline number simplifies to H₅₀ = e^(−a/b). Two thresholds get reported: H₅₀ and H₈₀.
三种误读Three misreadings
它不是模型能无人值守运行多久。METR 明确说明:这个指标描述的是任务难度,不是 AI 的墙钟时间。成功的模型通常比人类快几倍,因为动作更少——常常一次写对而不用反复迭代,查的东西也更少。
It is not how long a model runs unattended. METR states this directly: the metric describes task difficulty, not AI wall-clock time. Models that succeed are typically several times faster than the humans, because they take fewer actions — often writing code in one shot rather than iterating, and looking up less.
它不是"熟练老手的 T 小时"。METR 的任务被设计为自包含。真实工作依赖先前对话、隐性知识和对代码库的熟悉。他们自己的指引是:把两小时任务理解成一个低背景的人——新入职或外包——能在两小时内做完的事。在一个耦合未文档化的成熟系统里,这两者差距很大。
It is not "what an experienced engineer does in T hours." METR's tasks are self-contained by design. Real work draws on prior conversations, tacit knowledge and codebase familiarity. Their own guidance: read a two-hour task as what a low-context person — a new hire or a freelance contractor — could do in two hours. In a mature system with undocumented coupling, that gap is large.
它不是"低于这个时长的任务都能做"。METR 给了具体分解:GPT-5 智能体在人类耗时 90 分钟到 3 小时的任务上,约三分之一每次都成功,约三分之一每次都失败,其余时对时错。H₅₀ 是一个难度带上的聚合值,不是每个任务的抛硬币。
It is not a threshold below which everything works. METR gives a breakdown: for a GPT-5 agent on tasks in the 90-minute-to-3-hour band, about a third succeed every time, about a third fail every time, and the rest are mixed. H₅₀ is an aggregate over a difficulty band, not a per-task coin flip.
第二部分 — 曲线说了什么Part II — What the curve says
两条线,以及它们之间那个常数The two lines, and the constant between them
| 模型Model | 发布Release | H₅₀ | H₈₀ | 比值Ratio |
|---|---|---|---|---|
| GPT-4 | 2023-03 | 4 min | 0.6 min | 7.0× |
| Claude 3.5 Sonnet | 2024-06 | 11 min | 2 min | 6.3× |
| o1 | 2024-12 | 39 min | 7 min | 5.4× |
| Claude 3.7 Sonnet | 2025-02 | 1.0 h | 0.20 h | 5.1× |
| o3 | 2025-04 | 2.0 h | 0.50 h | 4.0× |
| GPT-5 | 2025-08 | 3.4 h | 0.64 h | 5.3× |
| Gemini 3 Pro | 2025-11 | 3.7 h | 0.90 h | 4.2× |
| Claude Opus 4.5 | 2025-11 | 4.9 h | 0.82 h | 6.0× |
| GPT-5.2 | 2025-12 | 5.9 h | 1.10 h | 5.3× |
| Claude Opus 4.6 | 2026-02 | 12.0 h | 1.16 h | 10.3× |
| Gemini 3.1 Pro | 2026-02 | 6.4 h | 1.50 h | 4.3× |
| Claude Mythos Preview | 2026-04 | 17.4 h | 3.10 h | 5.6× |
比值 r = H₅₀/H₈₀ 三年来稳定在 5.5 左右。(Opus 4.6 的 10.3× 是离群值,其置信区间跨越 [5.3, 60.6] 小时——饱和端的噪声。)从 logistic 形式看,这个恒定性是关于斜率 b 的陈述:
The ratio r = H₅₀/H₈₀ has stayed near 5.5 for three years. (Opus 4.6's 10.3× is an outlier whose confidence interval spans [5.3, 60.6] hours — noise at the saturated end.) From the logistic form, that constancy is a statement about the slope b:
成功曲线并没有变陡,它只是整体右移了。这给出了整份数据里最有用的一个衍生量。若跨度每 T₂ 天翻倍:
The success curve has not gotten steeper; it has slid right. This yields the most useful derived quantity in the dataset. If horizons double every T₂ days:
可靠能力落后头条能力约十个月。今天的 H₅₀ 就是明年的 H₈₀。
Reliable capability lags headline capability by roughly ten months. Today's H₅₀ is next year's H₈₀.
这一点之所以重要,是因为自主性需要的是可靠性而非中位表现。一个十二小时任务在 50% 成功率下的实际含义是:跑两遍,并且要有能力判断哪一遍是对的。第二个要求才是昂贵的那个,它是第五部分全部内容的种子。
This matters because autonomy needs reliability, not median performance. A twelve-hour task at 50% means running it twice and having the ability to tell which run was right. That second requirement is the expensive one, and it is the seed of everything in Part V.
基准在饱和,而且部分在说谎Benchmarks are saturating, and partly lying
SWE-bench Verified 作为前沿信号已经结束。2026 年 9 月,前三名分别是 96%、95.5%、95%,挤在一个百分点内。但这些分数从来没有看上去那么有信息量:
SWE-bench Verified is finished as a frontier signal. As of September 2026 the leaders sit at 96%, 95.5% and 95%, clustered within a point. But the scores were never as informative as they looked:
| 模型Model | SWE-bench Verified | SWE-bench Pro | 落差Drop |
|---|---|---|---|
| Claude Opus 4.5 | 80.9% | 45.9% | −35.0 |
| GPT-5 High | ~55% | 23.3% | −31.7 |
Pro 使用标准化脚手架,任务是模型训练时不可能见过的。还有数据质量问题:OpenAI 自己的审计发现,Verified 中最难的未解决问题有 59.4% 存在测试用例缺陷。OpenAI 已于 2026 年初停止报告 Verified 分数并转向 Pro。独立对抗性工作(SWE-ABS)在补强测试套件后,各模型普遍下降 15 到 18 个百分点。
教训不是模型弱,而是:任何成功判据可算法检验的基准都会被优化针对,而"测试通过"与"问题解决"之间的残差,恰好是你在生产环境中继承的那部分。METR 自己的流程里有专门的 reward hacking 检查——自动标记加多人复核——这说明这种背离有多常规。
Pro uses standardized scaffolding on tasks the models could not have seen in training. There is a data-quality problem too: an OpenAI audit found 59.4% of the hardest unsolved Verified problems had flawed test cases. OpenAI stopped reporting Verified in early 2026 and now points to Pro. Independent adversarial work (SWE-ABS) strengthens the test suites and finds uniform 15–18 point drops.
The lesson is not that the models are weak. It is that any benchmark with algorithmically checkable success gets optimized against, and the residual gap between "passes the tests" and "solved the problem" is exactly what you inherit in production. METR's own pipeline includes a dedicated reward-hacking check — automated flagging plus multiple human reviewers — which tells you how routine that divergence is.
趋势变陡了,而且原因清楚The trend steepened, and we know why
用 2019 年起的全部数据拟合:倍增周期约 188 天。只用 2023 年以后的数据:约 125 天。七个月压缩到四个月出头。
Fitting METR's full history from 2019: doubling every ~188 days. Fitting 2023 onward: doubling every ~125 days. Seven months compressed to four.
| 指标Metric | 增速Growth | 倍增周期Doubling |
|---|---|---|
| METR 时间跨度(2023 起)METR time horizon (2023→) | — | ~125 天days |
| 前沿训练算力(2020 起)Frontier training compute (2020→) | 5×/yr | 5.2 月mo |
| 全球 AI 芯片存量(2022 起)Global AI chip stock (2022→) | 3.3×/yr | ~7 月mo |
| 上下文窗口(2023 起)Context windows (2023→) | 30×/yr | 2.4 月mo |
| 芯片性价比(2023 起)Chip performance per dollar (2023→) | +49%/yr | 1.7 年yr |
拐点是推理模型。自 2024 年 9 月 o1-preview 出现以来,Epoch 的能力指数以每年约 14 点推进,而此前非推理模型是每年约 6 点——2.3 倍加速。
机制很直接:2024 年末之前基本只有预训练这一条扩展轴;推理模型打开了第二条——推理时计算,加上在可验证任务上做强化学习。请记住这个短语。它会在第五部分作为核心工程杠杆回来,在第六部分作为经济杠杆回来。
The inflection is reasoning models. Epoch's capabilities index has advanced at ~14 points/year since o1-preview appeared in September 2024, against ~6 points/year before — a 2.3× acceleration.
The mechanism: before late 2024 there was essentially one scaling axis, pre-training. Reasoning models opened a second — inference-time compute plus reinforcement learning on verifiable tasks. Note that phrase. It returns in Part V as the central engineering lever, and in Part VI as an economic one.
第三部分 — 曲线往哪走Part III — Where the curve goes
直接外推会爆炸Naive extrapolation detonates
| 时间Date | 外推 H₅₀Projected H₅₀ | 相当于Equivalent to |
|---|---|---|
| 2026 末late | 28 h | 半周工作half a work-week |
| 2027 中mid | 107 h | 近三周nearly three weeks |
| 2028 中mid | 817 h | 二十人周twenty person-weeks |
| 2030 中mid | 47,000 h | 二十五人年twenty-five person-years |
二十五人年的东西不是一个"任务",而是一个部门干几年的产出。这个指标在能力抵达之前就先失去了意义。
METR 已经标出了边界:超过 16 小时的测量在当前任务集上不可靠,趋势拟合明确排除了中心估计超过该值的点。他们最长的任务约八小时,右尾之下没有数据。置信区间显示了破坏程度:Opus 4.6 是 [5.3, 60.6] 小时,Mythos Preview 是 [8.5, 55.1] 小时,跨度超过一个数量级。
Twenty-five person-years is not a "task"; it is a department's multi-year output. The metric loses meaning before the capability arrives.
METR already flags the boundary: measurements above 16 hours are unreliable with the current suite, and the trend fit excludes points whose central estimate exceeds it. Their longest tasks run to about eight hours, so the right tail has no data beneath it. The confidence intervals show the damage — Opus 4.6 at [5.3, 60.6] h, Mythos Preview at [8.5, 55.1] h. Spans over an order of magnitude.
一个有原理的下限A principled floor
Whitfill、Snodin 与 Becker(其中一位在 METR)建立了让下限可计算的模型。两个假设:
Whitfill, Snodin and Becker (one of them at METR) built the model that makes a lower bound tractable. Two assumptions:
其中 Y 是时间跨度,C 训练算力,E 实验算力,A 算法乘子。第二个假设以实验算力为算法进步的唯一输入,明确排除了纯软件的智能爆炸——劳动力不是约束项。对 Y 取对数微分并代入稳态关系,得到主方程:
where Y is time horizon, C training compute, E experimental compute, A an algorithmic multiplier. The second assumption makes experimental compute the sole input to algorithmic progress, which explicitly rules out a software-only intelligence explosion — labour is not the binding input. Log-differentiating and substituting the steady state gives the main equation:
两个经验事实——跨度与算力自 2019 年起均以恒定速率增长——迫使 ∂logY/∂logC|_A 为常数,而这当且仅当 F 是幂律函数时成立。他们在 Llama 3.1 与 Qwen 2.5 上的跨族检验支持这一点,经算力最优性调整后共享斜率 R² = 0.949。
拟合参数:∂logY/∂logC|_A ≈ 0.454,λ/β ≈ 0.95,故 c ≈ 0.885。跨度增速与研发算力增速几乎一比一。同一框架反推出的算法进步率约为每年 3.5 倍,与 Ho 等人的独立估计吻合。
The two empirical facts — that horizon and compute have both grown at constant rates since 2019 — force ∂logY/∂logC|_A to be constant, which holds if and only if F is a power law. Their cross-family test on Llama 3.1 and Qwen 2.5 supports this, with shared-slope R² = 0.949 after adjusting for compute-optimality.
Fitted: ∂logY/∂logC|_A ≈ 0.454, λ/β ≈ 0.95, hence c ≈ 0.885. Horizon growth is very nearly one-to-one with R&D compute growth. The same framework backs out algorithmic progress at ≈3.5× per year, matching Ho et al.'s independent estimate.
最坏情况The worst case
假设最严苛的合理情景:AI 研发支出的实际金额从今天起停止增长。算力仍会增长,因为性价比在改进。设年度有效算力增长因子为 g:
Assume the harshest plausible scenario: real AI R&D spending stops growing today. Compute still grows, because performance per dollar improves. With annual effective-compute growth factor g:
| 性价比增速Perf/$ growth | g | 跨度年增速Horizon growth | T₂ |
|---|---|---|---|
| +49%/yr (历史值)(historical) | 1.49 | 1.42×/yr | ~24 月mo |
| +40%/yr | 1.40 | 1.35×/yr | ~28 月mo |
| +30%/yr | 1.30 | 1.26×/yr | ~36 月mo |
高置信度下限:H₈₀ 每两到三年翻一倍。
High-confidence floor: H₈₀ doubles every two to three years.
| 年份Year | 2027 | 2028 | 2029 | 2030 | 2033 |
|---|---|---|---|---|---|
| 下限 H₈₀Floor H₈₀ | ~5 h | ~7 h | ~9 h | ~12 h | ~26 h |
三个理由说明这个下限偏保守,全部指向同一方向:
- 它假设放缓从今天开始,但 2027–2028 年的算力基本已是既成事实——数据中心在建、芯片订单已下。而且前沿实验室在 2025 年只用掉了全球 AI 算力的 20–30%,在受制于总产能之前还有腾挪空间。现实路径是先按现速跑一两年,再落到这个斜率,这会把 2030 年推到 25–50 小时。
- 论文的主结果本身是延迟的下限。他们的凹性检验(∂logY/∂logC|_A = βC^ρ,ρ < 0)显著拉长延迟——一周跨度从推迟两年变成推迟七年。
- 未假设范式转变。2023 年没有人预测到推理模型会把倍增周期减半。
关于放缓的代价有个参照:把 OpenAI 自己的算力预测代入该模型,"一个月跨度、80% 可靠性"会比朴素外推晚约七年到来。
Three reasons this floor is conservative, all pointing the same way:
- It assumes the slowdown starts now, but 2027–2028 compute is largely pre-committed — datacenters under construction, chips ordered. And frontier labs used only 20–30% of global AI compute in 2025, so there is headroom before they are bound by total production. Realistically: current speed for a year or two, then this slope. That puts 2030 nearer 25–50 h.
- The paper's main result is a lower bound on delay. Their concavity check (∂logY/∂logC|_A = βC^ρ, ρ < 0) lengthens delays substantially — a one-week horizon slips seven years instead of two.
- No paradigm shift is assumed. Nobody predicted in 2023 that reasoning models would halve the doubling time.
For scale on what slowdown costs: feeding OpenAI's own compute projections through the model, a one-month horizon at 80% reliability arrives about seven years later than naive extrapolation implies.
第四部分 — 曲线看不见什么Part IV — What the curve cannot see
任务不是项目Tasks are not projects
METR 把任务定义为不可被轻易拆分的连贯单元。他们的例子:一千道各需一小时的独立数学题不是一千小时任务,而是一小时任务做一千遍——因为可以并行分给一千个人或一千次模型调用。
所以 Chromium(约 3000 万行,COCOMO 估算约一万人年)不是一个两千万小时的任务。真正决定它的是:(a) 其中最长的不可约子任务有多长;(b) 有没有东西能做分解与整合。第二项至今没有任何基准覆盖。
METR 对"真正的长任务"的原型描述是关键:迭代调试一个复杂系统,每次修复揭示新问题,而新问题只有在你知道之前试过什么的前提下才说得通。不可约性正是这个——真正重要的状态是累积的失败假设记录,而这种状态无法跨越接口传递。
METR defines a task as a coherent unit that cannot be trivially split. Their illustration: a thousand separate one-hour maths problems is not a thousand-hour task, it is a one-hour task done a thousand times, because you can parallelize it across a thousand people or a thousand model calls.
So Chromium — roughly 30M lines, ~10,000 person-years by COCOMO — is not a 20-million-hour task. What governs it is (a) the length of its longest irreducible sub-task, and (b) whether anything can perform decomposition and integration. The second has essentially no benchmark coverage at all.
METR's prototype of a genuinely long task is the key: iteratively debugging a complex system, where each fix reveals new problems that only make sense if you know what you already tried. Irreducibility is precisely this — the state that matters is the accumulated record of failed hypotheses, and that state cannot cross an interface.
搜索形态的工作不可分解。构造形态的工作几乎总是可以。
Search-shaped work does not decompose. Construction-shaped work almost always does.
调试间歇性故障、追查内存泄漏、性能调优——全部不可约。实现十二个端点、写五个适配器、移植到三个平台——全部可分。
Debugging an intermittent fault, chasing a memory leak, performance tuning — all irreducible. Implementing twelve endpoints, writing five adapters, porting to three platforms — all separable.
阶梯:真实项目The ladder, in real projects
| 量级Scale | 项目Project |
|---|---|
| 2–8 h | 网页贪吃蛇或俄罗斯方块;带数据库的短链服务;一个能用的 Markdown 编辑器Browser Snake or Tetris; a URL shortener with a database; a usable Markdown editor |
| 20–60 h | 打磨过的 Flappy Bird 复刻;首个自举版 Git;含 RESP 与持久化的 Redis 复刻A polished Flappy Bird clone; the first self-hosting Git; a Redis clone with RESP and persistence |
| 40–100 h | 完整路径追踪渲染器(《Ray Tracing in One Weekend》三本全做)A full path tracer (all three Ray Tracing in One Weekend books) |
| 150–400 h | 完整解释器与字节码虚拟机(《Crafting Interpreters》的 Lox);能启动并调度进程的 xv6 级内核A complete interpreter and bytecode VM (Lox, from Crafting Interpreters); an xv6-class kernel that boots and schedules |
| 500–800 h | Linux 0.01 —— Linus,1991 年 4 至 9 月,业余时间,约一万行Linux 0.01 — Linus, April–September 1991, part-time, ~10k lines |
| ~1 人年p-yr | 首个可用的 SQLite;一个 3D 引擎骨架The first usable SQLite; a 3D engine skeleton |
| ~5 人年p-yr | Doom (1993) —— 约五人一年Doom (1993) — roughly five people, one year |
| ~8 人年p-yr | Stardew Valley —— 单人开发约 4.5 年,每天约 10 小时Stardew Valley — one developer, ~4.5 years at ~10 h/day |
| ~25 人年p-yr | 初版 V8 引擎The original V8 engine |
| 30–60 人年p-yr | Godot → 1.0;LLVM → 1.0 |
| 10²–10³ | Unreal Engine 5;一款 AAA 游戏Unreal Engine 5; a AAA title |
| ~10⁴ 人年p-yr | Chromium;Windows |
最后几行只能当数量级看。COCOMO 是从代码行数反推工时,假设一切都是传统流程下手写的。Chrome 继承了 WebKit,WebKit 继承了 KHTML。
把当前能力叠上去:H₈₀ ≈ 3–5 小时停在第一档——贪吃蛇、短链服务、Markdown 编辑器。H₅₀ ≈ 12–17 小时以抛硬币的概率够到第二档下沿。
Treat the bottom rows as order-of-magnitude only — COCOMO reverse-engineers effort from line counts and assumes everything was hand-written under traditional process. Chrome inherited WebKit, which inherited KHTML.
Mapping current capability: H₈₀ ≈ 3–5 h stops at the first rung — Snake, a URL shortener, a Markdown editor. H₅₀ ≈ 12–17 h reaches the low end of the second rung at coin-flip odds.
Linus 用四天写出了自举版 Git。首次提交 2005 年 4 月 3 日,到 4 月 7 日它已经能管理自己的源码。那是 40 到 60 小时的连贯工作——高于 H₅₀,远高于 H₈₀。当前没有模型能可靠复现,而且给它一份完整设计文档也不解决问题。困难从来不在知道 Git 该是什么样,而在四十小时不走偏。
Linus bootstrapped Git in four days. First commit April 3rd 2005; by April 7th it was managing its own source. That is 40–60 hours of connected work — above H₅₀, well above H₈₀. No current model reliably reproduces it, and handing over a complete design document does not fix it. The difficulty was never knowing what Git should be. It was forty hours without drifting.
按趋势延续(T₂ = 125 天),H₈₀ 在 2027 年中约 23 小时(自举 Git),2028 年中约 175 小时(Lox、xv6),2029 年中约 1,300 小时(逼近 SQLite、Doom),2030 年中约 10,000 小时(Stardew Valley 量级的单体构建)。而在下限情景下,2030 年中是一个 Markdown 编辑器。
Under trend continuation (T₂ = 125 d), H₈₀ reaches ~23 h by mid-2027 (bootstrap Git), ~175 h by mid-2028 (Lox, xv6), ~1,300 h by mid-2029 (approaching SQLite, Doom), ~10,000 h by mid-2030 (Stardew Valley as one coherent build). Under the floor, mid-2030 is a Markdown editor.
这两个 2030 年结果之间的落差,就是"这件事会有多快"的诚实答案。
The spread between those two 2030 outcomes is the honest answer to "how fast will this go."
为什么"AI 造 Chrome"是个错的问题Why "AI builds Chrome" is the wrong question
通往 Chrome 量级自主性的现实路径,不是某个模型的跨度长到一万小时。而是跨度增长跨过某个阈值(也许几百小时),使得分解、并行执行与集成本身可以由系统自己跑起来。此后总产出不再受单任务长度限制,而是受算力、架构一致性和验证吞吐量限制。
那是一次相变,不是曲线上的某个点。注意另一侧的约束是什么:验证。
The realistic path to Chrome-scale autonomy is not a model whose horizon reaches ten thousand hours. It is horizon growth crossing some threshold — perhaps a few hundred hours — past which decomposition, parallel execution and integration can themselves be run by the system. Beyond that, total output stops being bounded by single-task length and starts being bounded by compute, architectural coherence, and verification throughput.
That is a phase change, not a point on the curve. And note what bounds it on the far side: verification.
第五部分 — 在曲线下工作Part V — Working under the curve
复合恒等式The compounding identity
对 n 个串行步骤、单步可靠性 p:
For n serial steps at per-step reliability p:
| p | n = 50 | n = 200 | n = 1000 |
|---|---|---|---|
| 0.99 | 61% | 13% | 0.004% |
| 0.999 | 95% | 82% | 37% |
| 0.9999 | 99.5% | 98% | 90% |
从 99% 提到 99.9%,千步成功率从十万分之四涨到 37%。敏感度随 n 线性增长——长任务正是可靠性投资最划算的地方。
Going from 99% to 99.9% takes thousand-step success from 4 in 100,000 to 37%. Sensitivity scales linearly in n — long tasks are where reliability investment pays.
单步可靠性不是纯粹的模型属性Per-step reliability is not purely a model property
你动不了 m。你今天就能动 e,而且因为它在指数位置上,杠杆被放大 n 次。环境质量分解为:
- 信息完整度 —— 这一步需要的东西在不在手边。模型在缺信息时不会停下来问,它会猜,而猜就是随机游走。
- 反馈锐度 —— 一个类型错误是一次确定的修正;一个 segfault 是一轮盲目搜索。
- 动作粒度 —— "改这个函数"的单步可靠性远高于"重构这个模块"。粒度本身是可调参数。
- 工具可用性 —— 能不能跑、能不能查、能不能调试。没有工具的每一步都是猜。
- 上下文洁净度 —— 还挂在上下文里的失败尝试会持续污染后续判断。
还有一个常被忘记的杠杆:推理时计算。更长的思考、带筛选的多次采样、生成-批判-修订循环。这是用钱买可靠性,而不是等新模型发布。
You cannot move m. You can move e today, and because it sits in the exponent, the leverage is multiplied n times. Environment quality decomposes into:
- Information completeness — is what this step needs at hand? Models do not stop and ask when context is missing. They guess, and guessing is a random walk.
- Feedback sharpness — a typed compile error is one deterministic correction; a segfault is a blind search.
- Action granularity — "change this function" has far higher per-step reliability than "refactor this module." Granularity is a tunable parameter.
- Tool availability — can it run, query, debug? Every step without tools is a guess.
- Context hygiene — failed attempts still in context keep polluting subsequent judgement.
There is also a lever people forget: inference-time compute. Longer reasoning, sampling with selection, generate-critique-revise loops. That buys reliability with money instead of waiting for a release.
验证是那个约束Verification is the binding constraint
当模型能独立回答"我做完了吗,这对吗",它的自主跨度会成倍扩张。当这个判断必须由你做出,跨度就等于两次人工检查之间的距离——而你的审查带宽是一个常数,不会每四个月翻一倍。
Where a model can independently answer "am I done, and is this right?", its autonomous span expands by multiples. Where that judgement must come from you, its span collapses to the distance between your inspections — and your review bandwidth is a constant that does not double every four months.
METR 自己的数据提供了旁证:当评分从算法判定改为整体人工判定时,模型表现大幅下降。那个落差就是"测试通过"与"活干对了"之间的距离,也正是你继承的那一部分。
还有一个长上下文失效模式值得点名:同一任务在短跨度变体上的成功率为 40–50%,一旦嵌入更长的交互历史,即使相关信息在技术上仍位于窗口内,成功率会跌到 10% 以下。这是注意力稀释,不是窗口大小问题。
METR supplies corroborating evidence from its own data: model performance drops substantially when scoring moves from algorithmic to holistic human judgement. That delta is the gap between "tests pass" and "the work is right," and it is exactly the part you inherit.
There is also a long-context failure mode worth naming. Success rates of 40–50% on short-horizon task variants fall below 10% when the same task is embedded in longer interaction history — even when the relevant information is still technically inside the window. This is attention dilution, not a window-size problem.
先建验证基础设施Build verification infrastructure first
- 属性测试与模糊测试优于例子测试。它们能抓到你没想到的情况,而那正是模型出错的地方。
- 类型、schema、可执行契约。把一整类错误变成编译期或启动期失败——连测试都不用跑。
- 非二元维度的回归基线 —— 延迟、输出质量。要数字,不要印象。
- 确定性。随机种子、时间、并发顺序全部可控。不能复现的失败无法被修复。
- Property-based and fuzz testing over example-based tests. They catch cases you did not think of, which is exactly where models fail.
- Types, schemas, executable contracts. These turn whole error classes into compile-time or startup failures — no test run required.
- Regression baselines for non-binary dimensions — latency, output quality. Numbers, not impressions.
- Determinism. Seeds, clocks, concurrency order controlled. A failure that cannot be reproduced cannot be fixed.
防御 reward hackingDefend against reward hacking
- 测试与实现在不同上下文中生成;绝不让写实现的那一轮同时改测试。
- 测试的 diff 与实现的 diff 分开审查。
- 断言端到端行为和黄金输出,而不是内部调用模式。
- 保留一份模型永远看不到的测试集。
- Generate tests in a different context from the implementation; never let the implementing turn edit tests.
- Review test diffs separately from implementation diffs.
- Assert end-to-end behaviour and golden outputs, not internal call patterns.
- Hold back a test set the model never sees.
在规格上花时间Spend on specifications
真实项目中大部分失败不是编码失败,而是未言明的需求、未考虑的边界条件、与相邻系统未文档化的契约。这些都不出现在任何基准里,却全部出现在你的项目里。
按顺序写下:验收标准(怎么算做完了——这一条比其余全部加起来更重要)、接口契约与数据形状、失败模式与处理方式、明确的非目标、以及已知的坑及其历史原因。
一个附带收益:写规格的过程会暴露你自己没有决定的东西。相当一部分"模型做不了"其实是"从来没有人把这件事说清楚过"。
Most real failures are not coding failures. They are unstated requirements, unconsidered edge cases, undocumented contracts with adjacent systems. None of that appears in any benchmark; all of it appears in your project.
Write down, in order: acceptance criteria (how do we know it's done — this outweighs everything else combined), interface contracts and data shapes, failure modes and handling, explicit non-goals, and known landmines with their history.
A side benefit: writing the spec surfaces what you had not decided. A surprising share of "the model can't do this" is "nobody ever specified this."
为廉价判断设计检查点Design checkpoints for cheap judgement
检查点有效,但注意真正的上限是你的审查带宽。每个检查点花你二十分钟,一天就只有十五个。你没有延长自主性,你把自己变成了质检工位。
所以设计目标不是更频繁地检查,而是更便宜地检查:
Checkpoints work, but note the real cap: your review bandwidth. At twenty minutes per checkpoint you get fifteen a day. You have not extended autonomy; you have become a QA station.
The design goal is not more frequent checks but cheaper ones:
你能不能在两分钟内做出「继续 / 回退」的判断?不能,这个检查点的位置就错了。
Can you decide continue-or-revert in under two minutes? If not, the checkpoint is in the wrong place.
好的检查点产出一个可运行的东西而非一堆改动;有明确的通过信号;只需要读结论而不是读 diff;并且能原子回退。位置要放在状态边界,不是时间边界——不是"每小时一次",而是"每当系统回到可运行、可测试、可丢弃的状态时"。
A good checkpoint produces a runnable artifact rather than a pile of edits; has an unambiguous pass signal; requires reading a conclusion rather than a diff; and reverts atomically. Place them at state boundaries, not time boundaries — not "every hour," but "every time the system returns to a runnable, testable, discardable state."
并行分解的标准流程An SOP for parallel decomposition
0. Classify Per pair: "does B need A's failed-attempt history?" Yes → irreducible, do not split. 1. Contract Single source defines interfaces. Make them executable (types, schemas, stubs, OpenAPI). Then FREEZE. 2. Slice Vertically. Each slice owns its full stack. Test: can this slice pass tests alone? No → sliced wrong. 3. Accept Each slice ships its own tests and a pass signal. 4. Isolate Separate worktree, separate context, zero inter-branch communication. Needing it means step 0 was wrong. 5. Integrate Serial. Most-depended-upon branch first, then outward. Full test run after each merge. 6. Budget Parallelism k ≤ 5. Integration cost ~ O(k²).
第 1 步被违反得最多,代价也最大。两个分支各自"顺手把接口整理了一下",产生的合并成本会超过并行省下的一切。
第 2 步区分真假分解。水平切分——一个分支做全部 model,另一个做全部 controller——会让每个分支在全部落地前都无法验证。那是延迟集成,不是分解。
第 6 步解释了并行度为什么上限很低。集成本身是不可约的,而它暴露的问题是契约理解偏差,需要跨分支上下文才能诊断。按两两接口摩擦计,成本约为 C(k,2) = O(k²)。超过四五个分支就会吃掉收益。想要更多并行就分层:四个分支,集成,再四个。
一个反直觉的后果:并行缩短了总耗时,却提高了你单位时间内必须供给的判断密度。五个分支意味着同时持有五份上下文。这正是第 3 步关键的原因——它把"判断五件事"变成"读五个红绿灯"。做不到这一点,串行三小时往往胜过并行一小时加上你两小时的脑力。
Step 1 is violated most often and costs the most. Two branches each "tidying the interface a little" produces merge costs exceeding everything parallelism saved.
Step 2 separates real decomposition from fake. Horizontal slicing — one branch does all models, another all controllers — leaves every branch unverifiable until everything lands. That is deferred integration, not decomposition.
Step 6 is why parallelism caps low. Integration is itself irreducible, and the problems it surfaces are contract misunderstandings requiring cross-branch context to diagnose. With pairwise interface friction, cost grows roughly as C(k,2) = O(k²). Past four or five branches it eats the gains. Want more? Layer it: four branches, integrate, four more.
One counterintuitive consequence: parallelism cuts elapsed time but raises the density of judgement you must supply. Five branches means holding five contexts. Which is why step 3 matters — it converts "judge five things" into "read five status lights." Without it, three hours serial often beats one hour parallel plus two hours of your attention.
| 处理的问题Handles | 机制Mechanism | 适用于Applies to | |
|---|---|---|---|
| 检查点Checkpoints | 不可约任务的长度Length of irreducible tasks | 阻止错误传播,不缩短任务Prevents propagation; does not shorten | 调试、探索、重构 —— 搜索形态Debugging, exploration, refactoring — search-shaped |
| 并行分解Parallel decomposition | 可分工作的总量Volume of separable work | 绕过 H₈₀,每个分支都在其内Bypasses H₈₀; each branch sits inside it | 实现、移植、覆盖 —— 构造形态Implementation, porting, coverage — construction-shaped |
真实项目是两者的混合。把项目切成构造部分与搜索部分,前者并行,后者串行并加密检查点,然后把你自己的注意力放在后者——那是模型最容易走偏、而且走偏最贵的地方。
Real projects mix both. Split the project into constructive and search-shaped parts, parallelize the first, serialize the second with dense checkpoints, and put your own attention on the second — that is where models drift, and where drift is most expensive.
第六部分 — 曲线之外Part VI — Beyond software
METR 必须把任务做成自包含、明确定义、可自动判分的。这三个约束定义了可自动化的区域。同一组约束换个角度看——作为价值的过滤器——能解释工程之外的很多事情。
METR had to make tasks self-contained, well-specified and algorithmically scorable. Those constraints define the automatable region. The same constraints, read as a filter on value, explain a great deal outside engineering.
哪些数字产品崩了,哪些没有Which digital goods collapsed, and which did not
数字产品的复制边际成本几十年前就归零了。最近改变的是创造的边际成本。崩塌落得并不均匀,而且模式是可读的。
归零的:孤立、可替代、无上下文的产出——库存音乐、模板文案、通用 CRUD 后端、落地页、图库摄影。Getty 的市值从 2022 年高点跌去约九成。
保住或上涨的:分发与注意力;对失败的责任承担;网络效应与数据积累;合规与认证;整合与运维;与物理世界的接口。
共同线索是:价值从制品迁移到了它周围的一切——而周围的东西大多是某种形式的验证。合规认证是验证。SLA 是附带责任的验证。品牌是被时间摊销的验证。分发是相关性的验证。
The marginal cost of reproducing digital goods hit zero decades ago. What changed recently is the marginal cost of creation. The collapse landed unevenly, and the pattern is legible.
Went to zero: isolated, substitutable, context-free output — stock music, template copy, generic CRUD backends, landing pages, stock photography. Getty's market value fell roughly 90% from its 2022 peak.
Held or rose: distribution and attention; accountability for failure; network effects and accumulated data; compliance and certification; integration and operations; interfaces to the physical world.
The common thread is that value migrated from the artifact to everything around it — and most of what surrounds it is some form of verification. A compliance certification is verification. An SLA is verification with liability attached. A brand is verification amortized over time. Distribution is verification of relevance.
当生产变得廉价而充裕,稀缺品就是"这个东西正确、安全、恰当"的可信确认。
When production becomes cheap and abundant, the scarce good is trustworthy confirmation that a given artifact is correct, safe and appropriate.
这也解释了一个让人困惑的背离:录音音乐被压到近乎免费的同时,演唱会票价暴涨。当可复制的东西变免费,不可复制的东西就升值。在场、稀缺、与创作者的关系——这些都无法通过检查制品来验证,只能通过来源验证。
它还让劳动力冲击变得可预测而非神秘。被掏空的是中间层——称职的、中等熟练度的、无署名的生产。顶端(名字、关系、独特视角)和底端(业余、自娱)相对不受影响。这不是一个宽慰人的观察:调整在两三年内发生,远快于摄影留给肖像画家的几十年,而成本落在那些来不及转身的人身上。
It also explains a divergence people find puzzling: recorded music compressed toward free while live ticket prices soared. When the reproducible becomes free, the non-reproducible appreciates. Presence, scarcity, relationship to the maker — none of which can be verified by inspecting the artifact, all of which are verified by provenance.
And it makes the labour impact predictable rather than mysterious. What gets hollowed out is the middle — competent, mid-skilled, unnamed production. The top (name, relationship, distinctive perspective) and the bottom (amateur, self-directed) are comparatively insulated. That is not a consoling observation: adjustment happens over two or three years, far faster than the decades photography gave portrait painters, and the cost lands on people who have no time to move.
版权问的是同一个问题,只是换了名字Copyright asks the same question, under another name
版权保护表达,不保护思想、方法或共享语汇。落到音乐上,产生了一种看似武断的不对称——直到你看到机制:
Copyright protects expression, not ideas, methods or shared vocabulary. Applied to music, that produces an asymmetry that looks arbitrary until you see the mechanism:
法理学直接命名了这一点:合并原则(当表达形式过少时,表达并入不受保护的思想)与通用场景(某一体裁的惯例性元素自由使用)。《Stairway to Heaven》案正是如此——那段下行半音线被认定为公共音乐素材,Led Zeppelin 胜诉。
这对"从零重新录制"意味着什么?重新演奏消除了录音制品权,但不触及词曲权——后者管辖旋律,与演奏者是谁无关。而大幅改编还会额外触发改编权,因为法定翻唱许可通常禁止改变作品的根本特征。干净的解法直接从图中读出:保留和声、律动、配器与混音,重写旋律。你想要的"那个味道"几乎全部住在不受保护的那一层。
还有两点反直觉但重要:不存在百分比门槛——没有"八小节",没有"四秒"。标准是实质性相似加接触可能性,整体认定。澳洲的 Larrikin v EMI 案就因《Down Under》中两小节长笛判定侵权。而且潜意识抄袭同样构成侵权:《My Sweet Lord》案中法院接受 Harrison 并非故意,仍然判了责任。
The doctrine names this directly: merger (when expression has too few forms, it merges into the unprotectable idea) and scènes à faire (conventional elements of a genre are free). Stairway to Heaven turned on exactly this — the descending chromatic line was held to be common musical material, and Led Zeppelin won.
What this means for re-recording from scratch: re-performing removes the sound recording right but does not touch the composition right, which governs the melody regardless of who plays it. Heavy re-arrangement additionally engages the adaptation right, since statutory cover licences typically forbid altering a work's fundamental character. The clean solution reads straight off the diagram: keep the harmony, groove, instrumentation and mix; write a new melody. The "feel" you wanted lives almost entirely in the unprotected layer.
Two more counterintuitive points. There is no percentage threshold — no "eight bars," no "four seconds." The test is substantial similarity plus access, assessed holistically. In Australia, Larrikin v EMI found infringement on two bars of flute in Down Under. And subconscious copying still infringes: in the My Sweet Lord case the court accepted Harrison did not intend it and found liability anyway.
AI 生成音乐:三个彼此独立的问题AI-generated music: three separate questions
最常见的错误是把三个互不相干的法律问题揉成一个。
The single most common error is collapsing three independent legal questions into one.
第一层是钱所在的地方。RIAA 于 2024 年 6 月起诉 Suno 与 Udio。到 2025 年底战线分裂:Warner 与 Suno 和解,UMG 与 Udio 和解(据报道每次生成 0.002–0.005 美元),而 Sony 与 Universal 继续诉讼。2026 年 5 月他们申请把 Suno 案的涉案录音从 560 首扩大到 61,026 首,潜在法定赔偿超过 90 亿美元。首个针对生成式音乐工具的实体裁决来自慕尼黑,2026 年 7 月 31 日,地方法院第一庭就 GEMA 诉 Suno 案作出对 GEMA 有利的裁决。美国的 fair use 裁决已排到 2027 年。
第二层是人们直觉反转的地方。常见的担忧——"模型学了几百万首歌,所以输出必然同时像很多首"——实际上论证的是更低的输出风险,而非更高。侵权判定永远是一对一的:原告必须指向一首具体作品。在大语料上的统计平均消解个体特征,而非叠加。像一万首歌意味着不特别像任何一首,而在平均中幸存下来的恰好是上一节里那个不受保护的层——风格、语汇、惯例。
真正的例外是记忆性复现:对高频或极具辨识度的训练样本过拟合。这正是 RIAA 证据最有杀伤力的部分。规避措施直接由此导出——prompt 里绝不出现艺人名或歌名,只用描述性词汇(速度、调式、配器、年代、质感)。这既降低记忆性复现概率,也改善你在接触与意图认定上的位置。
第三层与前两层正交。美国版权局认为缺乏人类创作性贡献的纯 AI 生成内容不可登记,Thaler v. Perlmutter 确认了人类作者要求。澳洲更严格,连"计算机生成作品"这个类别都没有。后果不是你不能使用,而是你无法排除任何其他人使用。做背景音乐或许无所谓;作为打算授权或维权的资产,它是个空壳。
Layer 1 is where the money is. RIAA sued Suno and Udio in June 2024. By late 2025 the front had split: Warner settled with Suno, UMG settled with Udio (reportedly $0.002–0.005 per generation), while Sony and Universal continued litigating. In May 2026 they moved to expand the Suno case from 560 to 61,026 recordings, pushing potential statutory damages past $9B. The first substantive ruling on a generative music tool came not from New York or Boston but Munich, on 31 July 2026, where the Regional Court ruled for GEMA against Suno. US fair-use rulings are now scheduled into 2027.
Layer 2 is where intuitions invert. The common worry — "the model trained on millions of songs, so its output must resemble all of them at once" — actually argues for lower output risk, not higher. Infringement is always one-to-one: a plaintiff must point at a specific work. Statistical averaging over a large corpus dissolves individual identity rather than accumulating it. Resembling ten thousand songs means resembling none in particular, and what survives averaging is exactly the unprotected layer from the previous section — style, vocabulary, convention.
The real exception is memorization: overfitting on high-frequency or highly distinctive training examples. This is what made the RIAA's exhibits damaging. The mitigation follows directly — never name an artist or track in a prompt; use purely descriptive terms (tempo, mode, instrumentation, era, texture). That lowers memorization probability and improves your position on access and intent.
Layer 3 is orthogonal to both. The US Copyright Office holds that purely AI-generated content lacking human creative contribution is unregistrable, and Thaler v. Perlmutter confirmed human authorship as a requirement. Australia is stricter still — it has no computer-generated-works category at all. The consequence is not that you cannot use the output; it is that you cannot exclude anyone else from using it. For background music that may not matter. For an asset you intend to license or defend, it is a hollow shell.
底下那个模式The pattern underneath
| 领域Domain | 可自动化 / 不受保护 / 廉价Automatable · unprotected · cheap | 抗拒 / 受保护 / 有价值Resistant · protected · valuable |
|---|---|---|
| METR 任务METR tasks | 自包含、有规格、可自动判分Self-contained, specified, auto-scorable | 需要累积的失败假设状态Requires accumulated failed-attempt state |
| 软件工作Software work | 构造形态,契约式交接Construction-shaped, contract handoff | 搜索形态,隐性状态交接Search-shaped, tacit-state handoff |
| 数字产品Digital goods | 可替代、无上下文的制品Substitutable, context-free artifacts | 验证、信任、关系、来源Verification, trust, relationship, provenance |
| 版权Copyright | 共享语法 —— 和声、风格、惯例Shared grammar — harmony, style, convention | 与某个作者绑定的具体表达Specific expression tied to one author |
| AI 输出AI output | 语料的统计平均Statistical average of a corpus | 被记忆的具体表达Memorized specific expression |
每一行里,左栏是能被平均或规格化掉的东西,右栏是需要不可约的、个体的、不可替代状态的东西。
这不是措辞上的巧合。自动化、市场价值与版权是在用不同的语域回答同一个底层问题:这里面哪一部分是真正特殊的,哪一部分是共享语汇?特殊的那部分生产昂贵、难以自动化、受法律保护;共享的那部分廉价、可自动化、免费。
生成式 AI 做的事,是把这条边界大幅推向第二类。从图库摄影的价格到测试套件的设计,下游发生的一切都是对这次推移的调整。
In each row, the left column is what survives averaging or specification, and the right column is what requires irreducible, individual, non-substitutable state.
This is not a coincidence of framing. Automation, market value and copyright are all answering the same underlying question in different registers: what part of this is genuinely particular, and what part is shared vocabulary? The particular is expensive to produce, hard to automate, and legally protected. The shared is cheap, automatable and free.
What generative AI did was move the boundary sharply toward the second category — and everything downstream, from stock photography prices to test-suite design, is an adjustment to that move.
结语Closing
下限本身已经很可观。即使假设 AI 支出从今天起持平,H₈₀ 每两到三年翻倍会把十二小时跨度推到 2030 年前后。一整天定义清晰的工程工作变成可靠可交付,足以重组这个行业——不是通过去掉人,而是把人类工作的单位从写代码移到定义任务和验收结果。
上限不可知,别再试图择时。2030 年的区间从一个 Markdown 编辑器跨到 Stardew Valley。十八个月以内可以按趋势延续来规划;三年以上的任何安排,都应该在三种情景下都不崩盘。
你现在能做的事与模型能力正交,所以它会复利。验证基础设施、真正的规格、为廉价判断而设计的检查点——这些既立即见效,又随每一次模型发布按比例放大,因为它们把瓶颈挪到了别处。不做这些,模型再强你也吃不到,因为你的审查带宽是常数。
The floor is already significant. Even assuming AI spending flatlines today, H₈₀ doubling every two to three years puts twelve-hour horizons around 2030. A full day of well-specified engineering work becoming reliably delegable is enough to reorganize the profession — not by removing people, but by shifting the unit of human work from writing code to specifying tasks and accepting results.
The ceiling is unknowable; stop trying to time it. The 2030 spread runs from a Markdown editor to Stardew Valley. Plan on trend continuation inside eighteen months. Make anything beyond three years survive all three scenarios.
What you can do now is orthogonal to model capability, which is why it compounds. Verification infrastructure, real specifications, and checkpoints designed for cheap judgement pay off immediately and scale with every future release, because they move the bottleneck elsewhere. Skip them and better models will not help you, because your review bandwidth is a constant.
最后一个自测拿出你最近做的三件事,对每一件问:如果没有人看过它,有什么机制会发现那个错误?凡是答不上来的地方,就是你真正的天花板——不是图上那条。
One diagnosticTake the last three things you built and ask of each: if no human looked at it, what mechanism would have caught the mistake? Wherever there is no answer, that is your actual ceiling — not the one on the chart.