ModificationGate 必须 anchor 在 verifiable evaluation
Evaluation modality
Spec-levelA spec-motivation / governance borrow. Evaluated by spec review + contract tests, not A/B or ablation.
- Primary owner
- —
- Phase-A verdict
- —
- Shadow profile
- —
- Source papers
- Novikov/Balog/Kohli 2025 AlphaEvolve(A2)+ AlphaDev/AlphaTensor
- Specs
- docs/specs/credit-and-self-modification.mddocs/specs/evaluation.md
Blind spot (现状盲点)
[`docs/specs/credit-and-self-modification.md`](../specs/credit-and-self-modification.md) 的 `claim_rare_heavy_net_benefit` 当前是否要求 [`docs/specs/evaluation.md`](../specs/evaluation.md) 的多家族评估能给出**可重复、可对照、可 counterfactual 的**提升信号?如果只是"模型 confidence 高"或"几次 turn 表现好"就开门,那么任何自修改本质都是赌博——AlphaEvolve / AlphaDev / AlphaTensor 的所有成功都建立在"答案可硬验证"之上,我们如果跳过这一步就是路径错位。
Adoptable suggestions (可落地动作)
- 1.在 [`docs/specs/credit-and-self-modification.md`](../specs/credit-and-self-modification.md) 加入硬约束章节"Eval-Anchored Gate Opening":**ModificationGate 不能开启**,除非 evaluation 的 6 大家族(task / interaction / relationship / learning / abstraction / safety)能给出可重复、可对照、可 counterfactual 的提升量。PROPOSED
Not a runnable A/B candidate — evaluated by the path above, not ablation.
- 2.在 [`docs/specs/evaluation.md`](../specs/evaluation.md) 增补 "counterfactual evidence requirement" 段落:定义"可 counterfactual"的最小验证范式(matched-control replay、A/B replay、cross-generation winrate 等)。PROPOSED
Not a runnable A/B candidate — evaluated by the path above, not ablation.
- 3.评估证据先行:先把 6 大家族中**最薄弱的"learning quality"和"abstraction quality"**做硬,再开 rare-heavy artifact 自修改。PROPOSED
Not a runnable A/B candidate — evaluated by the path above, not ablation.
Traceability
No plugins / runs linked yet. Scaffold a suggestion to start.
Expected benefit (预期收益)
- 避免重蹈"自修改没有可验证目标 → 系统漂移"的常见坑。 - 把 evaluation 完备性作为 R10 的**前置门槛**而不是事后补丁——这与 [`.cursor/rules/first-principles-not-patches.mdc`](../../.cursor/rules/first-principles-not-patches.mdc) 一致。 - 给"我们什么时候才允许 self-modify"提供清晰可审计的判定标准。
Cited paper (引用论文)
**A2. Novikov A, Vũ N, Eisenberger M, ..., Kohli P, Balog M (DeepMind). *AlphaEvolve: A coding agent for scientific and algorithmic discovery*. arXiv:2506.13131, 2025.** - 文档位置:[`research/papers/dm/alphaevolve-coding-agent-scientific-engineering-discovery-2506.13131.pdf`](../../research/papers/dm/alphaevolve-coding-agent-scientific-engineering-discovery-2506.13131.pdf) - 摘要原文(精炼): > In this white paper, we present AlphaEvolve, an evolutionary coding agent that substantially enhances capabilities of state-of-the-art LLMs on highly challenging tasks such as tackling open scientific problems or optimizing critical pieces of computational infrastructure. AlphaEvolve orchestrates an autonomous pipeline of LLMs, whose task is to improve an algorithm by making direct changes to the code. Using an evolutionary approach, continuously receiving feedback from one or more evaluators, AlphaEvolve iteratively improves the algorithm. ... AlphaEvolve developed a more efficient scheduling algorithm for data centers, found a functionally equivalent simplification in the circuit design of hardware accelerators, and accelerated the training of the LLM underpinning AlphaEvolve itself. ... Notably, AlphaEvolve developed a search algorithm that found a procedure to multiply two 4×4 complex-valued matrices using 48 scalar multiplications; offering the first improvement, after 56 years, over Strassen's algorithm in this setting. - 关键观点:self-improvement 必须 anchor 在 verifiable eval 上——AlphaEvolve 的所有成果都来自"答案可验证"的领域(数学/算法/工程指标)。我们必须先把 evaluation 做硬,否则任何 ModificationGate 都是空中楼阁。 **B6. Mankowitz D J et al. (DeepMind). *Faster sorting algorithms discovered using deep reinforcement learning* (AlphaDev). Nature 618:257-263, 2023. DOI: [10.1038/s41586-023-06004-9](https://doi.org/10.1038/s41586-023-06004-9).** - 文档位置:仅 Nature,无 arxiv(代码:[github.com/google-deepmind/alphadev](https://github.com/google-deepmind/alphadev))。 - 关键观点:把汇编级 sort 算法发现表述为单人游戏,reward = 算法正确性 + 实测 CPU 指令延迟。**Sort3-Sort8 + VarSort3-VarSort5 算法已被纳入 LLVM 标准 C++ sort 库**——AI 发现的算法首次替换核心 library 组件。强化"verifiable eval 是 R10 自修改的硬前提"这一论点。 **B7. Fawzi A et al. (DeepMind). *Discovering faster matrix multiplication algorithms with reinforcement learning* (AlphaTensor). Nature 610:47-53, 2022. DOI: [10.1038/s41586-022-05172-4](https://doi.org/10.1038/s41586-022-05172-4).** - 文档位置:仅 Nature,无 arxiv。 - 关键观点:基于 AlphaZero 把矩阵乘法算法发现表述为 tensor decomposition 单人游戏,**4×4 modular arithmetic 上首次超越 Strassen 算法(50 年来)**。AI 在"硬可验证目标 + 大规模 search"下能触达人类几十年没解决的优化——R10 自修改的远端 ceiling 参考。 ---