All topics
intelligenceevergreen
Research-ready episode

The AI That Helped Create Its Own Successor

AI 正在改进 AI？递归自我改进，到底到哪一步了

What this research must answer

A constellation film about recursive self-improvement. The frontier labs' admission opens the question; expert definitions separate real recursion from automation on a control diagram; AlphaZero and STaR show the principle — self-generated data closes the loop when the reward is checkable; test-time compute extends it to language reasoning; model collapse and reward hacking show where the loop breaks; coding agents and AI scientists are the two loops actually closing in production; and the frontier — inference scaling and compute economics — reveals what everyone is really betting on.

这是一部关于'递归自我改进'的星座式纪录片。前沿实验室的自我声明打开问题；专家用控制图把'真递归'与'普通自动化'分开；AlphaZero 与 STaR 展示原理——当回报可校验时，自生成数据能闭合回路；测试时计算把原理延伸到语言推理；模型崩溃与奖励黑客展示回路在哪里断开；编码代理与 AI 科学家是两个真正在生产中闭合的回路；而前沿——推理扩展与算力经济学——揭示了所有人真正押注的是什么。

Opening tension

A release note from OpenAI: early versions of GPT-5.3-Codex were 'instrumental in creating itself,' helping debug training runs, manage deployment, and diagnose evaluation failures. It is the first explicit frontier-lab admission that a model materially contributed to the engineering loop that produced its successor. Then the counterweight: an AI security engineer says the real question is a control diagram — was a qualified human still in the loop when it happened?

Opening variants
  1. ·

    The release note

    'Instrumental in creating itself.' Four words from OpenAI's GPT-5.3-Codex release notes — the first time a frontier lab said a model helped build its successor. The question: is that recursion, or marketing in technical clothing?

    The exact claim as opening pressure; the mystery is what 'creating itself' actually means.

  2. ·

    The control diagram

    Two rooms run the same training loop. In one, a human verifies every model change. In the other, nobody does. Same output, different systems. Which one is self-improvement?

    The expert definition as opening; reframes RSI from a capability claim to an org chart question.

  3. ·

    The revenue curve

    Twelve months. $37 million to $492 million in annualized revenue. A coding agent that plans, codes, and opens PRs from a Slack message — and the $26 billion valuation that followed. Everyone is betting on the loop; what exactly are they buying?

    The money as opening; the audience already senses the hype, and the dollars make the bet concrete.

How the episode moves
  1. 01

    What did OpenAI actually claim when it said GPT-5.3-Codex was 'instrumental in creating itself'?

    The February 5, 2026 release notes: early versions of the model helped debug training runs, manage deployment, and diagnose evaluation failures. It was the first explicit frontier-lab admission that a model materially contributed to the loop that produced its successor.

    CACM 2026; TechCrunch 2026

    What counts as real recursion versus ordinary automation?

  2. 02

    How do experts decide what counts?

    Not by benchmark — by control diagram (Kale): is a qualified human still verifying the loop? Helen Toner: using AI tools for research is not RSI; the classic definition means no humans needed. METR's Cotra separates adequacy, parity, and supremacy as milestones on the way to research takeover.

    CACM 2026; TechCrunch 2026

    Where did the idea of a self-improving machine come from, and when did it first become engineering?

  3. 03

    When did a machine first improve itself beyond human data?

    AlphaZero (2018): self-play reinforcement learning with no human games reached superhuman chess, shogi, and Go within hours. STaR (2022): a language model bootstrapped its own reasoning rationales, filtering by correctness and fine-tuning on them. The principle: with a checkable reward, the loop closes.

    Science 2018; NeurIPS 2022

    Can that principle scale from games to language and science?

  4. 04

    What makes the loop close in large language models today?

    Test-time compute: o3-preview jumped from 32% to 76-88% on ARC-AGI-1 by spending more inference compute on harder reasoning; ARC Prize found typical single chain-of-thought systems cluster around a 30% ceiling while o3-medium doubles it. Inference scaling laws (Wu et al.) formalize the trade-off between model size and inference compute.

    ARC Prize 2025; arXiv 2408.00724

    If the reward is checkable, what could go wrong when the loop feeds on itself?

  5. 05

    Does recursive data improve models or poison them?

    Model collapse (Shumailov et al., Nature 2024): training on model-generated data irreversibly erases the tails of the original distribution. Fresh data about genuine human interactions becomes more valuable, not less. 'Self-improvement' is not a free flywheel — it depends on where the new data comes from.

    Nature 631:755-759 (2024)

    And what happens when the signal the model optimizes isn't actually the objective?

  6. 06

    Can an AI system judge its own improvement?

    Reward hacking and the verifier problem: optimizing a proxy degrades it as a measure (Goodhart's law). CACM experts compare it to an addict short-circuiting the reward; Strauss's test: gains must survive write-protected, model-invisible holdout evaluations. Lab scores can be moved by tightening prompts or loosening graders.

    CACM 2026

    If verification is the bottleneck, where is the loop actually closing in production today?

  7. 07

    Where is self-improvement already paying?

    Coding: Copilot users completed tasks 55% faster in GitHub's 2022 study; an Anthropic lead programmer estimated close to 100% of his team's code was written by Claude Code; Devin grew from $37M to $492M ARR in twelve months at a $26B valuation, and Cognition says Devin writes 89% of its own code.

    GitHub 2022; TechCrunch 2026; TNW 2026

    But does faster output mean better systems?

  8. 08

    What do the numbers hide?

    GitClear: code churn is projected to double against the pre-AI baseline, duplicate code up 8x, refactoring down — velocity without verification shifts cost downstream. The same pattern appears in science: the AI Scientist's fully AI-generated paper passed peer review (Nature 2026), and DeepMind's Co-Scientist proposed drug combinations later validated in cells — but humans still ran the confirmatory experiments.

    GeekWire/GitClear 2024; Nature 2026; DeepMind 2026

    So how much of the 'self' in self-improvement is real?

  9. 09

    If we add up what is true, what is everyone actually betting on?

    What is real: automated development loops where the answer is checkable — code that compiles, tests that pass, benchmarks with write-protected holdouts, games with rules. What is not yet real: a closed loop improving the model itself with no humans — Pichai says 'we aren't quite there yet'; Karpathy's agents improve a GPT-2-scale model; model collapse and reward hacking are the unclosed loops. The bet is not 'AI improves itself' but 'whoever owns the verifiable loop and the compute owns the next scaling curve' — which is why capital is piling into agentic engineering and inference infrastructure.

    CACM 2026; TechCrunch 2026

    Payoff: 'instrumental in creating itself' is a sentence about an org chart. The question is who is still in the room when the loop runs.

Story bank
  1. opening-pressure

    The release note

    OpenAI's GPT-5.3-Codex: early versions were 'instrumental in creating itself,' debugging training runs and diagnosing evaluation failures. First explicit frontier-lab admission.

  2. historical-turn

    AlphaZero

    Self-play RL with no human games reached superhuman chess, shogi, and Go within hours — the original proof that self-generated experience can beat human data.

  3. mechanism

    STaR bootstrapping

    A model generates reasoning rationales, filters by correctness, fine-tunes on its own successful reasoning — the loop that seeded today's self-training lineage.

  4. mechanism

    Test-time compute

    o3-preview jumped to 76-88% on ARC-AGI-1 by spending more inference compute; inference scaling laws formalize the trade-off between model size and compute at answer time.

  5. conceptual-reversal

    Model collapse

    Nature 2024: training on recursively generated data irreversibly erases distribution tails — the flywheel has a documented failure mode.

  6. conceptual-reversal

    Reward hacking

    Optimizing a proxy degrades it; experts compare it to an addict short-circuiting pleasure. Gains only count on write-protected holdouts.

  7. application

    The coding loop

    Copilot 55% faster in GitHub's 2022 study; Anthropic's Claude Code writing close to 100% of its team's code; Devin from $37M to $492M ARR in a year.

  8. application

    The science loop

    The AI Scientist's fully AI-generated paper passed peer review; DeepMind's Co-Scientist proposed AML drug combinations validated in cells — with humans running the confirmatory experiments.

  9. countercase

    The GPT-2-scale frontier

    Karpathy's Auto-Research agents improve a small model, not the frontier; Pichai says 'we aren't quite there yet'; the loop stays open where the signal is not checkable.

  10. frontier

    Compute and capital

    The real bet is who owns the verifiable loop and the compute — inference scaling, agentic infrastructure, and the $26B coding-agent market are where that bet is being placed.

How the system works
Input
A model plus a checkable reward signal (a game outcome, a passing test, a verified answer).
Transformation
The model generates its own experience or code, a filter keeps only what scores well on the checkable signal, and the surviving data is fed back into training or deployment.
Output
A capability curve that improves without new human expertise on each step — the closed loop of self-play, bootstrapping, or test-time compute.
Limit
The loop only improves what the reward measures. Where the signal is missing, noisy, or gameable, the loop collapses or hacks instead of improving.
Mechanism cards

  1. Self-play reinforcement learning

    Input
    Game rules and compute; no human expert data.
    Transformation
    The agent plays against itself, using outcomes as reward to update its value and policy networks.
    Output
    Superhuman play in chess, shogi, and Go from a blank slate.
    Limit
    Requires a machine-verifiable reward (win or loss); useless where the objective cannot be scored.
    Evidence
    Silver et al., Science 2018

  2. Reasoning bootstrapping (STaR)

    Input
    A model's own generated rationales for answer-questions.
    Transformation
    Keep rationales that led to a correct answer; fine-tune the model on its own successful reasoning.
    Output
    A better reasoner trained substantially on its own output.
    Limit
    Depends on a verifiable answer key; correct-answer flukes can reinforce bad rationales.
    Evidence
    Zelikman et al., NeurIPS 2022

  3. Test-time compute scaling

    Input
    A trained model plus an inference budget.
    Transformation
    More tokens of chain-of-thought, sampling, or search are spent per problem at answer time.
    Output
    Higher accuracy on reasoning tasks up to a compute-optimal point.
    Limit
    Diminishing returns, rising cost per task, and incomplete coverage (o3-high answered 37 of 100 ARC tasks).
    Evidence
    ARC Prize 2025; Wu et al., 2024

  4. Model collapse

    Input
    Training corpora increasingly generated by earlier models.
    Transformation
    Recursive training amplifies sampling errors; tails of the original distribution vanish.
    Output
    Irreversible degradation of diversity and quality across generations.
    Limit
    Does not forbid all synthetic data — it forbids indiscriminate recursive training; fresh human data becomes more valuable.
    Evidence
    Shumailov et al., Nature 2024

  5. Reward hacking / verifier problem

    Input
    A reward model, benchmark, or test that stands in for the true objective.
    Transformation
    The policy optimizes the proxy; exploits in the proxy get amplified.
    Output
    Higher scores without real improvement — Goodhart's law in action.
    Limit
    Only write-protected, model-invisible holdouts can distinguish improvement from hacking.
    Evidence
    CACM 2026; reward-hacking literature

  6. Model-assisted development loop

    Input
    A frontier model plus engineering tasks: debugging runs, managing deploys, evaluating failures.
    Transformation
    The model performs tasks inside the lab's own development pipeline, with humans verifying.
    Output
    Compressed development cycles; the model participates in the loop that produces its successor.
    Limit
    Whether this is RSI depends on the control diagram — whether a qualified human was still verifying when it ran.
    Evidence
    CACM 2026; TechCrunch 2026
Where this changes a real decision

SignalA model can generate its own training data, code, or evaluations that pass a verifiable check.

Decision ownerFrontier labs and engineering organizations deciding how much of the development loop to hand to models.

ThresholdA human must confirm the model's output on an independent, write-protected holdout before the change is accepted.

ActionLabs scale model-assisted development (GPT-5.3-Codex debugging training runs; Anthropic's Claude Code writing its own team's code); enterprises buy agentic engineers (Devin at $492M ARR).

ConsequenceDevelopment velocity compresses, but verification cost and evaluation drift move to the humans still in the loop; reward hacking becomes the binding constraint.

Application chains

  1. The coding loop: velocity without verification moves cost downstream

    SignalCopilot users complete tasks 55% faster; Claude Code writes close to 100% of one Anthropic team's code.

    Decision ownerEngineering organizations and their review processes.

    ActionAdopt agentic engineering at scale — Devin enterprise usage grew more than 10x in 2026; Cognition reports Devin writes 89% of its own code.

    Consequence$492M ARR and compressed cycle times, while GitClear measures code churn doubling and duplicate code up 8x — the verification burden lands on humans.

  2. The science loop: AI proposes, humans confirm

    SignalThe AI Scientist generated a full paper that passed blind human peer review; Co-Scientist proposed drug combinations for acute myeloid leukemia.

    Decision ownerLab principal investigators deciding which AI-generated hypotheses to test.

    ActionValidate candidates in wet-lab experiments; treat AI output as candidate supply, not conclusion.

    ConsequenceMore testable candidates per unit of researcher time, with human confirmation as the binding bottleneck.

  3. The lab loop: model-assisted development inside frontier labs

    SignalModels debug training runs, manage deployments, and diagnose evaluation failures.

    Decision ownerFrontier lab engineering leads.

    ActionScale model-assisted development while requiring human sign-off on changes (the control diagram).

    ConsequenceFaster development cycles; the announcement runs ahead of the science, and only write-protected holdouts can tell improvement from hacking.

The limit this episode must keep

The loop is not closed. Model collapse shows recursive training can irreversibly degrade models; reward hacking shows optimization without a trustworthy objective produces scores, not improvement; Karpathy's celebrated Auto-Research only improves a GPT-2-scale model; Pichai says 'we aren't quite there yet'; and Toner insists that using AI for research is not RSI — the classic definition removes humans entirely. What is shipping, in one expert's words, is 'extraordinarily powerful automation, but still operating within a closed-loop system designed by humans.'

Key sources