Self-Improving AI — Self-Play, Co-Evolution, and Generated Curricula

What has to be true for a model to get better without anyone adding data? This article pulls three conditions out of AlphaZero's self-play, shows exactly which one breaks first for language models, explains how co-evolution and generated curricula try to patch the gap, and ends with why self-improvement claims are unusually easy to evaluate wrong.

Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

Why "getting better by yourself" is a special claim

The default way to improve a model is to add data. A human labels it, a human writes the answer, a human judges the output. Performance scales roughly with the human effort you pour in.

The family of methods called self-improvement tries to cut that proportionality. The model gets stronger using its own output as raw material, with no new data from outside. If it works, the ceiling on performance is set by compute rather than by headcount — which is an enormous economic difference.

And there is one case where it worked perfectly. In 2017, AlphaZero was handed nothing but the rules of Go, chess, and shogi, played against itself, and learned all three without ever seeing a human game.

Try the same trick on a language model and the results split cleanly into it works and it quietly breaks. The breaking mode is the nasty part: your internal metrics keep improving while the actual capability degrades. This article extracts what was true for AlphaZero and no longer holds for language, then follows how co-evolution and generated curricula attempt to buy those properties back.

An analogy: when does hitting against a wall count as practice?

A tennis wall is a free opponent that will play forever. But wall practice only improves your stroke mechanics — it never makes you win matches, because the wall always returns the ball the same way.

Now consider shadow swinging, where there is no opponent at all. It still counts as practice, but only because something supplies judgment: a mirror, a coach, your own trained feel. Without judgment, ten thousand swings reinforce a bad form ten thousand times.

Those are the two ingredients. An opponent that matches your current level and rises as you rise, and a judge that can tell good output from bad more reliably than you can produce it. Whether those two are present decides whether self-improvement is a machine for unbounded growth or a machine for amplifying your own mistakes.

Taking AlphaZero apart

The training loop is almost embarrassingly simple. Play one game against yourself with the current network. Observe who won. Record every position, the move played there, and the final result, then update the network on that record. Repeat.

The easily missed part is that during self-play, moves are not sampled straight from the network. They come from Monte Carlo Tree Search running on top of the network. Search spends extra time looking ahead, so it plays better than the raw network does. The post-search move distribution is therefore a teacher one notch stronger than the student that produced it. Training is the act of copying that stronger distribution back into the raw network.

Symbol by symbol: $s$ is a board position, $z$ is how that game ended ($+1$ for a win, $-1$ for a loss), and $v_\theta(s)$ is the network's guess at how good the position is. $\boldsymbol{\pi}(s)$ is how often search actually visited each move, $\boldsymbol{p}_\theta(s)$ is the move distribution the network produces with no search at all, and $c\lVert\theta\rVert^2$ keeps the weights from blowing up.

In plain words the loss says two things: make your prediction of the outcome match the real outcome, and make your snap judgment match the judgment you reach after thinking. The first half learns value; the second half is distillation — turning deliberate search into intuition. The full arc from AlphaGo to AlphaZero is covered in AlphaGo from scratch; here we only need the self-improvement skeleton.

The three conditions

That decomposition gives us three requirements.

Condition 1: an improvement operator. You need a procedure that produces better output from the same model. For AlphaZero it was tree search: pay more compute at play time and you get a stronger move without any additional training. If there is no headroom you can buy with compute, there is nothing for the model to learn from itself.

Condition 2: a verifier that is more accurate than the policy, and cheap. Deciding who won a game of Go is settled by the rules, never wrong, and costs essentially nothing. That is what made millions of free training signals possible.

Condition 3: difficulty that tracks the learner. In self-play the opponent is a copy of yourself, so it improves exactly as fast as you do. The match is permanently even — which is to say, permanently at the right difficulty. Nobody had to hand-design a curriculum; the setup produced one for free.

The importance of the third one is clearest in reverse. Pin the opponent to a fixed weak program and you soon win every game; the win/loss signal saturates and learning stops. Same reason the tennis wall stops teaching you.