Paper Walkthrough: S³Gym — Does Self-Testing and Self-Judging Add Up to Self-Improvement?

Can an LLM agent really learn from its own experience? A benchmark that separates permissive exploration from strict held-out evaluation across seven games, comparing raw history, summary memory, and parameter training under one protocol.

S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?

Plenty of experience, no improvement

An LLM acting as an agent calls tools, runs code, and reads what the environment sends back. Loop through observe → reason → act → feedback a few dozen times and you have a pile of behavioral records. Yet most agent evaluations still measure the model as a fixed policy. They tell you how well it scores today; they say nothing about whether it can look at its own failures and do better tomorrow.

The paper here goes straight at that gap. Its original title is "S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?" (arXiv:2608.31100, published 31 August 2026, by authors at ByteDance Seed, M-A-P and TokenWave.AI). S³Gym stands for Self-Testing, Self-Judging, and Self-Improvement Gym.

Here is the abstract in plain terms. The paper decomposes experience-driven learning into three coupled capabilities — Self-Testing (probing your own behavior to gather evidence), Self-Judging (scoring that experience yourself), and Self-Improvement (letting the score change future decisions) — and builds S³Gym to measure them. It separates permissive exploration from strict held-out evaluation, instantiating the protocol in seven text-based games with executable environment verifiers. Three pathways for absorbing experience are compared under one protocol: raw History ICL, score-conditioned Summary Memory, and parameter Training. The verdict is that self-improvement is neither automatic nor uniform. Summaries help when experience compresses into reusable strategic rules, but often lose to raw history when success hinges on precise, state-contingent information. Parameter training produces substantial gains on some tasks alongside unstable improvement and severe negative transfer on others. Recognizing a good action is not enough; the agent has to turn feedback into executable, transferable policy.

The analogy: driving range versus the course

Picture a golf driving range. Flat mat, unlimited balls, no cost to a bad swing. Hit a hundred and you have accumulated experience. The actual course has slopes, a finite number of shots, and every mistake lands on your card.

S³Gym is built the same way. The exploration phase uses permissive settings — smaller boards, weaker penalties, extra lives. The evaluation phase uses strict ones: larger search spaces, one-mistake-and-you-are-out termination. The two use disjoint random seeds, and evaluation trajectories never enter history, memory, or training data (§4.4). Only what actually transfers shows up in the number.

And on this driving range, there is no scoreboard. The verifier-computed reward $r_{x,i}$ is retained benchmark-side and withheld from the agent during exploration (§4.2). The agent has to declare for itself how good each move was. That declaration is Self-Judging, and the paper measures it as a capability in its own right.

One cycle, formalized

The paper writes a single self-improvement cycle as five stages (§4.1):

In words: try things, score yourself, consolidate what you learned, absorb it, then get measured under stricter rules. Here $t$ indexes the cycle and $p$ names the pathway — History, Memory, or Training.

At every step the model emits an action and a self-judged score together (§4.2):

Which says: given the current observation $O_{x,i}$, the within-episode history $H_{x,i}$, and the experience $Z_{t}^{(p)}$ carried over from earlier exploration, return an action $a_{x,i}$ and a self-judged score $s_{x,i}$. In the implementation the model's final line follows a fixed contract, Answer: <ACTION>, Score: <SELF-JUDGED SCORE>. (Appendix A.1), and the paper is explicit that $s$ is meant to be the immediate reward the action should earn under the stated rules, not a free-form confidence value. The environment returns the next observation, agent-visible feedback, the verifier reward, and a termination flag — but only the reward is hidden. That withholding is the whole mechanism.