LLM Agents from Scratch — Designing the Tool-Use Loop

Think, reach for a tool, look at what came back, think again. Working only from the text of the ReAct paper (Yao et al., 2022): why adding language to the action space is the whole idea, what function calling really implements, and the failure modes the authors actually measured — runaway loops and hallucinated tool calls.

ReAct: Synergizing Reasoning and Acting in Language Models

Where one shot of generation cannot reach

Picture a model asked how many rooms a hotel has right now. Answering from what it read during training is a guess. The ReAct paper puts it this way: chain-of-thought reasoning is "a static black box" — the model generates thoughts from its own internal representations and is not grounded in the external world (§1). Ungrounded, you get hallucinated facts and errors that propagate down the chain.

Now the opposite. What about an agent that reaches for tools without thinking? Figure 1 of the paper shows one: it can run the searches, but when the moment comes to assemble the retrieved pieces into a final answer, it fails (§2).

Humans do both, and that is the paper's starting point. Cooking, we reason in language between actions: "everything is chopped, so now heat the water" (tracking progress), "no salt, so soy sauce instead" (handling an exception), "I don't know how to make dough — let me look it up" (realising external information is needed). ReAct — Reason + Act — carries that back-and-forth into a language model (§1).

The loop, written down

Start from the standard framing. At step $t$ the agent receives an observation $o_t$ from the environment and takes an action $a_t$ under a policy $\pi(a_t \mid c_t)$, where $c_t=(o_1,a_1,\dots,o_{t-1},a_{t-1},o_t)$ is everything seen and done so far (§2). Read $\pi$ as "the function that picks the next move given the history" and $c_t$ as that history. The more implicit the mapping from context to action, the harder that policy is to learn.

ReAct's answer fits on one line (§2):

$\mathcal{A}$ is the original action space (run a search, click a button) and $\mathcal{L}$ is the space of language. Equation (1) says: add, to the menu of actions, the action of merely writing words. Spelled out in words: the hat marks an enlarged menu, and the union sign is the whole trick — everything the agent could already do to the world stays on the menu, and exactly one item is added to it, which is say something. The paper calls an action in that language space a thought, and gives it one decisive property. A thought does not change the external environment, so no observation comes back. Instead the context grows to $c_{t+1}=(c_t,\hat a_t)$ and supports the reasoning and acting that follow.

So "thinking", for an agent, is a side-effect-free action wired into the loop. That is all the Thought → Action → Observation sequence in every agent framework really is.

One more property you cannot design around. Every step lengthens $c_t$, and the model re-reads all of it each time. If the input per step grows linearly, the total tokens read across a run grows with the square of the step count.

What the paper actually measured

The experiments keep PaLM-540B frozen and drive it with few-shot prompts: six hand-written trajectories for HotpotQA, three for Fever (§3.2). The toolbox is three actions against a Wikipedia-like API — search[entity] returns the first five sentences of the page (or the top five similar entities), lookup[string] returns the next sentence containing that string (browser Ctrl+F), and finish[answer] ends the task. The paper openly calls this "significantly weaker than state-of-the-art retrievers"; the point was to force retrieval through explicit reasoning in language (§3.1).

The results are not tidy. In Table 1 (PaLM-540B), HotpotQA exact match is 28.7 for Standard, 29.4 for CoT, 27.4 for ReAct — ReAct alone loses to CoT. On Fever it wins: 60.9 against CoT's 56.3. And the best numbers come from switching between them: ReAct→CoT-SC scores 35.1 on HotpotQA, CoT-SC→ReAct scores 64.6 on Fever (§3.3).

The switching rule is blunt. If ReAct has not answered within a step budget (7 for HotpotQA, 5 for Fever), fall back to CoT-SC; if the CoT-SC majority answer occurs less than half the time, fall back to ReAct (§3.2).

On decision-making tasks the picture is far cleaner. On ALFWorld — a text game of household tasks, 134 unseen instances — ReAct reaches a 71% success rate best-of-6, against 45% for action-only Act and 37% for the imitation-learned BUTLER. ReAct's worst trial, 48%, beats the best trial of either (§4, Table 3). On WebShop, Act gets a 30.1% success rate and ReAct 40.0% (§4, Table 4).

One more result worth carrying around. Fine-tuning smaller models on 3,000 correct ReAct trajectories flips the ranking: fine-tuned PaLM-8B beats every prompted PaLM-62B method, and fine-tuned PaLM-62B beats every prompted 540B method (§3.3).

What function calling is doing

What follows is my reading, built on the paper's description. The paper's search[...] was one line of free text. Modern function calling does the same job in JSON.

ctx = [{"role": "user", "content": question}]
for step in range(MAX_STEPS):               # a cap is mandatory (the paper used 7/5)
    msg = model.generate(ctx, tools=tools)  # a thought, or a tool_call
    ctx.append(msg)
    if msg.tool_call is None:
        break                               # the finish action
    obs = registry[msg.tool_call.name](**msg.tool_call.args)
    ctx.append({"role": "tool", "content": obs})   # feed the observation back

What changed is the format: structured enough for a machine to validate the arguments. What did not change is the shape — thoughts, actions and observations stack up in one context, and the model re-reads all of it to pick the next move. MAX_STEPS is not decoration. The paper capped its own runs and reports that among trajectories with correct answers, only 0.84% on HotpotQA and 1.33% on Fever ever used the full budget (§3.2, footnote 3). The cap costs almost no accuracy and stops the runaway cases.

The failure modes were measured

The authors hand-labelled HotpotQA trajectories (§3.3, Table 2). This is the most useful table in the paper.

• Hallucination: 56% of CoT's failures. ReAct: 0%. False positives among the successes also drop — 14% for CoT, 6% for ReAct. This is where grounding in an external source shows up.
• Reasoning error: the reverse — 47% for ReAct against 16% for CoT. Interleaving thought, action and observation is a structural constraint, and the paper attributes the loss of flexibility in formulating reasoning steps to exactly that.
• Runaway loops: inside that reasoning-error bucket sits a pattern specific to ReAct, in which the model regenerates its previous thoughts and actions over and over. The paper files it as failing to reason about the proper next action and jump out of the loop, and in a footnote suspects the sub-optimal greedy decoding procedure, suggesting better decoding such as beam search might help (§3.3, footnote 6).
• Search result error: 23% of failures come from a search returning nothing useful. Once that happens the trajectory derails and rarely recovers.

Being told to suspect the decoder is a gift. When an agent keeps emitting the same move, the first thing to inspect is the sampling configuration, not the prompt.

Limits and open questions

• The prompting setup limits which reasoning and acting behaviours are expressible. Tasks with large action spaces need more demonstrations, and those easily exceed the input length limit of in-context learning (§1, §6).
• Nowhere near human experts. On WebShop, ReAct scores 40.0% against a human expert's 59.6%; humans explore far more products and reformulate their queries repeatedly (§4).
• Benchmark labels themselves go stale. In the hotel-rooms example, only ReAct retrieved the up-to-date answer — and it disagreed with the dataset label (Appendix A.2).
• The main model is not public. Reproducibility rests on the prompts in the appendix and the GPT-3 runs. Appendix A.1 reports GPT-3 (text-davinci-002) at 30.8 on HotpotQA and 78.4 on ALFWorld, above PaLM-540B's 29.4 and 70.9.
• Wiring a model to the world is itself dangerous. The Ethics Statement notes the experiments were confined to Wikipedia and WebShop, with an action space that cannot really buy anything or edit anything. "Keep the agent's permissions narrow" was already written down here.

How this shows up on the job

The people who build this loop are application engineers shipping internal knowledge search or first-line support, and the platform engineer who has to run it in production.

The knob names. On OpenAI-style APIs: tools, tool_choice, parallel_tool_calls, plus temperature and max_tokens. On Anthropic-style: tools and the stop_reason == "tool_use" branch. In frameworks: LangGraph's recursion_limit, AutoGen's max_consecutive_auto_reply. If you roll your own, you need three caps: MAX_STEPS, a token budget per run, and a wall-clock budget. The paper's 7- and 5-step cutoffs, which cost almost no accuracy, are your justification.

Three traps. One: the same thought, regenerated. If an identical tool_call appears twice in a row, stop that step and divert — ReAct→CoT-SC is precisely that design. Two: calls to tools and arguments that do not exist. Validate against a JSON Schema, and return validation failures as observations rather than raising, so the model can correct itself on the next move. Three: quadratic cost. An uncapped loop is one you find out about on the invoice.

The design-review question is "is ReAct better than CoT?" Quote the numbers: 27.4 against 29.4 on HotpotQA (a loss), 60.9 against 56.3 on Fever (a win), and the best result from switching between them. ReAct trades hallucination down to 0% and reasoning errors up to 47%. It is a choice about leaning on internal versus external knowledge, not a ranking — say that and you have shown you understand what adding tools costs.

The instinct to fetch before answering is continuous with RAG fundamentals. The difference is that RAG fetches once and answers, while an agent reasons about whether and what to fetch, repeatedly. And because the context lengthens with every step, the KV cache stops being a theory topic and becomes your operating cost.

Summary

• ReAct only adds language to the action space; a thought is an action that returns no observation (§2, equation (1))
• The results are not uniform — a loss to CoT on HotpotQA, a win on Fever, a large win on decision tasks (ALFWorld 71% vs 45%)
• 0% hallucination (CoT: 56%) is bought with 47% reasoning errors (CoT: 16%); runaway loops point at greedy decoding (§3.3, Table 2, footnote 6)
• Step caps cost almost nothing — the authors capped their own runs at 7 and 5

Original paper: Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models" (arXiv:2210.03629, 2022 / v3)