LLM Agents from Scratch — Designing the Tool-Use Loop

Think, reach for a tool, look at what came back, think again. Working only from the text of the ReAct paper (Yao et al., 2022): why adding language to the action space is the whole idea, what function calling really implements, and the failure modes the authors actually measured — runaway loops and hallucinated tool calls.

ReAct: Synergizing Reasoning and Acting in Language Models

Where one shot of generation cannot reach

Picture a model asked how many rooms a hotel has right now. Answering from what it read during training is a guess. The ReAct paper puts it this way: chain-of-thought reasoning is "a static black box" — the model generates thoughts from its own internal representations and is not grounded in the external world (§1). Ungrounded, you get hallucinated facts and errors that propagate down the chain.

Now the opposite. What about an agent that reaches for tools without thinking? Figure 1 of the paper shows one: it can run the searches, but when the moment comes to assemble the retrieved pieces into a final answer, it fails (§2).

Humans do both, and that is the paper's starting point. Cooking, we reason in language between actions: "everything is chopped, so now heat the water" (tracking progress), "no salt, so soy sauce instead" (handling an exception), "I don't know how to make dough — let me look it up" (realising external information is needed). ReAct — Reason + Act — carries that back-and-forth into a language model (§1).

The loop, written down

Start from the standard framing. At step $t$ the agent receives an observation $o_t$ from the environment and takes an action $a_t$ under a policy $\pi(a_t \mid c_t)$, where $c_t=(o_1,a_1,\dots,o_{t-1},a_{t-1},o_t)$ is everything seen and done so far (§2). Read $\pi$ as "the function that picks the next move given the history" and $c_t$ as that history. The more implicit the mapping from context to action, the harder that policy is to learn.

ReAct's answer fits on one line (§2):

$\mathcal{A}$ is the original action space (run a search, click a button) and $\mathcal{L}$ is the space of language. Equation (1) says: add, to the menu of actions, the action of merely writing words. Spelled out in words: the hat marks an enlarged menu, and the union sign is the whole trick — everything the agent could already do to the world stays on the menu, and exactly one item is added to it, which is say something. The paper calls an action in that language space a thought, and gives it one decisive property. A thought does not change the external environment, so no observation comes back. Instead the context grows to $c_{t+1}=(c_t,\hat a_t)$ and supports the reasoning and acting that follow.

So "thinking", for an agent, is a side-effect-free action wired into the loop. That is all the Thought → Action → Observation sequence in every agent framework really is.

One more property you cannot design around. Every step lengthens $c_t$, and the model re-reads all of it each time. If the input per step grows linearly, the total tokens read across a run grows with the square of the step count.

What the paper actually measured

The experiments keep PaLM-540B frozen and drive it with few-shot prompts: six hand-written trajectories for HotpotQA, three for Fever (§3.2). The toolbox is three actions against a Wikipedia-like API — search[entity] returns the first five sentences of the page (or the top five similar entities), lookup[string] returns the next sentence containing that string (browser Ctrl+F), and finish[answer] ends the task. The paper openly calls this "significantly weaker than state-of-the-art retrievers"; the point was to force retrieval through explicit reasoning in language (§3.1).

The results are not tidy. In Table 1 (PaLM-540B), HotpotQA exact match is 28.7 for Standard, 29.4 for CoT, 27.4 for ReAct — ReAct alone loses to CoT. On Fever it wins: 60.9 against CoT's 56.3. And the best numbers come from switching between them: ReAct→CoT-SC scores 35.1 on HotpotQA, CoT-SC→ReAct scores 64.6 on Fever (§3.3).