Designing Agent Memory — Short-Term, Long-Term, Episodic

An LLM remembers nothing. Conversations only look continuous because you re-send the whole history every turn. Building up from zero: the three-layer split, the function that decides what gets recalled, and the part almost nobody designs — forgetting.

MemGPT: Towards LLMs as Operating Systems

The colleague who arrives every morning with no memory

Picture a shift handover. The person taking over is excellent — knows the machines, knows the runbook cold. One problem: they have no recollection whatsoever of what happened overnight. All they know is what's written in the notebook left on the desk.

An LLM agent is in exactly that position. Enormous general knowledge is baked into the weights, but those weights froze when training ended, and yesterday's conversation with you isn't in there. A conversation looks continuous only because every turn, you make the model read the entire exchange again from the top. It isn't remembering. It's re-reading.

Which means "designing agent memory" is not a job of making the model smarter. It is the job of deciding what goes in that notebook, what stays out, when it gets rewritten, and when it gets thrown away. This article builds up those three decisions — write, select, forget — in order. The agent loop itself (think, reach for a tool, look at the result, think again) is covered in LLM Agents from Scratch; here we only care about what that loop carries around with it.

Split memory three ways

Borrowing the categories used in human memory research makes the design far easier to see. Agent memory works best split into three kinds and handled separately.

• Short-term memory (working memory) — the papers spread on the desk right now. Recent turns held raw, unsummarized. Lifetime: a few turns.
• Episodic memory — a record of when something happened. "Aug 3: this user said the invoice PDF wouldn't open; we offered a reissue and that resolved it." Timestamp and context travel together.
• Semantic memory (long-term) — a stable fact distilled out of events and detached from time. "This user bills to Company A." "The config file is TOML, not YAML."

Mix these and something will break, because the write frequency and the shelf life of correctness differ by orders of magnitude. Short-term memory is rewritten every turn; semantic memory might get updated once a month. Conversely, a single wrong entry in semantic memory keeps acting on every conversation forever. There is no reason to store them in one place under one policy.

Some systems add a fourth: procedural memory — knowledge about how things are done, like "deploys for this user always go through staging." Its rewrite rate differs again, so it usually lives in the system prompt or a dedicated instructions file, managed apart from the rest.

Why "just include everything" fails

The naive implementation resends the full history every turn. With a long enough context window that looks sufficient. It runs into two walls.

The first is the cost wall. Resending the whole history each turn is more expensive than it looks. Let $m$ be the tokens added per turn (your message plus the agent's reply) and $b$ the fixed part such as the system prompt. The cumulative input tokens sent through turn $T$ are:

The same thing in words: the running bill $C$ after $T$ turns is the fixed preamble $b$ paid once per turn, plus the conversation itself, where every sentence typed at turn 3 is still being paid for again at turn 40. The $T(T+1)/2$ is just the arithmetic of "1 copy of the first turn, 2 copies of the second, 3 of the third, …" added up.

Read it like this: double the number of turns and you pay roughly four times the tokens. What you send at turn $t$ grows in proportion to $t$, and you add that up $T$ times, so the total grows with the square of $T$. A conversation twice as long costing four times as much is counterintuitive, but that's what the $T(T+1)/2$ on the right side of equation (1) says.

The second is the attention wall. The assumption that anything placed in context will actually be used does not hold. Liu et al.'s "Lost in the Middle" (arXiv:2307.03172) measured accuracy as the needed information was moved between the start, the end, and the middle of a long context, and reported a U-shaped pattern in which information in the middle is used least. So if stuffing the whole history buries an important fact in the middle, it is present and never consulted. From the memory-design side the conclusion is one line: context is a desk, not a warehouse.

Summarization is compression — the design is what you throw away

Once short-term memory overflows the desk, something has to compress. The standard move is a rolling summary: fold the oldest $k$ turns into one summary, then carry the summary plus the recent turns.

There's one failure here that's hard to notice. Summarizing summaries makes degradation compound. Same shape as re-saving a JPEG: compression is lossy, so the second summary takes the first summary as input, not the original conversation. A carefully negotiated spec detail thins out, over five foldings, into "the user talked about configuration." And the thinning is invisible from the thinned side.