Designing Agent Memory — Short-Term, Long-Term, Episodic

An LLM remembers nothing. Conversations only look continuous because you re-send the whole history every turn. Building up from zero: the three-layer split, the function that decides what gets recalled, and the part almost nobody designs — forgetting.

MemGPT: Towards LLMs as Operating Systems

The colleague who arrives every morning with no memory

Picture a shift handover. The person taking over is excellent — knows the machines, knows the runbook cold. One problem: they have no recollection whatsoever of what happened overnight. All they know is what's written in the notebook left on the desk.

An LLM agent is in exactly that position. Enormous general knowledge is baked into the weights, but those weights froze when training ended, and yesterday's conversation with you isn't in there. A conversation looks continuous only because every turn, you make the model read the entire exchange again from the top. It isn't remembering. It's re-reading.

Which means "designing agent memory" is not a job of making the model smarter. It is the job of deciding what goes in that notebook, what stays out, when it gets rewritten, and when it gets thrown away. This article builds up those three decisions — write, select, forget — in order. The agent loop itself (think, reach for a tool, look at the result, think again) is covered in LLM Agents from Scratch; here we only care about what that loop carries around with it.

Split memory three ways

Borrowing the categories used in human memory research makes the design far easier to see. Agent memory works best split into three kinds and handled separately.

• Short-term memory (working memory) — the papers spread on the desk right now. Recent turns held raw, unsummarized. Lifetime: a few turns.
• Episodic memory — a record of when something happened. "Aug 3: this user said the invoice PDF wouldn't open; we offered a reissue and that resolved it." Timestamp and context travel together.
• Semantic memory (long-term) — a stable fact distilled out of events and detached from time. "This user bills to Company A." "The config file is TOML, not YAML."

Mix these and something will break, because the write frequency and the shelf life of correctness differ by orders of magnitude. Short-term memory is rewritten every turn; semantic memory might get updated once a month. Conversely, a single wrong entry in semantic memory keeps acting on every conversation forever. There is no reason to store them in one place under one policy.

Some systems add a fourth: procedural memory — knowledge about how things are done, like "deploys for this user always go through staging." Its rewrite rate differs again, so it usually lives in the system prompt or a dedicated instructions file, managed apart from the rest.

Why "just include everything" fails

The naive implementation resends the full history every turn. With a long enough context window that looks sufficient. It runs into two walls.

The first is the cost wall. Resending the whole history each turn is more expensive than it looks. Let $m$ be the tokens added per turn (your message plus the agent's reply) and $b$ the fixed part such as the system prompt. The cumulative input tokens sent through turn $T$ are:

The same thing in words: the running bill $C$ after $T$ turns is the fixed preamble $b$ paid once per turn, plus the conversation itself, where every sentence typed at turn 3 is still being paid for again at turn 40. The $T(T+1)/2$ is just the arithmetic of "1 copy of the first turn, 2 copies of the second, 3 of the third, …" added up.

Read it like this: double the number of turns and you pay roughly four times the tokens. What you send at turn $t$ grows in proportion to $t$, and you add that up $T$ times, so the total grows with the square of $T$. A conversation twice as long costing four times as much is counterintuitive, but that's what the $T(T+1)/2$ on the right side of equation (1) says.

The second is the attention wall. The assumption that anything placed in context will actually be used does not hold. Liu et al.'s "Lost in the Middle" (arXiv:2307.03172) measured accuracy as the needed information was moved between the start, the end, and the middle of a long context, and reported a U-shaped pattern in which information in the middle is used least. So if stuffing the whole history buries an important fact in the middle, it is present and never consulted. From the memory-design side the conclusion is one line: context is a desk, not a warehouse.

Summarization is compression — the design is what you throw away

Once short-term memory overflows the desk, something has to compress. The standard move is a rolling summary: fold the oldest $k$ turns into one summary, then carry the summary plus the recent turns.

There's one failure here that's hard to notice. Summarizing summaries makes degradation compound. Same shape as re-saving a JPEG: compression is lossy, so the second summary takes the first summary as input, not the original conversation. A carefully negotiated spec detail thins out, over five foldings, into "the user talked about configuration." And the thinning is invisible from the thinned side.

Two things to do about it in practice.

• Keep the raw log separately and make the summary a pointer into it. Attach the originating turn IDs to each summary item and you can pull the original text back only when you need it. The summary is an index, not a replacement for the record.
• Count the foldings. Freeze a span once it's been summarized and add new spans as separate summaries (hierarchically). Nothing gets compressed twice, so degradation stops there.

And what to keep in a summary is not decided for you. Tell a model "summarize this conversation" and it writes a plot synopsis. What an agent needs is three things — decisions made, open questions, and constraints the user stated explicitly — so it's safest to pin those as the output schema of the summarization prompt.

External memory — recall is a function

Semantic and episodic memory live outside the context, in a database. What you design there isn't storage but recall: the function that decides which memories go on the desk right now.

The naive version is embedding similarity search — the same machinery as RAG, and the chunking and top-k material from RAG Fundamentals and Design Patterns applies directly. Memory recall differs from document search in one respect: recency and importance matter. "Last week" and "a year ago" are worth different amounts even at identical semantic similarity.

Park et al.'s Generative Agents (arXiv:2304.03442) turned that intuition into a three-term sum. The score of memory $m$ against the current situation $q$ takes roughly this form:

The score in words: $s$ is the shelf position that memory $m$ earns in the situation $q$, and it is three separate marks added together — how well it matches the topic, how fresh it is, how much it mattered when it was written — with $\alpha, \beta, \gamma$ deciding how loudly each of the three gets to speak.

Term by term. $\mathrm{rel}$ is relevance — how close the memory's embedding sits to the current topic. $e^{-\lambda \Delta t}$ is recency, where $\Delta t$ is the time since that memory was last retrieved and $\lambda$ sets how fast it fades; it shrinks exponentially as time passes. $\mathrm{imp}$ is importance, assigned at write time, e.g. by having the model itself rate the memory on a 1–10 scale. The weights $\alpha, \beta, \gamma$ combine them, and implementations normalize each term to 0–1 before adding.

Put plainly, the formula says: surface memories that are close to the current topic, recently used, and important to begin with. Ranking on any one of them alone fails. Relevance alone and a similar conversation from a year ago shows up every time; recency alone and the last bit of small talk elbows out the actual subject.

Granularity is a design choice too. One conversational turn per record loses meaning when recalled, since the surrounding context is severed; make records too large and irrelevant material rides along onto the desk. A workable landing point is to normalize to "one memory = one statable fact (or one event)" and keep a link back to the originating turn.

Designing forgetting — the part nobody builds

Ninety percent of the effort in a memory system goes into writing and retrieving. What breaks in production is almost always failing to forget.

The classic incident: in March, "lives in Tokyo" was written. In August the user says "I've moved to Osaka." The straightforward implementation appends a new record. Now two contradicting facts sit side by side, recall puts one of them — or both — on the desk, and the agent quotes a different address in every conversation.

Forgetting isn't deletion. Four implementable forms:

• Overwrite (keyed memory) — force semantic memory into an (attribute key, value, updated-at) shape and keep exactly one live record per key. For facts that ought to be unique, like address or employer, contradiction becomes structurally impossible.
• Decay — use the $e^{-\lambda\Delta t}$ from equation (2) not only for scoring but for retention, demoting memories that fall below a threshold to a colder tier. Demote rather than delete (hot → cold → archive) and you can still audit later.
• TTL — facts with an inherent expiry, like "this week's deadline is Friday," get one at write time. Expired records drop out of the recall candidates automatically.
• Contradiction check — at write time, compare against the existing value for the same key. On conflict, update instead of appending, and keep the old value on the episodic side as "when it changed."

Making all four work requires provenance on each memory. Whether the user stated something or you inferred it should change how it's handled: inferences held as hypotheses you can discard at any time, statements retained until overwritten. Two extra columns — source_turn_id and status (stated / inferred) — are enough to draw that line.

A minimal memory layer

All of the above, in twenty lines:

def turn(user_msg, state, store):
    # 1. Recall: rank external memory against this message, put the top few on the desk
    recalled = store.recall(query=user_msg, k=5)   # rel + recency + importance

    # 2. Assemble: fixed -> long-term (stable) -> recalled -> summary -> recent turns
    prompt = [SYSTEM, state.profile, recalled, state.summary, *state.recent, user_msg]
    reply = llm(prompt)

    # 3. Update short-term memory and compress the overflow
    state.recent += [user_msg, reply]
    if tokens(state.recent) > BUDGET:
        old, state.recent = state.recent[:-KEEP], state.recent[-KEEP:]
        state.summary = summarize(state.summary, old)   # decisions / open / constraints only

    # 4. Write: only stated facts, keyed and overwritten
    for fact in extract_facts(user_msg, reply):        # {key, value, status}
        store.upsert(fact, ts=now, source_turn_id=state.turn_id)
    return reply

The ordering on the second step is deliberate: what doesn't change goes first, what changes goes last. That's not a readability point, it's a billing point. Prompt caching only works on a shared prefix, so putting the current time or this turn's recall results at the very top knocks every token after it out of cache (details in Prompt Caching and Context Design). Where you inject memory is decided together with the memory system, not after it.

There is also a line of work that hands the context editing itself to the model. MemGPT (arXiv:2310.08560) frames it as OS virtual memory, with the model paging content in and out of its own context through function calls. Reflexion (arXiv:2303.11366) stacks verbal self-reflections on failed attempts into episodic memory and mixes them into the next attempt's input.

How this shows up on the job

Who touches it, and when. Typically an engineer running an internal assistant or support automation, right after complaints land: "it asks me the same thing over and over," "it forgot what we decided last week." The first move is not to add a memory feature — it's to dump the current context verbatim and read it. More often than not the information is in there, buried in the middle or erased by summarization.

Parameters and tools you'll actually touch. The context budget (the remaining-token fraction that triggers summarization; thresholds around 0.6–0.8 are common), KEEP (how many recent turns stay raw), recall top_k, the decay constant $\lambda$ (easier to reason about as a half-life: a 7-day half-life is $\lambda = \ln 2 / 7$), and TTL. Off-the-shelf memory layers include Letta (MemGPT), Mem0, Zep, and LangGraph's checkpointer; rolling your own means adding key / value / status / ts / source_turn_id columns to pgvector or whatever vector store you already run.

Pitfalls that turn into incidents.

1. Compounding summary loss — implementations that re-summarize repeatedly lose information silently. Make the folding count a monitored metric.
2. Cache-busting bills — splice recall results into the head of the prompt and the shared prefix breaks every turn. If the bill climbs while per-token prices haven't, suspect this first.
3. Injection through memory — an instruction smuggled into a "fact" extracted from user input becomes a persistent attack that fires in a later session. You want both halves: reject imperatives at write time, and always present memories as delimited data.
4. No right to be forgotten — once you store personal information, you need a path that answers deletion requests. A design without source_turn_id cannot satisfy "delete everything you know about this person."
5. Adding memory without evaluation — nothing guarantees more memory is better. Recall dragging in noise and making things worse is common, so build the regression test that compares memory-on against memory-off on the same conversation logs first.

How it gets asked in a design review. "When a user says they've moved, how do you update the stored address?" — overwrite by key rather than append, keep the old value as history on the episodic side. "If context windows get 10× longer, does the memory system become unnecessary?" — no. Cost still scales with the square of length, the middle still gets ignored, and length can't substitute for a mechanism that forgets.

Wrapping up

• The LLM remembers nothing. Continuity is re-reading. What you design is what goes on the desk.
• Split into short-term, episodic, and semantic — because write frequency and shelf life differ by orders of magnitude.
• Recall is a function. Rank on relevance, recency, and importance together; never on one alone.
• Production breaks on not forgetting. Build overwrite, decay, TTL, and contradiction checks in from the start.

Before adding memory, dump what your agent is actually reading right now and look at it once. The design starts there and nowhere else.