Paper Walkthrough: SpeakerMem-R1 — A Two-Track Memory That Remembers Who Said What, About Whom, and When

A from-scratch look at SpeakerMem-R1, a long-term memory for group conversations. It keeps verbatim messages alongside person- and group-level states, and trains a small note-taking model (Qwen2.5-3B) with reinforcement learning that scores memory owner by owner.

SpeakerMem-R1: Speaker-Centered Dual-Track Memory for Multi-Party Dialogue

A minute-taker who drops the names is useless

Picture a neighborhood group chat with fifty members. Festival prep, trash-duty rotation, and membership fees all scroll by at once. A month later someone asks: "So who ended up on tent duty?" or "Was Sato actually in favor of Yamada's plan?" What you need isn't relevant text. You need who said it, who it was about, and how it was finally settled.

The paper this article covers is "SpeakerMem-R1: Speaker-Centered Dual-Track Memory for Multi-Party Dialogue" (Zheng et al., Zhejiang University, arXiv:2609.26780, September 2026).

Here is what the abstract claims. Long-term memory for multi-party conversation needs more than retrieving related content: it has to tell apart who said what, whom each statement concerns, how people perceive one another, what the group shares, and how states change over time. Existing general-purpose LLM memory systems tend to lose person and group relations, or fail to integrate clues scattered across members, groups, and time. The authors name two core bottlenecks — message attribution with relational understanding, and state reconstruction from interleaved histories — and propose SpeakerMem-R1. It stores speaker-labeled verbatim messages alongside derived states organized into person-level and group-level views, then combines evidence from both tracks by entity, event, and time at query time. The component that writes the structured memory, the Writer, is trained with SpeakerLevenshtein and speaker-conditioned GRPO so it can run locally as a small model. On GroupMemBench, SocialMemBench, and EverMemBench the reported binary accuracies are 47.9%, 69.2%, and 61.9%.

We will build up to it from zero. For the general design space of agent memory, see Designing Memory for Agents.

Why a better retriever alone doesn't fix this

The paper splits the problem in two (§1):

1. Message attribution: who said it, whom it concerns, and whether it's personal or shared by the group.
2. State reconstruction: rebuilding a current or past state from clues spread across members, groups, and time.

Standard RAG grabs the top-k passages most similar to the question. The authors point out that global relevance retrieval gives no guarantee that a low-frequency member, or the right topic branch, even makes it into the candidate set (§1). Lean instead on summaries, fact aggregation, or generic memory graphs, and you risk losing the source and owner, the person-vs-group scope, or which version of a state you're looking at. So the answer is not "summarize more" or "add more graph edges." What's needed is verifiable evidence that carries participant relations, plus local, query-specific state reconstruction.

The multi-party benchmarks the paper cites report that general-purpose memory systems degrade substantially in these settings, while BM25 or dense retrieval stay competitive in some configurations (§1).

The record format: separating source from owner

A conversation is written as (§3.1):

In words: the dialogue $D$ is a sequence of $T$ messages, and each message $u_t$ is a 4-tuple of text $x_t$, speaker $s_t$, time $\tau_t$, and channel $c_t$.

From this stream the Writer produces derived records of the form:

In words: store the content $x$ with who provided it (src) kept apart from whom it's about (own), and attach a Person-or-Group scope, the event, the time, the state, and a pointer back to the source message. For a self-report, src and own are the same person. But "Alice believes Bob has agreed" gives src = Alice and own = Bob (§3.1). That distinction is the heart of multi-party memory.

Two tracks, five layers

SpeakerMem-R1 keeps memory in two complementary tracks (§3.2, Appendix Table 5):

• System 1 (verbatim track): the only verbatim layer. Every message is stored as-is with text, speaker, time, and channel. No language model is involved; deterministic code just appends.
• System 2 (derived track): four structured layers.
  • Person-scoped Core (stable identity, facts, stances, recurring behavior) and Profile (observations about a person, and how people perceive each other)
  • Group-scoped Interaction (cross-speaker events, relations, decisions) and Insight (group norms, consensus, exceptions)

Every derived record carries from_ids linking it to the verbatim messages that support it. The verbatim track supplies exact wording and local context; the derived track supplies per-person and per-group state.

Updates are handled carefully too (§3.1, Appendix B.2). For each segment the Writer returns Add, Update, or Noop. An Update cites an existing entry_id, appends a new node, and links old to new via links / superseded_by. Nothing is overwritten, so you can later ask for either the current state (head) or the whole history (full).

You can feel what global top-k retrieval drops in the widget below. Drag the query around and you can get the long vectors — the talkative members — to crowd the top slots while the quiet member's record gets pushed out.