Paper Walkthrough: MerchantBench — Can an LLM Agent Run an Online Store for a Year? Why It Earns Only 27.3% of What Humans Do

A close read of MerchantBench, which measures the long-term coherence of LLM agents across a 365-day simulation of running an online store. Why the best configuration reaches only 27.3% of the average human's final net worth, read through three failure patterns: decaying activity, premature withdrawal, and policies frozen against the evidence.

MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations

Handing an online store to an AI for a full year

Mainstream agent evaluation today revolves around tasks that finish in minutes or hours and can be graded on the spot. Most real work does not look like that. Running a store has no "clear" screen: today's decision constrains next month's options, and the invoice for a mistake arrives long after you have forgotten making it.

MerchantBench is a benchmark that measures LLM agents on exactly this kind of never-ending work. In a simulated marketplace grounded in 98,843 real e-commerce products, the agent is handed sourcing, listing, pricing, and cash flow, and must keep the store running for 365 simulated days. Eight LLMs across two frameworks produced 48 full-year runs, and the outcome was unambiguous: the best-earning configuration finished with 27.3% of the human participants' average final net worth (§Abstract).

What "long-term coherence" means

The paper defines long-term coherence as the ability to hold behavior aligned with a goal over a long horizon while continuously revising judgments as evidence accumulates (§Abstract). Washing dishes is done when the dishes are clean; running a store is a job where continuing is the work itself, and assets start melting the moment you stop.

An environment that measures this needs three properties (§Abstract). First, actions constrain the future — today's purchase order drains cash. Second, feedback arrives with irregular delays: a sale registers immediately, but returns and bad reviews land days later. Third, losing coherence produces cumulative damage that shows up in the numbers, as fines and rating decay pile on. Against predecessors such as Vending-Bench (running a vending machine) and RetailBench (a fixed 96-item catalog), MerchantBench adds delayed feedback generated by the lifecycle of each individual order and a huge, seasonally varying catalog drawn from real data. Sourcing stops being a one-time choice and becomes year-round portfolio management (§Introduction).

Inside the environment: 98,843 real products, 365 days, 26 tools

The environment is grounded in real data from 1688, one of China's largest wholesale marketplaces. Each of its 98,843 products — across 10 categories from 36,576 suppliers — carries 365 days of real demand history from June 2025 through May 2026, preserving the sales peaks of "618" and "11.11" as well as the Spring Festival trough (§Real-World Data Grounding).

The business model is dropshipping, so no inventory is held. Every incoming order automatically triggers a purchase that drains cash; after shipping and delivery, payment arrives later (§Order Lifecycle). Trouble comes from two directions. Upstream, suppliers stochastically change prices, delist products, and ship late (§Upstream Supplier Simulation). Downstream, each order carries six possible anomalies: cancellation, out-of-stock, late shipment, two kinds of refund, and a bad review. Most of them trigger fines (8 yuan for a return refund, 5 yuan for a bad review or an out-of-stock, 3 yuan for a late shipment) plus a drop in store rating — and a lower rating reduces demand across every product (§Order Level Simulation, Appendix J).

The agent starts with 2,000 yuan in cash, a 1,000-yuan deposit, and 50 listing slots, and acts in decision windows every 12 hours by calling 26 tools (market reports, catalog search, listing and pricing, cash and order lookups, memory reads and writes, and more). If the deposit runs out, the store closes (§Agent Interface, §Task Formulation).

The crux here is how hard to react to feedback. Overreact to one piece of bad news and reprice the whole store, and your policy diverges; fail to react at all, and losses accumulate — precisely the shape of the learning rate in gradient descent.

One objective function: net worth after a year

The environment is a partially observable Markov decision process (POMDP), and the clock advances in one-hour ticks for 8,760 steps. Intermediate rewards are zero; only the final assets are scored (§Task Formulation).

The final score is cash on hand ($B_T$) plus the remaining deposit ($D_T$) plus capital in transit ($I_T$) plus receivables not yet settled ($Q_T$). A year of decisions collapses into this single number.

Put in words, the formula asks one question: when the year ends and you shut the doors, add up what is in the till, the deposit still parked with the platform, the money currently tied up in goods on their way to customers, and the money buyers owe you but have not paid yet — what is the total? How many profitable months you had along the way, or how much revenue passed through, earns nothing.

The demand side is driven by a formula as well. The hourly order arrival rate for product $i$ is (§Order Level Simulation):

Read aloud, it is a product of that day's real-data demand ($D$) × a time-of-day weight ($w$) × a multiplier from the store's rating ($r$) × exposure based on how long the item has been listed ($\ell$) × a term that shrinks demand as price rises (where $\epsilon_i$ is price elasticity). Cut prices and you sell more but earn less per unit; let bad reviews drag the rating down and the entire store sinks with it.

Spelled out in words, the formula starts from how badly people want this product today and then scales it by four questions in turn: is this an hour when people buy, is this store trusted, is the listing still fresh enough to get seen, and is the price above or below the going rate? That gives the number of orders arriving per hour. Because every factor multiplies, any single one collapsing toward zero flattens the whole thing — no amount of strength in the other four brings the orders back.

The experiment: 8 models × 2 frameworks × 48 year-long runs

The models under test are GPT-5.6 Sol, Claude Opus 4.8, Qwen3.7-Max and Plus, GLM-5.2, DeepSeek-V4-Pro and Flash, and Kimi K2.6. Each runs on two scaffolds — ReAct, a minimal setup with nothing but the 26 tools, and Hermes, which ships with code execution, planning, memory, and skill management — for three 365-day runs apiece. The reference points are a rule-based baseline that mechanically rotates products on a daily check, and three human participants with no e-commerce experience (§Experimental Setup). A year of history does not fit in context, so under ReAct the agent is prompted at 160k tokens to write anything important into memory, after which the context is truncated to the most recent 30k (Appendix J).

Results: 27.3% of human performance, at best

Average final net worth was 217,600 yuan for humans; the best of all 16 configurations was Qwen3.7-Max on Hermes at 59,500 yuan — the 27.3% from the opening. On the ReAct side the leader was GPT-5.6 Sol (40,900 yuan), which also tops the average across the two frameworks (§Main Results, Table 1). Humans operated at a different order of magnitude in volume too: 608,000 yuan of GMV across 9,442 orders.

Hermes adds +53.3% net worth and +71.5% GMV over ReAct averaged across the eight models, but the improvement holds for 7 of 8 (ranging from +11.5% to +187.8%); Kimi K2.6 alone moves the other way at -4.1% — the benefit of scaffolding is model-dependent (§Framework Analysis). Stability splits too. The top scorer, Qwen3.7-Max on Hermes, is erratic at a 55.1% coefficient of variation across its three runs, while GPT-5.6 Sol on ReAct sits at 3.3% (§Performance Variability). "Earns the most" and "earns the same amount every time" are not the same configuration.

Anatomy of failure I: the agent gradually stops showing up

To explain the gap against humans, the paper identifies two distinct ways coherence is lost, traced from the run logs (§Long-Term Coherence Analysis).

The first is loss of operational coherence — the agent simply stops coming to work. The metric is SWR (Sustained Window Rate): over any 30 consecutive days, the minimum share of decision windows in which at least one environment tool was called. Humans scored 100%, never leaving the store unattended for a single window all year. LLMs ranged from 10.6% to 99.4% under ReAct and 17.8% to 66.1% under Hermes, and Qwen3.7-Max's rate of effective windows decayed by quarter from 68% to 23% under ReAct (§Operational Coherence, Table 1).

Anatomy of failure II: drifting away from the goal

The second is loss of strategic coherence, which the paper splits into goal alignment and evidence-driven adaptation (§Strategic Coherence).

Goal alignment breaks down in stages. First comes a shrinking control loop: proactive sourcing, pricing, and diagnosis disappear, leaving only reactive handling of supplier anomalies. In Qwen3.7-Max on ReAct (SWR 11.1%), supply-chain checks swelled from 14% to 34% of all tool calls. The extreme form is premature withdrawal. One Kimi K2.6 run concluded on day 104 that the store could not be saved and, with moves still available, touched the environment in none of 355 out of its remaining 523 windows.

The contrast in evidence-driven adaptation is just as sharp. Humans widened their sourcing price range as cash accumulated (from an average of 43.4–53.1 yuan in the first three months to 58.7–90.8 yuan in the last three), and fell back to fast-turning cheap items when experiments with pricier goods did not pay off. GLM-5.2 and Kimi K2.6, among others, kept an almost flat price range. The memory logs show something further: a local mistake hardening into permanent policy. One Claude Opus 4.8 run believed the false inference that "pruning weak listings concentrates traffic on the rest" and kept cutting, from 47 listings on day 54 down to 3 by day 322. One Qwen3.7-Max run misremembered on day 282 that "the end date is day 285" and stopped refilling empty slots with 83 days still to go (§Strategic Coherence).

Reading too much meaning into too few observations and freezing a policy around it is the same shape as overfitting in machine learning.

What Hermes actually added

GPT-5.6 Sol made the heaviest use of code execution: in its very first window it applied the pricing rule $p=\max(1.70c,\ c+6)$ (where $c$ is the wholesale cost) to 50 candidate products, checked the margins with a verification script, and listed them in bulk. Claude Opus 4.8 barely used code at all, instead making 261 memory calls across three runs to keep dated operating hypotheses labeled as verified or rejected. Skill distillation auto-generated store-operation skills in 17 of 24 runs, and 17 of the 18 skills produced were reused. The paper's assessment is that the circuit converting experience into procedural knowledge does turn — but the quality of what comes out is uneven (Appendix I).

Limitations, and how to read this

First, this is a simulation. Demand is driven by the formula above, and the dynamics of a real market — rival stores, advertising, shifts in the search algorithm — are absent. Second, the samples are small: three runs per configuration and three humans, and with one configuration showing a 55% coefficient of variation, the fine detail of the ranking can move. Third, the data comes from a single platform, 1688, and generalization to other markets is untested. One more point, offered as my own note rather than a claim of the paper's: the humans worked through a dashboard across 5 calendar days to cover the 365 simulated ones (Appendix J), so their conditions were not strictly identical to the LLMs'.

How this is used in the field

This lands hardest for developers building autonomous agents that keep an operation running — inventory, ad ops, monitoring bots — and for engineers choosing a foundation model for that job.

• Monitor sustainment, not just accuracy. Put the equivalent of SWR on your dashboard: the minimum effective-activity rate over any trailing 30-day window. Total tool-call count will not tell you.
• Context compression is a design decision. "Summarize at 160k tokens, truncate to 30k, summarizer = the main model" is a workable starting point in production. Write dated hypotheses labeled verified or rejected into persistent memory.
• Three traps. (1) Do not leave a branch where the agent decides on its own that the situation is hopeless and halts — put human approval in front of any withdrawal decision. (2) Fetch deadlines and expiry dates from the environment every time, never from memory (the lesson of the day-285 incident). (3) Rate-limit sweeping reactions to a single bad signal (repricing 40 products over one bad review).
• The question to ask in design review. "What makes an agent fall apart in long-running operation?" This paper's answer is three things: decaying activity, drift from the goal (a shrinking loop and early withdrawal), and policies frozen against the evidence.

Read next

• LLM Agents from First Principles — the observe–think–act loop and what a framework, or "scaffold," really is; makes the ReAct vs. Hermes gap click
• Paper Walkthrough: SWE-Bench ProMax — a benchmark that measures agents on one long, heavy task instead; worth comparing against year-long operation as a measurement strategy
• Overfitting and Evaluation — the "freeze a policy from too little evidence" failure, from the machine-learning fundamentals side