Paper Walkthrough: MerchantBench — Can an LLM Agent Run an Online Store for a Year? Why It Earns Only 27.3% of What Humans Do

A close read of MerchantBench, which measures the long-term coherence of LLM agents across a 365-day simulation of running an online store. Why the best configuration reaches only 27.3% of the average human's final net worth, read through three failure patterns: decaying activity, premature withdrawal, and policies frozen against the evidence.

MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations

Handing an online store to an AI for a full year

Mainstream agent evaluation today revolves around tasks that finish in minutes or hours and can be graded on the spot. Most real work does not look like that. Running a store has no "clear" screen: today's decision constrains next month's options, and the invoice for a mistake arrives long after you have forgotten making it.

MerchantBench is a benchmark that measures LLM agents on exactly this kind of never-ending work. In a simulated marketplace grounded in 98,843 real e-commerce products, the agent is handed sourcing, listing, pricing, and cash flow, and must keep the store running for 365 simulated days. Eight LLMs across two frameworks produced 48 full-year runs, and the outcome was unambiguous: the best-earning configuration finished with 27.3% of the human participants' average final net worth (§Abstract).

What "long-term coherence" means

The paper defines long-term coherence as the ability to hold behavior aligned with a goal over a long horizon while continuously revising judgments as evidence accumulates (§Abstract). Washing dishes is done when the dishes are clean; running a store is a job where continuing is the work itself, and assets start melting the moment you stop.

An environment that measures this needs three properties (§Abstract). First, actions constrain the future — today's purchase order drains cash. Second, feedback arrives with irregular delays: a sale registers immediately, but returns and bad reviews land days later. Third, losing coherence produces cumulative damage that shows up in the numbers, as fines and rating decay pile on. Against predecessors such as Vending-Bench (running a vending machine) and RetailBench (a fixed 96-item catalog), MerchantBench adds delayed feedback generated by the lifecycle of each individual order and a huge, seasonally varying catalog drawn from real data. Sourcing stops being a one-time choice and becomes year-round portfolio management (§Introduction).

Inside the environment: 98,843 real products, 365 days, 26 tools

The environment is grounded in real data from 1688, one of China's largest wholesale marketplaces. Each of its 98,843 products — across 10 categories from 36,576 suppliers — carries 365 days of real demand history from June 2025 through May 2026, preserving the sales peaks of "618" and "11.11" as well as the Spring Festival trough (§Real-World Data Grounding).

The business model is dropshipping, so no inventory is held. Every incoming order automatically triggers a purchase that drains cash; after shipping and delivery, payment arrives later (§Order Lifecycle). Trouble comes from two directions. Upstream, suppliers stochastically change prices, delist products, and ship late (§Upstream Supplier Simulation). Downstream, each order carries six possible anomalies: cancellation, out-of-stock, late shipment, two kinds of refund, and a bad review. Most of them trigger fines (8 yuan for a return refund, 5 yuan for a bad review or an out-of-stock, 3 yuan for a late shipment) plus a drop in store rating — and a lower rating reduces demand across every product (§Order Level Simulation, Appendix J).

The agent starts with 2,000 yuan in cash, a 1,000-yuan deposit, and 50 listing slots, and acts in decision windows every 12 hours by calling 26 tools (market reports, catalog search, listing and pricing, cash and order lookups, memory reads and writes, and more). If the deposit runs out, the store closes (§Agent Interface, §Task Formulation).

The crux here is how hard to react to feedback. Overreact to one piece of bad news and reprice the whole store, and your policy diverges; fail to react at all, and losses accumulate — precisely the shape of the learning rate in gradient descent.

One objective function: net worth after a year

The environment is a partially observable Markov decision process (POMDP), and the clock advances in one-hour ticks for 8,760 steps. Intermediate rewards are zero; only the final assets are scored (§Task Formulation).

The final score is cash on hand ($B_T$) plus the remaining deposit ($D_T$) plus capital in transit ($I_T$) plus receivables not yet settled ($Q_T$). A year of decisions collapses into this single number.