merchantbench

Definition

A named benchmark for evaluating models or agents on commerce tasks. The word merchant points at the seller's side of a marketplace rather than the shopper's, so the task suite targets operational work a vendor does. Unlike general reasoning benchmarks, domain benchmarks like this score correctness against business rules and costs, so the exact task list and scoring rubric have to be read from the paper that defines it.

Explainers using this term

• Paper Walkthrough: MerchantBench — Can an LLM Agent Run an Online Store for a Year? Why It Earns Only 27.3% of What Humans DoMerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations