Paper Review: MetroLLM-Bench — What Happens When an LLM Runs the Ticket Machine. Why a 2.6 GB 4B Model Tied GPT-5.6, and Why Fine-Tuning Hurt at 27B

A 955-case benchmark that asks how well a language model can serve as the decision layer of a transit ticket kiosk. We read, from the paper alone, how a 4B QLoRA student beat GPT-5.6 on the deterministic tier and why PEFT gains shrink with model size until they turn negative at 27B.

MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes

Replacing the kiosk's code with prose

A transit ticket kiosk carries its fare tables, route topology, and disruption responses as a programmed state machine. Every time a station closes, the change has to pass through a ticket, a patch, regression tests, and a release window.

The paper "MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes" (Remco Hendriks, arXiv:2609.10016, published September 2026) measures an alternative: replace that decision logic with a language model that reads a natural-language system description (the paper calls it a framebook), calls structured tools, and emits a terminal state the kiosk hardware can render directly.

In summary, the abstract says this. MetroLLM-Bench is a 955-case benchmark built on six real metro systems (37 to 414 stations) across eleven categories including routing, fares, disruptions, accessibility, and adversarial input. In each case the model must call structured tools and submit a machine-renderable terminal state containing an outcome, a per-ticket fare quote where applicable, and a kiosk action. Scoring has two tiers: fourteen deterministic components (Tier 1) and eight semantic-quality components (Tier 2, six of them judged by a language model). A stratified 75/25 split reserves 717 cases for training-data generation and 238 for held-out evaluation. Twenty-six models from six vendors were evaluated and twenty-three ranked. A 4B Qwen 3.5 student trained with PEFT exceeds both GPT-5.6 tiers on Tier 1 (91.3 against 90.6 and 90.0) and matches GPT-5.4 full at maximum reasoning effort (91.4), with a 2.6 GB Q4_K_M footprint. The 9B and 27B students do not improve on the 4B, and the PEFT gain falls monotonically from +7.03 points at 2B to −0.91 at 27B. A deterministic rule-based baseline reaches 84.6 on Tier 1; the remaining language-model advantage is concentrated in policy adaptation, compound scenarios, accessibility, and temporal reasoning. Muse Glimmer 30B leads the composite ranking, and serving configuration alone moves the Qwen 3.5-to-3.8 comparison by 2.7 points.

An analogy: a new hire with a binder

Imagine you start today at a station's service window. You have one binder (the framebook) listing this system's terminology, currency, operating hours, and local rules, plus six internal phone lines: routing, fares, station facilities, line information, disruptions, and the rulebook. When a passenger arrives you call whichever desks you need and finally hand in a fixed-format slip saying "show this route, this fare, this banner." If the slip is malformed the terminal rejects it and you redo it.

MetroLLM-Bench measures how well that new hire follows the binder. The paper is explicit that it tests whether a model can follow supplied rules, not whether it remembers a network from pretraining (§2.1).

What is in the benchmark: six cities, eleven categories, 955 cases (§2.1)

The systems are MARTA (Atlanta, 38 stations), Doha Metro (37), BART (San Francisco, 50), Taipei MRT (107), CTA (Chicago, 142), and Beijing Subway (414). They cover three fare models (flat, distance-based, flat with exceptions) and four currencies.

The eleven categories are A Routing, B Fare, C Disruption, D Accessibility, E Cultural, F Policy, G Multi-turn, H Adversarial, I Temporal, J Tool-Hallucination, and K Compound Stress. cases/generator.py combines category templates with each system's station pairs and disruption metadata, and derives route and fare ground truth mechanically from the benchmark's graph and fare engines. An independent annotator validated a 50-case sample of the answer key against the underlying network data (Appendix B.7).

One case: six tools and a submit tool (§2.2)

The model enters a ReAct-style loop with a budget of twenty tool rounds, calling route_planner, fare_calculator, station_info, line_info, disruption_feed, and knowledge_base, and ends the case with submit_assistant_state.

That terminal tool is the kiosk's render contract. Every submission must carry one of five outcomes (route_and_fare_ready, advisory_only, service_unavailable, request_declined, policy_answer_only), a kiosk action with a reason code, and a passenger-facing message; a fare quote with per-ticket line items is required when a fare is ready. A mock server validates the structure with Pydantic and returns HTTP 422 with field-level errors on inconsistency, which the runner feeds back so the model can correct itself within the remaining rounds.

for round in range(20):
    call = model.next_action(prompt, history)        # a tool call or a submit
    if call.name != "submit_assistant_state":
        history.append(tools[call.name](**call.args)) # one of the six tools
        continue
    ok, errors = validate(call.args)                  # Pydantic validation
    if ok: break                                      # terminal state accepted
    history.append({"http": 422, "errors": errors})   # let the model fix it

In the paper's Figure 1 example, a BART passenger travels during a Transbay Tube closure; the model reads the closure from disruption_feed, re-plans with route_planner to a route ending at West Oakland, and submits an advisory_only state pointing to the free bus bridge.

Scoring: a deterministic Tier 1 and a judged Tier 2 (§2.3, §2.4)

• Tier 1 (14 deterministic components): route and fare correctness, tool-call accuracy and no-hallucination, renderable-state validity, outcome and reason-code correctness, disruption detection, advisory issuance, and so on. Computed without a model judge, and reused as the PEFT reward signal.
• Tier 2 (8 semantic-quality components): framebook conformance, advisory content, policy acknowledgement, safety, accessibility, temporal accuracy, no-data-fabrication, scope adherence. Six of them use Claude Haiku 4.5 as judge.

The paper repeats that no headline claim rests on Tier 2 alone; the central comparison uses Tier 1. That separation keeps the quirks of an LLM-as-a-Judge from deciding the conclusion. The judge was calibrated against humans on 100 case-rubric pairs: author–judge agreement is κw = 0.53, higher than the agreement between the author and an independent second annotator (κw = 0.25). Under prevalence-robust Gwet AC2, all three pairings land between 0.81 and 0.89.

How the students were built: QLoRA at rank 16 (§3.1, §4)

The teachers are Qwen 3.5 27B-dense and 35B-A3B MoE, run only on the 717-case training partition. Traces scoring at least 90 percent on Tier 1 are kept and deduplicated, giving 600 examples. No held-out case enters the teacher pool. The students come in four sizes (2B, 4B, 9B, 27B) and use QLoRA: a low-rank adapter on top of a 4-bit-quantised base.

In words: the original weight matrix $W_0$ stays frozen, and only the product of two thin matrices ($B$ and $A$, each 16 wide) is learned and added on. $d$ and $k$ are the dimensions of the original matrix; $r$ is the adapter width. Whether a width of 16 is enough is something you can feel in the figure below by trading error against parameter count.