Paper Walkthrough: From Production Traffic to Post-Training — Folding 200 Internal Apps Into One Self-Hosted LLM

How a company forced to self-host by data-residency rules collapsed a sprawling model fleet into one checkpoint: classify production failures by hand, train one GRPO expert per weak axis, then merge the weights with SLERP. Includes three real reward-hacking failures.

From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix

A growing model zoo quietly eats your GPUs

The paper behind this article is "From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix" (arXiv:2609.01572, T-Tech, 1 September 2026).

Here is the abstract restated. Data-residency constraints force enterprises to self-host LLMs, but because new models keep getting adopted while old ones are never decommissioned, the serving fleet expands and fragments a finite GPU pool. The authors consolidate traffic from over 200 internal applications onto a single model by closing the quality gaps found through production error analysis along three axes: instruction following, function calling, and the internal task distribution. Quality is tracked with offline benchmarks stratified to production traffic and scored either by deterministic verifiers or by calibrated LLM judges. Because optimising all objectives jointly introduces cross-domain reward interference, they train a separate GRPO expert per axis and merge them with two-stage SLERP. Each expert's reward exposes a distinct failure mode — semantic collapse, over-calling, and verbosity hacking — each needing its own fix. In non-reasoning mode the recipe beats a baseline roughly 7× larger by total parameters on the in-house Arena, 69.6 to 65.8; on instruction following, 0.85 to 0.83; and on function calling, 0.79 to 0.77, while also lifting general dialogue benchmarks. The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.

Think of a canteen that keeps adding serving lanes. If the only reason the noodle lane exists is that the curry lane could not serve noodles, then teaching the curry lane to serve noodles collapses the two back into one. Find, from production logs, what was blocking migration; fix exactly that with post-training. This is an operations report on an internal LLM platform, not a new architecture.

Counting what is actually missing, by hand

The paper does not start from a hunch. It samples $n{=}2{,}500$ responses from platform traffic and has three annotators assign each one its single primary failure among six categories, taking the majority vote (inter-annotator agreement Cohen's $\kappa=0.62$, §A.1).

Classification is the biggest single bucket, but formatting plus non-format instruction violations together come to 37.9%, which beats it — and those violations can be checked mechanically by deterministic verifiers. Largest and automatically scorable, so it earns a dedicated expert.

What they decline to do is argued just as explicitly. Classification failures are a heterogeneous downstream symptom rather than one capability, and in non-reasoning mode they rarely expose a clean rewardable signal. Knowledge-base failures track the backbone's parametric capacity, which a constraint-style reward cannot move. Function calling is handled separately: tool-equipped requests are about 12% of traffic, and human review traces the damage to Russian-language tool descriptions (out of distribution for English-centric models) and to failure to fill arguments in Russian, rather than picking the wrong function (§A.1).

Building a benchmark that still looks like production

Internal traffic is dominated by templated requests. The paper frames the tension with two measures (§3): diversity, the mean pairwise TF-IDF cosine distance within the selected set, and representativeness, the Jensen-Shannon distance from the production pool along four dimensions (queried model, prompt length, service, task taxonomy). Naive methods fail in opposite directions. Uniform random sampling preserves the distribution but inherits near-duplicate structure (diversity 0.653). Greedy max-min chases diversity and wrecks representativeness (JS distance on the service dimension degrades to 0.683).

Their template-aware sampler works like this. A markdown-aware parser splits each prompt into structural segments; variable tokens — numbers, opaque identifiers, long string literals — are masked to produce a normalised sequence. Near-identical sequences are grouped into templates by locality-sensitive hashing, greedy max-min then runs within each template over the variable spans only, and the per-template budget scales as $\sqrt{\text{count}}$ — or, in words, a family with 100× the requests gets only 10× the budget, a brake that stops frequent templates from either dominating or vanishing. The result is diversity 0.953, the highest of any method tested, with far lower JS distances than pure diversity sampling (Table 1).

One judging recipe does not fit all tasks

Scoring the benchmark with Arena-Hard-Auto and DeepSeek-V3-0324 as judge, a uniform side-by-side (SBS) judge agreed with expert annotators only at $\kappa=0.62$ (§3). On objective tasks like classification and information extraction, judge and human should be comparing against a shared reference answer, but SBS only ever asks which response is better. On open-ended tasks like summarisation and content generation, "good" is underspecified, so the two raters weigh different criteria.

So every request is routed through an LLM task classifier — matching human consensus in 90.6 to 99.6% of cases — and each segment gets its own scheme. Classification and information extraction, roughly 63.2% of traffic, are scored reference-based against gold answers generated by Kimi-K2.5 and verified by annotators (97.7% and 85.2% accepted unmodified). Open-ended tasks keep pairwise SBS but add RubricHub-style per-example checklists. The interesting part is that the best recipe differed by task (Table 12).

Same judge model; only the question changed. Overall, $\kappa$ rose from 0.63 to 0.88 on the reference-based slice and from 0.57 to 0.72 on open-ended content generation (Table 2). The lesson to carry away: when you adopt an LLM judge, measure its $\kappa$ against humans first (see LLM-as-a-Judge).