Structured Output and Constrained Decoding — How to Stop an LLM from Breaking Your JSON

Instead of asking a model nicely to return JSON, you can drive the probability of every grammatically illegal token to exactly zero before it is ever sampled. A from-scratch walkthrough of constrained decoding — logit masks, vocabulary indexing, what function calling really does, and why syntax guarantees are not content guarantees.

Efficient Guided Generation for Large Language Models

Why "please reply in JSON" keeps failing

You can write "RESPOND ONLY WITH VALID JSON" three times in capitals and still get garbage back a few times in every few hundred calls. A missing closing brace. A trailing comma. The whole object politely wrapped in a Markdown code fence. A cheerful "Sure, here you go!" bolted onto the front.

The reason is that generation runs left to right, one token at a time, with no way back. A human writes the object and then counts the braces. A model has no such step. The discipline of "close the brace I opened five lines ago" is a statistical habit picked up from seeing enormous amounts of well-formed JSON during training — not a mechanical guarantee.

And "usually right" is the worst possible property in production. Three records out of a thousand fail to parse, and those three cost you a retry path, an alert, and someone's evening.

The metaphor: crossing a maze without lifting the pen

Think of generation as crossing a maze in one continuous stroke, pen never leaving the paper. You cannot go back over a line you have drawn. Walk into a dead end and you are simply stuck.

There are only two ways out of this, in principle. One is to check the drawing afterwards and start over if it is wrong — validate and retry. It is trivial to implement, but success is a coin flip and every failure throws away the whole attempt.

The other is to wall off the corridors that lead to dead ends before the pen can enter them. That is constrained decoding, also called guided generation. With the walls in place, the routes that fail to cross the maze are not merely unlikely — they are unreachable.

The intuition: delete the illegal moves before they are played

At every step the model assigns a score to every token in its vocabulary — often on the order of a hundred thousand of them. Those raw scores are the logits. Normally you push them through a softmax to get probabilities and draw one token from the result.

Constrained decoding fits in a single sentence. Before the softmax, push the score of every token the grammar currently forbids down to negative infinity. That is the whole idea. A token whose probability is exactly zero will not be picked at any temperature, under any random seed.

This changes the kind of problem you have. It is no longer "how do I lower the rate of broken JSON," it is "broken JSON cannot be emitted." In operation, that distinction turns out to matter a great deal.

The mechanism: the logit mask

What the grammar produces, at each step, is a mask — an array the same length as the vocabulary, filled with ones and zeros.

Here $z_v$ is the model's raw score for token $v$, $m_v$ is the flag that says "the grammar allows $v$ right now" (1) or "it does not" (0), and $\tilde{z}_v$ is the corrected score. Since $\log 1 = 0$, an allowed token keeps its score untouched; since $\log 0 = -\infty$, a forbidden token falls to negative infinity. One addition switches each token between "unchanged" and "annihilated."

Whatever is removed gets redistributed

The corrected scores then go through an ordinary softmax.

$$p_v = \frac{\exp(\tilde{z}_v / T)}{\sum_u \exp(\tilde{z}_u / T)}$$

$T$ is the temperature (the knob that flattens the distribution as it rises) and $p_v$ is the probability of picking $v$. Because $e^{-\infty} = 0$, every forbidden token lands at exactly zero probability, and the mass they vacated is automatically redistributed among the survivors. A candidate the model gave 2% to will take the full 100% if everything else is banned.

This trick — adding $-\infty$ before the softmax — is precisely the causal mask from the Transformer (Attention from Scratch covers it as the device that stops a token from peeking at the future). Different job, identical tool.

One implementation note. If you literally assign -inf and a bug or a malformed grammar ever masks every token, the softmax denominator becomes zero and your probabilities quietly turn into nan. That is why many implementations use a large finite negative number such as -1e9 instead — and why you should explicitly detect "the allowed set is empty" and raise, rather than letting it slide.

Grammars are measured in characters, models in tokens

Here is where the real engineering starts. A JSON grammar is written over characters — {, ", , — but the model emits tokens, and the BPE tokens produced by a tokenizer do not respect character boundaries.

Real vocabularies contain {" as a single token, and chunks like ": or ", as single tokens too. So the grammar's statement "the next character must be a quote" has to be translated into "here is the set of tokens that are legal next."

The naive way is to test all hundred thousand vocabulary entries at every step, asking the parser whether appending each one would violate the grammar. A hundred thousand parse attempts per emitted token. That turns constrained generation into the slowest part of a serving stack whose entire selling point is speed.

UTF-8 makes it worse. A single non-ASCII character spans several bytes, and a token boundary can land in the middle of that byte sequence. An automaton stepping over characters will see a truncated byte and choke on it. This is exactly why serious implementations advance their state byte by byte rather than character by character.

Index the vocabulary up front

The paper that showed how to remove the brute-force scan is Outlines' Efficient Guided Generation for Large Language Models (arXiv:2307.09702). The idea is straightforward: move the work out of the generation loop.

1. Compile the regular expression or schema into a finite-state machine (FSM)
2. For every token in the vocabulary, work out ahead of time which FSM states can consume it
3. Store the result as an index from "state" to "set of allowed tokens"

At generation time you do a lookup. The scan proportional to vocabulary size disappears from the hot loop, and building a mask becomes a table read.

The cost has not vanished — it has moved to compile time. A new schema pays for its index once. Which makes caching the schema and its index mandatory in practice: if you assemble a schema dynamically and ship it with every request, that compilation reappears as a per-request tax.

JSON is not a regular language

An FSM alone is not always enough. JSON nests without bound, and whether you may emit } right now depends on how many { you have opened so far. Counting open braces is not something a finite set of states can do. JSON is not regular; it is context-free, and handling it exactly requires an automaton with a stack — a pushdown automaton.

XGrammar (arXiv:2411.15100) is a structured-generation engine that tackles this head on, maintaining a persistent stack while splitting the vocabulary into tokens that can be decided independently of context and tokens that have to be checked against the current stack.

The practical rule of thumb is easy to remember. If your schema pins the nesting depth — something like {"name": string, "tags": string[]} — the states collapse into a finite set and an FSM is sufficient. The moment you introduce a recursive $ref (a tree, a comment thread, anything self-referential), you need a genuine context-free grammar.

Writing it out

The skeleton of a constrained generation loop is this short:

def constrained_generate(model, grammar, max_tokens=256):
    state, out = grammar.start(), []
    for _ in range(max_tokens):
        logits  = model.next_logits(out)          # (vocab,) raw scores
        allowed = grammar.allowed_tokens(state)   # table lookup into the index

        mask = np.full_like(logits, -np.inf)
        mask[allowed] = 0.0
        tok = sample(logits + mask)               # softmax lives inside sample

        if tok == EOS:
            if not grammar.is_accepting(state):
                continue                          # can't stop while braces are open
            return out
        state = grammar.step(state, tok)
        out.append(tok)
    raise Truncated(out)                          # <- the ONLY path to broken JSON

The last few lines are the ones that matter operationally. EOS is subject to the grammar too: while a brace is still open, the end-of-sequence token is masked out and the model is not permitted to stop. Conversely, when the loop hits max_tokens, a half-finished string comes back as-is.

Constrained decoding guarantees that the output is correct if it finishes — not that it finishes. That distinction is the number one source of incidents below.

What function calling actually does

Tool calling is a thin layer sitting on top of this machinery.

1. The tool's schema (input_schema / parameters) is rendered into the prompt in a form the model can read
2. The model has been fine-tuned to emit special tokens or a specific format signalling "I am calling a tool"
3. While the arguments are being generated, a grammar compiled from that schema masks the logits
4. The resulting string is parsed and re-validated against the schema before it reaches your application

Strict mode, in whatever form a given provider exposes it, is essentially a switch for step 3. With it off, you are relying on step 2 alone: usually correct, never guaranteed.

When a strict mode imposes conditions like "every property must be in required" or "additionalProperties must be false," that is not pedantry — it is the grammar compiler protecting itself. Optional keys create an "either present or absent" branch per key, and the number of combinations explodes the grammar and its index.

Syntax is guaranteed; content is not

This is the misconception that causes the most damage. The mask guarantees exactly one thing: the output parses. {"age": 999999} and {"city": "Atlantis"} are both flawless JSON.

Anything with a finite set of choices — an enum — can be enforced, because the choices themselves become the grammar. But constraints that a regular grammar cannot express, such as numeric bounds (minimum / maximum) or string length, are silently ignored by some implementations. Adding constrained decoding is not a reason to remove the validator that runs after you receive the response.

There is a subtler statistical problem underneath. What you actually want is the model's distribution conditioned on the set $L$ of strings the grammar accepts:

$x$ is an output string, $L$ is the set of all strings the grammar accepts, and $p$ is the model's probability. In words: take the probabilities the model wanted to assign, throw away the invalid outputs, and rescale the rest while preserving their ratios.

What step-by-step masking actually produces is a product of locally renormalized terms:

$$q(x) = \prod_t \frac{p(x_t \mid x_{<t})}{\sum_{v \in A(x_{<t})} p(v \mid x_{<t})}$$

$A(x_{<t})$ is the set of tokens the grammar allows given everything generated so far, and $x_t$ is the token at step $t$. In words: at each step, rescale within the surviving candidates only, then multiply those steps together.

Because that denominator differs at every step and along every path, $q$ and $p(\cdot \mid x \in L)$ do not coincide in general. Making them agree would mean weighting each candidate by how much valid completion mass lies beyond it — enumerating all possible futures. In practice everyone accepts $q$, but it is worth knowing that a constraint does not merely narrow the model's thinking; it distorts the distribution.

The distortion becomes visible when the schema fights the way the model wants to write. The single most effective countermeasure is field order. Most implementations generate properties in the order the schema declares them, and because generation is left to right, later fields are decided while conditioning on earlier ones. {"answer": ..., "reason": ...} forces the verdict first and then rationalizes it; {"reason": ..., "answer": ...} lets the model think before committing. Put the thinking field first and the same model with the same prompt gets measurably more of them right.

How this shows up on the job

Who touches it, and when. Application and backend engineers wiring an LLM into a pipeline; data engineers running extraction batches; anyone maintaining an agent platform. The test is simple: is the output read by a human or by a machine? If it goes into a database, an API call, or a UI component, constrain it. If a person is going to read the prose, don't bother.

What you actually configure.

• Hosted APIs: the strict response_format mode that takes a JSON Schema; the input_schema / parameters of a tool definition
• Self-hosted: vLLM's structured-output options (with selectable backends such as outlines or xgrammar); llama.cpp's GBNF grammars (--grammar-file, with json_schema_to_grammar.py for conversion)
• Libraries: Outlines, Guidance, lm-format-enforcer
• Always check: max_tokens, and whether the schema is being cached

The failure modes, roughly in order of how often people hit them.

1. Truncation at max_tokens. Constrained or not, stopping halfway produces broken JSON. Estimate a worst-case length from the schema, leave headroom, and make truncation raise a distinct error. If every parse failure is filed under "the model is dumb," this cause never gets found
2. Runaway whitespace. JSON grammars permit unlimited whitespace between elements, so a model can fall into a loop emitting spaces and newlines forever. The reliable fix is to restrict whitespace to zero or one character in the grammar itself
3. Schema compilation cost. Dynamically assembled schemas passed fresh on every request pay first-compile cost every time. Make schemas constant and cache them
4. Silently dropped keywords. minimum, pattern, oneOf, and recursive $ref are ignored or rejected depending on the implementation. Keep validating after you receive the response
5. Field order. A schema that leads with the conclusion denies the model any room to reason
6. Retries are still needed. Constrained decoding guarantees syntax; validation and retry guarantee meaning. They cover different failures, so neither replaces the other

The question that comes up in reviews. "If we want JSON, why not just clean it up with a regex afterwards?" Because post-processing repairs something that has already been emitted. There are unboundedly many ways to be broken, and nothing guarantees that your repair matches what the model meant. A decode-time mask makes the broken output have probability zero in the first place. Repair-after versus forbid-before are not the same strength of guarantee. The throughput and concurrency consequences belong to the LLM serving side of the story.

Takeaways

• Constrained decoding is, mechanically, adding $-\infty$ to logits before the softmax so forbidden tokens land at exactly zero probability
• The hard part is the mismatch between grammars (characters) and models (tokens); index the vocabulary in advance so the hot loop becomes a table lookup
• JSON nests, so it is not a regular language — you need a stack, unless the schema collapses the depth to something finite
• Only the syntax is guaranteed: truncation still breaks it, values are never checked, and the resulting distribution is not the true conditional one