Structured Output and Constrained Decoding — How to Stop an LLM from Breaking Your JSON

Instead of asking a model nicely to return JSON, you can drive the probability of every grammatically illegal token to exactly zero before it is ever sampled. A from-scratch walkthrough of constrained decoding — logit masks, vocabulary indexing, what function calling really does, and why syntax guarantees are not content guarantees.

Efficient Guided Generation for Large Language Models

Why "please reply in JSON" keeps failing

You can write "RESPOND ONLY WITH VALID JSON" three times in capitals and still get garbage back a few times in every few hundred calls. A missing closing brace. A trailing comma. The whole object politely wrapped in a Markdown code fence. A cheerful "Sure, here you go!" bolted onto the front.

The reason is that generation runs left to right, one token at a time, with no way back. A human writes the object and then counts the braces. A model has no such step. The discipline of "close the brace I opened five lines ago" is a statistical habit picked up from seeing enormous amounts of well-formed JSON during training — not a mechanical guarantee.

And "usually right" is the worst possible property in production. Three records out of a thousand fail to parse, and those three cost you a retry path, an alert, and someone's evening.

The metaphor: crossing a maze without lifting the pen

Think of generation as crossing a maze in one continuous stroke, pen never leaving the paper. You cannot go back over a line you have drawn. Walk into a dead end and you are simply stuck.

There are only two ways out of this, in principle. One is to check the drawing afterwards and start over if it is wrong — validate and retry. It is trivial to implement, but success is a coin flip and every failure throws away the whole attempt.

The other is to wall off the corridors that lead to dead ends before the pen can enter them. That is constrained decoding, also called guided generation. With the walls in place, the routes that fail to cross the maze are not merely unlikely — they are unreachable.

The intuition: delete the illegal moves before they are played

At every step the model assigns a score to every token in its vocabulary — often on the order of a hundred thousand of them. Those raw scores are the logits. Normally you push them through a softmax to get probabilities and draw one token from the result.

Constrained decoding fits in a single sentence. Before the softmax, push the score of every token the grammar currently forbids down to negative infinity. That is the whole idea. A token whose probability is exactly zero will not be picked at any temperature, under any random seed.

This changes the kind of problem you have. It is no longer "how do I lower the rate of broken JSON," it is "broken JSON cannot be emitted." In operation, that distinction turns out to matter a great deal.

The mechanism: the logit mask

What the grammar produces, at each step, is a mask — an array the same length as the vocabulary, filled with ones and zeros.

Here $z_v$ is the model's raw score for token $v$, $m_v$ is the flag that says "the grammar allows $v$ right now" (1) or "it does not" (0), and $\tilde{z}_v$ is the corrected score. Since $\log 1 = 0$, an allowed token keeps its score untouched; since $\log 0 = -\infty$, a forbidden token falls to negative infinity. One addition switches each token between "unchanged" and "annihilated."

Whatever is removed gets redistributed

The corrected scores then go through an ordinary softmax.

$$p_v = \frac{\exp(\tilde{z}_v / T)}{\sum_u \exp(\tilde{z}_u / T)}$$

$T$ is the temperature (the knob that flattens the distribution as it rises) and $p_v$ is the probability of picking $v$. Because $e^{-\infty} = 0$, every forbidden token lands at exactly zero probability, and the mass they vacated is automatically redistributed among the survivors. A candidate the model gave 2% to will take the full 100% if everything else is banned.

This trick — adding $-\infty$ before the softmax — is precisely the causal mask from the Transformer (Attention from Scratch covers it as the device that stops a token from peeking at the future). Different job, identical tool.

One implementation note. If you literally assign -inf and a bug or a malformed grammar ever masks every token, the softmax denominator becomes zero and your probabilities quietly turn into nan. That is why many implementations use a large finite negative number such as -1e9 instead — and why you should explicitly detect "the allowed set is empty" and raise, rather than letting it slide.

Grammars are measured in characters, models in tokens

Here is where the real engineering starts. A JSON grammar is written over characters — {, ", , — but the model emits tokens, and the BPE tokens produced by a tokenizer do not respect character boundaries.

Real vocabularies contain {" as a single token, and chunks like ": or ", as single tokens too. So the grammar's statement "the next character must be a quote" has to be translated into "here is the set of tokens that are legal next."