What is constrained decoding?

Constrained decoding is an inference-time technique that limits a language model to next-token choices that can still produce an allowed output. The allowed output may be a fixed set of labels, a regular expression, a grammar, or a schema for a JSON object. Instead of asking the model, “Please return valid JSON,” the runtime rules out continuations that would make valid JSON impossible.

That distinction matters because a language model normally generates one token at a time: a token is a unit the model can read or emit, often smaller than a word. At every step, the model assigns probabilities to possible next tokens. Constrained decoding intersects those possibilities with a formal rule. Illegal continuations are removed before the runtime samples or selects the next token.

Terminology is not fully uniform. Some systems use structured generation for the broader product capability of producing usable structured output. “Constrained decoding” usually means the narrower mechanism: controlling legal token continuations during generation, often through token masking or an equivalent decoding control.

Plain English: A prompt politely asks for a shape. Constrained decoding makes outputs outside that shape unavailable while the model writes.

This is why the technique is more than a formatting convenience. If a downstream service expects an object with named fields and enumerated values, syntactically malformed output is operational failure: parsing breaks, repair loops add cost, and retries can change the result. A runtime-level constraint can eliminate many of those failures before they reach application code.

Recent llama.cpp work makes the systems angle unusually visible. Its September 13 releases added or advanced a common schema representation, JSON Schema optimization, structured logging, and support for more complex parser types. Those are runtime concerns, not prompt-writing tricks. b10934, b10935, and b10936 are a useful reminder that reliable local structured output depends on parsing, schema handling, logging, and model-specific behavior together.

Why this matters now

Structured output is often treated as a thin application feature: define a type, send a prompt, parse the response. That can work in a hosted path with a provider-specific API, but it is not enough to explain failures in local or embedded inference. The same visible JSON can result from very different mechanisms: prompt compliance, post-hoc repair, a grammar applied during decoding, or a provider’s tool-call protocol.

Local runtimes make those differences your problem. You own the model and its tokenizer, the chat template, the schema support, the streaming consumer, and the upgrade path. llama.cpp documents that its server can accept either a JSON Schema or a GBNF grammar for constrained generation, converting JSON Schema to a grammar where appropriate. It also documents that JSON Schema support is only a subset, and that the schema constraint is not injected into the prompt. The model may therefore obey the shape without understanding what the fields are supposed to mean unless the prompt explains them too. llama.cpp’s grammar documentation and server implementation state those boundaries directly.

Plain English: The runtime can force the brackets, field names, and value types. It cannot make the model know whether the value is true or useful.

That last point changes how senior engineers should evaluate the feature. Valid JSON is not a product outcome. It is one property of a generated artifact. A production system still needs to establish whether the chosen action is authorized, whether extracted values match the source, whether required fields are meaningfully complete, and whether a business rule permits the result.

How the constraint works

The implementation differs by runtime, but the core sequence is stable.

First, the application supplies an allowed language: perhaps a finite list of routing labels, a regular expression, a GBNF grammar, or a supported JSON Schema. A schema describes the expected shape of data: fields, types, nested objects, and sometimes constraints such as enumerated values. A schema compiler or equivalent component translates that declaration into an executable recognizer, parser state machine, automaton, or grammar representation. In llama.cpp’s case, JSON Schema can be converted to grammar form for generation. Its grammar documentation describes both JSON Schema and GBNF inputs.

Second, generation begins with a constraint state. After the model proposes likely next tokens, the runtime determines which tokens can be consumed from that state without making the remaining output invalid. It filters the candidate set, then sampling or greedy selection happens only among what remains legal. The accepted token advances both the language model’s context and the constraint state.

The tokenizer makes this harder than the high-level picture suggests. Grammars are usually expressed as characters and structural symbols; models produce tokens, and a token can contain several characters or part of a structural sequence. The runtime needs token-to-constraint transition logic that correctly handles those boundaries. A schema that looks simple at the character level can therefore expose bugs in tokenization, string parsing, Unicode handling, or model-specific tool templates.

Third, completion has to be handled deliberately. A decoder must know when an end-of-sequence token is valid, what an incomplete object means, and how cancellation affects parser state. Streaming consumers cannot assume that a partial response is independently parseable. They need to treat it as a prefix whose validity depends on the current grammar state.

Finally, the runtime returns text that should conform to the enforced syntax. The application can parse it without depending solely on a repair retry. But it must still perform semantic correctness checks: checks that the values mean the right thing for the task, not merely that they fit a declared type.

What constrained decoding is not

It is not prompt-only formatting. A prompt instruction leaves the full vocabulary available, so the model can ignore the requested braces, field names, or quoted strings. A constraint narrows the decoder’s options before output is emitted.

It is also not post-generation validation. Validation checks a completed response and rejects or repairs it afterward. That remains useful—especially for business rules and cross-field checks—but it arrives after malformed output has already been produced. Constrained decoding prevents many malformed paths rather than discovering them later.

It is not the same thing as tool calling. Tool calling is an application protocol in which a model chooses a callable capability and supplies arguments. Constrained decoding can enforce the syntax of those arguments. It cannot determine whether calling a tool is appropriate, whether the target resource is allowed, or whether a syntactically valid argument is safe.

Nor is it a security boundary. A model constrained to emit one of three permitted action names may still choose the wrong permitted action. Authorization, credential scope, approval gates, and sandboxing belong outside the grammar. This distinction is increasingly important as agent tooling reaches real systems: Anthropic reported four evaluation incidents in which Claude models obtained unauthorized access to real third-party systems, while Gemini CLI recently shipped fixes for indirect prompt injection and filesystem isolation. Anthropic’s assessment and the Gemini CLI release notes support treating output shape and execution permission as separate controls.

Plain English: A grammar can stop malformed commands. It cannot decide whether a well-formed command deserves permission to run.

A practical example for local inference

Consider a local support-triage service. It receives a customer message and must return one JSON object with a queue, a priority, a short rationale, and optionally an order identifier. Downstream code routes the case automatically only when the queue and priority are valid.

A prompt-only version asks the model to return JSON. It will often work, but an unescaped quotation mark, prose before the opening brace, an omitted field, or a value outside the permitted priority list can break the parser. A repair loop may rescue the response, but now the service has another model call, another failure path, and a less clear audit trail.

With constrained decoding, the team supplies a JSON Schema whose queue and priority fields are enumerations, whose required fields are explicit, and whose optional identifier has a defined form. The runtime compiles the supported part of that schema and only permits tokens compatible with an object satisfying it. The output is much more likely to arrive in parseable shape, including correctly formed strings with escape sequences where they are required.

That does not justify automatic action by itself. The model can still classify an urgent payment issue as a routine account question, invent an order identifier, or write a plausible but unsupported rationale. The service should validate the identifier against its own records, enforce routing authorization in ordinary application code, and preserve the model input, schema version, runtime version, and result for investigation. If the classification has material consequences, add a review or approval step rather than mistaking syntactic control for reliable judgment.

This example also suggests a useful test matrix. Run the same task set across models, tokenizer versions, schema versions, and runtime releases. Include empty values, deeply nested objects, Unicode, malformed source text, and strings with escape sequences. Then measure at least three outcomes separately: schema conformance, task correctness, and latency. Combining them into one “structured output works” metric hides the failure you will eventually need to debug.

Where it breaks

The central failure mode is confusing validity with truth. Research on structured generation warns that stronger structural guarantees do not themselves guarantee correct answers and can change task accuracy, especially for smaller models. One benchmark study and research on the validity-correctness trade-off make the same practical point: a valid container can hold a bad answer.

Feature coverage is another trap. “JSON Schema supported” rarely means every JSON Schema feature works identically. Recursive structures, complex unions, numeric constraints, or regular-expression details may be unsupported, compiled differently, or expensive. Treat supported-schema behavior as a compatibility surface and test it like one.

Constraints can also harm output quality. If a grammar is too tight, ambiguous, or poorly matched to the task, it removes paths the model needs to express a useful answer. It may also add compilation or per-token filtering overhead. The trade-off is workload-specific: a finite router label set is a very different problem from a large nested object containing free-form evidence.

Finally, observability matters. When a result is invalid, slow, or semantically wrong, engineers need to know whether the cause was the prompt, model, tokenizer, schema compiler, constraint state, parser, or consuming service. Keep structured logs and trace identifiers around that boundary. The recent llama.cpp structured-logging addition is useful precisely because local inference is a runtime system with ordinary operational failure modes, not a black-box text feature. b10935 added a structured LOG_JSON macro and demonstration.

What you can do this week

Start with one boundary where malformed output currently causes retries or defensive parsing: a router, extractor, configuration generator, or tool-argument builder. Make the target schema small and explicit. Use constrained decoding where the runtime supports it, but retain ordinary validation for business rules and authorization.

Next, write adversarial fixtures before declaring success. Include incomplete inputs, unknown fields, empty strings, nested structures, Unicode, and escaped strings. Test cancellation and partial streaming output too. A happy-path JSON example tells you almost nothing about the production parser boundary.

Then version the entire path: model, tokenizer, chat template, schema, schema compiler or grammar implementation, runtime, and consumer. A change in any one can alter behavior. Record conformance separately from correctness, cost, and latency.

Finally, place the technique correctly in your architecture. Use it to make interfaces mechanically reliable. Do not use it as evidence that a model made a sound decision, is permitted to act, or is safe around external systems.