Why grammar constraints matter for small language models
June 3, 2026
Ask a 1.5B parameter model to return JSON and, often enough, it will return something that looks like JSON — a missing brace, a trailing comma, a field name it invented instead of the one in the schema. At small scale, this isn't a rare failure mode; it's a routine one. And it's exactly the failure mode that breaks agentic systems, where a single malformed tool call can cascade into a broken pipeline.
The standard fix is prompting harder: more examples, more emphatic instructions, maybe a retry loop that re-asks the model when parsing fails. All of that helps at the margins and none of it eliminates the problem, because the model is still free, token by token, to sample something invalid.
Grammar-constrained decoding removes that freedom at the source. Instead of letting the model sample from its full vocabulary at every step, the decoder intersects the model's distribution with a formal grammar (in practice, something like GBNF) that encodes the exact schema the output must satisfy. If a closing brace is the only valid next token, that's the only token the model is allowed to choose — regardless of what probability it assigned to anything else.
The result isn't that the model becomes smarter. It's that the space of possible outputs shrinks to exactly the space of valid outputs. This matters disproportionately for small models, because their remaining errors — is the value semantically right, is the tool call reasonable — become the only errors left to worry about. Structural errors, the kind that used to require retry logic and defensive parsing, simply stop occurring.
There's a real cost: constrained decoding adds latency, and grammar complexity is itself something you have to design and tune, especially on CPU-only hardware where every extra check has a visible cost. But for small models doing agentic or tool-calling work, that tradeoff is usually worth it — you're trading a fixed, predictable overhead for the elimination of an entire, unpredictable failure class.