LLMStructuredOutput
Guaranteed 100% schema-valid JSON from an LLM — same model, same prompt, only the decoding changes.
Companion repo for the YouTube video: Why 90% of LLM JSON Outputs Fail (and the Fix That Guarantees 100%)
A 1.7B model (Qwen3-1.7B, running on an 8 GB consumer GPU) extracts structured
data from messy support tickets:
| Approach |
Valid outputs |
| Naive prompt ("give me JSON") |
1/10 — markdown fences, "N/A" instead of null, string order IDs |
| Careful prompt engineering |
9/10 — better, but a tail risk remains and it degrades on nested schemas |
| Constrained decoding (with the naive prompt!) |
10/10 — guaranteed, not lucky |
Constrained decoding doesn't hope for valid JSON: the server compiles your
schema into a grammar and masks every token that would violate it while
generating. Format is no longer the model's job — the constraint owns the
syntax, the prompt only carries the semantics.
What's inside
Quickstart
# 1. Start the vLLM server (first start: ~8 GB image pull + ~2.2 GB model download)
docker compose up -d
docker compose logs -f vllm # wait for "Application startup complete"
curl http://localhost:8000/v1/models
# 2. Python environment for the notebook
python -m venv .venv && .venv/bin/pip install -r requirements.txt
# (or: uv venv && uv pip install -r requirements.txt)
# 3. Open structured_output_demo.ipynb with the .venv kernel
Tested with vLLM v0.26.0 on an RTX 3070 (8 GB, Windows / Docker Desktop / WSL2).
The API — current state (things changed!)
guided_json is removed since vLLM v0.12 (deprecated in 0.10/0.11). Requests using it are rejected.
client.beta.chat.completions.parse is legacy — parse() moved out of the beta namespace in openai-python 1.92.0.
The current forms:
from openai import OpenAI
from pydantic import BaseModel
from typing import Literal
class Ticket(BaseModel):
customer_name: str
category: Literal["billing", "technical", "account", "other"]
priority: Literal["low", "medium", "high"]
order_id: int | None
summary: str
client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
model = client.models.list().data[0].id
# (A) Recommended: OpenAI-standard parse() with a Pydantic model
completion = client.chat.completions.parse( # NOT client.beta....
model=model,
messages=[{"role": "user", "content": f"Extract the ticket:\n\n{raw_text}"}],
response_format=Ticket,
)
ticket = completion.choices[0].message.parsed # validated Ticket instance
# (B) vLLM-native (shows what's happening under the hood)
completion = client.chat.completions.create(
model=model,
messages=[...],
extra_body={"structured_outputs": {"json": Ticket.model_json_schema()}},
)
# (C) Through LangChain — json_schema is the constrained path
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model=model, base_url="http://localhost:8000/v1", api_key="-")
extractor = llm.with_structured_output(Ticket) # method="json_schema" is the default on >= 1.x
ticket = extractor.invoke(prompt)
Server side: no flag is required — the schema arrives with the request and
--structured-outputs-config.backend defaults to auto. The compose pins it
to xgrammar for reproducibility.
Model choice & measured failure rates
Qwen/Qwen3-1.7B — official, Apache-2.0, ungated, BF16 (~3.4 GB weights,
~5-6 GB total with KV cache). Qwen3 is a hybrid reasoner with thinking ON by
default — the compose disables it server-wide (--reasoning-parser qwen3 +
--default-chat-template-kwargs '{"enable_thinking": false}').
Measured on this exact setup (the result shaped the video's storyline —
prompt quality dominates, not model size):
| Model |
Prompt |
Schema |
Unconstrained failures |
| Qwen3-4B-AWQ |
strict |
simple |
0/10 |
| Qwen2.5-3B-Instruct-AWQ |
strict |
simple |
0/20 |
| Qwen3-1.7B |
strict |
simple |
0/20 |
| Qwen3-1.7B |
strict |
complex/nested |
2/20 (enum cross-contamination) |
| Qwen3-1.7B |
naive |
simple |
18/20 |
Constrained decoding: 0 failures in every configuration — including the
naive prompt and the nested schema.
Troubleshooting (Windows / Docker Desktop / WSL2)
RuntimeError: UVA is not available crash loop at startup: vLLM 0.26's Model Runner V2 allocates pinned mapped host memory, which WSL2 GPU paravirtualization doesn't support. Fix: VLLM_USE_V2_MODEL_RUNNER=0 in the compose environment (falls back to the V1 runner).
unrecognized arguments: --swap-space: removed in current vLLM — don't copy it from older tutorials.
--model as a flag is deprecated for vllm serve — pass the model as a positional argument.
- OOM at engine init despite a small model: lower
--gpu-memory-utilization (vLLM budgets against total VRAM, but Windows/DWM already holds ~0.5-1.5 GB).
Data
data/support_tickets.json — 10 synthetic English support messages for the
fictional SaaS "Nimbus". Deliberately messy: names hidden in sign-offs, order
IDs in mixed notations (#84721, ORD-2231, one ambiguous 84721-B), 4
tickets without any order ID, priorities only implied by urgency language.
Sources