On November 12, 2026, researchers at ITMO University reported that the choice of structured output format directly controls the rate of undetectable semantic errors in fine-tuned sub-billion language models, demonstrating that natural-word encodings cut silent parsing failures by more than half compared to standard symbolic and JSON targets.
Detectable Failures Versus Silent Errors in Edge Parsing
Sub-billion parameter models are widely adopted for targeted semantic parsing in institutional data workflows. Deployed locally on laptops or edge devices, these lightweight parsers deliver rapid fine-tuning and negligible inference costs. However, evaluating structured prediction systems requires separating two fundamentally different failure classes when transforming unstructured input into structured data.
The first category consists of detectable failures: syntax malformations and explicit model refusals. When an output fails validation, downstream software can trap the exception, trigger retries, or alert human reviewers. The second, more dangerous category consists of silent errors: the model generates a syntactically valid parse tree that misrepresents user intent. Because the output parses cleanly, downstream systems cannot flag the discrepancy, allowing corrupted records or wrong actions to execute unnoticed.
While constrained decoding techniques eliminate syntax failures by restricting generation logits, they cannot prevent silent misparsings: a constrained decoder guarantees a well-formed tree, but cannot ensure that the tree matches user intent.
The Six-Format Benchmark: Why Shorter Is Not Better
To evaluate how target format governs silent failures, ITMO University researchers Umar Ahmed, Maxim Dunaev, Mohammad Nour Al Awad, Sergey Ivanov, and Ivan Khodnenko tested two sub-billion models—Qwen3.5-0.8B and LFM2.5-350M—fine-tuned with LoRA across 300 runs. As reported in their arXiv preprint (arXiv:2611.17169v1), the team constructed a ladder of six lossless encodings representing the same semantic parse tree on the PIZZA benchmark, evaluated on 1,357 human test orders across five pool sizes and five seeds.
The representations included native s-expressions (exr), JSON (json), abbreviated s-expressions (exr-short), key-value pairs (tagged), a symbolic grammar (dsl), and an ordinary word encoding (words) mirroring the DSL with natural words.
| Model and Target Format | Tokens | Exact Match (%) | Silent Error Rate (%) | Median Latency (ms) |
|---|---|---|---|---|
| Qwen3.5-0.8B: words | 10.9 | 87.5 ± 0.8 | 4.5 ± 1.5 | 317 |
| Qwen3.5-0.8B: dsl | 11.3 | 79.7 ± 1.9 | 8.3 ± 2.3 | 325 |
| Qwen3.5-0.8B: tagged | 22.2 | 85.9 ± 2.9 | 11.6 ± 2.4 | 616 |
| Qwen3.5-0.8B: exr-short | 32.3 | 75.2 ± 3.6 | 17.4 ± 4.1 | 903 |
| Qwen3.5-0.8B: exr (native) | 44.3 | 85.8 ± 1.6 | 10.3 ± 0.8 | 1171 |
| Qwen3.5-0.8B: json | 44.1 | 81.6 ± 1.3 | 14.9 ± 1.3 | 1220 |
| LFM2.5-350M: words | 12.0 | 77.4 ± 3.2 | 10.0 ± 1.6 | 186 |
| LFM2.5-350M: dsl | 11.4 | 72.0 ± 3.0 | 13.1 ± 2.6 | 178 |
| LFM2.5-350M: tagged | 24.4 | 71.6 ± 2.5 | 21.4 ± 3.8 | 395 |
| LFM2.5-350M: exr-short | 35.6 | 70.8 ± 0.7 | 16.9 ± 2.6 | 571 |
| LFM2.5-350M: exr (native) | 52.4 | 74.0 ± 2.5 | 17.9 ± 2.8 | 821 |
| LFM2.5-350M: json | 57.8 | 70.7 ± 3.6 | 19.1 ± 1.2 | 926 |
The findings contradicted the assumption that shorter token sequences reduce mistakes. Renaming constructors in exr-short cut tokens by a quarter but reduced exact match by 10.6 percentage points on Qwen3.5-0.8B. Across configurations, shortening alone showed no positive correlation with accuracy.
Natural Words Versus Symbolic Markers and the Negation Trap
Surface naturalness proved decisive. In the matched-length pair of words and dsl, the word format outperformed the symbolic DSL across all ten model-by-size conditions, gaining 7.8 percentage points in exact match on Qwen3.5-0.8B and 5.4 points on LFM2.5-350M at full pool size. On LFM2.5-350M, words required slightly more tokens than dsl (12.0 versus 11.4) yet achieved higher accuracy.
Crucially, words reduced the silent error rate by 2.3× on Qwen3.5-0.8B (from 10.3% to 4.5%) and by 1.8× on LFM2.5-350M (from 17.9% to 10.0%) compared to native s-expressions. Error analysis revealed negation as the primary driver: in schemas encoding negation via nested NOT operators or sigils, negation caused 19% to 34% of silent errors. Expressing negation in natural words dropped negation failures to 5% of silent errors on Qwen, cutting overall silent negation mistakes from 2.0% of orders to 0.2%.
Because subword tokenizers split artificial symbols while preserving common words, selecting an appropriate metadata format aligned with pretraining distributions avoids token fragmentation.
Guardrail Costs and Statistical Verification
Every run incorporated guardrails against off-domain inputs and spoken self-repairs. Models refused 95.9% to 99.0% of held-out negative inputs. However, strict refusal rejected 2.1% to 10.1% of real human test orders, primarily because human speech includes polite greetings (“please”, “thank you”) omitted by synthetic grammars. On self-corrections, words led all formats, applying repairs in 85.0% of cases on Qwen.
Using a hierarchical bootstrap across 10,000 resamples and pre-registered Holm corrections, the study provides a rigorous standard for repeatability and reproducibility in small-model evaluation.
Implications for Institutional Data Infrastructure
These findings provide clear guidance for research offices using local models to extract technical metadata from unstructured documents. Defaulting to rigid JSON schemas or terse custom DSLs degrades accuracy, inflates silent errors, and multiplies inference latency by up to 4.4×. Designing target schemas around natural vocabulary offers an immediate engineering lever to improve pipeline reliability.
Sources
- Ahmed, U., Dunaev, M., Al Awad, M. N., Ivanov, S., and Khodnenko, I. (2026). Parsed, Plausible, Wrong: Output Format Controls Silent Errors in Small-Model Semantic Parsing. arXiv:2611.17169v1 [cs.CL]. https://arxiv.org/abs/2611.17169 (Accessed 2026-11-15).
- Arkoudas, K., et al. (2022). PIZZA: A Spoken Dialogue Dataset for Semantic Parsing. https://arxiv.org/abs/2209.00000.
- ITMO University Research Code and Model Artifacts: https://github.com/UmarGit/coffee-llm.
