Skip to main content
v2026.11,986 entries · CC-BY 4.0

Editorial · CASRAI · AI and ML research outputs

SO-Arena Preprint Benchmarks Capability and Calibration of System One Decision Models

SO-Arena benchmarks 21 System One AI decision models across 11 domains, showing that speed and accuracy do not guarantee decision reliability.

Published 14 Nov 2026· 9 minute read

Ask CASRAI · free to try

Ask about this story

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

CASRAI is the reference for research administration — bookmark it for the next question.

On November 11, 2026, researchers released the SO-Arena benchmark preprint (arXiv:2611.15777), introducing an evaluation suite designed to assess the capability spectrum, operational latency, and decision reliability of non-generative System One decision models across eleven core domains. Led by Lichen Li and Xinchao Wang, the study reveals that while fast decision models achieve sub-second execution speeds that rival or exceed heavy generative models on direct semantic tasks, high classification accuracy does not guarantee decision reliability under option reordering or missing-answer scenarios.

Benchmarking Non-Generative System One Architectures Across 11 Domains

Modern enterprise and academic AI workflows increasingly separate cognitive workloads into two distinct regimes: multi-step autoregressive reasoning systems (often styled as “System Two” architectures) and single-step, low-latency decision models (“System One” architectures). While large reasoning models excel at complex mathematical derivation and programmatic synthesis, their high computational cost and multi-second latencies make them impractical for high-throughput, repetitive tasks such as document classification, query triage, policy enforcement, and live action dispatch. In response, open-weight developers and commercial providers have released specialized decision models engineered to produce rapid discrete choices, ordinal scores, and probability estimates without generating long textual rationale tokens.

However, evaluating these models has historically relied on narrow classification benchmarks or general generative leaderboards that fail to measure operational reliability. To establish rigorous empirical baselines, the SO-Arena preprint constructs an evaluation suite derived from 40 established public benchmarks, encompassing over 3,000 natural tasks organized across five capability families and eleven functional domains:

  • Semantic Understanding: Covering sentiment and emotion understanding (200 tasks from sources including Amazon Reviews 2023, GoEmotions, SST-5, DynaSent, and ARTS) and user intent and routing (300 tasks from CLINC150, BANKING77, SGD, NatCS, and NLU++).
  • Knowledge and Reasoning: Spanning mathematical problem solving (200 tasks from GSM8K-Platinum, MATH500, and AIME 2025–2026), code understanding (200 tasks from CodeJudgeBench, LiveCodeBench, and LeetCodeDataset), scientific reasoning (200 tasks from ARC-Challenge, MMLU-Redux, and GPQA Diamond), and general commonsense knowledge (200 tasks from MMLU-Redux, HellaSwag, and CommonsenseQA).
  • Safety and Policy Compliance: Evaluating content filtering, jailbreak detection, and policy adherence (500 tasks from RAGTruth, ReasoningShield, HarmThoughts, DynaBench, Aegis 2.0, and OR-Bench).
  • Evidence Assessment: Measuring document retrieval and relevance reranking (300 tasks from TREC Deep Learning, BRIGHT, and Touché 2020).
  • Agentic Decision-Making: Evaluating direct environmental interaction in sandboxed environments, including task completion verification (109 tasks from BrowseComp-Plus and VirtualHome), tool and API invocation (182 tasks from BFCL and ToolSandbox), and multi-step web interaction (237 tasks from Mind2Web).

To eliminate label noise and ensure rigorous measurement, the authors subjected all candidate tasks to evidence-grounded human and model verification, removing items with ambiguous or disputed ground-truth references. Tasks were further stratified into explicit difficulty tiers—Easy, Medium, Hard, and Very Hard—allowing researchers to assess where architectural scaling genuinely overcomes task complexity within AI and machine learning research outputs.

Latency and the Capability Spectrum: Where System One Matches System Two

The preprint reports comprehensive benchmark evaluations across 21 frontier models: 16 open-weight System One models ranging in scale from 0.4 billion to 31 billion parameters (including Laya, Kev-4B, JevAny-4B, H2O Lightning, StartLux Instant, Decider Flash, SII-4B, Jeva-4B, Clef-Flash-9B, Kev-27B, JevAny-27B, Clef-27B, GLiDE, Decider-27B, KnowLine-31B, and Sieve-31B-v3), one dynamically switching model (StartLux Auto), two closed decision APIs (TypeSafe Jev API 1.14 and OpenAI Decisions), and two state-of-the-art closed System Two reference models (OpenAI GPT-6 Sol and GPT-6 Luna).

Timing experiments conducted on a single NVIDIA H100 GPU demonstrate dramatic latency advantages for System One models across 1,109 timed decision tasks:

  • Sub-50 Millisecond Execution: Compact 4B parameter models delivered mean latencies between 25.2 ms and 40.6 ms (Kev-4B: 25.2 ms; JevAny-4B: 25.2 ms; H2O Lightning: 26.5 ms; StartLux Instant: 35.1 ms; Decider Flash: 39.8 ms). The ultra-compact 0.4B model Laya recorded a mean latency of just 6.0 ms (90th percentile: 10.2 ms).
  • High-Capacity 27B and 31B Models: Larger frontier open-weight decision models maintained mean execution times between 90.3 ms and 265 ms (Kev-27B: 90.3 ms; JevAny-27B: 93.9 ms; Clef-27B: 108 ms; GLiDE 27B: 113 ms; Decider-27B: 153 ms; KnowLine-31B: 177 ms; Sieve-31B-v3: 265 ms).
  • Comparison with Generative Reasoning: In contrast, the hybrid StartLux Auto model averaged 3,084 ms when invoking test-time deliberation, while full System Two reasoning models required multi-second execution windows that are orders of magnitude slower.

Beyond latency, empirical performance across the benchmark revealed three distinct capability regimes:

  1. Direct Semantic Judgment: In domains requiring immediate classification without symbolic derivation—such as sentiment analysis, intent routing, commonsense knowledge, and safety classification—top System One models performed comparably to, or even surpassed, GPT-6 Luna. For example, GLiDE 27B scored 90.0 on Sentiment and Emotion (+4.8 points above GPT-6 Luna) and 95.0 on Knowledge and Commonsense (+5.0 points above Luna). On safety filtering, GLiDE reached 99.0 on easy and 84.0 on medium tiers.
  2. Fine-Grained Grounded Decisions: In domains requiring grounded environmental assessment—such as retrieval reranking, tool parameter selection, web interface interaction, and task verification—System One models lagged initially at smaller parameter scales but exhibited massive performance leaps when scaling backbone architectures from 4B to 27B/31B parameters. Backbone upgrades from Qwen3.5 to Qwen3.8-27B delivered gains of +25.6 points in tool use, +23.9 points in web interaction, and +17.4 points in task completion verification.
  3. Multi-Step Derivation: In domains demanding symbolic logic and algorithmic chaining—specifically mathematics, code execution analysis, and scientific reasoning—System One models fell substantially behind System Two deliberation. On the medium and hard tiers of Math, GLiDE lagged behind GPT-6 Luna by 26.7 points, while StartLux Instant lagged by 38.3 points. Enabling thinking in StartLux Auto provided a massive +41.7 point gain in math, demonstrating that pure single-pass feedforward architectures cannot yet replicate deliberate scratchpad reasoning on formal proofs.

The Reliability Paradox: Why Benchmark Accuracy Fails to Ensure Consistency

The most consequential discovery documented in the SO-Arena preprint centers on decision reliability. Standard academic evaluations rank models exclusively by aggregate accuracy or leaderboard scores. However, in production pipelines, a model that achieves 80% accuracy may still be functionally dangerous if its choices fluctuate based on option order or if it forces an incorrect classification when none of the provided choices are valid.

The authors formulated two rigorous, complementary metrics to test decision robustness:

  • Option Order Stability (S): Each test item is presented in its original sequence and across 12 distinct permutations of the candidate options. The metric S measures whether the model identifies the correct semantic entity consistently regardless of position, normalized against chance agreement.
  • Null Discrimination (N): Measures the model’s capacity to recognize missing correct options. By evaluating items in both intact condition and with the correct option excised, the metric assesses whether the model’s estimated probability of the absence of a valid choice (p_null) correctly discriminates incomplete option sets, evaluated via ROC area under the curve.
  • Combined Reliability Score (R): Defined as the geometric mean of stability and null discrimination (R = sqrt(S * N)), requiring a model to perform above chance on both dimensions simultaneously.

When evaluating the 18 System One models across these criteria, Li and colleagues uncovered a striking reliability paradox: model capability does not imply decision reliability. The top-performing open-weight model on the capability leaderboard, GLiDE (score 70.2), ranked only seventh in overall reliability. Conversely, the two models demonstrating the highest decision reliability across the suite—JevAny-27B and Jev API 1.14—ranked only eighth and sixth in raw capability score.

Detailed analysis of failure modes demonstrated that while option order stability generally tracks model size and overall accuracy (14 of 18 models achieved S > 0.80), null discrimination behaves erratically. Highly capable models frequently exhibited severe overconfidence, arbitrarily picking a plausible distractor rather than signaling that no valid answer existed. This behavior aligns with broader methodological concerns documented in HELM benchmark methodology and related agentic AI benchmark frameworks, where uncalibrated confidence scores undermine system autonomy.

Implications for University Classifiers, Routing Networks, and Research Administration

The findings from SO-Arena carry immediate, practical relevance for university IT departments, research offices, and scholarly communications teams currently integrating automated AI classifiers into operational infrastructure. Academic institutions process vast streams of semi-structured text where high throughput and privacy are paramount, making local, open-weight System One models an attractive architectural choice.

Common institutional use cases directly impacted by these findings include:

  • Grant Proposal Routing and Discipline Classification: Sponsored projects offices frequently deploy routing models to assign incoming research proposals to appropriate review committees, compliance panels, or departmental administrators. If a routing model lacks option-order stability or fails to identify an out-of-scope proposal (the null discrimination failure), interdisciplinary applications risk misrouting or silent administrative neglect.
  • Automated Research Compliance and Ethics Triage: Institutional Review Boards (IRBs) and Export Control offices increasingly test fast classifiers to screen protocol submissions for restricted keywords, dual-use technologies, or foreign collaboration indicators. In safety and policy compliance domains, where SO-Arena showed open-weight models matching closed frontier APIs, System One models offer high speed and on-premises data confidentiality. However, their tendency to select an arbitrary category when an edge case arises necessitates human-in-the-loop escalation paths.
  • Research Data Repositories and Metadata Enrichment: Data curation teams utilize classification models to verify subject ontologies, validate file formats, and check compliance with computational reproducibility standards and reproducible AI experimentation guidelines. Fast decision models can rapidly categorize thousands of supplementary datasets, but verification pipelines must validate probability calibration before committing metadata to permanent catalogs.

Strategic Recommendations for Research Computing and Institutional AI Deployment

To avoid operational disruptions while capitalizing on the cost and latency advantages of System One decision architectures, research computing leaders should incorporate the following evidence-backed practices:

  1. Audit Models for Permutation Invariance: Never procure or deploy a routing model based on top-1 accuracy on static test splits alone. Conduct localized permutation tests by shuffling candidate labels across incoming batches to quantify decision stability under operational variance.
  2. Implement Explicit Null Option Detection: Because off-the-shelf decision models struggle to recognize missing correct options, production pipelines must incorporate explicit confidence thresholding, entropy filtering, or fallback handlers that flag low-margin decisions for human review rather than forcing a low-confidence match.
  3. Adopt Tiered Hybrid Architectures: For high-stakes institutional tasks, combine System One and System Two capabilities hierarchically. Use lightweight System One classifiers (such as 4B or 27B parameter open-weight models) to handle high-volume, straightforward triage at sub-100 millisecond latency, while routing ambiguous, low-confidence, or multi-step symbolic tasks to deliberative reasoning models or expert human reviewers.

Sources

  • Li, L., Dong, Z., Feng, S., Li, Q., and Wang, X. (2026). “SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models.” arXiv:2611.15777 [cs.CL], submitted November 11, 2026. Checked: 2026-11-14.
  • National University of Singapore Learning and Vision Lab. “SO-Arena Evaluation Framework and Benchmark Repository.” Checked: 2026-11-14.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →