Skip to main content
v2026.11,986 entries · CC-BY 4.0
CASRAIAsk CASRAI · AI assistant for researchAsk a research question. Answers come with their sources.Funder rules, compliance, methods and publishing. Answers link the sources they rest on.

Editorial · CASRAI · AI and ML research outputs

Preprint Evaluates Decision Models as Actors, Judges, and Teachers

A November 2026 preprint shows typed decision models excel at step-level RL supervision in agents despite struggling to act on multi-step dependent tasks.

Published 15 Nov 2026· 4 minute read

Ask CASRAI · free to try

Ask about this story

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

CASRAI is the reference for research administration — bookmark it for the next question.

On November 12, 2026, researchers Haochen Liu, Qianhao Zhu, Zixuan Wang, and colleagues at Zhejiang University and Baidu released a preprint, Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents. The study systematically evaluates typed decision models across actor, judge, and teacher roles, showing that models that struggle during active execution can provide dense, highly effective reinforcement learning supervision to train smaller autonomous research agents.

What happened

Decision models return typed outputs—such as discrete choices with normalized probabilities or scalar confidence scores—instead of open-ended conversational text. Because they avoid autoregressive token generation, they execute rapidly at low computational cost. Prior research has evaluated decision models primarily as actors selecting the next environmental step, where they frequently fail on multi-step tasks by skipping prerequisites or completing subgoals out of sequence.

To test whether execution failures reflect an inability to evaluate or supervise tasks, the researchers evaluated three dedicated decision models (TypeSafe AI’s JEV, Perplexity AI’s open-weights Decider v1.2, and OpenAI’s GPT Decisions) alongside three general LLMs (prompted Qwen3.8-27B, constrained Qwen3.8-27B, and GPT-6 Luna). They analyzed performance across three distinct operational roles: acting online in interactive environments, judging completed trajectories post-hoc, and serving as dense action-level reward teachers during Group Relative Policy Optimization (GRPO) training.

Methodology and key findings

The team benchmarked the models on ALFWorld, a text-based household environment spanning six task categories with and without sequential dependencies, and WebShop, a multi-page web shopping environment. The investigation established three core findings:

  • Actors fail on dependent subgoals: When tasks require prerequisite steps (such as cleaning an object before placing it on a cart), all six models suffered sharp accuracy drops. JEV achieved 77.1% on single-object Pick and 70.8% on Pick2, but collapsed to 11.1% on Clean, 18.8% on Heat, and 16.0% on Cool, averaging 42.9% overall. Actors were consistently pulled toward easier downstream subgoals, and general prompt hints failed to resolve the ordering error.
  • Judges succeed in hindsight: When evaluating trajectories as judges, models were evaluated across prefix contexts (past steps only) versus hindsight contexts (the full trajectory). Asking whether each action contributed to goal progress cleanly separated productive actions from useless exploration. Under hindsight, judges accurately credited early prerequisite actions that they overlooked when acting.
  • Teachers provide potent dense supervision: Using JEV as a teacher to provide centered action-level rewards during GRPO training boosted a compact Qwen2.5-1.5B-Instruct student policy from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop. Notably, the 1.5B student supervised by JEV outperformed a 7B parameter baseline (Qwen2.5-7B-Instruct with GRPO at 77.6% on ALFWorld and 66.1% on WebShop), with the largest gains on the exact dependent tasks where JEV failed as an actor.

Why it matters for research practice

Evaluating models strictly through end-to-end task completion benchmarks risks discarding effective evaluators. Step-level credit assignment in agent training has traditionally required manual annotation or expensive API calls to frontier LLMs at every step. Fast typed-output decision models offer a practical, low-cost alternative for researchers conducting reproducible AI experiments and testing policies across synthetic benchmarks.

Because the teacher model functions strictly as an offline supervisor and is discarded at test time, it introduces zero inference overhead during runtime deployment. Compact student policies trained with dense hindsight rewards can match or exceed the performance of much larger models trained on sparse outcome rewards alone.

What changes for researchers

The study provides actionable principles for research groups deploying reinforcement learning and designing agent architectures:

  • Decouple actor evaluation from judge selection: Weak execution does not imply weak evaluation; retrospective judgment benefits from full trajectory visibility that online actors lack.
  • Screen evaluators offline on logged trajectories: Offline judging metrics reliably predict downstream policy gains at a fraction of full training costs.
  • Score action progress rather than terminal value: Asking whether a specific action helped achieve the goal generates informative step-level gradients, whereas querying overall task success collapses into sparse outcome rewards.
  • Support agent verification in frontier AI safety: Dense trajectory evaluation provides rigorous audit mechanisms for semi-autonomous agents within frontier AI safety frameworks.

Sources

  • Liu, H., Zhu, Q., Wang, Z., Li, D., Chen, T., Qiu, P., Qu, Y., Lu, W., Xiao, J., and Shen, Y. (2026). Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents. arXiv preprint arXiv:2611.17167 [cs.AI]. https://arxiv.org/abs/2611.17167
  • Zhejiang University REAL Group. Decision Agents Source Code. GitHub repository: https://github.com/ZJU-REAL/decision-agents

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →