Skip to main content
v2026.11,986 entries · CC-BY 4.0

Editorial · CASRAI · AI and ML research outputs

JustRUA Study Evaluates GPT-6 Robot-Use Agents Across Embodied Manipulation Tasks

A study of GPT-6 robot-use agents shows lean harness design and depth perception raise RoboDojo manipulation benchmark scores from 29.7 to 50.3.

Published 14 Nov 2026· 4 minute read

On November 11, 2026, researchers from Huazhong University of Science and Technology, Tsinghua University, Peking University, ModelBest, and Tencent released JustRUA: A Controlled Study of GPT-6 Robot-Use Agents (arXiv:2611.16009), presenting the first systematic empirical breakdown of foundation-model agent harnesses in bimanual robotic manipulation. Testing OpenAI’s GPT-6 Astra model on the 42-task RoboDojo simulation benchmark, the authors revealed that a lean execution harness combining full episode context, bounded motion steps, and predicted scene depth improved overall manipulation performance from 29.7 to 50.3 out of 100 without requiring specialized policy fine-tuning or complex multi-agent reflection loops.

Deconstructing the Robot-Use Agent Stack

Modern embodied artificial intelligence increasingly deploys foundation models as robot-use agents (RUAs). Unlike specialized vision-language-action (VLA) models that require extensive end-to-end imitation training, RUAs employ general-purpose multimodal models to observe workspace camera feeds, reason over task instructions, and output direct Cartesian targets for downstream motion planners such as CuRobo. Despite rapid progress on leaderboards, recent RUA architectures have frequently bundled numerous design modifications simultaneously—ranging from custom action abstractions to iterative self-reflection—making it difficult for researchers to isolate which components drive genuine performance gains.

To establish rigorous baseline comparisons, the research team adopted the official Level-3 (L3) Inspect EEF harness from the RoboProbe benchmark. Under the RoboProbe taxonomy, an L3 agent interacts with robotic hardware strictly through a fixed, non-learned harness without intermediate learned policies. The benchmark evaluation encompassed 42 distinct bimanual manipulation tasks categorized across five core capability groups: Generalization, Precision, Long-horizon, Memory, and Open environments.

Empirical Findings Across Four Architectural Axes

The study evaluated design choices across four fundamental axes: harness memory, action interface design, experience reuse, and scene geometry representation.

1. Harness Loop and Closed-Loop Perception

The standard baseline harness called the foundation model API while preserving only the two most recent visual observations, causing models to lose visual track of occluded objects. By retaining complete episode histories within a persistent context window, task scores rose from 29.7 to 34.7. Furthermore, the team observed that motion planners occasionally output unstable trajectories that execute unchecked in standard open-loop frameworks. Imposing a strict 20-step execution cap (approximately 0.8 seconds of physical motion) forced frequent visual re-observation, preventing runaway positioning errors and lifting the benchmark score to 38.0.

2. Action Primitives and Command Structuring

The authors systematically tested alternative action interfaces, including relative coordinate targets, high-level semantic macro-operations (such as pick-and-place wrappers), and chained move sequences. Contrary to common architectural assumptions, complex semantic abstractions degraded overall performance to 34.6, frequently failing when unexpected physical contacts occurred. Plain world-frame end-effector coordinates remained the most dependable control primitive for multimodal models.

3. Experience Reuse and In-Context Trajectories

Supplying models with visual demonstrations or text trajectories from previous episodes yielded mixed results. While visual demonstrations increased task completion in certain multi-stage workflows, overall scores reached 39.9 at the expense of higher token costs ($5.11 per episode compared to $4.94 for the default harness). In addition, demonstrations transferred systematic failure modes across runs, including repeated collision trajectories during bimanual handovers.

4. Scene Geometry and Monocular Depth

Integrating compact proprioceptive readings—such as real finger separation distances and fingertip contact forces—helped the model confirm successful grasps, lifting performance to 39.8. Providing spatial depth representations delivered the largest single performance surge. Because benchmark governance rules prohibit direct simulator depth extraction, the researchers paired camera frames with monocular depth maps predicted by Depth Pro from raw RGB imagery. Combining uncalibrated predicted depth with a code-execution tool that solves 3D coordinates from pixel inputs elevated the final JustRUA configuration score to 50.3, with task success rates reaching 43.3% at an average inference cost of $12.53 per episode.

The Limits of Precomputed Affordance Maps

A notable negative result in the study involved precomputed visual affordances. Overlaying affordance heatmaps and keypoint predictions onto camera observations consistently lowered evaluation scores (falling as low as 30.9). Rather than aiding spatial reasoning, precomputed affordances introduced visual clutter, distracted attention heads from fine contact points, and caused models to hallucinate clearances in cluttered workspaces.

Implications for Academic Robotics Laboratories

The JustRUA findings offer substantial operational guidance for university robotics facilities and research offices managing emerging AI and ML research outputs. For academic teams operating under constrained hardware budgets, the study demonstrates that state-of-the-art manipulation capabilities do not inherently demand costly domain-specific retraining or opaque prompt scaffolding. Instead, laboratories can achieve robust closed-loop execution by optimizing sensor telemetry, maintaining short control horizons, and equipping models with geometric tool-use capabilities.

These findings also underscore the necessity of standardized benchmarking protocols. Methodological variations in context length, motion execution limits, and visual depth extraction can substantially skew reported model performance, highlighting the importance of conducting each reproducible AI experiment against open, verifiable baselines to preserve computational reproducibility. As embodied intelligence accelerates across physical lab automation and human-robot collaboration, rigorous empirical harness evaluations will remain central to broader frontier AI safety standards.

Sources

Zefan Wang, Yufan Dang, Xuanwei Shi, Mingqi Gao, Cheng Yin, SikYuen Tam, Hanyu Liu, Yasheng Wang, Yaxi Lu, Songyang Zhang, Xin Cong, and Yuan Yao. “JustRUA: A Controlled Study of GPT-6 Robot-Use Agents.” arXiv preprint arXiv:2611.16009 [cs.RO], submitted November 11, 2026. Checked 2026-11-14.

Ask CASRAI · full access

Ask about this story

Full access: large monthly usage, up to 10 source passages behind each answer, and MCP and API access for Claude Code, Codex and other CLIs. Cancel any time.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →