On November 11, 2026, researchers from Huazhong University of Science and Technology, Tsinghua University, Peking University, ModelBest, and Tencent released JustRUA: A Controlled Study of GPT-6 Robot-Use Agents (arXiv:2611.16009), presenting the first systematic empirical breakdown of foundation-model agent harnesses in bimanual robotic manipulation. Testing OpenAI’s GPT-6 Astra model on the 42-task RoboDojo simulation benchmark, the authors revealed that a lean execution harness combining full episode context, bounded motion steps, and predicted scene depth improved overall manipulation performance from 29.7 to 50.3 out of 100 without requiring specialized policy fine-tuning or complex multi-agent reflection loops.
Deconstructing the Robot-Use Agent Stack
Modern embodied artificial intelligence increasingly deploys foundation models as robot-use agents (RUAs). Unlike specialized vision-language-action (VLA) models that require extensive end-to-end imitation training, RUAs employ general-purpose multimodal models to observe workspace camera feeds, reason over task instructions, and output direct Cartesian targets for downstream motion planners such as CuRobo. Despite rapid progress on leaderboards, recent RUA architectures have frequently bundled numerous design modifications simultaneously—ranging from custom action abstractions to iterative self-reflection—making it difficult for researchers to isolate which components drive genuine performance gains.
To establish rigorous baseline comparisons, the research team adopted the official Level-3 (L3) Inspect EEF harness from the RoboProbe benchmark. Under the RoboProbe taxonomy, an L3 agent interacts with robotic hardware strictly through a fixed, non-learned harness without intermediate learned policies. The benchmark evaluation encompassed 42 distinct bimanual manipulation tasks categorized across five core capability groups: Generalization, Precision, Long-horizon, Memory, and Open environments.
Empirical Findings Across Four Architectural Axes
The study evaluated design choices across four fundamental axes: harness memory, action interface design, experience reuse, and scene geometry representation.
1. Harness Loop and Closed-Loop Perception
The standard baseline harness called the foundation model API while preserving only the two most recent visual observations, causing models to lose visual track of occluded objects. By retaining complete episode histories within a persistent context window, task scores rose from 29.7 to 34.7. Furthermore, the team observed that motion planners occasionally output unstable trajectories that execute unchecked in standard open-loop frameworks. Imposing a strict 20-step execution cap (approximately 0.8 seconds of physical motion) forced frequent visual re-observation, preventing runaway positioning errors and lifting the benchmark score to 38.0.
2. Action Primitives and Command Structuring
The authors systematically tested alternative action interfaces, including relative coordinate targets, high-level semantic macro-operations (such as pick-and-place wrappers), and chained move sequences. Contrary to common architectural assumptions, complex semantic abstractions degraded overall performance to 34.6, frequently failing when unexpected physical contacts occurred. Plain world-frame end-effector coordinates remained the most dependable control primitive for multimodal models.
3. Experience Reuse and In-Context Trajectories
Supplying models with visual demonstrations or text trajectories from previous episodes yielded mixed results. While visual demonstrations increased task completion in certain multi-stage workflows, overall scores reached 39.9 at the expense of higher token costs ($5.11 per episode compared to $4.94 for the default harness). In addition, demonstrations transferred systematic failure modes across runs, including repeated collision trajectories during bimanual handovers.
4. Scene Geometry and Monocular Depth
Integrating compact proprioceptive readings—such as real finger separation distances and fingertip contact forces—helped the model confirm successful grasps, lifting performance to 39.8. Providing spatial depth representations delivered the largest single performance surge. Because benchmark governance rules prohibit direct simulator depth extraction, the researchers paired camera frames with monocular depth maps predicted by Depth Pro from raw RGB imagery. Combining uncalibrated predicted depth with a code-execution tool that solves 3D coordinates from pixel inputs elevated the final JustRUA configuration score to 50.3, with task success rates reaching 43.3% at an average inference cost of $12.53 per episode.
The Limits of Precomputed Affordance Maps
A notable negative result in the study involved precomputed visual affordances. Overlaying affordance heatmaps and keypoint predictions onto camera observations consistently lowered evaluation scores (falling as low as 30.9). Rather than aiding spatial reasoning, precomputed affordances introduced visual clutter, distracted attention heads from fine contact points, and caused models to hallucinate clearances in cluttered workspaces.
Implications for Academic Robotics Laboratories
The JustRUA findings offer substantial operational guidance for university robotics facilities and research offices managing emerging AI and ML research outputs. For academic teams operating under constrained hardware budgets, the study demonstrates that state-of-the-art manipulation capabilities do not inherently demand costly domain-specific retraining or opaque prompt scaffolding. Instead, laboratories can achieve robust closed-loop execution by optimizing sensor telemetry, maintaining short control horizons, and equipping models with geometric tool-use capabilities.
These findings also underscore the necessity of standardized benchmarking protocols. Methodological variations in context length, motion execution limits, and visual depth extraction can substantially skew reported model performance, highlighting the importance of conducting each reproducible AI experiment against open, verifiable baselines to preserve computational reproducibility. As embodied intelligence accelerates across physical lab automation and human-robot collaboration, rigorous empirical harness evaluations will remain central to broader frontier AI safety standards.
Sources
Zefan Wang, Yufan Dang, Xuanwei Shi, Mingqi Gao, Cheng Yin, SikYuen Tam, Hanyu Liu, Yasheng Wang, Yaxi Lu, Songyang Zhang, Xin Cong, and Yuan Yao. “JustRUA: A Controlled Study of GPT-6 Robot-Use Agents.” arXiv preprint arXiv:2611.16009 [cs.RO], submitted November 11, 2026. Checked 2026-11-14.







