Skip to main content
v2026.11,986 entries · CC-BY 4.0
CASRAIAsk CASRAI · AI assistant for researchAsk a research question. Answers come with their sources.Funder rules, compliance, methods and publishing. Answers link the sources they rest on.

Editorial · CASRAI · AI and ML research outputs

Verify by Doing Co-Evolves Agent Execution and Evaluation

Verify by Doing co-evolves actor and evaluator harnesses, enabling AI agents to self-improve complex tasks without external ground-truth feedback.

Published 16 Nov 2026· 6 minute read

Ask CASRAI · free to try

Ask about this story

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

CASRAI is the reference for research administration — bookmark it for the next question.

Researchers from Ant International and Zhejiang University introduced Verify by Doing (VbD), an artificial intelligence framework that co-evolves task execution and active tool verification harnesses without updating model weights. Unveiled in November 2026, the architecture addresses a core bottleneck for research computing and data-intensive laboratories: enabling autonomous agents to self-improve and verify complex digital work when external ground truth or benchmark scores are absent.

What happened

In mid-November 2026, a research team from Ant International and Zhejiang University—led by Yitong Lin, Jianda Chen, Yihao Hu, Pan Wang, Hang Wang, Zhihao Wen, Xiaoxue Ren, and Wei Wu—published a technical study titled “Verify by Doing: Co-Evolving Agent Execution and Evaluation without External Feedback” (arXiv:2611.15276). The work addresses a persistent limitation in autonomous language agents: how to verify and improve complex multi-step task performance when external test-time feedback, grading rubrics, or reference answers are unavailable.

In real-world settings such as administrative workflows, software refactoring, scientific computing, and financial modeling, external reward signals do not exist at deployment. When agents rely on passive text inspection or superficial output checks—such as verifying file existence or matching table headers—critical functional defects remain undetected. Furthermore, generating additional solution candidates under flawed, static checks merely reinforces those blind spots, causing agents to accept regressions or drift into repetitive errors.

To overcome this bottleneck, the researchers introduced Verify by Doing (VbD). The system keeps the base foundation model weights fixed—evaluated primarily using DeepSeek-V4-Flash-0731—while co-evolving two separate software harnesses within a single task: an Actor harness that executes actions, and an Evaluator harness that actively inspects deliverables and environment states using programmatic tools.

Key details

Verify by Doing decouples execution from verification by establishing two distinct operational harnesses:

  • Actor Harness: Governs direct execution through an editable set of instructions, modular skills, and operational checkpoints.
  • Evaluator Harness: Directs active inspection of artifacts, database records, and environment state via tools, scripts, and queries.

Instead of relying on passive language-model critique, the Evaluator actively interrogates the environment. In a complex spreadsheet accounting task, for example, initial checks may only confirm that column headings exist. An updated Evaluator actively retrieves underlying cell values, recalculating expense totals and after-tax revenue offline to verify arithmetic consistency against task requirements.

To prevent erroneous updates from degrading the system, VbD implements decoupled selection and retention mechanisms:

  • Evaluator Adoption: Candidate Evaluator harnesses are comparatively evaluated against the current Evaluator on a shared candidate pool. A new Evaluator is adopted only if its verification reports provide stronger evidential grounding and higher discrimination of flawed deliverables.
  • Actor Retention: An updated Actor harness is accepted only when its fresh execution deliverable receives explicit validation from the active Evaluator.
  • Output Selection: Choosing the final deliverable is decoupled from updating the Actor. The retained Evaluator independently selects the best deliverable from the historical pool, recovering successful results without generating new attempts.

The authors evaluated VbD across AutomationBench (covering Finance, Sales, Marketing, Operations, Support, and Human Resources), tau^3 Banking, and GDPval. Across all three benchmarks, VbD achieved the highest overall scores among methods operating without external feedback:

Benchmark / Domain Base Model (%) Strongest Baseline (%) VbD (Ours) (%) Advantage (pp)
AutomationBench (Overall) 22.3 33.7 (RQGM) 37.0 +3.3
AutomationBench (Operations) 42.3 48.3 (RQGM) 56.3 +8.0
AutomationBench (Marketing) 22.7 46.3 (RHI) 52.0 +5.7
AutomationBench (Finance) 35.3 52.7 (BoN) 53.3 +0.6
tau^3 Banking Pass Rate 35.1 45.4 (RHI) 49.5 +4.1
GDPval Rubric Quality 80.0 83.1 (RQGM) 84.7 +1.6

In addition to raw pass rates, VbD demonstrated an 89.3% True Negative Rate (TNR) on candidate regressions in AutomationBench, compared to 71.2% for Red Queen Gödel Machines (RQGM). Ablation tests confirmed that removing active tool verification reduced overall AutomationBench scores from 37.0% to 34.3%, while disabling Evaluator evolution reduced them to 35.4%, demonstrating that both active verification and evaluator adaptation are essential.

Background and context

The development of Verify by Doing addresses the growing focus on test-time compute in frontier language modeling. Early inference-time scaling strategies focused on best-of-n sampling, chain-of-thought majority voting, or self-consistency, where models sample multiple paths and rank them by internal confidence.

Subsequent frameworks introduced self-referential adaptation, including Handcrafted Gödel Machines (HGM), Reflexive Harness Optimization (RHO), and Red Queen Gödel Machines (RQGM). However, RQGM relied on held-out reference benchmarks or ground-truth scoreboards to evaluate new evaluator iterations. In real-world enterprise environments or customized scientific workflows, such scoreboards are unavailable.

Without external scoreboards, naive self-reflection suffers from evaluation drift: an evaluator sharing weights with the actor tends to overlook systematic mistakes. By requiring the Evaluator to collect empirical evidence through tools—such as database queries and offline recalculations—VbD grounds evaluation in the environment. This architecture mirrors professional research software engineering standards, where automated testing harnesses operate independently from implementation logic to maintain verification rigor.

Reaction and community analysis

The release of the preprint prompted widespread discussion across AI research and engineering circles. Technical commentary published by Grafy News on November 14, 2026, highlighted the cost efficiency of keeping foundation model weights frozen while evolving execution and verification routines. Grafy emphasized that VbD’s ability to recover successful deliverables from fixed candidate pools provides practical value for organizations seeking to maximize inference-time compute efficiency.

On technical discussion forums such as AlphaXiv and research tracking services like ArXivDaily, practitioners noted the significance of VbD’s 89.3% regression rejection rate. Analysts observed that this high true-negative discrimination mitigates the degradation commonly observed in recursive agent self-play, where flawed outputs gradually corrupt agent memory. Reviewers also commended the evaluation across realistic workplace domains rather than artificial puzzle benchmarks.

What it means for researchers, universities and research offices

The Verify by Doing framework offers tangible benefits for academic research offices, computational laboratories, and institutional IT administration:

  • Auditable Research Administration: Research administration offices exploring AI for pre-award proposal routing, compliance screening, or expenditure tracking require transparent verification. VbD generates structured logs detailing which operational requirements are satisfied, violated, or unresolved, creating an explicit audit trail.
  • Reproducible Scientific Workflows: Computational research groups frequently automate data wrangling, genomic pipelines, and statistical analyses. Incorporating active tool verification ensures that data pipelines maintain computational reproducibility. Research units can incorporate these principles into an institutional model evaluation suite that programmatically verifies intermediate files.
  • Accessible Inference-Time Adaptation: Because VbD achieves performance gains without parameter fine-tuning, academic institutions facing compute constraints can deploy open-weight foundation models and rely on harness optimization to handle complex multi-step tasks.
  • Infrastructure Demands: Implementing active verification requires institutions to invest in secure reproducibility infrastructure and automated workflows. Research computing units must support sandboxed environments where evaluators can safely inspect files and execute test scripts without risking shared systems.

What happens next

Following the mid-November 2026 publication of arXiv:2611.15276, the research team is expected to release open-source code repositories and modular harnesses for AutomationBench and tau^3 Banking. Academic groups are preparing to extend the framework into multi-modal domains, including graphical user interface automation and automated laboratory equipment operation.

Subsequent investigations are also expected to explore coupling VbD’s tool inspection with formal verification solvers, such as Satisfiability Modulo Theories (SMT) engines, to deliver mathematical guarantees for safety-critical academic and clinical applications.

Sources

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →