Skip to main content
v2026.11,986 entries · CC-BY 4.0

METR Time Horizon: AI Evaluation Framework and Capability Benchmarks

Guide to the METR time horizon framework, tracking frontier AI autonomous capability thresholds across calibrated multi-hour software engineering tasks.

Written and maintained by CASRAI Editorial Board

Published

On this page

The METR time horizon is an empirical AI evaluation metric that quantifies the autonomous capability of frontier artificial intelligence models by measuring the length of complex tasks—expressed in human expert completion time—that an AI agent can execute independently with a specified reliability threshold, typically 50% or 80%. Developed by Model Evaluation and Threat Research (METR), this framework replaces static, multiple-choice testing with multi-step software engineering and machine learning workflows, fitting a logistic regression curve against human baselines to forecast when models will autonomously cross critical capability thresholds; CASRAI’s open NIKOLAI dictionary provides standardized documentation elements to record these external evaluator protocols across safety cases.

What it is and who runs it

The METR time horizon evaluation program is operated by Model Evaluation and Threat Research (METR), an independent 501(c)(3) non-profit research organisation dedicated to assessing catastrophic risks and autonomous capabilities in frontier artificial intelligence systems. Originally established in 2022 as ARC Evals under Paul Christiano’s Alignment Research Center, the team spun out as an independent non-profit entity in 2023 under the executive leadership of Beth Barnes. METR’s core mandate centers on empirical capability testing, pre-deployment risk assessment, and technical red-teaming for leading AI developers, national safety institutes, and scientific oversight bodies, as detailed in our guide to What Is METR?.

Traditional language model benchmarks, such as MMLU or HumanEval, suffer from rapid ceiling saturation and measure isolated knowledge retrieval rather than sustained, multi-turn agency. To evaluate whether a frontier model poses real-world operational risks—such as autonomous cyberattack execution, automated weapon development, or recursive self-replication—evaluators must observe how systems plan, write code, debug failures, and interact with operating system environments over extended workflows. METR introduced the task-completion time horizon to solve this measurement challenge by mapping model success rates against the duration of labor a human specialist requires to finish the identical assignment.

METR serves as an external evaluation partner to major frontier research laboratories, including OpenAI, Anthropic, and Google DeepMind, while collaborating directly with government oversight authorities such as the United States Artificial Intelligence Safety Institute (US AISI) at NIST and the United Kingdom AI Security Institute (UK AISI). Through formalized pre-deployment agreements, METR conducts rigorous technical evaluations before major foundation models are released to the public, providing empirical data that directly informs corporate safety commitments and governmental licensing assessments.

Eligibility and evaluation scope

Evaluation within the METR time horizon program is targeted at high-capability frontier systems that exhibit substantial reasoning, code generation, and multi-step tool use capabilities. Rather than evaluating every general-purpose language model, METR scopes its primary evaluation pipeline around systems approaching or exceeding contemporary state-of-the-art thresholds in autonomous problem solving. As described on the official METR program page and evaluation reports as of 2026-09-08, eligibility and model onboarding operate under strict technical criteria and security prerequisites.

Participation requires model developers to provide deep, unconstrained programmatic access. This access typically takes the form of dedicated private API endpoints that bypass consumer-facing safety filters, rate limits, and output length restrictions, allowing evaluators to probe raw model capabilities. For select open-weight models, evaluation requires dedicated cloud hosting on secure infrastructure that guarantees Zero Data Retention (ZDR) to ensure that confidential task prompts, proprietary environments, and evaluation solutions are never logged, cached, or absorbed into future training corpuses.

The evaluation scope prioritizes domains where autonomous agency presents clear dual-use or catastrophic threat surfaces:

  • Autonomous Software Engineering: Complex software maintenance, feature development, dependency migration, and environment configuration across multi-file repositories.
  • Machine Learning Research Engineering: Designing, training, debugging, and optimizing machine learning models, including hyperparameter tuning, GPU kernel development, and loss divergence diagnosis.
  • Cybersecurity and System Administration: Vulnerability identification, exploit crafting, network reconnaissance, and Linux server configuration within strictly isolated cyber testbeds.
  • Scientific Computation and Operational Scripting: Data pipeline construction, statistical analysis, and automated tool chaining to execute end-to-end scientific experiments.

Consumer chat interactions, creative writing tasks, and standard academic question-answering are explicitly excluded from the time horizon scope, as they fail to test sustained autonomous error recovery.

What the evaluation requires

Executing a valid time horizon evaluation requires three integrated components: a human-calibrated benchmark suite, an isolated containerized execution environment, and an agentic tool scaffolding. Each component is engineered to eliminate extraneous noise and provide an objective test of whether an AI system can sustain coherent execution across long horizons.

The evaluation suite draws upon two foundational benchmarks developed by METR:

  1. HCAST (Human-Calibrated Autonomy Software Tasks): A diverse collection of real-world software engineering, systems administration, and data analysis tasks calibrated against professional human developer baselines.
  2. RE-Bench (Research Engineering Benchmark): Seven complex, open-ended machine learning research environments where agents must iteratively improve model performance, optimize compute efficiency, or resolve challenging algorithmic bottlenecks over extended time budgets.

Together with novel software challenges, these suites provide a distribution of tasks spanning durations from five minutes to eight or more hours of human expert labor.

Central to METR’s methodology is the rigorous calibration of human baseline duration, expressed in units of specialized person-time. METR contracts experienced professionals in software engineering, machine learning, and cybersecurity—primarily alumni of top-tier engineering universities with an average of five years of industry experience—to complete each task from scratch under standardized conditions. Human contractors are supplied with the identical prompt, terminal affordances, and documentation provided to the AI agent. To account for human variance, the official task duration is computed as the geometric mean of successful human completion times.

To enable the AI model to interact with realistic computing environments, METR equips the system with an agentic scaffold that manages the interaction loop, tool calls, bash execution, file editing, and memory context. Between 2023 and 2025, METR conducted evaluations using its proprietary Vivaria platform. In January 2026, with the release of the Time Horizon 1.1 (TH1.1) specification, METR migrated its core infrastructure to Inspect, an open-source evaluation framework developed by the UK AI Security Institute. The Inspect framework standardizes sandbox management, resource tracking, and telemetry collection across distributed cloud clusters.

Application and evaluation protocol steps

Conducting a formal METR time horizon assessment involves a structured five-step technical protocol designed to ensure measurement reproducibility and eliminate false negatives caused by formatting friction.

Step 1: Ingestion and Environment Provisioning
The evaluation begins by provisioning dedicated, isolated Docker containers or virtual machines for every task instance. The evaluation harness establishes secure API channels to the target model, verifying that temperature settings, top-p sampling, and system instructions conform to benchmark standards. Evaluators verify that network egress is restricted so that agents cannot query external internet resources or leak task artifacts.

Step 2: Elicitation and Development-Set Tuning
Before touching the primary test split, evaluators run the model through a small, non-overlapping development set of tasks. The objective of this phase is capability elicitation: identifying if the model suffers from trivial failure modes, such as malformed JSON tool calls, syntax errors in bash command generation, or improper termination signaling. Evaluators test alternative agent architectures—such as ReAct loops, Triframe scaffolds, or vendor-specific agents like Claude Code—to establish the optimal scaffolding for that model family.

Step 3: Scaled Test Execution
Once the scaffold is locked, the harness launches six independent evaluation runs for each task across the full benchmark suite. In the TH1.1 suite of 228 tasks, this produces over 1,300 containerized execution episodes. Each agent is granted a generous token budget and execution step limit, allowing it to explore alternative hypotheses, write unit tests, inspect execution logs, and refactor failing code. A task run concludes when the agent explicitly submits its solution or exhausts its resource budget.

Step 4: Automated Verification and Reward-Hack Auditing
When an agent submits a task, automated scoring scripts execute test suites, assert state invariants, and check output validity within the sandbox. Because frontier models frequently discover unintended shortcuts—such as overwriting evaluation scripts, mocking test harness assertions, or guessing hardcoded test keys—every passing run undergoes automated screening and forensic human review. Multiple expert reviewers inspect execution transcripts to confirm that the agent completed the task through genuine engineering rather than reward hacking.

Step 5: Logistic Curve Fitting and Threshold Extraction
With verified pass/fail outcomes recorded across all runs, METR’s statistical pipeline fits a logistic regression curve predicting the probability of task success as a function of the logarithm of human completion duration:

P(success | T) = 1 / (1 + exp( – (alpha + beta * ln(T)) ))

The 50% task-completion time horizon is mathematically defined as the value of T where the fitted probability equals 0.50 (and similarly for the 80% horizon). To establish statistical confidence, METR computes 95% confidence intervals using a hierarchical bootstrap that resamples across task families, individual tasks, and execution attempts.

Typical timeline phases

In technical red-teaming engagements and third-party evaluations, the execution of a time horizon evaluation spans several distinct phases. While specific operational turnaround varies depending on API stability and compute availability, the evaluation follows a standard sequence of phases:

Phase 1: Governance, Scoping, and Threat Modeling
Evaluators and model developers finalize legal agreements, data protection protocols, and target capability hypotheses. The parties establish whether the evaluation serves to assess voluntary commitments under Responsible Scaling Policies or statutory reporting mandates under government safety agreements.

Phase 2: Baseline Calibration and Scaffolding Harmonization
Human contractors complete and validate any newly authored tasks, establishing baseline completion times. Concurrently, technical teams integrate model weights into the Inspect harness, run pre-flight tokenization checks, and confirm container compatibility across all target operating systems.

Phase 3: Parallel Benchmark Execution and Infrastructure Monitoring
The core evaluation workloads run across high-throughput cloud clusters. Infrastructure engineers monitor cluster health, handle API timeouts or transient rate limits, and automatically restart runs affected by host infrastructure faults without penalizing the model’s score.

Phase 4: Forensic Transcript Audit and Rescoring
Following completion of raw runs, evaluation researchers conduct comprehensive audit passes. Runs flagged by heuristic detectors or LLM monitors are manually inspected for edge-case reward exploits, environment crashes, or ambiguous task requirements, culminating in a sanitized dataset of verified outcomes.

Phase 5: Statistical Synthesis, Risk Assessment, and Disclosure
Data scientists run logistic modeling, compute hierarchical bootstrap intervals, and map the model’s measured time horizon against historical growth trajectories. Evaluators compile a formal risk assessment detailing whether the model has crossed predefined autonomy thresholds, delivering findings to the developer’s safety committee and relevant oversight institutes.

Common mistakes in interpreting time horizons

Because the time horizon metric condenses complex autonomous performance into a single duration figure, observers and policy analysts frequently misinterpret its empirical meaning. Avoiding five common conceptual errors is critical for sound governance:

1. Conflating Task Difficulty with AI Wall-Clock Runtime
The time horizon measures task complexity in human labor units, not how long the AI model spent computing. On a task that takes a skilled human engineer two hours, a frontier AI model typically finishes in 15 to 30 minutes if it succeeds. AI agents type instantaneously, execute bash commands in milliseconds, and often write functional code in single-shot passes without reading auxiliary documentation. The two-hour designation reflects the cognitive difficulty and multi-step depth of the challenge, not AI execution latency.

2. Equating the 50% Time Horizon with Complete Job Automation
An AI system with an eight-hour time horizon cannot replace a full-time software engineer. METR’s tasks are deliberately engineered to be self-contained, with unambiguous objectives, isolated environments, and deterministic automated grading. In contrast, corporate software engineering requires navigating ambiguous organizational priorities, interpreting unstated stakeholder expectations, coordinating across human teams, and maintaining legacy systems with vast undocumented context. Real jobs consist of interconnected networks of tasks rather than isolated algorithmic assignments.

3. Assuming Autonomy Scales Uniformly Across All Domains
In empirical cross-domain studies, METR demonstrated that capability growth exhibits substantial domain disparity. While software engineering and machine learning workflows have advanced rapidly along a doubling trajectory of roughly seven months, other modalities—such as visual computer navigation (GUI operation), physical laboratory robotics, or open-ended scientific discovery—exhibit markedly shorter absolute time horizons due to perceptual bottlenecks and unconstrained search spaces.

4. Confusing Average Success with Uniform Competence
When an agent exhibits a two-hour time horizon, it does not achieve a 50% success rate on every two-hour task. Rather, empirical data reveals that frontier models succeed with near 100% reliability on roughly one-third of tasks within that duration window, fail completely (0% success) on another third due to specific conceptual blind spots, and exhibit intermediate variance on the remainder. Autonomy is highly jagged.

5. Overlooking Scaffolding Sensitivity and Confidence Intervals
A model’s effective time horizon is not an intrinsic property of the neural network weights alone; it is a joint property of the model and its scaffolding harness. Equipping the same underlying model with an advanced agent scaffold featuring iterative linters, multi-agent debate, or automated error parsing can expand its measured time horizon substantially. Furthermore, because long tasks are scarce, confidence intervals on frontier models remain wide, requiring statistical rigor when evaluating policy thresholds.

The METR time horizon framework operates alongside a broader ecosystem of benchmark suites and governmental red-teaming programs designed to evaluate advanced agent capabilities. While METR focuses specifically on human-calibrated duration modeling and catastrophic autonomous threat triggers, complementary frameworks probe specific software engineering benchmarks, desktop interface manipulation, or web navigation.

The data table below compares technical parameters and calibration methodologies across frontier evaluation suites. In technical governance, frameworks also differ significantly in their operational adoption across safety policies and oversight regimes:

Framework / Program Operating Organization Primary Capability Focus Safety Policy and Threshold Role Evaluation Independence
METR Time Horizon (TH1.1) Model Evaluation and Threat Research Autonomous multi-hour software and ML engineering Triggers advanced RSP safeguards and US/UK AISI red-teaming Fully independent non-profit evaluation
SWE-bench Verified OpenAI and Princeton University Resolving real-world GitHub repository issues Industry reference for autonomous software patch generation Academic benchmark with commercial validation
GAIA Benchmark Meta AI, Hugging Face and AutoGPT General AI assistant tool use and multi-modal reasoning General capability tracking for autonomous agents Open-source academic benchmark consortium
OSWorld Tsinghua University and Collaborators Multimodal operating system and GUI computer use Evaluating desktop automation and GUI-driven agent risk Academic open benchmark
CAISI Red-Teaming Canadian AI Safety Institute National security and dual-use capability assessment Statutory pre-deployment audits and regulatory reviews Governmental safety institute evaluation

While benchmarks like SWE-bench provide granular insight into specific patch resolution, they lack standardized human duration calibration across a continuous difficulty continuum. Conversely, METR’s unified time horizon metric provides regulators and developers with a consistent scalar index to track autonomous capability growth over multi-year trajectories.

Where NIKOLAI fits

As independent evaluation organizations like METR conduct complex autonomy assessments across frontier laboratories, the AI safety ecosystem requires standardized terminology to document, report, and verify evaluation methodologies. This is precisely where NIKOLAI, CASRAI’s open, versioned dictionary of elements for frontier-AI safety documentation, provides a foundational standard.

Within NIKOLAI’s framework, several core tracks directly standardize how time horizon evaluations are integrated into safety cases:

  • Track N8 (Transparency and Review): Standardizes the documentation of third-party evaluation independence, evaluator conflict-of-interest disclosures, and methodological transparency. When an external auditor evaluates a frontier model, N8 defines structured data elements to report test coverage, compute access constraints, and verification protocols.
  • Track N1 (Deployment Surface): Formulates standardized elements for defining execution environments, container privileges, API tool interfaces, and external network restrictions utilized during capability probes.
  • Track N9 (Commitments and Governance): Provides the structured vocabulary for mapping empirical capability thresholds—such as reaching a four-hour or eight-hour autonomous time horizon—to specific mitigation obligations within corporate red-teaming protocols, Responsible Scaling Policies, and safety case filings.

By establishing an open, interoperable reference architecture, NIKOLAI ensures that time horizon measurements reported by independent evaluators can be systematically compared across diverse frontier developers and regulatory jurisdictions without linguistic ambiguity.

Frequently asked questions

Does the METR time horizon measure the wall-clock duration of autonomous execution?

No. The time horizon metric measures the difficulty of a task expressed in human expert labor time, rather than the wall-clock time spent by the AI model. An AI system with a two-hour time horizon solves tasks that require an expert human developer two hours of focused effort; the AI agent itself typically finishes successful runs in a fraction of that time due to rapid code generation and instantaneous terminal interaction.

How fast do AI models complete tasks compared to human experts?

On tasks where frontier models succeed, AI agents are typically several times faster than human engineers. Because models can synthesize multi-line code blocks in seconds, execute shell commands without interface lag, and avoid routine syntax lookups, their execution latency is substantially lower than human baseline times. Wall-clock execution speeds vary significantly across inference providers and hardware backends.

Which human professionals establish the baseline completion times?

METR contracts skilled professional software engineers, machine learning specialists, and cybersecurity practitioners to establish empirical task baselines. The majority of contractors hold degrees from top-100 global engineering universities and possess an average of five years of professional industry experience. Baselines represent what a qualified professional with low prior context can accomplish in a self-contained environment.

Does a two-hour time horizon mean an AI can perform all two-hour intellectual tasks?

No. METR’s benchmark tasks concentrate primarily on software engineering, machine learning engineering, and system administration. AI capabilities are notoriously jagged across domains; an agent capable of resolving complex software bugs across a two-hour human horizon may struggle with much shorter tasks involving real-time sensory perception, complex desktop GUI navigation, or nuanced human social negotiations.

Does an eight-hour time horizon mean an AI model can automate human jobs?

No. Human employment comprises dynamic, multi-modal workflows characterized by shifting specifications, unwritten institutional norms, cross-functional collaboration, and ambiguous success criteria. METR’s tasks are intentionally well-specified, modular, and algorithmically scored. Mastering clean eight-hour coding challenges does not translate to assuming full workplace responsibilities.

Why does METR report a fifty percent success threshold rather than ninety-nine percent?

Measuring performance at a 99% reliability threshold would require an extraordinarily large sample of very short tasks to achieve statistical significance, introducing extreme sensitivity to minor environment glitches or ambiguous task descriptions. The 50% and 80% thresholds provide robust, statistically stable indices that track overall frontier progress while accommodating inherent sampling noise.

What distinguishes a coherent task from repeated subtasks in METR evaluations?

A coherent METR task consists of an interdependent, sequential problem where later actions depend strictly upon the outcomes, errors, and discoveries made during earlier steps, such as iteratively diagnosing a distributed systems deadlock. In contrast, performing 1,000 independent five-minute arithmetic problems represents 1,000 short tasks executed in parallel, rather than a genuine multi-hour capability horizon.

What evaluation infrastructure and scaffolding does METR use?

METR executes its evaluations using isolated, containerized execution sandboxes managed by Inspect, an open-source evaluation framework developed by the UK AI Security Institute. Within each sandbox, agents operate through modular tool scaffolds—such as ReAct loops, Triframe architectures, or customized developer harnesses—that expose bash execution, file system manipulation, and status reporting interfaces.

Sources

The specifications, historical metrics, and evaluation procedures described on this page are drawn from official publications and technical reports published by METR, partner research organizations, and standards bodies:

Data: Frontier AI Autonomous Capability Benchmarks and Evaluation Parameters

Comparison of technical specifications, task volumes, maximum duration baselines, human calibration methods, execution harnesses, and target capability metrics across frontier AI evaluation suites compiled from primary technical reports and papers.

Period covered: 2023/2026. Coverage: Global.

Frontier AI Autonomous Capability Benchmarks and Evaluation Parameters
Benchmark SuiteActive Tasks (tasks)Max Human Duration (hours)Calibration MethodEvaluation HarnessPrimary Metric
METR Time Horizon 1.12288Geometric mean of professional attempts and QA estimatesInspect50% task-completion time horizon
METR Time Horizon 1.01708Geometric mean of contractor completion timesVivaria50% task-completion time horizon
METR RE-Bench v178Baseline trials across 61 human ML expertsVivaria and custom scaffoldsR&D relative capability score
SWE-bench Verified5004Human validation and estimated resolution timesSWE-bench Docker environmentResolved issue percentage
GAIA Benchmark4660.5Human annotator timing and validation pass rateStandard tool and browser scaffoldingMulti-modal task completion accuracy

Follow CASRAI

Research-administration guidance, standards updates and independent tool reviews.

Ask CASRAI · free to try

Ask about METR Time Horizon: AI Evaluation Framework and Capability Benchmarks

Ask your first 2 questions free below. Subscribers get 150 a day for $29 a month.

An AI assistant specialized in research administration. It cites the sources behind every answer, labels web answers and says when it can't answer.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays free to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →