Examples
Examples from published documents
- OpenAI treats the most common expert answer as correct and uses that consensus expert baseline of 80% as an indicative threshold on a tacit-knowledge biology evaluation; GPT-5.6 Terra scored 84.1% when refusals were counted as successes (GPT-5.6 system card, section 9.1.1.3)
- Anthropic reports that frontier models now surpass human baseline performance on most tasks in its task-based AI R&D evaluation suite, so it added an evaluation built on real internal R&D problems (Anthropic Risk Report, August 2026, section 3.4.3)
- METR reported a 50% time horizon of around 50 minutes for frontier models such as Claude 3.7 Sonnet on its task suite (METR, arXiv:2503.14499)
Counter-examples
Not to be confused with
- The control group in an uplift study measures people with and without AI help; a human baseline measures people doing the evaluation task itself, as a scale for reading the model's score.
Related terms
Related AI safety terms
Terms defined under this element that are not NIKOLAI elements themselves, each as the cited document uses it. Every term is also listed in the AI safety glossary.
- Time horizon
- A model's 50% time horizon is the length of tasks, measured by how long skilled humans take to complete them, that the AI model can finish with a 50% success rate; METR uses it to track agent capability over time.
- Source: METR, "Measuring AI Ability to Complete Long Software Tasks" (arXiv:2503.14499)
Source of record
Where this definition comes from
EU GPAI Code of Practice, Safety and Security Chapter, Measure 7.3(1)(e)
“(v) how the scores compare to human baselines (where applicable), across the model versions, and across the evaluation settings”
https://ec.europa.eu/newsroom/dae/redirection/document/118119Meta Advanced AI Scaling Framework v2, §2.2.1, p.8
“will include comparison to human expert baseline performance where available and relevant”
https://ai.meta.com/static-resource/Meta_Advanced-AI-Scaling-Framework-v2Anthropic Risk Report (August 2026), §3.4.3
“The task-based AI R&D evaluation suite we have reported on in our system cards has reached the point where frontier models surpass human baseline performance on most tasks.”
https://www-cdn.anthropic.com/f61d49fa5596956a5dec75fea0e973bf6a6a8378/Redacted%20Risk%20Report%20August%202026%20.pdf
Crosswalk
How named organisations use this concept
| Organisation | Their term, as published | Match & verification | Source |
|---|---|---|---|
| EUShadow mapping EU GPAI Code of Practice, Safety and Security Chapter | “Model Reports describe evaluation results including "how the scores compare to human baselines (where applicable), across the model versions, and across the evaluation settings" (Measure 7.3(1)(e)).” | closeCL confidence: high | EU GPAI Code of Practice, Safety and Security Chapter |
| MetaShadow mapping Meta Advanced AI Scaling Framework v2 | “Preparedness reports "will include comparison to human expert baseline performance where available and relevant" (§2.2.1, p.8).” | closeCL confidence: high | Meta Advanced AI Scaling Framework v2 |
| AnthropicShadow mapping Anthropic Risk Report (August 2026) | “"frontier models surpass human baseline performance on most tasks" (§3.4.3); on covert-capability benchmarks "we did not have direct human baselines to compare against" (§2.7); informative evidence includes "Evaluations that have a baseline of expert performance to compare to" (§4.4.1).” | closeCL confidence: high | Anthropic Risk Report (August 2026) |
| OpenAIShadow mapping GPT-5.6 deployment safety card | “"The consensus expert baseline of 80% is used as an indicative threshold (i.e. treating the most common expert response to each question as correct)" (§9.1.1.3); "The 80th-percentile expert performance of 36.4% is used as an indicative threshold" (§9.1.1.4).” | closeCL confidence: high | GPT-5.6 deployment safety card |
| Google DeepMindShadow mapping Gemini 3.7 Flash FSF report | “Gemini 3.7 Flash passed 3/4 stealth challenges, "exceeding the human baseline on Classifier Evasion and Strategic Rule Breaking" (p.39); cyber uplift compares the AI's cost "to the human baseline" across TAC tiers up to "95th+ percentile cybersecurity expert" (p.16).” | closeCL confidence: high | Gemini 3.7 Flash FSF report |
| METRShadow mapping METR, "Measuring AI Ability to Complete Long Software Tasks" (arXiv:2503.14499) | “"50%-task-completion time horizon. This is the time humans typically take to complete tasks that AI models can complete with 50% success rate." (abstract)” Uses human completion time as the scale for an AI capability metric, rather than reporting a human score on the same test. | narrowNR confidence: high | METR, "Measuring AI Ability to Complete Long Software Tasks" (arXiv:2503.14499) |
What do these codes mean?
- exact
- The source term is equivalent to this element
- close
- The source term is close but not equivalent to this element
- broad
- The source term is broader than this element
- narrow
- The source term is narrower than this element
- none
- No mapping claim — used for false-friend and declared-but-undefined rows
- EQ
- Equivalent
- CL
- Close
- BR
- Source is broader than the element
- NR
- Source is narrower than the element
- FF
- False friend — same or similar label, different meaning
- DU
- Declared but undefined by the source
- UV
- Unverified
Divergence
Where sources materially disagree
Sources set the human reference at different points: OpenAI uses a consensus-of-experts answer or an 80th-percentile expert score as an indicative threshold, Google DeepMind maps cyber performance to tiers from novice to 95th-percentile expert, Meta and the EU Code of Practice ask for comparison with human or expert baselines where available, and METR expresses capability as the length of tasks measured in human working time.
Gap
The research did not find a requirement to report how many people formed a baseline or how their expertise was verified; Anthropic reports that for some covert-capability evaluations it had no direct human baselines at all.
