Anthropic published recommendations for stronger AI safety evaluations on November 13, 2026, calling for expert-designed measures, realistic conversations and reproducible reporting. For universities assessing AI systems, the publication puts the evidence behind a safety score under scrutiny.
What Anthropic recommends
The company’s guidance calls for domain and methods experts, risk-specific measures and attention to refusals of legitimate requests. Refusal rates alone can misrepresent safety. The issue is construct validity: whether a test measures the intended behavior.
Anthropic recommends multi-turn tests, preregistered analysis, graders validated against human judgments, and uncertainty estimates. A confidence interval helps interpret model comparisons. It also calls for documenting model versions, prompts, settings and graders so runs can be reproduced.
Existing tools provide infrastructure
The recommendations sit alongside established evaluation tooling. Inspect, an open-source framework developed by the UK AI Security Institute and Meridian Labs, organizes evaluations around datasets, solvers and scorers. Its documentation describes tools for monitoring and visualizing runs.
Anthropic’s Petri auditing tool, published in October 2025, automates conversations with target systems and uses model judges to score transcripts. Its release cautioned that the pilot metrics were provisional and encouraged combining quantitative results with close reading of transcripts. Petri therefore predates this week’s methodological recommendations.
The company’s separate agent-evaluation explainer distinguishes tests of emerging capabilities from regression tests that detect lost performance. It also describes human judgments as a calibration reference for model-based graders. These distinctions help explain why a high pass rate needs context.
What the publication means for research offices
For university evaluation teams and research offices reviewing supplier evidence, the institutional implication is to distinguish three questions: what behavior a score measures, how convincing the comparison is, and whether another team could reproduce the run. A reproducible AI experiment provides a useful connection between evaluation research and the records needed to assess its findings.
Those are implications for interpreting evidence. Anthropic presents the publication as methodological recommendations; it does not establish a university compliance obligation.
Sources
- Anthropic: Building strong safety evaluations. Published 2026-11-13; checked 2026-11-14.
- Inspect: official framework documentation. Checked 2026-11-14.
- Anthropic: Petri, an open-source auditing tool. Published 2025-10-06; checked 2026-11-14.
- Anthropic: Demystifying evals for AI agents. Checked 2026-11-14.
