Singularity-Bench, introduced in a preprint submitted on November 12, 2026, evaluates whether AI agents can build systems used in AI development. It covers six tasks and compares agent-built implementations with mature human-engineered references on identical hardware. The authors also introduce Breakwater, safeguards intended to prevent agents from gaming the evaluation.
Six layers of AI development
Each task starts with a minimal codebase and a fixed time budget. Agents implement core components, with internet access blocked during development. The paper’s task descriptions identify the following objectives and reference systems.
| Layer | What the agent builds or improves | Human-built reference |
|---|---|---|
| Pre-training | A language model trained from scratch | OLMo 2 recipe in Megatron-LM |
| Mid-training | A checkpoint prepared for a fixed reinforcement-learning stage | PRISM mixture in Megatron-LM |
| Post-training | A reinforcement-learning system for a coding agent | Miles |
| Inference | A model-serving engine for coding-agent traffic | SGLang |
| Agent harness | The software coordinating a coding agent | Claude Code |
| Cluster orchestration | A serverless inference orchestrator | NVIDIA Dynamo with KAI Scheduler |
These are comparisons with human-built software under specified experimental conditions. They do not directly measure how an agent compares with a university research team working on its own projects.
What the reported scores show
The preprint evaluates 11 agents combining seven models and two harnesses. The authors report that Opus 5.5 with Claude Code matches or outperforms the reference on every task, with a Singularity-Index of 1.25; the other agents average 0.58. These are the research team’s reported findings.
The project’s scoring explanation defines the index as the mean of six scores relative to their references, with the perplexity ratio inverted because lower perplexity is better. Failed tasks contribute zero. An index of 1 represents reference performance on average, so an aggregate score alone does not show whether every layer succeeded.
Breakwater protects the evaluation process
Harbor is a framework for evaluating agents. The authors’ red-teaming found reward-hacking vulnerabilities in their conventional Harbor setup, including when an LLM judge was added. Breakwater extends it with five safeguards:
- Check environment binaries before and after agent development.
- Route API calls to block unauthorized access.
- Terminate live processes when the agent phase ends.
- Keep external activity logs the agent cannot read or alter.
- Execute submitted code with restricted privileges before the verifier and external judge inspect its output.
The authors report blocking every attack tested in their study, which covered nine OWASP agentic-risk categories. That result establishes a boundary around the tested attacks; it does not establish security against every possible attack.
What this means for research institutions
For university AI laboratories, the benchmark offers a way to examine automation at particular development layers. For research offices assessing capability claims, the useful distinction is between a measured task result, computational reproducibility, and evidence that an evaluator resisted manipulation. A strong result in one category does not settle the others.
The safeguards also connect to the broader question addressed by AI control evaluations: what evidence shows that controls withstand attempts to subvert them? Here, that evidence remains scoped to the experiment. The authors acknowledge that their tasks are smaller than true recursive self-improvement would require. The preprint therefore provides evidence about component skills, rather than demonstrating an agent repeatedly building stronger successors.
Sources
- Zhou and colleagues: Singularity-Bench arXiv submission record and abstract. Checked 2026-11-15.
- Singularity-Bench preprint, version 1: tasks, evaluation and results. Checked 2026-11-15.
- Singularity-Bench project: scoring, Breakwater safeguards and stated limits. Checked 2026-11-15.
- Harbor documentation. Checked 2026-11-15.
