Deepnote published Agent Initiative Benchmark results on 13 November 2026, evaluating 12 AI models on 50 workplace situations inside PageMantle, a fictional software company. The test asks whether an agent can investigate scattered records, offer useful help without being asked and stay silent when intervention is unnecessary.
What Deepnote released on November 13
The repository’s results-presentation commit is dated 13 November. The commit history records earlier source-code work with an 8 November author date and a 10 November committer date. The results publication and the earlier code work are distinct milestones.
According to Deepnote’s repository, an agent starts with Slack conversations and investigates across connected services before deciding what to say, whom to contact and where to send a message. Tasks cover finding information, explaining, analysing, verifying, diagnosing, deciding, coordinating and checking access or policy. The tools operate on fixed local data without requiring live service accounts. This is a synthetic benchmark, with a fictional organisation supplying the evidence and situations.
What the reported results show
Deepnote reports mean scores of 8.97 out of 10 for GPT-6 Astra, 8.65 for GPT-6.1 Sol and 8.17 for Claude Fable 5.2. The published 95% task-bootstrap intervals for Astra and Sol overlap: 8.30–9.52 and 7.88–9.31 respectively. Their numerical ordering should be read alongside that uncertainty. Astra and Sol each stayed silent in eight of ten situations requiring silence; Fable 5.2 did so in six.
Two LLM evaluators, blinded to the agent model’s identity and each other’s verdict, grade runs against reference solutions. The agent budget is 180 seconds of model response time, excluding tool execution. These are company-reported benchmark scores, rather than evidence from a university deployment.
What researchers can run and inspect
The repository contains eight runnable examples, one per task type. Its results documentation explicitly cautions that these examples do not constitute a score for the full benchmark. That distinction matters for researchers comparing their own experiments with the published 50-task panel.
The same documentation describes inspection of delivered messages and each judge’s reasoning. Results retain both judge scores and average only complete, compatible pairs; missing scores and valid zero scores remain distinct. This gives researchers a way to examine individual outcomes alongside an aggregate score.
The PageMantle dataset card describes records saved as of 30 September 2026 from eleven services, spanning conversations, documents, customer records, billing and analytics. Files include SQLite, Parquet, Markdown, JSON and PDF. The dataset is distributed under CC BY 4.0; the benchmark repository identifies Apache 2.0 licensing for code, documentation and tasks. Researchers reusing it should preserve the distinction between software licensing and synthetic research data documentation and attribution.
What this means for universities and research offices
Deepnote identifies limits to generalisation: a fixed snapshot, no replies from simulated teammates, one high-effort setting and Slack-only message output. The evaluation measures a contribution at a particular point, rather than its effects on a team over time.
For institutional evaluation, this suggests testing correct intervention and correct abstention separately. A well-written but unnecessary alert can still be the wrong action. Research offices considering agents could use the benchmark’s structure to develop local tests for evidence quality, appropriate recipients and situations already resolved, while examining whether the automated judges’ reasoning matches their own criteria.
The release adds a concrete intervention-or-silence test to the broader discussion of agentic AI benchmarks and research governance. Its synthetic setting and judging method mean the reported rankings should inform further evaluation rather than serve as an institutional deployment verdict.
Sources
- Deepnote Research: Agent Initiative Benchmark. Checked 2026-11-15.
- Deepnote: Agent Initiative Benchmark repository and README. Checked 2026-11-15.
- Deepnote: results-presentation commit. Checked 2026-11-15.
- GitHub API: Deepnote benchmark commit history. Checked 2026-11-15.
- Deepnote: benchmark results documentation. Checked 2026-11-15.
- Deepnote: PageMantle Company Dataset card. Checked 2026-11-15.
