Science Prediction Universe (SPU), introduced in a preprint submitted to arXiv on November 11, 2026, tests whether AI agents can rank experimental outcomes they have not seen. The UniPat AI-led team reports results on 100 public tasks and 16 private tasks, with substantial limits on prediction accuracy and on comparisons between systems.
The arXiv submission record dates the development to November 11; the PDF cover carries November 13. The findings are reported in a preprint, rather than presented here as established evidence of autonomous scientific capability.
What Science Prediction Universe measures
The benchmark asks agents to order candidates by a measured experimental outcome, rather than predict the numerical measurements themselves. Its public tasks draw on 85 papers in 11 Nature Portfolio journals across chemistry, materials science, biology and medicine. The private tasks come from four unpublished experimental series carried out with partner laboratories, according to the authors.
An isolated grader compares submitted rankings with withheld measurements using Spearman rank correlation. Agents can search and compute, but the paper’s information policy prohibits retrieving the source studies’ answers. This separates the intended scientific forecast from simply finding a published result.
Reported scores need task-coverage context
The authors evaluated 13 configurations spanning 12 models through the Codex and Claude Code agent harnesses. On public tasks, they report task-adjusted mean correlations from 0.227 to 0.478. Individual configurations exceeded task-specific chance thresholds on 34% to 52% of their scored tasks. These correlations are ranking scores, not percentages of experiments predicted correctly.
The highest public task-adjusted mean, 0.478 for Claude Opus 5.5, came from 44 scored tasks out of 100; safety refusals accounted for its missing coverage. The paper cautions that statistical adjustment cannot reveal performance in refused domains. Its main prediction analysis also recovers rankings from formatting-rejected files and excludes refusals and runs without recoverable submissions. Operational completion and scientific scoring therefore need to be read together.
On private tasks, the authors report that 12 configurations performed significantly above random rankings after adjustment for multiple comparisons. They found no statistically detectable pairwise differences between configurations on that small private set. Those findings support a narrower conclusion than a definitive ranking of models for laboratory work.
The public release differs from the paper’s evaluation
The released dataset card states that the public tasks include measurements, graders and reference solutions. It explicitly says these tasks are not a held-out test set. The 16 private tasks are not released.
This matters for data leakage and reproducibility: downloading the package does not reproduce every control used in the paper. The authors’ audited web tools and source-identifier screening are absent from the release, and the dataset card says results obtained without them are not directly comparable. Its reporting instructions call for the mean over scored tasks, task count, model version, reasoning effort and harness.
What universities and research offices can infer
SPU supplies a concrete test of experimental ranking, but its comparisons describe complete agent configurations. The paper’s limitations section says model, reasoning effort and harness effects are combined. Public-task selection also used pilot scores, mainly from GPT-6 Astra. These design choices qualify how broadly institutions can generalize the leaderboard.
For a university evaluating scientific AI, the practical implication is to request task coverage and evaluation controls alongside any headline score, and to record the system configuration used. A ranking result alone does not establish that an agent can replace experimental validation. The immediate development is a new benchmark and an accompanying public task package, with private evaluation and controlled harness comparisons still important to interpreting its results.
Sources
- Tu and colleagues, Science Prediction Universe: arXiv abstract and submission history. Checked 2026-11-14.
- Science Prediction Universe, version 1: methods, results and Appendix B. Checked 2026-11-14.
- Science Prediction Universe PDF, cover date. Checked 2026-11-14.
- UniPat AI: SPU public dataset card. Checked 2026-11-14.
- UniPat AI: SPU project repository. Checked 2026-11-14.
