FrontierMath: Epoch AI’s Research-Level Math Benchmark

FrontierMath is a benchmark of unpublished, research-level mathematics problems created by Epoch AI to test whether AI systems can do advanced mathematical reasoning. Its answers are exact and checked automatically, so a model is scored on whether it produced the right object, not on how convincing its explanation reads. As described on Epoch AI’s pages on 2026-11-12, it is built from original problems written with more than 60 mathematicians, and in the launch evaluation none of the six models tested solved more than 2% of them.

This guide explains what the benchmark is, how its answers are verified, how the family of related problem sets is organised, and what the funding and access disclosures mean for anyone using its results as evidence in an evaluation or governance setting. It deliberately does not quote current leaderboard scores, because those change and are published by Epoch AI itself on its benchmarking pages.

What FrontierMath is and who runs it

FrontierMath is run by Epoch AI, the research organisation described in our Epoch AI guide. The launch post, “FrontierMath: The Benchmark”, was published on 8 November 2024 by Tamay Besiroglu, Elliot Glazer and Caroline Falkman Olsson. The accompanying paper, “FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI” (arXiv:2411.04872), has 24 authors; its first version appeared on 7 November 2024 and its seventh on 23 December 2025.

The official page describes the benchmark as several hundred unpublished, highly challenging mathematics problems. Epoch AI does not state an exact count in the launch post, so this guide does not either. The problems are original and expert-crafted rather than collected from textbooks or competitions, which is the central design choice: because the problems have never been published, they cannot have been memorised from public text.

The paper’s abstract states the two design goals directly: new, unpublished problems and automated verification, both intended to limit data contamination. It also reports that state-of-the-art models solved under 2% of the problems at launch, and that a typical problem takes a researcher in the relevant field multiple hours, with the hardest taking multiple days.

Why a benchmark like this exists

The paper’s stated aim is to limit data contamination, meaning the risk that a model has already seen test problems in its training text. Using unpublished problems and automated checking is how the benchmark addresses that. This connects to the general idea of a frontier model evaluation, where the point is to measure capabilities at the edge of what systems can do. For contrast, a widely used knowledge test such as the MMLU benchmark measures a different thing: broad multiple-choice knowledge rather than novel research-level problem solving.

Tiers and related problem sets

Epoch AI organises the main benchmark into difficulty tiers and has added related sets that use different verification modes. As described on the official FrontierMath page on 2026-11-12:

• Tiers 1-3 cover undergraduate problems through exploratory problems suitable for an advanced graduate student.
• Tier 4 is research-level mathematics.
• FrontierMath: Open Problems is a broad collection of significant open research problems whose solutions are computationally verifiable.
• FrontierMath Erdos consists of problems posed or studied by Paul Erdos that were open as of August 2026, formulated in Lean. An AI system must write a complete Lean proof or disproof.

The data table below the article compares these components by what each contains and how an answer is accepted. The distinction matters when reading headlines: a score on the original problem tiers measures exact-answer problem solving, while the Open Problems and Erdos sets ask for something different, namely new mathematical results that a computer can check, either by computation or by formal proof in Lean.

Epoch AI keeps a version history on the FrontierMath page, and the original analysis was run on the version labelled FrontierMath_10-22-24. When you compare a reported score with another, check which version and which tier set each refers to, since the problem set is revised over time.

How answers are verified

Every problem in the main benchmark has a single, exact, automatically checkable answer. According to the benchmark page, answers are exact integers or other objects such as matrices or SymPy expressions, and a script checks them by exact match or by comparison with a known solution. There is no grader model and no human reading of a written argument.

The “guessproof” design

Epoch AI describes the problems as designed to be guessproof: the probability of producing the correct answer without doing the real mathematical work is stated to be under 1%. Because correct answers cannot be reached by luck, a low score reflects failure to do the mathematics.

Peer review and the error rate

Problems were peer reviewed for correctness, ambiguity and difficulty. A second review of a random subsample found that roughly 1 in 20 problems contained errors that needed correction. Epoch AI disclosed this figure itself, which is useful context: no large expert-written benchmark is error-free, and a small residual error rate bounds how far any score can be interpreted. The page also records, in an update dated 2026-09-22, that the error-bounty program is now closed.

Evaluation conditions

In the launch evaluation, models were given extended thinking time and a Python environment. The six models tested included Claude 3.5 Sonnet, o1-preview, GPT-4o and Gemini 1.5 Pro, and none exceeded 2%. Results from any later run should be read together with the conditions under which it was produced, including tool access and token budget, because those can change the outcome as much as the model does.

Funding, access and the holdout set

The benchmark page includes an explicit disclosure about OpenAI’s involvement, which is the most relevant part of the page for governance readers. The facts as stated on the official page on 2026-11-12:

• OpenAI commissioned 300 of the questions and fully owns them.
• OpenAI has access to the statements and solutions for those questions up to the version labelled FrontierMath_XX-XX-25, except that 50 solutions were randomly withheld as a holdout set.
• Epoch AI retains the right to conduct and publish internal evaluations using all questions.
• Epoch AI thanks OpenAI for its support.

Why this disclosure is worth reading closely

Questions of who funds an evaluation, who owns its test items and who can see the answers are exactly the issues covered in our guides on evaluator independence in the AI safety ecosystem and third-party AI auditing. The FrontierMath disclosure is a concrete, published example of the information an outside reader needs: the funder, the share of items it owns, what it can access, and what is held back. The 50 randomly withheld solutions are the part of OpenAI’s set that the funder has not seen.

This guide does not assess whether the arrangement affects any particular result. The point for assurance work is that the disclosure exists and is specific.

Getting access

Representative samples of the problems can be downloaded, and five problems with their solutions were released initially. Enquiries about running evaluations go to [email protected], per the benchmark page.

Reading FrontierMath results

Four checks make a FrontierMath claim easier to interpret:

1. Which set? Tiers 1-3, Tier 4, Open Problems and Erdos measure different things with different verification, as the table below shows.
2. Which version and date? The problem set has a version history; a score is tied to the version evaluated.
3. Who ran it and under what conditions? Epoch AI points to its Benchmarking page for the latest results. Tool access and thinking budget are part of the conditions.
4. What is the error margin of the benchmark itself? Epoch AI’s own review found about 1 in 20 problems needed correction.

A benchmark score is evidence about one narrow capability, exact-answer mathematical problem solving. It is not, on its own, a safety case; for how benchmark results fit into broader claims about a model, see our guide to what a frontier AI model is and the synthetic benchmark entry for how constructed test items differ from real-world data.

Related benchmarks and organisations

Item	What it is	Where to read more
Epoch AI	Research organisation that runs FrontierMath and the Epoch AI Benchmarking page	Epoch AI guide
MMLU	Broad multiple-choice knowledge benchmark, a different task from research-level problem solving	MMLU benchmark
Third-party evaluation	Outside assessment of model capabilities and risks	Third-party AI auditing

FAQ

What is FrontierMath?

FrontierMath is Epoch AI’s benchmark of several hundred unpublished, expert-written mathematics problems designed to test advanced mathematical reasoning in AI systems, with answers verified automatically.

Who created FrontierMath?

Epoch AI created it, working with more than 60 mathematicians. The launch post was written by Tamay Besiroglu, Elliot Glazer and Caroline Falkman Olsson, and the arXiv paper lists 24 authors.

How hard is FrontierMath?

The paper states that a typical problem needs multiple hours of a researcher in the field and the hardest need multiple days. At launch, none of six tested models solved more than 2%.

How are FrontierMath answers checked?

By script. Answers are exact integers or objects such as matrices and SymPy expressions, compared by exact match or against a known solution.

Did OpenAI fund FrontierMath?

Epoch AI’s page states that OpenAI commissioned 300 questions and owns them, has access to their statements and solutions except for a 50-solution random holdout, and that Epoch AI may still evaluate and publish results using all questions.

Can I use the problems myself?

Representative samples are downloadable. For evaluation access, Epoch AI directs enquiries to [email protected].

Sources

• Epoch AI, FrontierMath, read 2026-11-12.
• Epoch AI, FrontierMath: The Benchmark, read 2026-11-12.
• Glazer et al., FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI, arXiv:2411.04872, v7 of 23 December 2025, read 2026-11-12.