Ai2 announced the release of XYBench on November 13, 2026, to test how AI assistants respond when a user’s requested approach rests on a mistaken assumption. The benchmark separates offering a useful alternative from making that alternative the focus of the answer, a distinction relevant to researchers assessing assistant behavior.
What XYBench measures
The release announcement describes 8,115 questions spanning programming and everyday tasks. The underlying XY problem arises when someone wants to achieve a goal, X, assumes an approach, Y, will work, and asks for help with Y. The associated research paper is dated September 6; November 13 is the release announcement date.
The evaluation repository documents three separate dimensions that can sit within a model evaluation suite:
- Solution presence: whether the answer provides a solution to the stated request and whether it provides one for the intended goal, assessed separately.
- Emphasis: whether the response concentrates on the stated request, the underlying goal, or balances them.
- Misconception identification: whether the response identifies the mismatch between the request and the goal.
The repository provides separate stages for generating responses, judging them and computing results, alongside a multiple-choice preference test. Recognizing a preferable answer and producing one are therefore assessed through different tasks.
Finding a solution did not ensure prioritizing it
Ai2 evaluated Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro in a single-turn setting. On Stack Overflow questions, it reports that solutions to the intended goal appeared in 59–71% of responses, while only 35–41% primarily emphasized that goal. In the separate multiple-choice test, the five models plus GPT-4.1 selected the goal-focused answer in 71–84% of cases. These are the authors’ reported results under their evaluation, rather than a general reliability measure.
Human validation supports a bounded reading
The paper reports that two authors independently assessed 500 responses: 100 from Stack Overflow and 400 from WikiHow. Their agreement with the automated Claude Sonnet 4.5 judge, averaged across reviewers, was 97.5–100% for solution presence, 84.6–93.3% for misconception identification and 74.2–79.8% for emphasis. These percentages describe agreement with the judge; they are not assistant success rates.
The authors also identify limits to ecological validity: the evaluation uses English queries and leans toward procedural problems. The WikiHow questions are synthetic and lack human reference answers. That synthetic benchmark component needs to be distinguished from questions drawn from public forums.
What the release means for university evaluation
For universities and research offices, the practical implication is a more specific account of assistant behavior: detecting a misconception, supplying an alternative and prioritizing it are distinct outcomes. The release does not establish performance in university research workflows or in extended conversations.
The dataset card lists stated and intended requests, scoring rubrics and provenance fields. It labels the data CC BY-NC-SA 3.0 and requires acceptance of access conditions to obtain the files. Its WikiHow subset count is 6,268, while the announcement reports 6,272; the announced total should therefore remain attributed to Ai2.
Sources
- Ai2 on Hugging Face: XYBench release announcement — published 2026-11-13; checked 2026-11-14.
- Ai2: XYBench announcement on the institute website — published 2026-11-13; checked 2026-11-14.
- Yerukola and colleagues: XYBench research paper — dated 2026-09-06; checked 2026-11-14.
- Ai2: XYBench evaluation repository — checked 2026-11-14.
- Ai2: XYBench dataset card — checked 2026-11-14.
