CoolingResearch
Ai2's XYBench tests whether AI answers the real goal behind a flawed question
Ai2 published XYBench on Nov. 13: 8,115 questions testing whether assistants catch a user's mistaken premise and address the underlying goal. On Stack Overflow questions, models solved the stated request in 75 to 86% of answers but the underlying goal in 59 to 71%. Ai2 says humans beat all five models tested.
- HEAT
- OUTLETS
- 0
- FIRST SEEN
- LAST SIGNAL
Primary source
Hugging Facehuggingface.co/blog/allenai/xybenchCoverage · 1 article, oldest first
Friday, November 13, 2026
Hugging FacePrimary
XYBench: Helping with the problem behind the question