Definition
MATH-500 is MATH-500 is a math benchmark of 500 competition-style problems, a random sample from the MATH test set, used to score how well AI models solve multi-step math and show their work.
MATH-500 is a standard test for AI math ability. When a lab says a model scored a certain percentage on MATH-500, it means the model solved that share of 500 competition-style problems correctly.
The problems come from MATH, a dataset of competition math questions. For their 2023 paper "Let's Verify Step by Step," OpenAI researchers held out 500 problems from the MATH test set for evaluation. The project's repository says those 500 were selected uniformly at random and are believed to be representative of the whole test set. The name "MATH-500" came into common use afterward for that same subset.
Because the sample is random, it covers a mix of subjects and difficulty levels. It is not the 500 hardest problems.
Each problem has one correct final answer. A model is given the question, writes out its solution, and the final answer is compared with the key. The score is the percentage correct. A model that gets 250 right scores 50%.
Because only the final answer is checked, a model can be right for the wrong reasons. Scoring rules, such as how answers are matched or how many attempts a model gets, can also differ between labs, so scores are only comparable when the setup is the same. See pass@k for one common variation.
Math is useful to test because answers are checkable and need several steps of logic. It became a favorite for measuring reasoning models and the effect of chain of thought, where writing out steps tends to improve accuracy.
Like any AI benchmark, treat it as one signal. It is most useful for comparing small models, or for checking whether a change helped, and least useful for ranking the top systems.
GSM8K covers easier grade-school word problems and is often reported alongside it.
No. MATH-500 is a random 500-problem subset of the MATH test set, which is much larger.
MIT CSAIL researchers say a frontier LLM, used as is, read and rewrote neural network weights without gradient descent, lifting a small Qwen model's MATH-500 score from 7.9% to 48.1%. It is an early blog post, not a peer-reviewed paper, and still the strangest result of the week.