Definition

What is Pass@k?

Pass@k is Pass@k is a benchmark score that gives an AI model k attempts at each problem and counts the problem as solved if at least one attempt passes the tests, so pass@1 means it must succeed on the first try.

What pass@k measures

Pass@k is a way to score an AI model on tasks that have a checkable answer, most often code that must pass unit tests. For each problem, the model gets up to k tries. If any one of them passes, the problem counts as solved. The score is the share of problems solved this way.

The k matters a great deal. Pass@1 asks whether the model gets it right the first time, which is how most people use an assistant. Pass@10 asks whether the right answer appears somewhere in ten tries, which says more about what the model can do with a human or a test suite picking the winner.

Think of a basketball player at the free-throw line. Pass@1 is one shot. Pass@10 is "did you sink at least one of ten?" Both are real numbers, but they answer different questions.

How it is calculated

The metric was popularized by the 2021 paper Evaluating Large Language Models Trained on Code, which introduced the HumanEval coding benchmark. Running exactly k samples per problem gives a noisy result, so the authors generate more samples, n, per problem and count how many, c, pass. They then use an unbiased estimator: 1 minus the chance that k samples drawn from the n would all fail. A simpler shortcut, raising the single-attempt failure rate to the power k, consistently underestimates the true score.

You rarely need the formula. You do need to know which k a published number uses.

Why it matters when you read results

A lab can report a high pass@10 or pass@100 and look strong, while the number you will actually live with is closer to pass@1. When an AI benchmark result does not say what k it used, ask.

Pass@k also hides cost. Ten attempts means ten times the tokens and more time, plus you need a reliable way to tell which attempt was right. For code, tests can do that. For writing or analysis, nothing automatic can, so a high pass@k on those tasks is far less useful.

Some benchmarks report a variant, such as the share of problems solved on every attempt. That measures reliability instead of best-case ability, and it is usually much lower.

Pass@k is one scoring rule used inside evals, and it appears in coding benchmarks like SWE-bench and agent tests like Terminal-Bench. When you run your own evals, record how many attempts each task gets, and decide up front whether your real workflow allows retries.

Questions people ask

What is the difference between pass@1 and pass@10?

Pass@1 counts a problem as solved only if the model's first attempt passes. Pass@10 counts it as solved if any of ten attempts passes, so it is always equal to or higher than pass@1.

Who introduced pass@k?

The pass@k metric and the HumanEval coding benchmark were introduced in the 2021 paper Evaluating Large Language Models Trained on Code.