Definition
Terminal-Bench is Terminal-Bench is a benchmark that tests AI agents on realistic command-line tasks, where each task runs in its own terminal environment and is graded by automated tests.
Terminal-Bench is a way to measure how well an AI agent can do real work at a command line, the text window where developers install software, run programs and fix things by typing commands.
Each task in the benchmark comes with its own terminal environment, a human-written reference solution and automated tests. The agent is dropped into the environment and given a goal, such as setting up a service, repairing a broken build or processing a data file. It types commands, reads the output and tries again until it thinks it is done. Then the tests check the result. If the tests pass, the task counts as solved. If not, it does not, however reasonable the attempt looked.
The tasks are drawn from workflows people actually do, which is the point. The authors of the ICLR 2026 paper describing version 2.0 say earlier command-line tests were often too narrow or too easy for the strongest models. Version 2.0 has 89 tasks, and the paper reports that frontier models solved fewer than 65% of them. The dataset and the code that runs it are public at tbench.ai.
Most coding benchmarks hand a model a small, well-defined problem. Real work is messier: you have to explore an unfamiliar machine, notice what is missing and recover from your own mistakes. Terminal-Bench is closer to that. If you are choosing a model for an agentic coding tool or an AI agent that operates a computer, a terminal score tells you something a pure code-completion score does not.
An analogy: SWE-bench is like asking someone to fix one bug in a codebase you have already opened for them. Terminal-Bench is like handing them a fresh laptop and a to-do list.
Treat any single number with care, as you would with any AI benchmark.
Terminal-Bench belongs to a family of tests for agents that act, rather than just answer. It overlaps with computer use, which controls a graphical screen, while Terminal-Bench is text only.
Terminal-Bench is a benchmark of realistic command-line tasks for AI agents. Each task has its own environment and automated tests that decide whether the agent succeeded.
SWE-bench asks a model to fix a bug in an existing code repository. Terminal-Bench puts an agent in a terminal environment and asks it to complete a wider range of command-line work.
DeepSeek released V4.1-Flash-1017 under the MIT license on Oct. 17, and says it scores 77.0 on DeepSWE v1.1, up from 74.2, at an unchanged API price. The model is a modest step. The $0.75-per-1,000 search API, free in Claude Code and Codex until Nov. 1, is the part to try.
Moonshot AI released Kimi K3.1 Pro on Oct. 23, a 3T-class model with a 1M-token context, and says full weights arrive by Nov. 4. If the weights ship as promised, the gap between open and frontier is about to get small.
The Nanbeige team released Nanbeige4.3-33B under Apache 2.0, an agentic coding model it says leads rivals near its size on most benchmarks in its own table. The numbers are self-reported. They are good enough to earn an afternoon of your evals.