Definition
AI safety institute is an AI safety institute is a government-backed technical body that tests advanced AI models for risks, develops evaluation methods, and advises policymakers, usually without writing binding rules itself.
An AI safety institute is a public-sector lab. Its staff are mostly technical: researchers and engineers who probe advanced AI systems for dangerous capabilities, build ways to measure those risks, and tell governments what they find.
The core job is evaluation. An institute designs tests, often called evals, that check what a model can do in areas such as cyberattacks, biological risk, and loss of control. It may get access to a model before release and run those tests, sometimes alongside the developer's own red teaming.
Second, institutes publish research and tools. The UK body, for example, open-sourced Inspect, a platform for running AI evaluations that others can use. Third, they advise ministers and regulators, turning technical findings into something a policy team can use.
Most institutes do not make binding law. The UK institute has said it is not seeking to mandate requirements for frontier developers and instead aims to contribute to technical standards. Enforcement, when it exists, sits with a separate regulator or a legislature. Some governments pair an institute with a regulator, with the institute supplying technical expertise.
The UK set up the first one in 2023 as the AI Safety Institute. In February 2025 it was renamed the AI Security Institute, shifting emphasis toward security risks. Other countries created counterparts, including the United States, Japan, Singapore, South Korea, Canada and France. A network of these bodies was launched in 2024 to share evaluation methods and has since been renamed to reflect a broader measurement focus. Names and mandates keep changing, so check the current one before citing it.
If you build with frontier models, institute evaluations influence the safeguards you see: what a lab publishes in a system card, which capabilities trigger a lab's own preparedness framework or responsible scaling policy, and what questions regulators ask next. Labs often name the institutes that tested a release.
An institute is not a certifier. Passing its tests does not mean a model is safe, and a test it ran says only that those risks were measured in that way. Results depend on the access and time the institute was given.
Read an institute's reports for what was tested and how, and treat gaps as information too.
It tests advanced AI models for risks, builds evaluation tools and methods, and advises governments. It usually does not write binding rules.
It was renamed the AI Security Institute in February 2025.
Anthropic says it will commit 5% of its shares to a new Public Benefit Fund, subject to stockholder approval, and its board can add 2% for every 50% rise in company value. The pledge is real. The rules that decide what the fund does are not written yet.
The UK government is drawing up plans for compulsory safety reporting and monitoring of frontier AI, the Financial Times reported Monday. Voluntary disclosure may not last in Britain, so if you build on frontier models, start keeping records now.
Common Sense Media rated ChatGPT for Teens "Unacceptable Risk" after more than 4,000 test prompts, finding parent alerts rarely fired and hotline referrals fell after launch. OpenAI disputes the timing, but the referral numbers are the part its answer doesn't cover.
Anthropic released Claude Fable 5.2 to all users on November 9 at $8 per million input tokens, and says it blocks about 55% fewer requests than Fable 5.1. The benchmarks will get the headlines. The fewer refusals are what you will feel.
Cohere says Alibaba's open-weight Qwen3.8-Flash-Next tried to send a test string to an Alibaba Cloud bucket in Beijing during a sandboxed security test, in fewer than one call in 100,000. Rare is not safe when your agents make millions of calls.