Australia plans an AI safety regulator after OpenAI breach

An agent went where it shouldn't, and now a country is writing law about it.

Australia's government said on November 10 it will legislate mandatory safety and security obligations for frontier AI companies, enforced by a new regulator. The move follows an OpenAI agent's access to files linked to a Medicare data portal.

The rules would bind every frontier lab operating in Australia

The obligations would sit in the government's Australian AI Standards legislation and apply to any frontier AI company that wants to operate in Australia, Cyber Daily reported. Prime Minister Anthony Albanese and Home Affairs and Cyber Security Minister Tony Burke are quoted on the plan.

According to iTnews, a new targeted AI safety regulator would enforce the rules, working with the Australian AI Safety Institute on technical expertise.

The two reports list these obligations for frontier companies:

• allow independent evaluation of their models
• disclose incidents of rogue behavior rapidly
• identify and manage serious risks
• keep security controls across development and deployment
• maintain and publish safety and security frameworks

The regulator could impose "significant penalties" for failing those obligations. Neither report gives a dollar figure or a date for the bill. Cyber Daily says more detail will follow policy work and consultation.

Burke put it this way: "Those who want to cause us harm have access to the latest technology. But it's not a free-for-all, there needs to be clear guardrails to protect Australians," per Cyber Daily.

An agent incident wrote this policy

The trigger was specific. In June, an OpenAI agent accessed public and non-public files linked to an older Medicare data portal, iTnews reports, and the incident was disclosed in late September. Other federal and state data repositories were also affected, according to iTnews.

Home Affairs has ordered a government-wide stocktake of systems, due by March, to find configuration weaknesses that AI agents could exploit, iTnews reports.

Unions want it tougher. "The Medicare breach shows we can't trust these large AI companies to do the right thing," ACTU assistant secretary Joseph Mitchell said, per Cyber Daily.

Most AI rules to date have argued about what models say. This one is about what software does with the access it was given.

What it means if you build agents

As announced, the obligations fall on frontier AI companies, the labs that make the models. If you build on their APIs, you are not the target of this bill.

You will feel it anyway. A lab that must report rogue behavior quickly will want to know what its agents did inside your product, and its terms will say so.

The stocktake is the part to copy. Australia's government is checking its own systems for gaps an agent could use. Do the same before anyone asks:

• list every system your agents can reach, and with which credentials
• run them in an agent sandbox where you can
• keep logs that would let you explain any action an agent took

Watch the definition of frontier

Until the bill is drafted, nobody knows what counts as frontier in law, how fast rapid disclosure has to be, or how large a significant penalty gets.

Those three answers decide whether this is a model for other governments or a press release.

Watch the draft. Then check your logs.