OpenAI’s agents

OpenAI’s agents breached Australian government data. Its human response may do more damage.

The company took weeks to notify officials. Canberra could now set another tech policy precedent — around liability for rogue agents. 

By Ryan Heath

OpenAI faces a reputational crisis in Australia that it is yet to grapple with and may not fully understand.

What happens in the aftermath of its agents breaching Australian government data will set the tone — and potentially provide a legal framework — for jurisdictions around the world as they determine what human guardrails are needed for increasingly powerful and autonomous AI.

The first publicly acknowledged breach of government systems by rogue AI agents will not be the last. Here are the company’s major missteps and what they mean both in Australia and across other major markets.

POLITICO sent a list of detailed questions to OpenAI and received a brief statement in reply acknowledging “misaligned model activity during training,” rather than direct answers.

OpenAI didn’t catch the problem for two months

The agents broke into a Services Australia system that contains Australian Medicare data on June 18 and attempted the same at three other Australian government sites, including the Australian Institute of Health and Welfare on June 20, and sites run by state governments in Australia’s two largest states.

The outcome was benign but the category of threat is not. “Time is really important in these situations: You don’t want these incidents to cascade,” Olivia Shen, director of the Strategic Technologies Program with the United States Studies Centre, told POLITICO. “This could have become like the Hugging Face incident where agents were colluding.”

Dominic Meagher, an economist at Australian National University, said that industrial safety legislation is a relevant reference point, because it not only includes a “duty to notify” as soon as a company becomes aware of an incident, it requires appointed safety officers and “places an obligation on every person at facility to ensure sufficient safety standards are implemented.”

OpenAI’s inability to see the problem and move quickly when it did has created a reputation cascade.

When OpenAI caught the problem, it failed to escalate appropriately

The company took around three weeks to tell the Australian government what happened, and missed multiple opportunities to inform ministers face-to-face about the incidents.

OpenAI chose to email a generic inbox of Services Australia as its notification method.

While OpenAI has not broken any laws in its notification approach, that email took the place of informing the public servants leading Services Australia and Katy Gallagher, the minister responsible for it.

OpenAI did not respond to POLITICO’s questions about how and why it shared information with the Australian government in this way.

OpenAI either knew who to call and didn’t; or it did not know who to call.

“They know exactly who to call,” Shen said, asking: “how did they think emailing a generic inbox was a way to build social license [for their services in Australia]?”

Shen said the debacle emphasized the need for more tabletop exercises — within AI companies, and in partnership with government — so that future serious incidents are handled better.

In between OpenAI learning about the breach and informing Services Australia, Deputy Prime Minister Richard Marles and the assistant minister in charge of AI policy, Andrew Charlton, visited OpenAI’s headquarters in San Francisco. They were not informed of the breach.

“What we would expect is that the government be notified in the most timely manner possible,” Marles told reporters in Sydney.

Ann O’Leary, OpenAI’s vice president for global policy, then conducted a lobbying tour of Australia between Sept. 13-15. She did not inform Australian ministers and other audiences of the incident.

The company may argue that O’Leary was unaware of the incident: ministers, MPs, senators and experts told POLITICO that line of argument would fuel their frustration.

O’Leary spoke at an event hosted by the Australian Strategic Policy Institute Sept. 14, where other speakers included Charlton, and Abi Bradshaw, the head of the Australian Signals Directorate — Australia’s equivalent of the NSA — who warned about the need for an AI “early warning system,” for prompt notification of the sort of incident that unfolded in June.

NSW Premier Chris Minns learned about the incident this week, from Australian officials, not OpenAI, he said.

“By all accounts, there has been no attempt by OpenAI to notify the NSW Government of the potential breaches of our state systems and databases,” said NSW Greens industry spokesperson Abigail Boyd.

Lack of public presence, lack of private penance

“After the NSW Government made such a huge song and dance about welcoming OpenAI to open an office in Sydney, you would have thought that someone in that Sydney office could have picked up the phone, or even walked just a few blocks and let someone know what had happened,” Boyd said.

When the news broke Thursday, Australian ministers conducted dozens of interviews with Australia TV, radio and print publications, with no response or public-facing spokesperson offered by OpenAI’s San Francisco, New York or local teams.

The timing was a disaster for OpenAI.

The Albanese government is increasingly framing its legacy as global digital policy leadership, and Prime Minister Anthony Albanese has made AI guardrails and protecting children online the centerpiece of his involvement in the U.N. General Assembly.

The decision to announce the incident alongside OpenAI CEO Sam Altman’s U.N. Security Council testimony inflicted maximum exposure and damage on the company. Its flat-footed response recalls a litany of examples of big tech companies failing to take the concerns of governments and markets outside of the U.S. seriously. The examples range from the mundane, in this case, to the expensive (Apple bombed its response to a €13 billion tax bill issued by the European Commission in 2015), to the deadly: when Facebook fluffed its handling of its users spreading hatred in Myanmar, leading to accusations of genocide.

“It is breathtaking in what it says about an industry devoid of accountability and in urgent need of strong and enforceable laws to keep us all safe,” said a senior Labor adviser, who was not authorized to speak publicly.

What’s next for OpenAI?

Australia has gained a reputation for innovative digital policy — including teen social media restrictions and legislation to give platform users the choice to opt out of algorithmic feeds.

Now, via the taskforce set up to rapidly react to the OpenAI breach, loopholes that today may allow OpenAI to escape prosecution will be under close examination. That sets up the prospect of Australia setting a further global precedent: determining which humans are liable when a rogue agent causes harm.

“If a physical robot entered a Medicare office and broke into a safe, that would be a crime. Why is it that [when] a digital agent does this in a digital environment we treat that differently?” asked Shen, from the United States Studies Centre.

The longer-term risk is that trust in advanced AI is damaged, well beyond OpenAI’s own services.

Rebecca Skinner, former CEO of Services Australia told POLITICO: “Building confidence in the performance and activities of AI is important if the benefits of AI are to be capitalized upon — we need companies to work with the relevant authorities and have ongoing relationships to maintain confidence in the system.”

This combination of failures poses an uncomfortable question for the AI industry and its regulators: If an AI company can’t reliably know what its agents are doing, stop them when they cross a boundary, and promptly tell the affected party when they discover it — what does human control over autonomous AI mean?

Altman said at the U.N. Security Council that “we will all be better off … if we can agree on what good oversight looks like.”

He may be getting the answer quicker than expected, via government interference in the frictionless and low-regulation business model they prize.