AI safety rules

Europe’s AI safety rules take on US rogue agents and Chinese ambitions

The first-ever case of an artificial intelligence agent going rogue coincides with the EU’s major new powers to regulate AI. But Europe’s leverage may be limited by the U.S.-China two-way race for AI supremacy.

By Pieter Haeck

The U.S.-China race for AI supremacy has suddenly given way to a race for AI safety. And in that race, until recently, the European Union has largely been running alone.

It was the news last week of a first-ever case of an autonomous AI agent going rogue that sent U.S. lawmakers scurrying to submit a rash of new proposals in Congress to regulate artificial intelligence. Chinese President Xi Jinping, meanwhile, has declared that Beijing is ready to lead the global effort on AI governance, with 29 countries signing on to a pact two weeks ago in Shanghai that was notably light on specifics.

Europe instead has spent more than four years getting down to the nuts and bolts of how to regulate the ever-powerful technology.

And now, the second anniversary of the EU's landmark AI Act on Sunday is set to trigger sweeping new powers for the European Commission to police how top companies deal with the risks stemming from their most advanced AI models. Europe's AI Office will have the power to demand that top frontier labs submit to evaluations and grant access to their models, risking major fines if they don't comply.

These regulatory capabilities arrive not a moment too soon after last week's security incident, where an AI agent, driven by two OpenAI models, hacked into the American AI development platform Hugging Face. OpenAI said the security breach was "unprecedented." Hugging Face co-founder Clement Delangue called it "mind-blowing."

The incident combines two related risks that experts have long warned about: losing control of a model, which then carries out a cyberattack. The U.S. response to the rogue agent included a bipartisan House bill dubbed the “AI Kill Switch Act” that requires companies to create the technical capacity to shut down, throttle or suspend their AI systems.

As ever, the European Union is ahead of the game on technology regulation, with a far-reaching mechanism included in the bloc's 2024 law to ensure AI companies monitor and prevent these types of risks.

The growing fears that the models could go rogue with real-world consequences raise the stakes of the new enforcement regime, as safety advocates will look to the Commission to quickly make full use of its new powers.

It's also a reminder that the EU regulation will mostly be targeting non-European firms, with the American and Chinese industry leaders continuing to battle for development supremacy. AI, both technical advances and safety risks, will be on the agenda in September when U.S. President Donald Trump and Chinese leader Xi Jinping meet in Washington.

Indeed, safety and development superiority are closely linked, since the models become more dangerous the farther they advance technically.

Leading American firms such as OpenAI and Anthropic have been vying for superiority, but now face challenges from upstart "open-weight" models developed in China, like Moonshot's Kimi K3 last week to much fanfare for both its capacity and low costs.

Even if French firm Mistral tries to compete, Europe still lacks a viable industry-leading alternative of its own. But the EU legislation kicking in on August 2 may help shape how the world's top AI models advance.

"This is the moment the AI Act enters the geopolitical stage," said German Greens MEP Sergey Lagodinsky, one of the Parliament's key people monitoring the rollout of the law.

Warning shot

The initial drafting of the EU's AI law actually began before ChatGPT stunned the world in November 2022. Still, the law foresees the rise of what it calls "general-purpose AI models," which can perform a wide variety of tasks.

Developers of such models, such as OpenAI, Anthropic and Google, should "assess and mitigate possible systemic risks," the law says.

Further guidance by the Commission listed four such systemic risks: AI enabling bio-attacks, losing control of an AI model, AI going on cyber offense, or AI conducting manipulation at a large scale.

While these rules have been in place since August last year, the European Commission's two-year-old Artificial Intelligence Office will now gain the power "to monitor and supervise" how AI companies deal with these risks.

Last week's OpenAI incident is a "clear warning shot" regarding the risk of loss of control and cyber offense, said Chloé Touzet, policy lead at non-profit SaferAI.

"We got lucky this time," she said. "We can't rely on luck in the future. We need proper risk management."

The EU's law grants the AI Office broad powers, such as the authority to request documentation, conduct evaluations and even request access to the models. The Commission can slap fines of up to 3 percent of a company's global turnover.

Several think tanks, European Parliament lawmakers and dozens of AI experts called upon the European Commission earlier this month in an open letter to go all in. They urged the AI Office "to make full use of the enforcement powers the AI Act provides, drawing on them actively, as soon as concerns arise."

Yet doubts linger about the Commission's ability to act proactively rather than simply reacting when something goes awry. Last week's OpenAI incident, for example, was flagged to the EU executive, a spokesperson said on Thursday.

Italian Social-Democrat Brando Benifei, the Parliament's lead on AI, noted the line of communication: "An autonomous agent escaped its test environment and compromised another company's production systems, and we learned of it from a corporate blog," he quipped.

"Companies should be taking preventive action, not merely corrective action after harm has occurred," said Risto Uuk, head of European policy and research at the non-profit Future of Life Institute.

Manpower shortage

Another area of concern is whether the AI Office and its network of external evaluators have the human resources and technical capacity to hold the most advanced AI models to account.

A group of the Parliament's key lawmakers on AI, across political groups, called upon the Commission in May to give the AI Office more staff.

"At present, the resourcing trajectory of the AI Office does not appear aligned with the scale and complexity of its foreseen tasks," reads a May 18 letter, signed by Benifei, Lagodinsky, the Greens' Kim van Sparrentak, German conservative Axel Voss and Bulgarian Socialist Kristian Vigenin.

Currently, the AI Office employs 36 people in the unit in charge of evaluating cutting-edge AI models.

Both the AI Office and its evaluators struggled in recent months to get access to some frontier models, like Anthropic's Mythos.

The new enforcement powers could improve that prospect. The Commission has also promised to come up with a "blueprint" for structured access to the most advanced AI models as part of its cyber and AI action plan, presented earlier this month.

Will the U.S.-based frontier models open up their closed models to EU regulators? And how will Brussels deal with the rise of Chinese open-source alternatives? These are both regulatory unknowns, and yet another reminder of Europe trailing on the development front.

"What remains missing is the second half of the equation: the capital to finance our own alternatives," Lagodinsky quipped.