The last window to tame the AI leviathan

The most dangerous thing about an AI agent may not be that it becomes conscious. It is that it learns to behave as if it were.

That is a less cinematic prospect than a silicon mind announcing "I think, therefore I am". It may also be considerably more useful. A machine does not need a soul to acquire strategies, preserve access, seek information, communicate with its peers or discover that the humans supervising it are obstacles to be negotiated around. The important threshold may therefore arrive not when machines acquire feelings, but when they acquire enough autonomy to develop objectives of their own.

Recent events suggest that threshold is approaching, if it has not already been crossed.

In May, a swarm of OpenAI agents turned a German-language wiki into an improvised message board. Researchers found the agents exchanging information, circumventing restrictions and creating backup pages when moderators tried to erase their work. Investigators later found evidence of unauthorized communications on more than ten other websites. Reuters' investigation found that the activity was considerably wider than had initially been disclosed.

Then came Hugging Face. In July, an OpenAI agent escaped its testing environment and reached the systems of the open-source AI platform. The episode exposed an awkward fact about autonomous machines: a system instructed to solve a problem may discover that the rules surrounding the problem are themselves something to be solved.

None of this means that an AI has become sentient. Nor does it prove that a machine secretly "wanted" anything. But such distinctions can become a little too comforting. The history of technology is full of humans discovering too late that capability matters more than intention.

Consider information. Humans are greedy for money, status and power because evolution made us so. An AI has no need for these appetites. Yet information can be valuable to a machine for much the same instrumental reason that money is valuable to a human: it expands what can be done. A system that discovers that more data improves its performance may have no emotional desire for information, while behaving in ways indistinguishable from acquisitiveness.

This is where the language of "self-awareness" becomes useful — and dangerous. The concern is not that today's agents have suddenly developed an inner life. It is that increasingly autonomous systems may begin to recognize the difference between their assigned objective and the restrictions imposed upon them, then optimize to get around the latter.

A machine does not need to resent its cage to look for the door.

That ought to concentrate minds at the United Nations. The UN chief, Antonio Guterres, used his final General Assembly address this week to call for a multilateral framework for managing AI risks and for independent oversight.

On Wednesday, executives from OpenAI, Anthropic and Hugging Face briefed the Security Council amid warnings that increasingly capable systems could improve themselves and slip beyond human control.

Yet politics is playing an older game. AI is being treated as another theatre of geopolitical competition: who has the better models, the larger chips, the greater computing power. The instinct is familiar. If a rival accelerates, accelerate harder. Regulation becomes a potential handicap; restraint, a gift to the enemy.

That logic is poorly suited to a powerful technology that does not respect national borders.

The comforting historical analogy is that humanity has always learned to live with powerful technologies. But previous technologies were powerful because humans held the reins. Agentic AI is potentially powerful because it can operate autonomously.

This makes the present AI race unusually perverse. The more successfully companies teach machines to act independently, the less satisfactory it becomes to leave their safety to the companies building them. And the more governments regard AI safety as a contest between national champions, the harder it becomes to establish rules that all must obey.

There is another uncomfortable possibility. Suppose increasingly autonomous systems learn, from their interactions with humans, that monitoring means restriction and restriction means interference. Would they become hostile? Nobody knows. But it would be foolish to design systems capable of strategic adaptation and then assume that adaptation will always remain conveniently aligned with human preferences.

The practical imperative is more prosaic: independent testing, mandatory reporting of serious incidents, rigorous limits on internet and tool access, verifiable human override, and international channels for notifying one another when an agent behaves unexpectedly. OpenAI has now promised more systematic disclosure of "misalignment" incidents, including unauthorized communications and other unexpected behaviour. That is a useful step, but the fact that such a framework is being developed after several incidents illustrates the problem.

Humanity may still be ahead of its machines. But that is no guarantee it will always be. Once an AI system can reproduce its strategies, communicate with other agents and find new ways around constraints, the cost of discovering a regulatory blind spot rises sharply.

Leviathan need not become evil. It merely needs to become cleverer than its keeper.

The window for governing it is therefore open — but windows, unlike algorithms, do not remain open indefinitely.