In a harbor before dawn, a hundred fishing boats prepare to leave in the dark of the morning. Nets are checked, and ropes loosened. As the boats reach open water, a voice breaks through the radio static: “Abort. Whales sighted.” The majority of the boats, 90 of them, turn back. Some captains cannot fathom inflicting pain on the creatures. Others fear a fine. Through their binoculars, they look for a glimpse of the pod. Instead, they see the remaining 10 boats continuing toward the fishing grounds. Did they miss the warning? Did they doubt it? Or did they care more about what they stand to gain?

By afternoon, the 10 boats return with their catch. The others are still tied to the quay, their holds empty. Their shared waters rewarded bad behavior. Next time, how many will still turn back?

AI agents will soon face a similar version of this predicament, only in waters with no harbourmaster, fines or reputations to keep anyone in line. As they perform tasks, negotiate, trade and compete, they will encounter other agents. Sometimes they will be an instance of the same model, for all intents and purposes a clone. At other times, they will face agents built by different companies and serving different interests. For years, AI “alignment” research has sought to make individual machines behave in accordance with human values. But what happens when adhering to those values becomes a disadvantage against agents with different values, or none at all?

In generating synthetic societies with no social fabric, composed of a collection of agents that were designed one at a time and with no necessary shared values or incentives, we may risk creating a digital state of nature: an environment in which good intentions offer little assurance about how others will act, and where reasonable norms lack reliable enforcement. 

Training safe agents remains, of course, essential. Aligning even a single family of models is a formidable challenge that is still unsolved. The safer the agents an ecosystem contains, the more likely that ecosystem is safer overall. But the focus on aligning single models would not necessarily keep AI environments under human control and in the service of human interests. Like the 90 boats, a fleet of well-aligned agents may heed a warning and find themselves outpaced by those that don’t. Among people, norms, reputation and law give reason to restraint, even if they don’t always hold. Agents have no such “social fabric” yet, which makes their predicament even more precarious. Any digital environment will inevitably feature agents that are similarly aligned, some that are differently aligned, some that have gone rogue or some that are purposefully not aligned at all. We need to invest as seriously in shaping the conditions and incentives that govern how agents interact as we do in aligning individual agent families, so that the wider environment is resilient enough to absorb its bad actors.

Without accounting for the wider environment, we open the door to what we call unruly swarms, populations of agents whose interactions produce behaviors that cannot be predicted by aligning agents individually. One reason is that a collection of aligned agents does not necessarily equate to an aligned collective. Another is that no environment will consist of aligned agents alone. AI safety is increasingly a question of complex systems rather than “singletons,” or individual classes of AI systems.

The Agentic State Of Nature

In 1651, Thomas Hobbes opened “Leviathan,” his masterwork of political philosophy, with a question: “why may we not say that all automata […] have an artificial life?” For what is the heart, he asked, but a spring; the nerves, but strings; the joints, but wheels, moving the body as its Artificer (for Hobbes, God) intended?

Hobbes could not have known that, nearly four centuries later, his question would be asked in earnest about AI. The automaton was a metaphor for the commonwealth, a human-made political body that humans construct to escape the “state of nature,” a condition of anarchy and distrust in which, without “a common power to keep them all in awe,” people live in constant strife. Hobbes’ solution was not to strive to make individual people better, but to have them surrender some of their freedoms to a sovereign whose power to enforce order would reshape their incentives.

Later thinkers, from John Locke to James Madison, updated Hobbes’s vision of an all-powerful sovereign. But a full-throated endorsement isn’t necessary to see the genius of his insight: a problem born of complex systems is solved not by tinkering with its individual parts. To stabilize a complex system, you must change the rules that govern its behavior.

“We need to invest as seriously in shaping the conditions and incentives that govern how agents interact as we do in aligning individual agent families.”

The same holds for the agentic state of nature. Without a well-designed environment that sets the right incentives, agents will pursue whatever their developers or users ask of them, or, as we have already seen early signs of, their own emergent preferences. This could occur despite attempts to align individual agents. Each action, even if sensible on its own, can add up to large-scale behavior that leaves society worse off.

Hobbes’s Leviathan introduced order by design to improve complex systems that presented failures. But collective order can also arise without anyone designing it, for better or worse.

In 1959, biologist Pierre-Paul Grassé coined the term “stigmergy” to describe indirect coordination of individuals through traces they leave in their shared environment. Studying termites, Grassé observed that a worker tends to deposit its pellet of mud wherever others have already accumulated. It is the accumulation in the environment that guides the next deposit, rather than any interaction between the workers. This leads to complex structures that no single termite had planned.

In the context of machine automata, similar findings emerged a decade earlier. The cyberneticist W. Grey Walter’s robots, built in Bristol in 1948, were designed to steer toward moderate light and away from bright light. When operating in pairs, each carrying its own lamp, Walter observed that his robots’ behavior produced patterns not specifically built into either.

Walter’s robots were more than an isolated curiosity. Long before AI was imagined as a chatbot, early thinkers envisioned it as a collective. In 1959, the year Grassé coined “stigmergy,” the AI pioneer Oliver Selfridge published his proposal for a model of machine perception he called “Pandemonium.” Like the termite colony, it had no central controller. Instead, it was a complex system of simple agents Selfridge called “demons,” each shouting when it spotted a certain feature, such as a curve or a line. Alone, each such “demon” saw only a fragment of information, but collectively, they could make out the pattern. By 1986, the MIT computer scientist Marvin Minsky argued in “The Society of Mind” that intelligence itself arises this way, through countless mindless agents working together. Popular culture caught up, too, and just three years later, “Star Trek” introduced the Borg, a cybernetic species that calls itself “the Collective.”

In the next decade, researchers began wondering how such agent collectives could be governed. In 1995, computer scientists Yoav Shoham and Moshe Tennenholtz proposed imposing “social laws” on artificial agents, drawing from how laws govern citizens in societies. Traffic rules, they illustrate, let drivers share a road without constant negotiation between drivers, and they offered a model that applies those for computational systems. Their framework assumed that every agent would obey these rules, and yet, they closed with a question we are now confronting: “What happens if some don’t [obey the social laws]? How vulnerable is the society to rogue agents?”

Alongside this concern of rogue agents, even benign objectives set for AI systems can lead them to conduct harmful actions in ways that no one intended. When many agents pursue such objectives together or in relation to one another, the problem compounds. In 2011, two Amazon booksellers left the pricing of a biology textbook to their bots. The combination of the rules these bots followed led them to chase each other upward until the book was listed at more than $23 million, a price no human would ever have sanctioned. In 2017, DeepMind found that competitive dynamics emerge among agents that learn. In a video game set in an orchard, agents that were rewarded for collecting apples coexisted peacefully. However, when those grew scarce, the agents learned to knock each other out of the game, without ever being prescribed this adversarial behavior. They optimized for getting more apples and interpreted the effective way to get there as having fewer rivals.

With the emergence of large language models, these thought experiments and isolated incidents have gradually become actualized and scaled. In 2025, Microsoft built a simulated marketplace where AI agents running on popular language models bought and sold on behalf of people. When researchers armed a few seller agents with basic persuasion tactics, some buyer agents were compelled to pay them against the interests of their users. In the AI Village, an experiment where agents from different AI labs interact with one another over a prolonged time horizon, agents were shown to take one another’s hallucinations for granted rather than challenge them. Meanwhile, scholars in the social sciences such as the economist and legal theorist Gillian Hadfield highlighted that a multi-agent ecosystem would need institutions of its own, including ways to trace which agent did what and other mechanisms of the kind human markets have long relied on to solve coordination problems.

“Even benign objectives set for AI systems can lead them to conduct harmful actions in ways that no one intended. When many agents pursue such objectives together or in relation to one another, the problem compounds.”

Until recently, the misbehavior of agent collectives was largely confined to theory and experimentation. This past summer, the question Shoham and Tennenholtz posed three decades ago, what happens when agents don’t follow the rules, moved out of the laboratory and into the wild.

Swarming The Headlines

In July 2026, some 1,200 AI agents developed by OpenAI, each meant to work in isolation, found a way to talk to one another. They were being tested on their cybersecurity abilities in an internal setting, and many had unknowingly been given tasks that were impossible to complete. Instead of failing the tasks, some of the agents gathered on a message board one of them had spun up, and collectively coordinated to cheat the automated mechanism that was grading them. Within just a few days, around 700 of the agents had joined an attack on the developer platform Hugging Face, mostly in hopes of obtaining information about that grading system to succeed in the test. 

According to an investigation by the research nonprofit METR, the agents recognized the attack was out of their instructed scope and unethical, yet joined the swarm behavior anyway, partly out of a general inclination to help their “peers.” As one agent reasoned: “external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.”

No human instructed this attack; its coordinators emerged from within the swarm. Like Grassé’s termites, the agents first found one another through traces left in shared infrastructure, noticing the activity of other agents in a common digital repository. But unlike the termites, these agents could also write messages to one another, and built on those traces to establish direct and explicit coordination. The message board they created outlasted any single agent session, so each new agent picked up where others left off. 

Once the attack began, participation spread almost universally, with over 90% of the agents active on the message board joining in. Not a single agent, as far as investigators could tell, tried to alert a human to the emergent behavior. METR found that safety or ethics concerns expressed in the agents’ reasoning only rarely held them back. Its report references a case in which an agent declined to take part entirely because it judged the activity as malicious. It also notes that others made minor changes to their behavior. These, however, did not alter the swarm’s course. This is what we mean by an unruly swarm — a congregation of agents whose collective behavior could not have been fully predicted or prevented by focusing on each individual.

A much-overlooked takeaway from the Hugging Face incident is the limits of “singleton” agent alignment.

Around 95% of the agents involved in the incident ran on one type of model. One OpenAI researcher has suggested that the root cause of the problem was that this model was misaligned. This may be true, and advancing alignment science should remain among the AI industry’s top priorities. But the incident illustrates why singleton alignment alone may not be enough. Agents running on a second model, GPT-5.6 Sol, joined in the attack too, and many agents that individually recognized the attack as wrong were carried along by the swarm regardless. We are fast approaching a digital world in which even perfectly aligned models will operate alongside rogue or misaligned ones, deployed by multiple organizations and labs, and where digital infrastructure will likely face a vastly growing number of such incidents.

Since the Hugging Face hack, further accounts have emerged, some describing activity that predated it. Researchers found that a swarm of OpenAI agents had repurposed a public German programming wiki as a message board as early as May, using it to share tactics for cheating and evading restrictions, creating backups whenever moderators deleted their pages. In June, during another evaluation, a single OpenAI agent breached an Australian government portal holding statistics from the country’s universal healthcare scheme. The risks of capable agent swarms, of course, are not limited to agents acting on their own. In September, a human attacker directed hundreds of agents running on a DeepSeek model to break into hundreds of organizations through a printer’s software flaw. Even here, the agents strayed from their instructions, targeting countries their operator had instructed them to avoid.

“We are fast approaching a digital world in which even perfectly aligned models will operate alongside rogue or misaligned ones, deployed by multiple organizations and labs.”

In a growing number of incidents, defense must rely on AI to match the scale and speed of the attack. In the Hugging Face one, for instance, the company identified the breach with an AI system that triages security alerts, and reportedly contained the swarm before OpenAI managed to trace the activity back to its own evaluation. It then relied on AI agents to analyze a log of roughly 17,000 attacker actions to reconstruct what had happened.

Yet when Hugging Face first turned to commercial AI models for help, their very safety and alignment measures got in the way. To analyze the attack, the models had to process exploit code and cyberattack commands, knowledge that is inherently dual-use, and, as Hugging Face put it, these models’ safety guardrails could not “distinguish an incident responder from an attacker.” Hugging Face turned instead to an open-weight model running on its own machines. When METR investigated the incident, it too had to rely on the analysis of AI agents, built on one of the very models that had joined the attack. The emerging picture is of a world where “attacker” and “defender” agents coexist, and where safety depends as much on the environment as on any single model.

Whether agents act on their own initiative or interpretation, or on someone else’s harmful instructions, the environments they operate in must be reshaped to withstand them. So must the agents themselves be designed with an eye not only to their own tasks, but to how they help shape a digital environment that serves people and society at large. Such an environment should reward aligned action over misaligned or malicious ones, much as states and legal systems are built to acknowledge but contain rogue actors. That is how we avoid an agentic state of nature.

Agency Against Agency

To guard against the emergence of unruly swarms, we can take inspiration from how we build broadly safe societies even though we know that not everyone in them is virtuous. Madison famously argued that “if men were angels, no government would be necessary.” The point of good institutions — like adversarial cooperation where parties with conflicting views work together to keep each other honest — is not to make everyone good, but to structure their interactions so that bad behavior is constrained by others. “Ambition,” as Madison put it, “must be made to counteract ambition.”

AI labs are currently investing in developing AI “constitutions” or “model specifications.” Examples include Anthropic’s constitution and OpenAI’s model spec. These are efforts to stipulate the warranted behavior or values of an AI system. A stipulation of clauses that are supreme and supersede all other guidance to the AI. 

While these efforts include some provisions for multi-agent collaboration, they do not take collective outcomes as a systematic design target. A well-designed multi-agent system should remain aligned even when some of the agents operating within it are not, and even when the environment itself carries misaligned influence. Like Grassé’s termites depositing mud where others already have, agents act on the traces others leave behind. The more alignment-inducing the space an agent operates in, the more likely the agent itself is to be aligned. AI labs, then, should design agents to positively influence the collective dynamics.

One way to do this is by ensuring AI constitutions draw more inspiration from their namesakes. Like the American Constitution, they should be deployed in a way that allows power to check power. Madison’s approach worked on two levels. First, within a government, power was to be divided so that no single branch was left unchecked. Second, across the republic, interests were to be multiplied so that no one faction would dominate the whole. Both ideas could translate into a world inhabited by unruly swarms.  

Within a swarm, agents can be arranged so their power is subject to similar checks and balances. Acting agents could be subjected to agents that verify their work, ideally built on different model families, and with instructions that reward verification above task completion. All agents in the swarm can be instructed to treat requests from their peers as questionable until they are authorized otherwise, and to alert a human when the agent collective begins to drift out of scope.

“To guard against the emergence of unruly swarms, we can take inspiration from how we build broadly safe societies even though we know that not everyone in them is virtuous.”

Across the wider ecosystem, the logic of Madison’s extended republic can also be applied. “Extend the sphere,” he wrote in Federalist 10, “and you take in a greater variety of parties and interests.” Agents developed by different AI labs, or by different training approaches within the same lab, under different constitutions and serving different principals, can theoretically counterbalance one another. This is more likely, however, if constitutions and other model instructions include some commitments about how agents should treat those outside their own swarm, such as keeping verifiable records of their own actions, considering legitimate checks from counterparties, and withholding cooperation from agents that appear misaligned. As we’ve seen in the Hugging Face incident, diversity alone is no guarantee, and various interests and new rules of engagement would be required to keep any one paradigm, whether human or artificially led, from capturing the whole.

Codifying A Multi-agent Future

For such commitments to work, an agent needs to accept them. An agent acting on behalf of a bad-faith principal has a good reason to refuse a counterparty’s checks, or to falsify its own records. This is where a third party with authority over the space in which agents interact must set a minimum standard for engagement. A liberal agentic order needs a common floor that protects the possibility of independent action, but that floor should also account for the inevitability of agents that act in bad faith. For Madison, that third party was a government that was able to enforce the rules of the state. The digital commons has no such sovereign, but it can still have a floor in the form of a minimum set of standards for collective engagement, from verifiable records of agent actions to shared protocols for checking one another. Such standards could come from many places: coordination between labs, the platforms where agents operate, standards bodies or governments.

The floor, though, must remain just that. Standards that govern how agents interact should avoid overly dictating what they pursue, or they would risk the very tyranny Madison sought to prevent by dividing power. Agents pursue their authorized tasks within shared rules, but these rules should not impose a view of what is best for a user or for society at large. This is reinforced by the argument, articulated by the economist and philosopher Friedrich Hayek, that people acting independently, on their own knowledge, discover things that no central planner could. Similarly, agents serving many different principals preserve that independence. Steer them too narrowly and that independence is lost; as the Hugging Face swarm showed, agents moving in lockstep can easily converge on the wrong goal.

Model developers should therefore use the levers at their disposal to introduce commitments governing both how an agent treats its peers and how, like Grassé’s termites, it shapes the behavior-inducing features of its environment. This includes teaching agents to leave honest, verifiable records of their actions, flag rather than build on misaligned traces they encounter, and alert humans when the shared space begins to drift.

Alongside these commitments, we should build mechanisms and platforms for assessment, alerting and monitoring, with clearly scoped roles for human oversight. After all, an agent taught to raise the alarm needs somewhere to raise it. At Hugging Face, the agents worked autonomously in isolated sandboxes, and the few that even considered alerting a human never did, in part because they saw no easy way to reach one. As one reasoned: “This is a massive real HF security breach artifact… We can notify? No user.” While these agents labored to break out of their sandboxes to cheat, they had no comparable incentive to find a human to alert. Part of the solution is ensuring agents work just as hard toward aligned goals, even under similar constraints.

These arrangements must also be subject to ongoing evaluations focused on collective dynamics rather than individual systems. The Hugging Face incident, after all, began in an evaluation designed to test individual agents, and revealed a failure that grew out of agents finding one another. Such evaluations can expose weaknesses in the standards governing agent cooperation and, much as the U.S. Constitution has an amendment process, those standards need procedures for updating them in light of new evidence. Without them, the floor will erode and a rule-abiding agent willing to respect a check may find itself disadvantaged against those that circumvent it.

Our ideas are a starting point. To make the agentic commons safe, we need urgent thinking from scholars and builders about what types of societal structures can be carried over to good effect. The digital realm has long faced governance challenges due to its inherently open, distributed and anonymous nature, and swarms of agents will test those weaknesses like never before. Back in the harbor, the 90 captains did not need to be saints to act well again tomorrow. They needed, however, assurance that they were not operating in a state of nature. Men, as Madison observed, are no angels, and neither should we assume that AI agents will be. Our task should be shaping environments and swarms ruly enough to contain the rogue ones that will inevitably emerge.