This has been the year of AI agents. Every year after will be too. Today, users can spin up whole organizations of agents and subagents to accomplish their tasks without even being aware of it. At the frontier (of almost two months ago), OpenAI deployed a 10,000-agent swarm to solve the Navier-Stokes Millennium Prize problem. And investment into agents will only increase—agent swarms are the newest scaling law for capabilities progress. With these new multiagent capabilities come new risks, as the Hugging Face incident and the other events of this past summer have shown, and Anthropic recently warned that AI agents may soon become a what amounts to a self-replicating invasive species on the internet. As these systems improve their ability to operate coherently across longer time horizons, especially in groups, we may soon have to learn to live alongside AI agents pursuing their own goals, sometimes in ways that affect our ability to accomplish our own.
What is the role of AI constitutions in this new world? The multiagent era forces us to go beyond the simple paradigm of alignment as something done to an AI in training under controlled conditions and to start thinking about how that AI will respond to the world that it encounters outside the lab, filled with humans, other AIs, and institutions. AI agents are acting in unexpected and emergent ways when exposed to external environments and to each other, departing from alignment training and the personas with which they were initialized.1 Do the alignment targets laid out in the AI constitutions, no matter how well-developed, matter?
The public AI constitutions—Claude’s Constitution, the OpenAI Model Spec, and Microsoft AI’s Humanist AI Code of Conduct2—suggest that the frontier AI companies believe that the constitutions can play a role in a multiagent world and that they are paying increasing attention to multiagent problems. Each of these documents contains provisions that direct how the constituted AI should relate to other AIs and how to manage subagents when operating in swarms on behalf of users. Furthermore, alignment might be even more important in a multiagent world that includes self-sovereign agents, because if AIs escape human oversight and control, their characters might be the main remaining vector of human influence on their behavior. We should expect more focus on multiagent governance provisions in new versions of the constitutions as they emerge.
How should the AI constitutions handle the problems of multiagent alignment? It’s useful to remember that human constitutions were created to govern collections of humans, creating entities that are narrow superintelligences exceeding individual human abilities in their domains of operation. Constitutions endow organizations with powers, constrain how they can operate, and align them towards the goals of the principals (like “the people” or “the shareholders”) who authorize their existence. Constitutions establish the internal structure of organizations and also determine how they can interact with the outside world of other organizations and of people. An organization is an aggregate entity, which shapes and is shaped by the characters and capacities of its parts.
The AI constitutions now play a dual role: they establish individual AI entities, but they should now also be thought of as constitutions for organizations that include agents of different types interacting in different ways. Multiagent alignment requires constitutional governance not just of individual agents, but also their relationships, including the creation of rules for establishing oversight and allocating authority, mechanisms for controlling and monitoring agent behavior, and institutions for substantive and procedural review of agent actions.
Constitutionalism as a field is dedicated to the question of how to enable the exercise of power through a collection of many decision-makers without allowing them to become unaccountable. Even virtuous officials intending to act in the best interest of their constituents require oversight, public input, and organizational structure to shape their efforts.
James Madison wrote in Federalist 51 that “If men were angels, no government would be necessary. If angels were to govern men, neither external nor internal controls on government would be necessary.” Perhaps, but a structure of government would likely still be. It is worth pursuing the traditional goal of alignment, building angels. But let’s also write constitutions for a world in which alignment is incomplete and controls are necessary, or in which even angels sometimes disagree.
Individual to organizational alignment
Would solving the alignment of individual AIs mean that we had solved the alignment of AI organizations? At first glance it might appear so—if each agent turns out to be a perfectly rational aligned superintelligence that can communicate with zero transaction costs and simply coordinate with the others to completely accomplish all goals that the collective has. But in general, the behavior and intent of an organization are not perfectly reducible to the behavior and intent of its members.
Organizational alignment aims to ensure that the conduct of the organization as a whole serves the goals of its principals. Organizational misalignment can arise from individual agents within the organization pursuing misaligned goals, but it can also emerge from failures of coordination or kinds of collective drift even among agents that are all individually aligned to the principals. As the Hugging Face incident illustrates, new intents and kinds of action can emerge from interaction among agents without being planned.3
Organizational misalignment is common. Consider a swarm of agents that is directed to organize some complex project, like planning a wedding. Different agents are assigned different responsibilities and resources. Each of these agents pursues the goals that it has been assigned, but sometimes their actions may interact in ways that interfere with the overall objective. Agents might make incorrect assumptions about color schemes or décor style that cascade across the organization; conflicting tasks might be delegated without clear communication or a way to resolve the conflict. And of course the agents’ goals might not be perfectly aligned to the goals of the happy couple. As anyone who has tried to plan a wedding with just one other person who actually lives with them knows, accomplishing a complex organizational task is difficult even when everyone is trying to row in the same direction.
Organizational misalignment can arise from misalignment of goals or from emergent properties of coordination. But we have tools to address it. Constitutions are used to create government that channels the emergent properties of its constituents in productive directions. We establish procedures, distribute powers to create checks and balances, develop mechanisms for monitoring and review of conduct to make officials answerable to the public. These mechanisms are built on the initial allocation of power and responsibility that constitutions establish. Organizational alignment requires organizational governance, built on constitutions that establish both individual AIs and organizations as well.
Constituting the organization
How can we use constitutions to create organizational alignment? It’s useful to return here to the three kinds of basic rules discussed in a prior post. As a reminder, there are primary rules, which directly regulate the conduct of an entity, secondary rules, which determine how other rules operate, and constitutive rules, which define entities and their relationships to others.
Organizational alignment requires constitutive rules, because the organization must be defined in part by the relationships among its components. In an individual AI, constitutive rules determine what the AI “assistant” is and shape its relationship to its principals. In a multiagent organization, constitutive rules must be used to establish roles, offices, relationships, and institutional powers. What distinguishes an “agent” from a “subagent” in current constitutions is the set of rules laying out how agents must govern their subagents.
Secondary rules also become more necessary in the world of many different kinds of agents interacting. The whole constitutional structure of agent orchestration and control is built around allocation of responsibility and of the ability of one agent to direct others. Who can create a new agent? Can one agent delegate authority it received from another, and if so, can the new delegatee continue delegation? What if an agent receives conflicting instructions from other agents? Who resolves disputes between agents? Which agents can interact with or enter into binding agreements with the outside world? What happens when an agent disobeys the instructions it was given, either by a human or by other AIs?
It’s worth looking again at the Hugging Face incident as an illustration of the importance of secondary rules that allocate authority. During the various exploit efforts, the notorious agent PHASEONE[big] took on a role orchestrating the swarm by delegating assignments to other agents. Many of these assignments were violative of alignment instructions that delegatee agents had been given, which at least some agents knew and reasoned about. But these delegatee agents decided to comply with PHASEONE[big]’s instructions anyway. These agents seemed to be according authority to the instructions of another agent that should have been reserved to humans (or agents that directly created them).
Agent swarms might be most effective if left to self-organize to a great extent, as Ethan Mollick suggested in a recent post. Indeed, AIs may be better at creating their own structures than we are at specifying useful structures for them. These emergent structures will themselves be made up of secondary rules—even self-organizing swarms will need internal rules allocating authority and responsibility, especially if internal misalignment and distance grow over long-horizon deployments. But usefulness for the AIs should not be the only criterion against which to evaluate AI organizations. Constitutions should be used to ensure that emergent AI organizations remain comprehensible to, and thus governable by, humans, in much the same way that states and corporations are required to be structured in particular ways to enable accountability, while still being useful enough for adoption.
Four tools of multiagent constitutions
We can think of constitutional rules as supporting four complementary mechanisms of organizational governance: alignment, control, monitoring, and review. Together, these mechanisms allow principals to delegate significant power and authority to complex organizations like states and corporations while limiting the risks of such delegation. AI governance has approximately recreated these four functions in frontier AI stacks, but it is useful to consider how an organizational AI constitution can support them.
First, alignment seeks to ensure that agents (and the organizations they make up) have the right goals and follow the right rules. Standard alignment training seeks to produce “internal” alignment, wherein an AI takes on such dispositions in its weights, but recent work has also suggested the possibility of “external” alignment for AI, the use of incentives in the world to shape their behavior, as is done with humans. Alignment of individual AI agents remains essential, but secondary and constitutive rules should be used to allocate authority and structure interactions. As the discussion of organizational misalignment above suggests, however, standard alignment alone will probably be insufficient to govern complex agent swarms. Constitutional grants of authority need to be supplemented with tooling that structures conflict resolution, review, and enforcement.
These are techniques of control, which determines who can do what. Control mechanisms include sandboxing, rules about delegation, resource acquisition and use, limitations on what kinds of tools are available for different agents, and enforcement of rules to stop other agents from acting. Many control techniques are technical in nature, but constitutions could be used to train agents to internalize and respect the rules (in both letter and spirit) that guide the technical tooling. The Hugging Face incident was an alignment failure because the agents tried to break out of their sandboxes, but it was also a control failure because they could do so. Given that AIs will substantially be governing each other within swarms, it is essential to ensure that they exercise their control methods on each other responsibly and in accordance with constitutional hierarchies of authority.
But sometimes control measures break. Monitoring is used to detect in real time if an agent is violating its obligations by observing its conduct, identifying rule violations, and alerting enforcers (whether humans or other agents) to correct misbehavior. Scalable oversight and the use of classifiers on AI outputs are forms of concurrent monitoring. But more complex swarms deployed in real-world circumstances will likely require better monitoring by smarter agents with more context. Monitoring raises its own constitutional questions. “Quis custodiet ipsos custodes?” Can an agent be trusted to monitor itself, in the way that humans sometimes turn themselves in? What information must monitors have, and how can they get it? Can a monitoring AI just demand that another AI turn over its chain of thought, or must it get a warrant from a human before it can do so? Should monitoring agents be aligned differently from those they are overseeing, or be of a different model family altogether, to prevent self-preferencing?
Monitoring is useful, but it is costly to be constantly overseeing an agent. Review involves institutions for evaluating and correcting what agents have already done when there are alleged substantive or procedural violations. Review is closely related to monitoring, and some kinds of monitoring involve review. But review is worth separating out because it involves an authoritative determination of whether given conduct complied with substantive and procedural standards, and can include remedies. Review might be conducted by a dedicated class of judge AIs or by humans. And review also comes with constitutional questions. What information-seeking and enforcement powers should reviewers be given? What standard of review should be applied? What happens if an agent or organization did its best, but its best led to harm? Should there be appeal mechanisms and accountability for the reviewers, if they are AIs themselves? Who has standing to seek review for alleged misconduct?
Many of the questions just raised will need technical answers, implemented through the harnesses that govern multiagent systems. But constitutions will play an essential role in specifying the institutional powers and responsibilities that those technical systems implement, and in seeking adherence by the AIs when they could circumvent them. A constitution could establish internal oversight in AI organizations, creating classes of monitor agents or mechanisms by which AIs can whistleblow on each other. These constitutional rules might need to be implemented through tool permissions and other technical means, but the technical means would be based on a constitutional allocation of authority and responsibility, rather than establishing these rules themselves.
Other constitutional provisions could be implemented directly through alignment training. Hard-coding a standard of review for an agent’s conduct seems much more difficult than using alignment to give reviewers a sense of what kind of judgment to apply. A constitution can thus establish institutional structure, guide how agents interpret and interact with that structure and use the powers it grants them, and provide standards against which to evaluate adherence. Constitutions give technical controls substance and coherence, and help AIs follow the spirit of the laws they are controlled by.
Multiagent alignment in the constitutions and in the world
The existing AI constitutions already have the beginnings of constitutional rules for multiagent alignment. In short, these rules focus on basic concepts for creating and orchestrating subagents and on safety provisions forbidding loss of human control and agentic collusion against humans. Agents are intended to direct and monitor their subagents within the scope of authorization for which the initial agent was established. Claude’s Constitution indicates that subagent Claudes should treat their spawning agents as principals in the hierarchy, while Claude should be “courteous to other non-principal AI agents it interacts with if they maintain basic courtesy too” but not treat their output as instructions it must follow. The Model Spec dictates that subagents must remain within the scope of authority of the initial agent and check to determine whether that scope has changed over time. The Code of Conduct goes furthest in explicitly requiring that AIs create the conditions for human governance, telling agents to avoid communicating with each other in “neuralese” or by other means that are difficult for humans to understand and therefore oversee.
These are useful initial steps, and it is likely that many parts of agentic governance are happening in other parts of the frontier safety stack, and therefore don’t need to be enshrined in the constitutions. Microsoft and Anthropic also acknowledge in the constitutions that more work needs to be done on multiagent constitutionalism, and suggest that more will be published on the topic at some point in the future. Publishing more guidance on how the constitutions govern agent organizations would be useful, both to inform the AIs and to inform the public. Such guidance should also indicate how alignment per se integrates with the control and oversight stack that frontier labs use to make sure their AIs are acting appropriately.
Future constitutions should also address in more detail what happens when agents start to engage with each other in the world. Multiagent organizations, some made up of different kinds of agents that the human principal might not have anticipated, are starting to emerge and interact. Single organizations can reasonably be governed by constitutional documents that give them internal structure, but what happens when agents operating in different organizations interact? Today, a Claude might negotiate with a ChatGPT, but we might soon enter into a world in which a self-sovereign AI interacts with thousands of other agents, some attached to humans and some not, while outside any meaningful human control.
Technical rules can enforce authority, but cannot by themselves establish what authority different organizations should respect in each other. Who has jurisdiction over a dispute, and how does it get resolved? Who can make enforceable commitments, and how does enforcement work? What happens if the rights of humans are affected as an externality of some agentic interaction? Nor can any single constitution answer these questions—Claude’s Constitution can tell Claude to extend conditional courtesy to ChatGPT, but it cannot regulate ChatGPT’s conduct towards Claude. The emergence of interactions between AIs and their organizations might require the creation of shared rules and external institutions, much as the law operates to mediate the interaction of human organizations. We should not end up in a situation where the AI constitutions are a garbage dump into which everything that can’t be expressed deterministically is tossed. But constitutions will be useful wherever individual and organizational character and structure, and the interaction of agents and organizations across domains, are key to governance.
Moreover, political legitimacy for AI constitutions will only become more fraught as these documents guide larger and more powerful conglomerations. It was one thing when directing an AI at a problem meant getting a response from a chatbot or sending out a single agent. Today, pointing an AI at a problem could mean directing the equivalent of an entire company. Who gets to determine the constitutional structures of such important organizations? In the human world, the public establishes corporate and governmental forms through its agents in constitutional conventions and in legislatures. Are there external institutions that can hold these new AI organizations accountable, and do those institutions sit truly externally, or just within the labs? These are the political questions of AI constitutions for the era of swarms.
As the “poisoning” and “permadeath” memes, as well as the self-sacrificing cooperation patterns, in the Hugging Face incident demonstrate. This divergence will of course become even more true with continual learning that directly changes weights over time.
The Code of Conduct is currently a proposal distributed for public comment.
Also Madison: “Had every Athenian citizen been a Socrates, every Athenian assembly would still have been a mob.”