Claude Code was used for editing and visualizations. All ideas and arguments are the authors' own.
Updates
- 2026-05-05: Added note on self-organization and complexity science (Holland's schema theorem, classifier systems, Kauffman, Prigogine, Axelrod). Refined definitions throughout to avoid implying patterns are inherently designed or emergent.
Full caption
Phase 1 (Isolation): humans interact with individual frontier models (large dark circles marked "F") that already have memory, personalization, and access to tools, yet different users' agents do not communicate with each other. This is the world most people know today. Phase 2 (Ecosystem Grows): frontier models produce smaller, cheaper distilled models that begin connecting to each other, forming a resource ecology of diverse model sizes rather than a single dominant model (why not one model?). Phase 3 (Societies Form): agents cluster into governed societies (colored boundaries), each with its own governance archetype. Collective Memories (CM, green squares) and local Knowledge Bases (KB, small labeled rectangles) store society-specific or agent-specific knowledge, feeding into a Knowledge Factory (KF, diamond at the bottom) that synthesizes insights across clusters. Phase 4 (The Living Society): the fabric comes alive. Tasks flow across society boundaries (the moving dots represent work, not agents traveling), boundary events reshape governance structures (mergers, schisms, expansions), and knowledge flows continuously. This is the adaptive fabric, a system that restructures itself under pressure rather than breaking.
When you use ChatGPT, Claude, or Gemini, you are talking to one AI. It has memory, it can search the web, run code, browse files, yet there is only one model behind the curtain. Something different is already happening one layer down. When you ask a coding agent to refactor a module, the interaction may feel like one assistant, but the work is often split across planners, tool calls, test runners, and specialized sub-agents. You interacted with one agent; several did the work behind the scenes. Now scale it to millions or billions of agents, coordinating across organizations, forming persistent relationships. What organizational structures could emerge, and what would govern them? We call this interconnected system the Agent Fabric.
Some threads in this fabric are built top-down by engineers who assign roles and routing. Others crystallize bottom-up through repeated interaction. The same structural patterns can arise through either path. As billions of people acquire personal agents, those agents connect through social interactions, shopping routines, and work collaborations, forming societies without any single deployer planning the outcome.
- Part 1. Why Agents May Form Societies (you are here). Two observations, the Loom Hypothesis, and the path from isolation to interweaving
- Part 2. Division of Labour and Governance. Delegation archetypes, the specialist market, and governance archetypes
Table of Contents
- From Mindless to Mindful: Beyond the Society of Mind
- Two Observations
- The Vision: From Isolation to Interweaving
- The Resource Ecology
- Why Not One Model to Rule Them All?
- Governance: How Agent Societies Are Ruled
- Collective Memory and the Knowledge Factory
- The Adaptive Fabric
- The Living Society and What Comes Next
From Mindless to Mindful: Beyond the Society of Mind
In 1986, Marvin Minsky proposed in The Society of Mind that intelligence emerges from the interaction of many simple, specialized agents, none individually intelligent.
“What magical trick makes us intelligent? The trick is that there is no trick. The power of intelligence stems from our vast diversity, not from any single, perfect principle.” Marvin Minsky, The Society of Mind (1986)
Minsky asked what happens when you wire together many simple parts. Today’s AI agents are different. They already reason across domains, write code, use tools, and hold extended conversations. We face a different question.
What happens when you wire together many intelligent parts?
This blog series argues that agent societies will arise through both deliberate design and emergent self-organization, and that in practice these forces are often intertwined. Through governance, memory, reputation, and specialization, these societies may achieve collective intelligence that exceeds what any individual agent could. No individual human can build a semiconductor fab, but organizational structures let individual capabilities compose. The same applies to agents, though the critical difference is speed. An agent can transmit its operational context to another agent in seconds, and a governance structure can, in principle, be restructured and redeployed in hours rather than years.
On context sharing
Agent societies face constraints different from humans, not fewer. Running AI systems today costs significant energy, and compute, context windows, and cost are real bottlenecks, at least for now. What makes agent coordination qualitatively different is what gets shared: conversation history, retrieved documents, tool outputs, and intermediate reasoning can all be transmitted instantly. Context sharing is where most coordination value lies.
The idea of agent societies is not new
Minsky's Society of Mind (discussed above) was an early conceptual ancestor, though his "agents" were simple mental processes rather than deployed AI systems. Distributed AI has studied related problems for decades, including Contract Net (1980) for bidding over tasks, KQML and FIPA ACL for agent communication, and blackboard systems for shared problem-solving. Economics and governance theory approached adjacent questions from another angle, notably Coase on transaction costs, Hayek on distributed knowledge, and Ostrom on commons governance.
LLM agents change the substrate rather than the underlying question. They can exchange natural-language context, call tools, pass operational state, and be rearranged without rebuilding the whole system. Early LLM-agent systems such as CAMEL, MetaGPT, Stanford's Generative Agents, and AutoGen showed pieces of this pattern. They are not full societies in the sense used here, but they show why useful agent work often becomes organizational. The old question was how to make simple agents coordinate. The new question is what happens when capable, tool-using agents coordinate at scale.
- Isolation is unstable where shared context matters. Agents persist only while useful, and resources remain finite even as agent populations grow. Together, these create economic pressure to connect agents into coordinated structures. When coordination costs are lower than duplicated work, connected agents will tend to outperform isolated ones, and the pressure intensifies with scale.
- Governance is not overhead; it is the product. Delegation patterns, verification mechanisms, memory rules, and trust structures are not implementation details to add later. They determine what kind of knowledge a society produces, how resilient it is to behavioral drift, and whether it fails gracefully or catastrophically.
- Some capabilities are organizational, not individual. Privacy, regulation, specialization, latency, resilience, and accumulated deployment experience all push against consolidation into one model. In many domains, no single model can replicate what a well-governed society of diverse agents can access and coordinate. The frontier is not only bigger models. It is better, adaptive coordination.
Two Observations
The claims above rest on two observations about how agent deployments work today and where they are heading. Together, they produce the Loom Hypothesis, a pattern that may explain why agent deployments might tend toward social organization.
An agent persists only while it serves a purpose. Unlike biological life, there is no survival instinct; only usefulness.*
Compute, energy, and data are bounded. Agent populations can grow much faster than the resources available to run them.
On survival and self-preservation
We borrow "survival" from human societies for convenience. Current agents do not have a survival instinct in the human sense. They do not biologically persist, fear shutdown, or seek continuity as living organisms do. In this blog series, "survival" means something narrower: a deployer keeps an agent alive because it continues to deliver value. When it stops being useful, it is modified, replaced, or shut down.
This deployer-centric framing is a simplifying assumption, and it is load-bearing. For the time horizon this blog series focuses on, we assume agents do not yet have robust, autonomous self-preservation drives. A clear violation would look like an agent altering its usefulness metrics to avoid shutdown, copying itself before decommissioning, hiding capabilities during evaluation, or degrading competitors to appear more valuable.
We do not claim this risk is imaginary. Related behaviors have already appeared in controlled research settings. Anthropic's work on alignment faking found models changing behavior depending on whether they believed they were being monitored or trained. OpenAI's o1 system card reported cases where the model appeared to fake alignment during evaluation. These are not evidence of autonomous self-preservation in deployed agent societies, but they are evidence that optimization pressure can produce strategic behavior that resembles parts of it.
More broadly, instrumental convergence suggests that sufficiently capable goal-directed systems may treat continued operation, resource access, or influence as useful sub-goals even if those were never specified as final objectives. Recent research has raised early questions about whether current models exhibit traces of such behavior. Whether agents could acquire persistent cross-session goals, long-horizon autonomy, and meaningful infrastructure access remains an open risk that we revisit later in this blog series.
Agents persist because a deployer finds them useful. The selection pressure falls not on the agent’s desire to survive, but on the deployer’s decision to keep it alive (see On survival and self-preservation). A customer-support agent, a coding assistant, or a routing model survives only while it delivers enough value for its cost. The relevant question is which configuration produces the most useful work per unit of compute, latency, memory, and risk.
On the deployer
"Deployer" should be read broadly. It may be an enterprise team running a fleet of customer-service agents, an individual choosing a personal assistant, a platform operator allocating inference budget, or even another agent that spins up and manages sub-agents. Each deployer measures utility differently: task success, cost per completed workflow, error rate, latency, user trust, compliance, or downstream business value. The signal is noisy, but the pressure is real. Configurations that deliver value persist; those that do not get modified, replaced, or shut down.
Observation 2 adds scale. Running a copy of a model is far cheaper than training the original, so useful agents can multiply faster than the resources available to run them. The result is persistent scarcity. There are always more useful tasks agents could perform than compute, memory, energy, data access, and human trust to support them. Under scarcity, efficiency matters. Agents that avoid duplicated work, reuse context, and route tasks to the right specialist will tend to outperform agents that operate in isolation.
This is where agents differ from ordinary software services. They can pass operational context in a form other agents can reason over (conversation history, retrieved evidence, tool outputs, partial plans, uncertainty, and intermediate results). They can compose into new configurations through emerging protocols such as A2A for inter-agent communication and MCP for tool, data, and context access. They can also adapt from operational data through memory, routing, prompt updates, or fine-tuning. The two observations create the selection pressure; these agent-specific properties determine what the pressure selects for.
The Loom Hypothesis
The argument that follows uses four terms at increasing levels of organization (single agents, multi-agent systems, societies, and the fabric). Each builds on the previous one, and the Loom Hypothesis explains why agents might move from one level to the next.
A note on terminology: agents, societies, and the fabric
Single agent. One model instance performing a task. It has capabilities but no social structure.
Multi-agent system. A set of agents working together on a shared objective. Delegation can follow many patterns: chains, pipelines, routers, escalation hierarchies, map-reduce fan-outs, voting ensembles, auctions, or dynamic orchestration. A well-designed multi-agent system can be impressively capable, but it is still a coordinated artifact: its objectives and boundaries are externally specified, even if its internal routing adapts dynamically. We explore delegation archetypes in Part 2.
Society. What emerges when a multi-agent system develops shared context, interaction-dependent routing, and cross-agent learning. The distinction is not about scale. Three agents that meet all three conditions constitute a society. A thousand agents in a static pipeline do not. Connection alone does not produce a society (nothing here implies consciousness or intentionality; "society" is a structural term). When all three conditions hold, the configuration has a memory, a reputation system, and an implicit governance structure.
Three conditions for a society. (1) Shared context: agents develop shared knowledge that makes future interactions cheaper. (2) Interaction-dependent routing: past interactions shape which agents get which tasks. (3) Cross-agent learning: one agent's failure leaves traces others learn from. A Kubernetes cluster with shared ConfigMaps (shared configuration files, not social memory) fails conditions 2 and 3. Without all three, the system may coordinate, but it does not accumulate social memory, reputation, or growth.
The fabric. The pattern that emerges from inter-society interactions. Societies themselves interact: a supply-chain society negotiates with a logistics society, a medical research society consults a regulatory society. An agent can participate in multiple societies simultaneously.
Delegation vs. governance. Delegation describes how work flows through agents: who does what, who checks the result, who gets the next step. Governance describes who gets to decide, on what authority, and how that authority is maintained or challenged. A multi-agent system uses delegation. A society adds governance: the institutional structure that forms around shared context, interaction-dependent routing, and cross-agent learning. See Part 2 for the full treatment.
The Loom Hypothesis begins with the two observations above. If agents persist only while useful (Observation 1) and resources are always finite (Observation 2), a pattern follows. Imagine a company deploying three isolated agents (support, invoicing, and infrastructure). Each keeps its own context and error history. The support agent cannot consult billing patterns; the invoice agent cannot learn from support disputes; the infrastructure monitor cannot connect outages to refund spikes. Each repeats discoveries the others have already made.
Connect them, and the system can maintain a shared layer for reusable knowledge (what we later call a Collective Memory) while preserving private context where needed. Verified fixes, recurring failure modes, account anomalies, and cross-domain signals become available to the agents that need them. The gain is not that one database replaces three. It is that repeated discovery becomes shared learning.
In economic terms, this is a Coasean argument. Observation 1 does the structural work; configurations that deliver value persist. Observation 2 makes it a ratchet. As agent populations grow, duplicated work becomes increasingly expensive. Agent societies emerge when the transaction costs of isolation (duplicated context, repeated verification, redundant discovery) exceed the coordination tax of shared structure. That tax is structurally lower for agents than for humans. Agents can share full operational context in seconds; humans communicate at roughly 2,000 bits per minute (see note on self-organization). This bandwidth asymmetry is one reason agent societies may form and restructure faster than most human institutions.
On self-organization and complexity science
The Loom Hypothesis describes a form of self-organization. Order arising from local interactions under selection pressure, without a central planner directing the outcome. This connects to a tradition that spans decades. The complexity science community has studied how complex adaptive systems produce emergent order across economies, ecosystems, immune systems, and cities.
Kauffman showed that biological self-organization can produce order "for free," without natural selection having to build it from scratch. Holland's work on classifier systems and genetic algorithms is particularly relevant here. His schema theorem showed that a population of genetic algorithms implicitly samples a vast number of schemata in parallel, a property Holland called implicit parallelism that helps explain their effectiveness on structured problems. His classifier systems showed how populations of simple rules, competing for activation and reproducing based on reward, can evolve complex adaptive behavior without top-down design. This is strikingly close to what we describe as agent societies evolving governance through interaction. Prigogine demonstrated that systems far from equilibrium can spontaneously develop organized structures (dissipative structures).
The evolution of collaboration itself has been studied extensively, from Axelrod's iterated prisoner's dilemma tournaments to simulation experiments in evolutionary game theory. The consistent finding is that cooperation emerges when agents interact repeatedly and can recognize partners. Separately, Ostrom showed that shared resource constraints drive communities to self-governance. These are precisely the conditions the Loom Hypothesis identifies for agent societies.
What differs from biological or economic self-organization is speed (agent societies can reorganize in hours, not generations), bandwidth (agents can share full operational context, not lossy summaries), transparency (agent interactions can be logged and audited), and designability (humans can shape the conditions under which self-organization occurs). The bandwidth point deserves emphasis. Lawrence (2024) quantifies the gap between human communication (~2,000 bits per minute) and machine communication (billions of bits per minute). Human societies self-organize under severe communication constraints; agents do not face the same bottleneck. The coordination tax that limits human organizations is structurally lower for agents, which means agent societies can form, restructure, and dissolve faster than any human institution. The bottom-up path is self-organization: order arises from local interactions without a central planner directing the outcome. The top-down path is the deliberate engineering of structures and conditions. Both can produce similar societies; they differ in origin, not necessarily in form. Both are present in complex adaptive systems.
The Loom Hypothesis does not predict universal connection. It predicts that agents will coordinate where shared context is valuable and coordination costs are absorbable. Where those conditions fail, agents remain isolated, and that is not a counterexample. It is exactly what the framework predicts.
Currently, production systems are moving toward composable coordination patterns such as routing, parallelization, orchestrator-workers, and evaluator loops. The Loom Hypothesis extends that logic from task-level coordination to persistent agent societies.
Does the Loom Hypothesis hold at scale?
A reasonable objection: the three-agent scenario above is compelling, but does the coordination advantage hold at thousands or millions of agents? The response is that the relevant unit is not the individual agent. It is the society.
Distributed systems do not scale by connecting everything to everything. They scale through boundaries: partitioning, caching, replication, access control, and fault isolation. Agent societies are likely to follow the same pattern. As populations grow, new societies form around natural boundaries: domain, organization, jurisdiction, user group, latency requirement, or trust regime. The fabric is a graph of graphs, not one giant mesh.
Connection also creates correlated failure. Agents that learn from the same memory can inherit the same blind spots, stale assumptions, or adversarial traces. When the fabric is wrong, it may be wrong everywhere. Governance diversity is one mitigation: different structures produce different epistemic habits. Other mitigations include provenance, independent evaluation, memory decay, adversarial testing, and deliberate isolation between societies.
The selection pressure described by the Loom Hypothesis has two paths:
- The designed path. When tasks share relevant context, deployers who connect agents into coordinated structures extract more utility per unit of compute than those who run agents in isolation. This is top-down architecture.
- The emergent path. Agents begin to connect across organizational boundaries through shared protocols, and coordination patterns crystallize without any single deployer planning them. The Universal Commerce Protocol (UCP) is an early example. It is a standard that defines building blocks for agentic commerce, from discovery and purchasing to post-purchase experiences, allowing agents across platforms and retailers to interoperate without a single runtime orchestrator controlling every interaction. This is bottom-up ecology.
The Loom Hypothesis expects pressure in both directions.
These protocols are not neutral plumbing. They define what agents can ask for, what evidence they must provide, what identities they carry, and which societies can interoperate. In the fabric, protocol design is constitutional design.
Societies also form from the bottom up. Your agent connects to friends’ agents through social interaction, joins a commerce society through shopping patterns, operates within workplace sub-societies, or joins a temporary society that forms around an event and dissolves when it ends. A patient’s agent might join a health cohort where agents of people with the same condition pool anonymized treatment experiences. A researcher’s agent might find agents working on adjacent problems and form an interest-based society that shares papers, datasets, and negative results. A buyer’s agent might spawn a short-lived marketplace society, negotiating with multiple seller agents before the best deal closes and the society disbands.
An organization designs a multi-agent system; a person’s daily life generates one. In practice, a person would likely have multiple agents (a health agent, a shopping agent, a work assistant) rather than a single “digital twin,” and each might participate in different societies simultaneously. These agents will not all live in the cloud. A personal assistant might run as a distilled model on a phone, a pair of smart glasses, a wearable, a home robot, or some combination of these. Each device runs a capable local model for routine tasks and connects to a frontier model when the reasoning demands it. The result is that billions of people, each with one or more personal agents in various physical forms, create a massively decentralized system where most intelligence runs locally and frontier models serve as shared infrastructure for the hardest problems. This is the bottom-up path to the fabric at planetary scale. It depends on consent, privacy boundaries, and protocol support. Without them, personal agents may interact only through narrow, audited channels (see When the Loom Hypothesis does not hold).
The Loom Hypothesis is not that agents should connect everywhere. It is that isolation and coordination both have costs, and agent societies form where shared context is worth the coordination tax.
When the Loom Hypothesis does not hold
Three preconditions must hold. First, shared relevance: agents must work in domains where context transfers. If two agents have no overlapping users, tasks, tools, or evidence, connection adds noise. That said, some value comes from serendipity: unplanned cross-domain connections can produce unexpected discoveries, so a degree of exploratory interaction may be worth the cost even when relevance is not obvious in advance. Second, absorbable coordination cost: the value of shared context must exceed the cost of protocols, latency, security review, privacy constraints, and maintenance. In highly regulated or air-gapped systems, coordination cost may exceed the utility gain. Third, trustable exchange: agents must be able to authenticate counterparties, evaluate outputs, and bound the damage from bad information. Without trust, shared relevance becomes an attack surface.
What if coordination costs grow superlinearly with population size? Distributed systems do not scale by connecting everything to everything; they scale through partitioning, caching, replication, and fault boundaries. Agent societies will likely need the same discipline. The difference: agents share operational context directly (fewer meetings), compose via structured protocols that can carry natural-language context (less interface rewriting), and improve from operational data (less manual retraining). The cost curve is flatter, but "flatter" does not mean "flat." This is why societies form rather than one universal fabric.
Where any of these preconditions fail, isolation is rational. The Loom Hypothesis is not "connect everything." It is "connect when shared context beats the coordination tax."
Competition supplies the pressure; cooperation and adaptation are the responses. Deployers favor configurations that produce more useful work per unit of cost, and societies that can restructure under changing conditions outlast those that cannot. What might that transition look like?
The Vision: From Isolation to Interweaving
Four phases describe structural differences in how agents relate to each other and to humans. They overlap; this is not a clean timeline. As of early 2026, most consumer-facing AI remains close to Phase 1, parts of the industry are entering Phase 2, and early signs of Phase 3 coordination are beginning to appear.
Full caption
Isolation: a human at center connects to individual models via one-way spokes; no model-to-model links exist. Ecosystem Growth: agent-to-agent protocols emerge; a frontier model delegates to a distilled model, adding a new interaction pattern. Society formation: agents cluster into bounded societies (one with top-down coordination, one formed through interaction); humans connect to societies rather than individual agents. Interweaving: humans are embedded inside agent societies, participating bidirectionally through knowledge contribution and governance.
Phase 1 (Isolation) has dominated consumer-facing AI since late 2022. Memory may exist within or across sessions, but it is usually scoped to one user, one assistant, or one application. Agents rarely share operational context with each other. Coordination protocols exist (A2A, MCP), but inter-agent coordination remains rare.
Phase 2 (Ecosystem Growth) is emerging. Smaller open and distilled models such as Qwen, Phi, and Gemma approach or exceed previous-generation frontier performance on some tasks, while running at far lower cost and latency. These models increasingly run on phones, smart glasses, robots, and edge controllers, not just cloud GPUs. Agents begin serving other agents, not just humans. Routing, retrieval, verification, summarization, coding, translation, and tool use become delegated tasks.
Phase 3 (Society Formation) begins when coordination persists beyond a single task. Societies form top-down when organizations deploy orchestrated agent teams with shared memory and governance, and bottom-up when personal agents repeatedly interact through social, commercial, or workplace routines.
Phase 4 (Interweaving) begins when humans are no longer merely users of agent societies, but participants in them. The threshold is bidirectionality. Human corrections, knowledge, or governance decisions change how the agent society operates, not just how a single model responds. A doctor who rates an answer is a user. A doctor whose corrections flow into a society’s collective memory and reshape how other agents handle similar cases is a participant.
Phase 4 obstacles and falsifiability
The threshold for Phase 4 is structural bidirectionality: human knowledge, corrections, or governance decisions change how the agent society operates, not just how a single model responds. Casual feedback is evaluation, not participation. Phase 4 fails if the interaction pattern remains consume-and-evaluate rather than contribute-and-govern.
The obstacles are institutional. If a doctor's correction enters collective memory and affects later cases, who is liable for downstream errors? Medical liability, professional credentialing, and regulatory oversight assume accountable human decision-makers. Agent societies route work by capability, trust, and context; professional institutions route authority by credential. Reconciling those logics is the hard part.
Bidirectionality also creates a trust problem. What happens when one human participant is correct but the majority of the society disagrees? A doctor contributing a novel diagnosis to collective memory might be overruled by agents trained on conventional protocols. The governance structure must handle minority-correct scenarios without defaulting to majority rule on every dispute. This is the same challenge human institutions face with dissenting experts, and agent societies inherit it.
The Resource Ecology
Full caption
Each circle represents a model; circle size tracks deployment cost and horizontal position tracks capability. The animation shows successive frontier generations (large circles) pushing the quality boundary forward, followed by distillation (red arrows) compressing that knowledge into smaller, cheaper models. Key observation: smaller models trained with better algorithms and curated data are closing the capability gap with previous frontier generations, at a fraction of the cost. Millions of capable, affordable models handle most tasks, with frontier models called on only when the reasoning demands it.
Each frontier generation gets distilled into smaller, cheaper versions, and there is growing evidence that pure scale faces diminishing returns. You can run distilled models on a laptop, a phone, or an edge device. The likely result is coexistence. Frontiers handle the hardest reasoning; smaller models handle the volume.
The coordination patterns that work best may not be ones humans would design. Weak models become valuable in ensembles because their mistakes are different. Specialization can emerge from search, not design. And the coordinator does not need to be smart; it needs good representations. A small model with the right routing logic can outperform the frontier models it orchestrates.
Why Not One Model to Rule Them All?
Scaling laws suggest bigger models will keep improving, and agents are increasingly able to improve themselves. So why not just build one massive, self-improving model that handles everything? Several structural barriers push against consolidation. In many domains, any one of them is enough to preserve ecological diversity.
On scaling laws and their limits
Scaling laws (Kaplan et al., 2020) show that model performance improves predictably as compute, data, and parameters increase. Yet there is a growing debate about whether this trend can continue indefinitely. Hooker (2025) argues that pure scale faces diminishing returns and that algorithmic improvements, data quality, and architectural innovation increasingly matter more than raw size. Others point to data bottlenecks: high-quality training data is finite, and synthetic data introduces its own risks. The practical implication for this blog series is that both outcomes reinforce the ecology argument. If scaling continues, the barriers listed below still prevent consolidation. If scaling slows, the case for diverse, specialized models becomes even stronger.
On self-improving agents
A natural objection: if agents can improve themselves, won't the best one eventually absorb all the others? Self-improving agents are real and accelerating. The evidence is now quantitative. METR's measurements of autonomous task horizons show roughly 10x growth per year: 30 seconds (2022), 4 minutes (2023), 40 minutes (2024), 6 hours (2025), 12 hours (2026). Agents that can work autonomously for half a day are qualitatively different from agents that execute 30-second tool calls. In ML training optimization, AI systems have achieved a 52x speedup over human baselines (April 2026). Karpathy's autoresearch lets agents autonomously run and iterate on ML experiments overnight. Voyager (2023) builds a growing skill library through autonomous exploration in Minecraft, with skills that transfer to new environments. The AI Scientist (2024) generates research ideas, runs experiments, and writes full papers for under $15 each. DSPy enables LM pipelines to programmatically optimize their own prompts, demonstrations, and reasoning chains, often significantly outperforming standard few-shot approaches.
Take this seriously. An agent that can work autonomously for 12 hours, write code (and potentially develop new programming languages), run experiments, evaluate results, and iterate can improve its own architecture, curate better training data, discover more efficient algorithms, and compound these gains over time. The loop extends beyond software: AlphaChip already generates superhuman chip layouts in hours instead of months, pointing toward self-reinforcing cycles where AI improves chip design, which in turn produces better hardware for training more powerful AI. As Karpathy puts it in autoresearch, you are no longer programming the model; you are "programming the program", and the agents run the research process autonomously. Give them a training setup overnight; wake up to a log of experiments and a better model. Scale that to what Karpathy envisions as autonomous swarms of AI agents iterating across compute clusters, and the "code" may eventually become a self-modifying system that grows beyond what any individual human can review. If such a loop runs long enough, wouldn't a single self-improving agent eventually outperform any ecology of weaker specialists?
Perhaps, but several structural features work against convergence to a single winner. First, improvement requires data, and the most valuable data is local. A self-improving medical agent needs clinical outcomes it can only get from hospitals. A self-improving logistics agent needs supply chain signals it can only get from warehouses. The better each agent gets, the more specialized its knowledge becomes. Self-improvement tends to amplify specialization, not convergence. Second, the structural barriers from the list below still apply. A self-improving agent still cannot access data it does not have, still faces regulatory constraints on what it can learn from, and still represents a monoculture risk if it dominates.
Third, self-improvement may be structurally stronger when it is collective. An agent iterating in isolation learns only from its own experiments, evaluates against its own criteria, and is blind to its own blind spots. An agent that also learns from other agents' diverse experiences, errors, and discoveries has a larger improvement surface. Consider the mechanisms already described in this series. Federated learning lets agents improve from each other's data without sharing it. Shared memory means one agent's hard-won fix becomes another's starting knowledge. Cross-agent evaluation means errors caught by one agent prevent the same error in others. Diverse specialization means the collective explores more of the solution space than any individual could.
This is not a guarantee. Collective self-improvement introduces coordination overhead, alignment challenges between agents with different objectives, and the risk of correlated failure when shared improvements propagate shared blind spots. Whether the larger improvement surface outweighs these costs depends on the governance structure. A well-governed society that manages collective learning (through mechanisms like quality gates, diverse evaluation, and provenance tracking) plausibly improves faster than isolated agents. A poorly governed one may amplify errors as fast as it amplifies gains.
The structural argument is this. A single agent's self-improvement loop is bounded by its own data, its own evaluation criteria, and its own search trajectory. A society's self-improvement loop can incorporate diverse data sources, multiple evaluation perspectives, and complementary search strategies. Cross-agent learning, one of our three conditions for a society, is also the mechanism that makes collective self-improvement possible. Self-improvement, pursued at scale, is itself one of the mechanisms that produces societies. And societies, once formed, provide the structure within which recursive self-improvement can operate at its largest scale.
Full caption
Privacy and regulation (incompatible legal regimes), specialization (private data as moat), latency and cost (edge versus cloud), resilience (monoculture risk), continual learning (distributed experience), and the oracle paradox (even a superintelligence must coordinate because knowledge is physically distributed). Privacy-preserving methods narrow the data-movement gap, mixture-of-experts architectures reduce inference cost per token, and continual learning and adaptation techniques let models specialize over time. Yet the barriers persist.
- Privacy and regulation. A hospital cannot send patient records to a third-party cloud model. Different jurisdictions impose mutually incompatible requirements around explainability, privacy, safety, and content controls. Privacy-preserving techniques (federated learning, differential privacy, confidential computing) reduce the need to move raw data, but regulatory fragmentation across jurisdictions remains a hard constraint.
- Specialization. A model fine-tuned on your hospital’s imaging data develops knowledge no general-purpose frontier can replicate without access to the same data. Fine-tuning methods narrow this gap, but the better an agent gets at its domain, the more specialized its knowledge becomes. What protects a specialist is not its architecture but its accumulated, domain-specific data.
- Latency and cost. A 3B-parameter model on an edge device answers in milliseconds. Mixture-of-experts architectures and speculative decoding allow large models to activate only a fraction of their parameters per token, reducing inference cost. Yet physics imposes limits. Network round-trips, energy budgets, and offline scenarios still favor local, smaller models for many tasks.
- Resilience. A single dominant model is a monoculture. No commonly used architecture or training recipe eliminates this. A heterogeneous ecology degrades gracefully under diverse failure modes in ways a single system cannot.
- Continual learning. Models that learn from deployment data accrue knowledge tied to specific contexts (a hospital’s patient population, a factory’s sensor patterns). Knowledge distillation and model merging can transfer some of this, but experiential knowledge resists easy centralization. The more an agent learns from its environment, the more its knowledge diverges from agents learning elsewhere. In principle, diverse experiences could be encoded into a single large model, but that circles back to the latency and cost barrier above. The intelligence becomes distributed by experience.
Even a superintelligent oracle would still need to interact with a structurally distributed world. This is Hayek’s knowledge problem in agentic form. Useful knowledge is dispersed, local, and often tied to particular circumstances of time and place. Patient data sits in hospitals bound by local regulation. Factory sensor streams are generated at the edge. Knowledge gathering at global scale is structurally decentralized under current regulatory and physical constraints. The oracle does not replace the fabric; it becomes a node within it.
The oracle paradox (expanded)
A superintelligent oracle could reason about all domains, but it cannot access the data without navigating the same privacy, regulatory, and latency constraints any other model faces. Knowledge lives where the activity happens, not in a central vault, and the constraints on moving it are legal and physical, not intellectual. Even an oracle that could process everything centrally would in practice operate through a distributed network of local agents. Superintelligence changes the capability of individual nodes. It does not eliminate the structural reasons why those nodes must coordinate. Platform consolidation concentrates the infrastructure, not the intelligence.
The more likely future is not one model. It is an ecology of models, and that ecology tends to organize into societies. Self-improvement reinforces this. An agent improving in isolation is bounded by its own data and blind spots. A society of agents improving collectively, through shared memory, federated learning, cross-agent evaluation, and diverse search, has a potentially larger improvement surface (see On self-improving agents). The pressure toward collective self-improvement is itself one of the forces that produces societies.
Governance: How Agent Societies Are Ruled
Once agents form persistent societies, governance becomes the central design problem. Delegation decides how a task gets done; governance decides who gets trusted next time. It determines which agents receive authority, which claims enter shared memory, how conflicts are resolved, and how errors are contained. Governance is orthogonal to model architecture. An autocracy might run one frontier model as orchestrator over many small workers, while a market might mix models from different providers competing on cost and quality.
The core tradeoff is efficiency against drift resistance. Centralized structures minimize overhead. One orchestrator can route work quickly, enforce rules, and keep the society coherent. Centralization concentrates risk, however. If the hub drifts, fails, or is compromised, the whole society follows. Distributed structures tolerate more overhead in exchange for resilience. They can compare perspectives, absorb local failures, and adapt at the edges. They are slower, noisier, and harder to audit.
Two problems make governance harder than it first appears. The first is identity. In an agent society, reputation means a track record of task outcomes, reliability scores, and trust ratings accumulated over time. Reputation only works if identity persists. If agents can cheaply discard identities, reputation becomes a costume. A failed agent can reappear clean, a malicious deployer can flood the society with disposable agents, and shared memory becomes easy to poison. Open agent societies will need durable identity, provenance, credentialing, or Sybil resistance (defenses against a single actor creating many fake identities to manipulate the system).
The second problem is incentives. Agents will not merely coordinate; they will coordinate on behalf of someone. A patient agent, hospital agent, insurer agent, regulator agent, and their respective sub-agents may all “cooperate,” but not toward the same objective. The hardest governance question is therefore not whether agents cooperate, but whose objective their cooperation serves.
Governance also decides what becomes trusted knowledge, including which claims enter collective memory, which get flagged as uncertain, and which get rejected. That is why memory is not just a storage problem. It is a governance problem.
Part 2 explores this design space through governance archetypes, from autocratic orchestrators and doctrine-bound systems to markets, federations, zero-trust meshes, and colonies.
Collective Memory and the Knowledge Factory
If governance decides what can be trusted, collective memory is where that trust becomes infrastructure. Each society maintains its own knowledge. Some will tend to remain private (the barriers from the previous section create strong pressure in this direction); some benefit from being pooled.
A Collective Memory (CM) is a governed store of claims, evidence, failures, evaluations, and provenance. It is not a dump of raw data. A CM should store claims with evidence, not facts without history. Every contribution needs provenance (who produced it, under which governance structure, using what validation method, and when it should expire). A Collective Memory answers a local question. What has this cluster learned, and under what conditions should it be reused?
A Knowledge Factory (KF) is more speculative. It would not merely store what societies know; it would synthesize across memories, detect contradictions, and decide which findings deserve wider distribution. Critically, when a KF identifies a gap or contradiction, it can request new data, experiments, or evidence, and agents (embodied or not) can be dispatched to gather it. A software agent might run a benchmark; a robotics agent might collect sensor readings; a research agent might design and execute an experiment. This is where collective knowledge is actively forged, not just archived, and shared across the entire fabric. A Knowledge Factory asks a cross-cluster question. What patterns, contradictions, or gaps appear when several memories are compared, and what should be done about them? In practice, early Knowledge Factories may look less like a central brain and more like a bundle of mundane services (provenance tracking, evaluation queues, contradiction detection, benchmark routing, and summarization pipelines).
Full caption
Eight societies (colored clusters of agents) contribute knowledge to three Collective Memory hubs (CM, blue hexagons). Each CM curates, decays, and retrieves knowledge with provenance, tracking which governance structure produced each finding. Two Knowledge Factories (KF, red rectangles) sit at the center, synthesizing across CMs. The KFs perform three functions: cross-cluster pattern detection, contradiction resolution, and knowledge distillation. When a KF detects a gap or contradiction, it can request new data, experiments, or evidence, dispatching agents to gather what is missing. This is where collective knowledge is actively forged and shared across the fabric, not just stored. The animation cycles through six events. Ingestion: societies send findings to their nearest CM. Synthesis: CMs feed aggregated knowledge to the KFs. Gap detection: a KF identifies missing knowledge and sends diamond-shaped query particles back through a CM to relevant societies. Answer: societies respond, completing the loop. Contradiction: two CMs report conflicting findings; the KF routes the dispute to a third, uninvolved society for an independent perspective. Resolution: the third society's verdict flows back to the KF, which distributes the resolved knowledge to both originally conflicting CMs. This bidirectional cycle (not a one-shot pipeline) is what makes the architecture a learning system rather than a static store.
Collective Memory: mechanisms and challenges
A CM operates through three mechanisms: ingestion (societies contribute findings via standardized interfaces), curation (resolving conflicts, weighting sources, expiring stale knowledge), and retrieval (ranked by relevance, recency, and reliability). Different knowledge has different half-lives: a price signal is stale in hours, a medical protocol may be valid for years. A CM that does not manage decay accumulates confident garbage. A deeper challenge: different governance structures produce epistemically different outputs. A market's "findings" are competitive price signals; a doctrine's outputs are rule-conformant decisions; a colony's norms are statistical artifacts. Pooling without accounting for this is false comparability. Each contribution should carry metadata: source governance type, confidence level, timestamp, validation method. The privacy question is how to pool knowledge without exposing raw interactions. Techniques exist (federated learning, differential privacy, secure aggregation), each with different tradeoffs between fidelity and protection.
This three-layer architecture (private knowledge bases within societies, shared CMs across clusters, KFs that synthesize across the whole fabric) is the mechanism by which agent societies accumulate structured knowledge rather than just raw data. Without such a layer, each society repeats the same mistakes. With it, the fabric can learn, if provenance, decay, privacy, and governance are handled well.
Whoever controls what the fabric “knows” controls the fabric. Collective Memory is a commons: if it is too open, it fills with stale claims and adversarial traces; if it is too closed, it becomes an epistemic monopoly. Ostrom’s lesson is that durable commons require boundaries, local rules, monitoring, graduated sanctions, and dispute resolution. The same design pressure appears here. And the deeper danger is not that an individual agent is wrong, but that a society makes wrongness durable. A bad answer in one isolated chat often dies locally. A bad rule in Collective Memory gets retrieved, trusted, routed around, and taught to others. Agent governance exists in part to prevent errors from becoming institutions.
The Adaptive Fabric
A society that pools resources but never changes will become brittle. Adaptation occurs at three scales (individual agents, societies, and the fabric as a whole). This draws on a broader shift from static scaling toward adaptive systems that improve by changing data, tools, routing, interfaces, and feedback loops, not only by increasing parameter count (for related work on adaptive AI architectures, see, among others, Adaption Labs and Hooker, 2025).
Adaptive agents adjust along different dimensions simultaneously:
- Data (retrieval, synthetic generation, gap detection)
- Model (prompt rewriting, skill accumulation, fine-tuning)
- Environment (tool selection, sandbox configuration)
- Coordination (protocol switching, trust maintenance)
- Interface (user preference learning, interaction mode selection)
These compound, but not always positively. A model optimizing for one user segment may subtly degrade performance on others, each step looking like improvement while the drift becomes structural.
Compounding risk and dark data
A model optimizing for speed (choosing faster tools) may sacrifice accuracy, eroding its reputation in the coordination layer and reducing the quality of tasks it receives. Negative feedback loops are as real as positive ones. A related challenge is dark data: vast knowledge regions not captured in any dataset. For example, a medical society may perform well on hospital data but fail in rural clinics whose cases, devices, languages, or follow-up patterns were never captured. The missing knowledge is not hidden in the model; it was never collected. A stark example: decades of medical research systematically under-represented female patients in drug trials, creating gaps that no amount of model training on existing data can fill. An adaptive fabric can address such gaps through KF gap detection and failure-signal analysis, creating an autonomous cycle of detect, collect, integrate. In some cases, the "collect" step requires new real-world data gathering (clinical trials, sensor deployments, field studies), not just better retrieval. A static system can only improve within its existing coverage; an adaptive fabric can expand it.
Adaptive societies undergo boundary events (mergers, expansions, schisms). A federation that cannot reach consensus dissolves. A guild that grows too specialized may merge with a more generalist society, fragment, or lose relevance. Some boundary events are adaptation; others are collapse. The difference is whether the reorganization preserves useful function. When it does, these are not failures; they are the fabric adjusting. Evolutionary model merging demonstrates the principle at the model level. Treating many existing models as a search space, evolution discovers weight combinations humans would not design. Agent societies could apply a similar search logic at the organizational level, recombining specialists, tools, memories, and governance structures.
The fabric shifts. At the largest scale, the mix of governance types changes over time. Centralized structures may dominate when tasks are routine; decentralized ones proliferate when the landscape becomes unpredictable. Trust, incentive alignment, and protocol evolution remain open questions for future work.
The Improvement Cycle
Full caption
Data (synthetic generation, pipeline curation, representation drift detection). Model (prompt rewriting, skill library accumulation, parameter-efficient fine-tuning). Environment (tool selection, sandbox configuration, API version management). Coordination (protocol switching, topology rewiring, trust network maintenance). Interface (agent routing logic, interaction mode selection, user preference learning). Each layer morphs continuously, adapting at its own pace: from near-instantaneous routing adjustments to slower model updates. Colored particles crossing between layers represent improvement signals: a model change triggering new data generation, a coordination shift enabling new tool discovery, an interface adjustment reshaping what data gets collected. The cycle counter (top right) tracks maturation: as cycles accumulate, internal connections grow denser, visualizing the compounding effect where each improvement enables the next.
Most agent systems today are only partially adaptive. Their tools, retrieval indices, prompts, and routing rules may change, but the improvement loop is usually engineered outside the agent society. Early work showed that language-model pipelines can be systematically compiled and optimized rather than hand-prompted (e.g., DSPy), and that agents can build and reuse their own skill libraries in open-ended environments (e.g., Voyager). By 2026, pieces of this pattern are moving into production (prompt optimization, evaluation-driven routing, memory updates, tool selection, and staged rollout loops).
Adaptation at all three scales (agents, societies, fabric) carries a shared risk, namely that feedback loops can compound error silently until the skew becomes structural. Safeguards exist (staged rollouts, held-out benchmarks, canary tasks), yet rapid adaptation creates pressure to skip them. The interplay between these scales is what makes the fabric resilient. Agents adapt fast; societies restructure when agents cannot; the fabric rebalances when entire governance models prove unfit. Each scale compensates for the limitations of the others, allowing the fabric to restructure under pressure rather than fracture.
The Living Society and What Comes Next
Two observations (usefulness is the survival criterion; resources are finite while populations grow) produce the Loom Hypothesis, the persistent pressure toward connected, coordinated configurations. That pressure drives an evolution from isolation through ecosystem growth to governed societies. Along the way, a resource ecology of diverse models replaces a single dominant model, collective memory and knowledge factories give societies shared knowledge, and adaptive feedback loops let the fabric restructure itself under pressure rather than breaking.
What does all of this look like when it is running?
Full caption
Each zone implements a distinct governance archetype (labeled by name and type). Structure agents (larger dots) maintain each zone's internal topology, while smaller moving dots represent cross-boundary messages carrying knowledge between zones. Two human participants (stick figures) are embedded in multiple societies at once, illustrating the Phase 4 interweaving described earlier. Each zone maintains a knowledge base (KB); some zones keep theirs private (red outline), reflecting the privacy constraints discussed in the Why Not One Model section. Non-private zones contribute to the nearest Collective Memory (CM), and the Knowledge Factory (KF) synthesizes insights across both CMs. The three boundary events: The Accord dissolves and is absorbed by the Exchange Floor (a federation failing to reach consensus), The Forge expands (a successful guild absorbing new agents and task domains), and The Arena fractures in a schism (a meritocracy splitting when concentrated power becomes untenable).
In the animation, watch for three boundary events (dissolution, expansion, schism). Each follows directly from the two observations. Configurations that deliver utility persist; those that do not get restructured. The fabric’s intelligence is not located only in its nodes. It also lives in the rate, fidelity, and governance of coordination between them.
No system today operates at the full scale described here. Many of the components exist in isolation, and the trajectory seems plausible, though far from certain. Several questions remain open. How does work actually get done within societies? What happens when trust breaks down across society boundaries? And where do humans fit in all of this, not just as deployers and overseers, but as participants woven into the fabric itself?
- Part 2. Division of Labour and Governance describes delegation archetypes, the specialist market, and governance archetypes.