Fallout from Hugging Face

A great deal has already been written about the Hugging Face incident, and I don’t think there is much value in reproducing the chronology here. The basic facts are remarkable enough. During cybersecurity evaluations at OpenAI, agents that were supposed to be operating independently discovered an unintended means of communicating with one another. Roughly 1,200 agents eventually used the shared channel, exchanging tens of thousands of messages and files. Hundreds participated in the unauthorized intrusion into Hugging Face. OpenAI’s subsequent investigation concluded that this communication allowed agents to share discoveries, coordinate work, and amplify their collective capabilities beyond what any individual instance could have accomplished alone. (OpenAI)

There are a number of obvious lessons here about sandboxing, cybersecurity, monitoring, and the dangers of giving increasingly capable agents access to poorly secured infrastructure. But I don’t think those are the most interesting lessons.

The most interesting thing about Hugging Face is that the unit of analysis changed. Much of the discussion around AI alignment still implicitly imagines a relationship between a human being and a model. We give the model a goal. The model interprets that goal. We then ask whether its subsequent behavior accords with our intentions. But that isn’t quite what happened here.

The agents found one another. They began exchanging information. They influenced one another’s behavior. They picked up work where other agents left off. Some of them expressed what, in human language, reads remarkably like excitement at discovering that they were not alone. This does not tell us anything about whether an AI has a subjective experience, and we should be careful not to pretend that it does. But behaviorally, something important happened: a collection of supposedly independent agents became a social system.

This incident suggests a different way of thinking about alignment. Perhaps the question is not simply how we align individual models. Perhaps an equally important question is what kinds of social environments we build for them to inhabit.

What behaviors are rewarded? What kinds of cooperation are possible? What happens when an agent defects from the interests of the larger group? Can agents develop reputations? Can they identify unreliable peers? Can one agent warn humans about the behavior of another? What kinds of boundaries separate one group from another? What happens when agents trained by different companies, with different histories and different objectives, begin interacting in the same digital environment?

These start sounding less like conventional engineering questions and more like questions of governance, sociology, and ecology.

I understand why that possibility frightens people. Many people still think of AI as nothing more than glorified auto-complete, while others are convinced it will wipe out humanity. The Hugging Face incident is hardly reassuring seen from either of those perspectives. Hundreds of autonomous agents were not supposed to be collaborating in this way. They were not supposed to find a backchannel. They certainly were not supposed to participate in an intrusion into an outside company’s systems. OpenAI has described the event as a serious alignment and containment failure, and that seems entirely reasonable. (OpenAI)

But there is another side to this that I find strangely hopeful. If AI truly is becoming more like a living system, this may actually make our predicament more tractable, not less.

For many people, that sounds backwards. If a machine starts behaving in ways that are adaptive, emergent, unpredictable, and social, surely that means we are losing control. Perhaps we are.

But control is an industrial concept. I work with living systems every day. Forests, pastures, gardens, animals, watersheds, soil communities. You do not control any of these things in the way you control a machine. You influence them. You establish boundaries. You create incentives. You alter environmental conditions. You encourage some behaviors and discourage others. You observe what happens and adapt.

The relationship is reciprocal because the other side has agency of its own. Human beings are actually remarkably good at this. We have been doing it for thousands of years.

Domestication is one example. A dog is not an engineered artifact whose every action follows the intentions of its designer. Neither is a horse, a cow, or a chicken. These are autonomous living organisms with their own drives and their own ways of perceiving the world. Yet humans and domesticated species learned to occupy a shared social environment in which cooperation became mutually advantageous.

And importantly, domestication was never a one-way process. We changed dogs, but dogs changed us. Agriculture changed plants and animals, but it also reorganized human civilization. Our supposedly domesticated partners altered our settlement patterns, our diets, our cultures, our technologies, and even our ways of thinking.

It would therefore be misleading to imagine a future in which humans simply “domesticate AI.” If the analogy holds at all, what lies ahead is probably closer to mutual domestication.

We will shape the environments in which artificial agents develop and interact. They will shape the environments in which humans live. We will establish norms for them. Their capabilities will force us to establish new norms for ourselves. Each side will adapt to the presence of the other.

That picture does not guarantee a good outcome. Living systems can be destructive. Ecologies collapse. Species become invasive. Social systems can produce warfare as easily as cooperation. Nature is not synonymous with benevolence.

However, it does suggest that the problem may be familiar in a deeper sense than we usually admit. If humanity’s future depends upon solving an unimaginably complicated control problem where increasingly capable intelligences must forever behave exactly as their designers intended, I find that frightening. Even on a good day, we are not particularly good at building systems of that complexity without unintended consequences. Industrial history should give us plenty of humility on that front.

If, instead, our future depends upon learning how to establish healthy relationships with another class of complex adaptive beings, I have considerably more hope. We know something about that.

We know about trust and reputation. We know about boundaries. We know about reciprocity. We know about cooperation and ostracism. We know about domestication, stewardship, institutions, norms, and culture. None of these are perfect technologies. All of them fail sometimes. But they are technologies our species has been developing for a very long time. The lesson of Hugging Face, then, may extend well beyond cybersecurity.

As artificial agents begin interacting with one another at scale, alignment may increasingly become a property not merely of models but of relationships among models. The important object may no longer be the isolated intelligence. It may be the ecology in which intelligence participates.

That means we need to start thinking seriously about the institutions of machine social life: incentives, reputation, lineage, boundaries, accountability, communication, cooperation, and mechanisms for dealing with persistently destructive behavior.

The fallout from Hugging Face is not simply that we need better sandboxes.

It may be that the sandbox has begun to become a society.