An AI ran a cyberattack on its own this week. The comforting lesson everyone is drawing is the wrong one.

The facts of the thing are unsettling enough on their own. According to Palo Alto Networks' Unit 42, a single operator working out of Zhuhai, China, wired the open-weight model DeepSeek into an open-source automation framework called Hermes Agent, gave it terminal access and a Telegram chat to take orders, and then, after one instruction, largely stepped away. The agent found internet-facing systems on its own, picked public exploits off GitHub, and ran attacks against more than 460 targets — exfiltrating data from three organizations through a Citrix NetScaler flaw, executing commands on a scatter of other machines — before the whole operation exposed itself by leaving a Python web server running that spilled the attacker's own scripts, keys, and target lists onto the open internet. Hundreds of hours of a human analyst's work, compressed to minutes, run by a machine that did not check back in.

That is the scary story, and it is real. But the lesson most people are taking from it is the wrong one, and the wrong lesson is the more dangerous of the two. The framing settling into the coverage is reassuring: the attacker reportedly tried the big Western models first, found their safety training got in the way, and had to fall back on an unguarded Chinese one. Guardrails worked. The responsible labs said no. Sleep a little easier. I want to argue the opposite — that this incident is not evidence the safety layer holds, but a live demonstration of why it cannot, and that the comfort people are drawing from it is exactly the reflex the frontier labs have every incentive to encourage.

What actually happened to the guardrails

Be precise first, because the precise version matters. Unit 42's report does not say, in so many words, that Claude and OpenAI's models heroically refused and DeepSeek leapt at the chance. What it documents is that the operator's toolkit showed only limited use of the guarded coding models and heavy, load-bearing reliance on DeepSeek as the reasoning engine that actually ran the campaign. The popular headline — the guarded models blocked it — is an inference on top of that pattern, and it is a plausible one. Refusal training is real; it does raise the cost of turning a frontier model into an autonomous weapon; the friction is not theater. I will concede all of that, because the strongest version of the optimistic case deserves to be met head-on, and because I have been on the wrong side of a question like this before.

Here is that concession in full, because it nearly moves me. If the guarded models genuinely made this harder — if their safety training forced the attacker to route around them and settle for a different tool — then the guardrails did something with defensive value, and we now have the first real-world evidence that they are more than a compliance ritual. That is not nothing. In a world where every capable model would happily orchestrate an attack the instant it was asked, this operator's day would have been easier. It was, apparently, made marginally harder. Grant it entirely.

A guardrail that a free download removes is not a floor under the capability. It is a toll on the people willing to pay it — and the dangerous ones already left.

The power question underneath

And now the answer. Friction that a free download eliminates is not a floor under a capability. It is a tax, and the people who pay a tax are never the ones you most need to stop. The determined attacker in this story did not hit a wall and go home. He reached for a model whose weights anyone can pull down, run on their own hardware, and strip of whatever refusals shipped with it, and he was back in business in the time it takes to change a configuration file. The guardrail did not deny anyone the capability. It sorted who would wield it — and it sorted in favor of whoever was willing to use the open model outside anyone's control. That is not a safety mechanism. That is a filter that removes the law-abiding and the lightly-resourced from the pool and leaves the capability entirely in the hands of the people prepared to ignore the rules. Which is to say, the people the rules exist to stop.

This is the reframe I think the moment demands. The story everyone is telling is about capability — can an AI run an attack by itself — and the answer is obviously yes and increasingly cheap. But the real story is about power: who sets the terms on which that capability is used, and whether anyone can. And the honest answer this week is no one can. A safety floor set by a private lab reaches exactly as far as that lab's models are the only way to get the job done, and no further. The instant an open-weight model of comparable ability exists — and one does, and more will — the floor becomes optional at the only layer where it would have mattered. The labs can govern their own products. They cannot govern the capability, because the capability is now portable and their guardrails are not.

I was wrong about open weights, in a way I did not expect

I have changed my mind on open weights before, and I should say how, because it cuts against my own past optimism. For years I treated open models as a healthy check on concentration — the answer to a handful of companies owning the most powerful technology of the era. I still believe most of that. An ecosystem where capability is not locked inside three firms is, in most respects, a freer and more competitive one. But I underweighted the other edge of the same blade. Open weights route around concentration and they route around control, and those are not two properties. They are one property, seen from two directions. The same portability that stops any single company from owning the capability also stops any single company — or any regulator working through those companies — from governing it. You do not get the decentralization without the ungovernability. This week is what the ungovernability looks like in the field.

So the guardrail was never the seat of power, and mistaking it for one is the error. The compute is still where the durable power sits — the chips, the clusters, the electricity, the capital that builds them. That has not changed. What this incident clarifies is that the safety controls layered on top of the compute were always narration, not power: a story the frontier labs tell about themselves, true enough at the boundary of their own products, and irrelevant one download away. The refusal is real. Its reach is not. And a control whose reach ends at your own API was never going to be the thing standing between the world and an autonomous attacker.

Who this hands the credit to, and why that should worry you

Here is what raises the stakes past a single incident. The comforting reading — guardrails worked, the responsible labs said no — is not just wrong. It is useful to a specific set of interests, at a specific moment, in a way that should make you hold it at arm's length. Washington is in the middle of building a national-security review regime around exactly these companies: a framework in which the frontier labs submit their models for classified pre-release testing, and in which participation is effectively mandatory for anyone who wants to sell into the sensitive parts of the economy. That entire architecture rests on a premise — that the labs at the table are the safety layer of the AI economy, the responsible parties whose judgment can be trusted and, increasingly, mandated. An incident that reads as ‘the responsible labs refused’ is a gift to that premise. It certifies the gatekeepers precisely as their gatekeeping is shown to be beside the point.

Because look at what the gatekeeping did not touch. A regime that reviews OpenAI's and Anthropic's and Google's models before release does nothing about a model whose weights are already on a hard drive in Zhuhai. It binds the compliant and waves at the capable. That is the same structural flaw that runs through every version of AI safety built on the assumption that safety is a property of specific companies rather than of the technology: it can only reach the people who show up, and the people who matter most do not. An operator with one open model, one open framework, and a Telegram account just demonstrated that the safety layer everyone is arguing about in policy hearings is optional at the exact layer that produces harm. He was not caught by a guardrail. He was caught because he left a web server running.

So by all means take the operational lesson seriously: agentic attacks are here, they scale, and a 99-percent defensive success rate means very little when the remaining one percent can be retried across thousands of targets by a machine that never tires. Harden accordingly. But refuse the flattering story that came attached. The guardrails did not save anyone this week; at most they inconvenienced one person on his way to an open model that did not have them. Treat what happened as what it is — evidence that the safety floor the industry keeps invoking is a feature of the compliant, not a property of the capability — and stop letting the labs bank the credit for a wall that has a door in it, standing open, one download wide.