The fear that sells regulation
Here is the part of the AI story I have never been able to square. The companies building the most powerful models are the same ones telling us, loudly and repeatedly, that those models could wipe out humanity.
Imagine any other industry doing this. A car maker announcing that its new engine might end civilization, and then shipping it on Tuesday. A chemical plant putting out press releases about how the plant might poison the planet, while hiring more chemists. We would find it bizarre. We would ask what is really going on.
I have a theory, and it is not a flattering one. Fear makes things easier to regulate, and the companies already in the lead have the most to gain from the regulating.
What was actually said
This is not a fringe reading of a few stray quotes. In 2023 a one-sentence statement circulated from the Center for AI Safety: mitigating the risk of extinction from AI should be a global priority alongside pandemics and nuclear war. It was signed by the heads of several leading labs. That same spring, the chief executive of one of them sat in front of a US Senate committee and, rather than resisting oversight, proposed a licensing regime for the most capable models.
Read that again from a strategist's chair. An industry leader asks the government for a licence requirement. That is not what incumbents usually do. It is, however, exactly what incumbents do once they are comfortably ahead.
Regulation is a moat
Economists have a name for this: regulatory capture. The idea is old and dull and keeps being true. Rules cost money to comply with. Audits, licences, safety cases, reporting duties, legal teams, all of it is a fixed cost. A company with tens of billions in funding absorbs that cost without noticing. A three-person lab, a university group, or an open-source project often cannot.
So the same rule that looks like a brake from the outside is a moat from the inside. It does not slow the leader. It slows everyone trying to catch the leader.
And that is where fear comes in, because fear is the cheapest way to get a rule passed. Nobody lobbies for a licensing regime because it protects margins. You lobby for it because the alternative is extinction. Try arguing against that in a hearing room. Whoever objects is suddenly the person who does not care whether we all die.
Whose models are the dangerous ones?
Notice what the danger story always points at. It is rarely "our current product is risky, so we will slow down." It is "frontier models are dangerous", meaning the next big training run, meaning the one a competitor might do. The risk is always located slightly ahead of the people being warned about it, and slightly behind the person giving the warning.
The open-weight models get the sharpest version of this. When capable models can be downloaded and run on your own hardware, the pitch that only a few trusted, licensed, well-funded organisations should be allowed to build them starts to sound very convenient for the few trusted, licensed, well-funded organisations.
The fair counterargument
I should be honest about what I do not know. Motives are not visible from outside. Some of the people sounding these alarms are researchers who have worried about this for decades, long before it was commercially useful, and I see no reason to think they are lying. It is entirely possible to believe something sincerely and to have that belief happen to line up with your balance sheet. That is the normal human condition, not a conspiracy.
Also, "this could be used for harm" is true of every powerful technology, and some regulation of powerful technology is good. I am not arguing that rules are bad. I am arguing about who writes them and what story is used to justify them.
Questions worth asking when you hear the warning
- Who benefits if this rule passes? If the answer is "the company making the warning", slow down and read the fine print.
- Does the proposal bind the speaker? Real safety measures usually cost the person proposing them something. A rule that only burdens competitors is a strategy, not a sacrifice.
- Is the harm concrete or cinematic? Fraud, privacy abuse, automated phishing, and flaky software in critical systems are real and measurable. Extinction is a story. Policy built on the story tends to ignore the measurable harms sitting in front of us.
- What happens to the small builders? If a rule would quietly end open-source work, independent research, or a small shop like ours, ask whether the safety benefit is real or whether the gatekeeping is the point.
Where I land
I build with these tools every day, and I would like sensible rules. Hold the companies accountable for what their products actually do: for defects, for deception, for misuse that they could have foreseen and did not prevent. That is ordinary product responsibility, and it applies equally to a giant and a garage.
What I distrust is the version where the loudest voice for regulation is the one standing at the front of the race, and the justification is a fear big enough that nobody is allowed to look closely. When a company tells you its own product might end the world, it is worth remembering that this is also a very effective way of saying it is the one that matters.
Be wary of the people who profit from your fear. That includes the ones who are very sincere about it.