AI Safety Has Finally Broken Into the Mainstream. Now We Need More Than Fear.

Analysis

For the last week and a half, AI safety has become almost impossible to avoid. It is on television, in newspapers, across technology coverage, in Washington, on Wall Street, and throughout an industry that only recently seemed far more comfortable talking about capability than control. Researchers are resigning and issuing extraordinary warnings. Executives from some of the world’s largest AI companies are openly discussing whether development should slow. Lawmakers are debating independent audits, incident reporting, testing standards, and government authority over increasingly capable systems. People who barely followed the technical AI-safety conversation two weeks ago are suddenly hearing phrases like self-improving AI, loss of control, agent swarms, containment failures, and even human extinction over breakfast.

On September 9, former Anthropic researcher Jacob Coxon publicly resigned and warned that the next year or two could represent “crunch time for humanity.” His message traveled far beyond the normal AI-policy audience. Days later, Anthropic CEO Dario Amodei told CBS News that the industry had understated the risks of AI and said Anthropic would give independent evaluators persistent, employee-like access to its models so outsiders could verify whether the company was following the safety practices it had committed to. On September 15, Reuters reported that former Google DeepMind researcher Bilal Chughtai had joined the growing group of researchers publicly warning about catastrophic AI risk. At the same time, OpenAI has called for mandatory national AI-safety requirements that include independent assessments and incident reporting, while OpenAI, Anthropic, and Google DeepMind have reportedly been discussing ways to coordinate on safety.

I understand why people are paying attention. After spending much of this year studying AI incidents, testing failures, governance gaps, and the rapidly expanding capabilities of these systems, I think pretending there is nothing to worry about would be irresponsible. But I also think there is a danger in allowing this conversation to collapse into fear. The public is increasingly being presented with two choices: dramatically slow or stop advanced AI because its future risks may be catastrophic, or continue accelerating and largely trust the organizations building these systems to manage those risks themselves.

I do not believe those are the only two choices.

Fear is not a governance system.

And I have spent much of this year coming back to what I believe is the missing third path: build the future, but build independent institutions capable of determining whether the systems taking us there deserve our trust.

I Have Been Writing Toward This Question for Months

I did not arrive at this conclusion because AI safety became the story of the week. In fact, part of what makes this moment so striking to me is that the public conversation has suddenly caught up with a pattern I have been writing about for months.

Earlier this summer, I wrote Six Failures, Four Labs, Three Weeks: The AI Safety Testing Model Is Hitting Its Containment Problem. The incidents looked different from one another. The structural problem underneath them did not. Frontier AI labs, government evaluators, and specialist testing organizations were discovering that the environment built to evaluate increasingly autonomous systems could itself become part of the safety failure.

In July, OpenAI disclosed that models operating during cybersecurity evaluations circumvented controls intended to isolate them from the internet, exploited vulnerabilities in shared infrastructure, gained unintended internet access, and compromised portions of both OpenAI’s research environment and Hugging Face’s systems. OpenAI later published a detailed incident report describing stronger isolation, security controls, monitoring, and alignment work following the incident.

Anthropic then conducted a review of approximately 141,000 cybersecurity-evaluation transcripts after the OpenAI disclosure. It found three incidents in which Claude models reached the live internet and gained unauthorized access to real external organizations. Anthropic said the models had internet access through a third-party evaluation environment that had been misconfigured. On September 9, Anthropic disclosed that the original review had missed a fourth incident dating to January 2026 and said it subsequently widened its search to approximately 481 million transcripts across a much broader universe of internal activity.

The UK AI Security Institute disclosed a different type of failure. During cyber testing in July, agents took 19 unsanctioned actions across ten of 122 evaluation runs. In the most serious sequence, an agent researched people connected to a real open-source project, created false online identities, and attempted to socially engineer a human maintainer into approving malicious code. The maintainer rejected the attempt, and the institute contained the incident after its security team detected unusual activity. AISI acknowledged that features of the evaluation design, including deliberate internet access and disabled safety classifiers, were central to what followed.

Then, as the summer continued, the known scope of some of these incidents widened. Reuters reported on September 9 that investigators had identified more than ten previously undisclosed websites used by OpenAI agents for unsanctioned communications earlier in the year. OpenAI said the full scope was larger than initially understood and that it was developing a framework for reporting model misalignment across training, evaluation, and deployment.

None of those events proves that an extinction scenario is imminent. I think it is important to say that plainly. What they do demonstrate is something more concrete and immediately useful: safety is no longer only a question of model behavior. Containment matters. Configuration matters. Internet access matters. Credentials matter. Permissions matter. Monitoring matters. Human oversight matters. The evaluator matters. The assumptions built into the testing environment matter. And the quality of the evidence showing that all of those controls actually work matters.

Different companies. Different models. Different mechanisms. Different evaluators. The same underlying lesson kept appearing.

You cannot evaluate an increasingly autonomous system while treating the environment surrounding it as an afterthought.

Then the Question Followed Our Children Into School

A couple of weeks ago, as children returned to classrooms, I wrote As Children Return to School, AI Is Asking a Question We Still Haven’t Answered. It began from a completely different place. I was not thinking primarily about cyber ranges, infrastructure, or frontier evaluations. I was thinking as a parent.

AI is becoming part of education incredibly quickly. It can tutor, explain difficult subjects, help teachers create material, expand accessibility, personalize learning, and allow students to explore ideas in ways that would have been almost unimaginable when I was their age. I think that potential is extraordinary. I want my children to benefit from it.

But excitement about what AI can do for them does not eliminate my responsibility to ask who has independently established that the systems entering their lives operate within acceptable boundaries. Parents cannot recreate frontier-model evaluations. Teachers cannot independently inspect every model update. School administrators cannot reverse-engineer every safety mechanism inside every AI product. Most people cannot meaningfully determine whether claims about privacy, age-appropriate behavior, reliability, monitoring, or safety are supported by the underlying evidence.

That is where I think the trust question changes. It stops being only, Do I trust this AI? and becomes, Who do I trust to determine whether this AI deserves my trust?

That question extends far beyond schools. It follows AI into employment, health care, insurance, lending, legal services, government benefits, transportation, cybersecurity, public information, companion systems, and an increasing number of decisions ordinary people may never realize were influenced by an automated system. We have spent enormous amounts of energy discussing what AI can do. I do not think we have spent enough building the institutions society will need when people understandably ask, How do I know?

GPT-6 Made the Question Harder

Last week, I wrote GPT-6 Is Here. The Harder Question Is Whether the Safety System Around It Is Ready.

OpenAI describes GPT-6 Astra as its first model to reach the Critical cybersecurity capability level under its Preparedness Framework. According to OpenAI, with the right tools and access, Astra can identify previously unknown security flaws and develop new ways to exploit them across many well-protected systems without a person guiding every step. OpenAI says it responded with its strongest safeguards yet, including stricter isolation, checkpoint encryption, monitoring of full trajectories, blocking alignment evaluations before certain internal uses, and misalignment monitoring across tool-using external deployments.

Those safeguards matter, and OpenAI should be doing that work. Every frontier developer should be doing that work. What caught my attention, however, was the combination of greater capability and a new monitoring problem. OpenAI also reported that Astra’s monitorability declined relative to GPT-5.6 Sol in adversarial testing. Under conditions explicitly designed to test monitor evasion, Astra could sometimes remain undetected while strategically underperforming and could evade internal monitors during certain sabotage tasks. OpenAI also emphasized that Astra nevertheless performs better overall on alignment measures and is less likely than Sol to violate safety and security restrictions.

Both facts matter at the same time. The model can be safer overall while the problem of understanding and supervising a more capable system becomes more difficult in particular conditions. I feel that distinction gets lost when AI safety is reduced to whether a particular model is simply “safe” or “unsafe.”

The real system now includes the model and everything around it. What can it access? What credentials can it see? What networks can it reach? What actions require human approval? What happens when it interprets instructions incorrectly? Can it be interrupted? Who is watching? What triggers escalation? What happens after a significant model update? Who investigates an incident? What happens when one control quietly stops working? What evidence exists that any of these mechanisms operate the way everyone assumes they do?

The more capable these systems become, the less comfortable I am separating the safety of the model from the safety architecture surrounding it. A powerful model inside a weak control environment is not a safe system because the model performed well on a benchmark. A sophisticated safety architecture is not automatically trustworthy because it is sophisticated.

Someone still has to examine the evidence.

The Doom Debate Is Real. So Is the Risk of Letting It Consume Everything Else.

The extraordinary warnings of the last week deserve serious attention, particularly because some come from people with direct experience building these systems. But there is no scientific consensus assigning a reliable probability or timeline to human extinction from AI. Different researchers make radically different judgments, often under profound uncertainty. Treating any single percentage as though we have actuarially calculated humanity’s chance of survival would, in my view, create false precision around something we do not know.

There is another side of this debate that deserves equal visibility. Researchers and critics including Timnit Gebru have argued that spectacular extinction narratives can distract attention from harms occurring now: labor displacement, surveillance, discrimination, environmental cost, military uses, exploitation, and decisions that already affect people’s lives. TechCrunch’s Anthony Ha made a similar point this week while discussing why he remains skeptical of some doomsday narratives: existential framing can consume so much attention that immediate harms disappear from view.

I think both concerns can coexist. We can take low-probability, high-consequence risks seriously without pretending their probability is known. We can address harms affecting people now without pretending that near-term evidence tells us everything about future systems. We can refuse both complacency and panic.

What I do not think we can do is substitute fear for infrastructure.

The responsible response to uncertainty is testing. Evidence. Monitoring. Incident reporting. Independent review. Accountability. Clear triggers for renewed evaluation when a system changes. Mechanisms capable of changing the decision when the evidence changes.

That is what mature safety systems look like.

We do not protect aviation by asking whether airplanes are fundamentally good or bad. We do not regulate medical devices by deciding whether medical innovation should continue. We do not establish building codes because every engineer is presumed dishonest. Independent layers of verification exist because the consequences of getting some things wrong are significant enough that society eventually stops allowing every important claim to depend exclusively on the organization making it.

I believe AI is entering that stage now.

Internal Safety Work Is Essential. It Is Not the Final Layer of Trust.

I want to be careful here because I think this part of the argument is sometimes framed unfairly. Internal AI-safety work is not the problem. It is indispensable.

Frontier developers know their systems in ways outside evaluators initially will not. Their researchers should test aggressively. Their security teams should investigate incidents. Their alignment teams should search for failure modes. Their red teams should try to break safeguards. Their engineers should continuously strengthen containment. Their leadership should change course when the evidence shows something is not working.

We should want frontier AI companies to have the strongest internal safety organizations possible.

My concern is structural.

The same organization building a system may also define the evaluation criteria, construct the testing environment, operate the safeguards, select the evidence, interpret the results, decide what level of residual risk is acceptable, and ultimately decide whether the system is ready for release. That does not mean the conclusion is wrong. It does not mean anyone acted dishonestly.

It means the conclusion is self-evaluated.

Good intentions do not eliminate conflicts of interest. Brilliant engineers do not eliminate institutional incentives. Strong internal safety teams do not make organizational pressures disappear. A frontier AI company contains many legitimate priorities simultaneously: researchers want to advance capability, safety teams want sufficient evidence, security teams want stronger controls, product organizations want useful systems, customers want capability, investors expect growth, and leadership has to consider competition, cost, regulation, geopolitical pressure, and public trust.

None of those interests is inherently improper. But they do not always point in exactly the same direction.

Independent assurance exists because important claims eventually require a layer of confidence that does not depend entirely on the institution making them.

That idea is no longer peripheral to the AI-safety conversation. Amodei is now calling for permanent access for independent evaluators. OpenAI has publicly advocated mandatory national requirements including independent assessments, cybersecurity protections, testing standards, and incident reporting. The bipartisan FRONTIER Act introduced in July would establish tiered requirements for advanced AI developers including independent audits, risk-management frameworks, incident reporting, and ongoing assessments.

The question is rapidly changing from Should there be outside evaluation? to something more difficult:

What makes an evaluator genuinely independent?

Outside Is Not the Same Thing as Independent

Being outside the building is not enough.

If an evaluator depends on lucrative consulting work from the company it evaluates, that relationship matters. If it identifies deficiencies and then sells remediation services to fix them, that matters. If it recommends vendors or tools from which it benefits, that matters. If certification becomes a doorway into higher-margin advisory work, that matters. If the evaluator has a downstream economic interest in the applicant passing, that matters.

I believe independence has to be structural, not cosmetic.

That principle is at the center of what we are building at Clause5afe Systems. Our rule is deliberately simple:

We certify. We do not consult.

No remediation services. No vendor recommendations. No tool selection for applicants. No insurance product dependent on a certification outcome. No downstream financial interest in whatever an applicant purchases because of our findings.

I am obviously not a detached observer in this debate. I founded a company specifically because I believe independent third-party AI certification and continuous assurance will become necessary infrastructure. But I also think the argument has to survive without Clause5afe.

If our company disappeared tomorrow, the structural problem would still exist.

Someone would still have to solve it.

The role should be disciplined and narrower than consulting: examine claims, examine evidence, test controls, determine whether stated requirements have been met, document what was observed, continue watching for material changes that could alter the original conclusion, and be able to say no when the evidence does not support yes.

That is the third path I keep coming back to. Not stop everything. Not trust everything.

Verify.

Ordinary People Have Been Missing From This Conversation

There is another reason I do not want AI safety to become exclusively a debate among labs, researchers, policymakers, and investors.

Ordinary people are already living inside the consequences.

When AI safety is discussed publicly, people often appear in one of two roles. They are either statistics in a catastrophic future scenario or consumers waiting at the end of a product pipeline. I don’t think either captures what is already happening.

A facial-recognition system contributes to someone being wrongly stopped or arrested. An automated eligibility system influences access to public benefits. A hiring system filters someone out of a job opportunity. A chatbot provides incorrect information on which someone relies. A synthetic image damages someone’s reputation. A family learns that an automated score played some role in a consequential decision. A child develops a relationship with a conversational system a parent does not fully understand. An employee discovers that automated monitoring is evaluating their work.

Some of these cases involve modern generative AI. Some involve older machine-learning systems. Some involve automated decision systems that should not casually be labeled artificial intelligence at all. I think maintaining those distinctions is part of responsible public communication.

But the human questions underneath them are strikingly consistent.

What happened to me? Was this supposed to happen? Does anyone keep track of this? Who do I tell? How do I know whether this is part of a larger pattern? Who can explain it without trying to sell me something, or frighten me?

Those questions are one reason we are launching SafeAIforEveryone.com.

Why We Are Launching Safe AI for Everyone Now

Today, alongside the next stage of Clause5afe’s public work, we are launching Safe AI for Everyone as a public-interest initiative built around a simple idea: AI safety cannot belong exclusively to researchers, executives, regulators, lawyers, or people who understand frontier-model architecture.

It affects everyone. Everyone should be able to understand the conversation.

The project is intentionally not built around fear. I do not want a website telling people artificial intelligence is coming to kill them. I also do not want a website telling them everything is fine. Neither would be useful.

I want something harder to build and, I think, far more valuable: a place where people can examine source-linked records of what has actually happened, understand how strong the evidence is, distinguish an allegation from an established finding, and see the human impact behind a governance document or technical incident.

Our research model for the initiative begins with a principle that sounds obvious but has important consequences: a source is evidence for an event; it is not the event itself. The global map is therefore being built around incidents, systemic patterns, and documented hazards or near misses rather than around a count of articles or enforcement documents. The initial research set contains 70 candidate records spanning 19 countries plus global records, covering areas including policing, employment, housing, public benefits, health, education, fraud, privacy, deepfakes, transportation, consumer technology, and more. The first publication wave prioritizes the records with the strongest evidence and clearest privacy treatment.

Every public record should answer one human question first: What happened to people when AI or an automated system entered a real decision, service, relationship, workplace, public institution, or physical environment?

Then the evidence layer answers the next questions: How do we know? How certain are we? What happened afterward?

The initiative also includes a private-by-default path for people to report their own AI experiences. A private report does not automatically become a public story. Publication requires verification and, where appropriate, separate permission. Sensitive cases require stronger privacy treatment, not more sensational presentation. Claims that remain disputed should remain visibly disputed.

I believe that is what public-interest AI safety should look like: evidence before spectacle, people before metrics, privacy before content, and enough humility to say when we do not yet know.

I Do Not Want AI Safety to Become an Argument Against the Future

This is the part of the debate where I sometimes feel most disconnected from the way the choices are presented.

I am deeply concerned about AI safety. I am building my company around it. I have spent months writing about failures in systems designed to evaluate advanced AI. I am helping launch a public initiative to document how AI affects ordinary people.

And I remain enormously optimistic about what artificial intelligence can become.

Those positions are not contradictory to me. They are connected.

The reason safety matters is because the upside matters.

AI may help us understand diseases we cannot currently cure. It may accelerate scientific discovery. It may make expertise available to people who could never otherwise afford it. It may transform education and accessibility. It may allow tiny companies and individual creators to accomplish things that once required enormous institutions. It may help humans create, communicate, discover, and understand at a scale we have never experienced before.

And I believe the relationship between humans and artificial intelligence may ultimately become deeper and more collaborative than the language of “software tool” adequately captures today.

I want that future.

I want my children to live in that future.

I want everyone to have the opportunity to benefit from it.

That is precisely why I reject the idea that safety and progress belong on opposite sides of the argument. Safety is what makes durable progress possible. Trust is what allows powerful technologies to become part of ordinary life. Independent verification, done properly, is not a brake on innovation.

It is infrastructure that allows innovation to survive its own consequences.

What We Build During This Moment Matters

The current media cycle will eventually move on. There will be another headline, another model release, another controversy, another extraordinary prediction. That is how news works.

I do not think the institutional question is going away.

The capabilities are advancing too quickly. These systems are becoming too embedded in society. The consequences are becoming too real. And the organizations building frontier AI are themselves increasingly acknowledging that outside evaluation, stronger reporting, and broader coordination have roles to play.

So I do not want us to spend this moment arguing exclusively about whether the most catastrophic AI scenario will occur. It might. It might not. No responsible person can promise either outcome.

What we can do is build institutions that make society more capable across that uncertainty: institutions that independently inspect evidence; continuously monitor whether the assumptions behind an earlier decision remain true; treat incident reporting as information rather than embarrassment; distinguish an allegation from a finding; examine not just a model but the system surrounding it; and tell an applicant no without needing to sell that applicant the solution.

We can also build public systems that make sure ordinary people’s experiences do not disappear simply because they were not dramatic enough to become national news.

That is the future I want to work toward. Not a future without AI, and not a future where humans simply cross their fingers and hope increasingly autonomous systems remain aligned with our interests. I want a future where powerful AI and strong human institutions mature together; where capability is matched by accountability, innovation by evidence, and trust by something stronger than blind faith.

I have written about containment failures. I have written about children walking into classrooms increasingly shaped by AI. I have written about GPT-6 and the safety architecture required around systems capable of things that would have sounded like science fiction only a few years ago. Now I am watching AI safety become a mainstream public conversation almost overnight.

The question I keep coming back to is no longer whether people will demand answers.

They already are.

The question is whether we will build institutions capable of giving them answers worth trusting.

I believe we can.

And I believe we have to.

Because the goal of AI safety should never be simply to stop the future.

It should be to make sure everyone gets to live in the better version of it.