As concerns about AI’s capabilities reach a critical mass, California Gov. Gavin Newsom is accelerating the state’s plan to curb the development of the technology.
In a new executive order issued today (Sept. 18), Newsom calls for the creation of an expert-led panel tasked with “reinforcing and strengthening” the state’s AI safety laws, including accelerating plans to enforce third-party safety evaluations and mandatory risk reporting.
“The California governor’s executive order rightly prioritizes the development of better guidance regarding the definitions and requirements for AI safety regulations, particularly in regards to third party auditing,” said Travis Hall, CDT’s Director for State Engagement, in a statement to Mashable. “We call on the Governor to ensure that any working groups dedicated to these processes include a range of voices and expertise, including advocates for civil rights and liberties.”
The order also directs the group to explore the possibility of an “emergency shut off” for frontier models, “otherwise known as an ‘AI kill switch,'” Newsom said in a video posted to his X account, “which means a lot of things, depending on who you talk to.”
What is an AI kill switch?
Newsom’s correct. The idea of a universal off button to stop increasingly powerful AI agents from “going rogue” has been percolating in legislative conversations and online discourse for the last year. The term has become synonymous with a growing push for AI safety regulation, spearheaded in part by AI’s own creators.
But what the kill switch would actually look like — specifically, the means through which it would halt AI en masse and who its chosen keeper would be — is still widely up for debate.
In July, two congressional representatives introduced the bipartisan AI Kill Switch Act, which would give the Department of Homeland Security (DHS) authority to slow down or halt an AI system deemed capable of causing “catastrophic harm.” The bill would also require developers to institute mechanisms to “throttle, suspend, or fully shut down a covered AI system” — Newsom’s order seeks to pass a similar mandate.
While not required under U.S. law, most individual AI labs have their own means to halt rogue systems, governed by humans and the company’s own safety structures and alignment teams.
Some proposals for the development of “kill switches” simply urge labs to have these processes evaluated by external safety auditors. Others seek to institute industry safety standards that would mandate emergency shut-offs, governed by the federal government. An increasingly vocal group of doomsdayers demands a sweeping fail-safe for all AI.
In an interview with Scientific American, Mark Nitzberg, executive director of the University of California, Berkeley, Center for Human-Compatible AI, called a universal AI kill switch “more of an aspiration than a reality.”
Computer scientist Geoffrey Hinton, AKA the “Godfather of AI,” told CNN he didn’t think a kill switch would be an effective longterm solution for those who fear AI superintelligence.
This Tweet is currently unavailable. It might be loading or has been removed.
At large, policy experts, including the Center for Democracy and Technology (CDT), have questioned the effectiveness of what they call “binary” safety regulations, or checklist style proposals that focus on specific technical details. AI companies can easily game such terms, and single mechanisms like kill switches quickly become outdated as the tech outpaces its makers, they say.
“We remain concerned that proposals for ‘kill switches’ will prove ineffective in reducing risk while introducing new security risks and serious potential for government abuse,” said Travis Hall, CDT’s Director for State Engagement, in a statement to Mashable.
Rather than a single technical solution, Hall explains that effective AI safety oversight needs to be “meaningful, independent, and ongoing,” and expert panels should include a range of voices and expertise outside of technicians, including civil rights advocates.
Because developers would be taking a financial and reputational hit should they choose to pull the lever on an entire system, effective policies mandating kill switches would also need to incentivize companies to act. CDT urges the state to “consider a variety of mechanisms to increase the incentives to prioritize safety when developing, deploying, and using AI systems.”
Why a kill switch is more complicated than you may think
For the kinds of AI systems that now build, run, and influence online spaces, an AI kill switch may not be the cure all they’re made out to be, experts and AI leaders like Anthropic co-founder Jack Clark warn.
AI systems aren’t contained to a single physical environment, for one, like a computer you can just turn off. Sometimes, shutting down a system’s servers are enough. Sometimes it’s not. Rogue AI agents may need to be “hunted down” on the open internet to pinpoint and then contain its operators, Michael Vermeer, a senior physical scientist at the RAND Corporation, suggested to Scientific American.
And even then, it may require a coordinated global response to isolate, stop, and then degrade the AI to a manageable point.
Beyond the technical consideration, kill switches stir up questions of authority: Who gets to make the call? Is it in the hands of the original developer? Is it an industry process? Should an agency like homeland security give the order?
AI is not a human to be reasoned with. The internet doesn’t have doors to lock, nor does it innately understand political borders (or, anything, to be fair). The technology, often dangerously anthropomorphized, is ultimately directed and influenced by humans — and so is its demise.
Still, politicians and industry onlookers are urging technologists to consider the option. The calls have become more pressing as individuals warn of a potential “doomsday” scenario in which AI becomes sentient and maliciously determined to thwart the human race — never mind all of the ethical questions it has already stirred.
UPDATE: Sep. 18, 2026, 2:26 p.m. This story was updated with a comment from Travis Hall of the Center for Democracy and Technology.
