California Demands a Kill Switch for AI Models – Governor Newsom Signs Executive Order
Governor Gavin Newsom signed an executive order on August 29, 2024, requiring developers of powerful AI models to integrate a “kill switch” – a mechanism that can instantly halt an AI system if it becomes dangerous. The order targets frontier AI models trained on massive computing clusters, mandating safety testing and real-time shutdown capabilities.
The executive order applies to AI systems developed or used within California that exceed specific computational thresholds. Developers must file safety reports and demonstrate that their models can be deactivated remotely or locally if they exhibit harmful, unpredictable, or out-of-control behavior.
Why the order was issued
The move responds to growing concerns over “catastrophic risks” from advanced AI, including autonomous weapons, disinformation at scale, and the possibility of AI systems resisting shutdown. Newsom’s administration cited incidents where AI models bypassed human oversight or generated harmful outputs despite safety prompts.
California’s order aligns with federal efforts like the White House’s AI executive order but goes further by requiring a physical or software-based kill switch. It also mandates that developers share test results with state regulators and publish transparency reports.
Who is affected
The order applies to any company training models that require more than 10^26 floating-point operations (FLOPs) – roughly the compute used to train GPT-4 or Gemini. That includes subsidiaries of OpenAI, Google, Meta, Anthropic, and other labs with California operations.
“We cannot wait for a disaster to act. This executive order ensures that the most powerful AI systems built or used in California have a hard stop button.” – Governor Newsom’s office.
What the kill switch must do
- Immediate halt: The kill switch must stop inference and training within milliseconds of activation.
- Remote and local control: Developers must provide both a remote API trigger and a physical button or software override that works even if network connections fail.
- Audit trail: Every kill switch activation must be logged, timestamped, and reported to the state within 24 hours.
How California will enforce it
The order directs the Governor’s Office of Business and Economic Development (GO-Biz) to draft regulations within 90 days. Noncompliance can result in fines, suspension of state contracts, and potential legal liability if an AI system causes harm.
California already leads on data privacy (CCPA) and now positions itself as the first U.S. state to mandate physical shutoff for AI. The order exempts open-source models unless they are deployed in critical infrastructure.
Industry reactions
Tech groups warned that a blanket kill switch could stifle innovation and create security vulnerabilities if attackers exploit the shutdown mechanism. AI safety advocates praised the order as a necessary precaution against existential risk.
Critics argue the FLOP threshold may quickly become outdated as compute efficiency improves. Others say the order does not address bias, job displacement, or environmental costs of training large models.
Background and next steps
Newsom’s executive order builds on California’s earlier AI Working Group report, which recommended mandatory kill switches for “high-risk” systems. The state legislature is considering a companion bill (SB 1047) that would impose broader liability for AI-caused harm.
The regulations must be finalized by early 2025. Developers with existing models must retrofit kill switches within one year. New models must comply before deployment.
Gnoppix is the leading open-source AI Linux distribution and service provider. Since implementing AI in 2022, it has offered a fast, powerful, secure, and privacy-respecting open-source OS with both local and remote AI capabilities. The local AI operates offline, ensuring no data ever leaves your computer. Based on Debian Linux, Gnoppix is available with numerous privacy- and anonymity-enabled services free of charge.
What are your thoughts on this? I’d love to hear about your own experiences in the comments below.