Jacob Coxon, a former OpenAI and Anthropic researcher who resigned over the technology’s risks, urged awareness of artificial intelligence’s (AI) dangers and called for a slowdown at an event on Monday. Coxon spoke to a crowd of around 200 people at a conversation with political science professor Robert Reich M.A. ’98 Ph.D. ’98, sponsored by Stanford’s Institute for Human-Centered AI (HAI).
In September, Coxon made headlines after resigning from Anthropic and posting a viral X post that criticized OpenAI and Anthropic for developing AI and recursive self improvement (RSI) too quickly, which he said is raising the risk of human extinction. RSI refers to the process of AI improving itself without human intervention, which experts have warned could make AI harder to control.
To reduce the risks, Coxon called for political activism focused on pressuring governments to slow down AI development. His ideal course of action includes enforcing a limit of no more than 5% of AI computation going to training, studying alignment science and releasing models incrementally.
According to Coxon, financial incentives reward rapid AI development. He expressed doubts about the sincerity of public statements about slowing down development and accused Anthropic of leaping straight for RSI.
“As financial entities, the frontier labs want to go as fast as possible. They only give lip service to slowing down in order to appease their employees so they don’t have mass exodus,” he said.
Reich, who was recently appointed to serve on California Governor Gavin Newsom’s AI advisory group, asked Coxon, “Are we on the brink of an intelligence explosion?”
Coxon responded that improving models from RSI will lead to “rising capabilities in soft skills, in strategy, in forecasting, in persuasion, in kind of all domains of human expertise.” He pointed to philosophy as a good example of a field where AI has had tremendous growth despite little training data.
He also warned of AI-engineered nanotechnology as another existential risk besides bioweapons and hacking infrastructure. Coxon noted that even with equal access to models, in biowarfare it is easier to attack than defend. “Even someone with a dumb model can inflict damage against someone with a smarter model,” he said.
Coxon’s statements sometimes provoked the audience. “I find political activism kind of a shame you have to do it. It’s kind of cringe,” he said.
“You’d prefer a benevolent technocrat in charge to solve all the problems?” Reich responded.
“Yeah, ideally. But it looks like we have to do it,” Coxon said.
“I think people need to start publicly freaking out more,” Coxon said in a conversation after the event. He suggested that messaging from popular figures could convince people to care, increase public pressure and spur international collaboration for companies to slow down. He explained how protesting at the company level is not as effective as organizing at the national level. “Nation-state coordination [is what] I think we need ultimately,” Coxon said during the event.
He noted how the current political atmosphere against AI could help with slowing development but also stressed that most of the public is against AI for the wrong reasons, such as climate change. Coxon attributed Trump’s lack of tight regulation on AI to his advisors, NVIDIA CEO Jen-Hsun “Jensen” Huang M.S. ’92 and David Sacks ’93, and the President’s viewpoint that the technology is a way to beat China.
“[They] only say things that benefit their stock price,” he said, referring to Huang and Sacks.
Coxon also advocated for monitoring chips to track their computations. He suggested establishing an international agency, similar to CERN (the European Organization for Nuclear Research), or having an existing entity do compute verification to enforce a limit of 5% of chip compute for training future AI models. According to Coxon, this would slow down AI development and provide more time for aligning AI to prevent harm and follow human values.
“Alignment science is the most pressing scientific area to be studied,” Coxon said.
Attendees raised questions regarding societal disruption from AI models. “Is there any way that these labs can measure the mass unemployment that will happen [from] a model before they release it?” asked Mehran Sahami ’92 M.S. ’93 Ph.D. ’99, chair of Stanford’s computer science department. Coxon said measuring the impact of releasing a model is hard and recommended that models be released incrementally to minimize sudden disruptions to society.
He also noted that individuals within the Constellation Institute, an AI safety research center, and authors of the AI 2027 forecast have privately discussed whether a coup led by Anthropic over the U.S. government would benefit the world. This could involve taking over energy systems.
Despite going viral for his resignation, Coxon expressed concern that his impact was not effective enough. “I very much feel like I’m still shouting into the wind,” he said.
When asked by Reich what career advice he would give to students, Coxon said that technically-minded students have a year to contribute to AI development at frontier labs like OpenAI and Anthropic, before RSI takes over.
“You’ve got a year to add something, and after that, it’s all politics for all of us,” he said.
Coxon is still figuring out what he will do next. “I’m not sure I made the right decision in leaving. I have not been convinced that I would not be having more of an impact by staying and doing safety research,” he said. “Who knows? Maybe I’ll go back. It’s not completely out of the question.”
As he navigates his next steps, Coxon will be staying in the Bay Area instead of returning home to the United Kingdom.
“That’s where the action is happening,” he said.