Experts weigh in as researcher says AI has more than 10% chance of 'killing all humans'

An artificial intelligence researcher quit his job at Anthropic on Tuesday and accused the company, and its chief rival OpenAI, of acting irresponsibly, igniting a frenzy of concern on social media about the rapid pace of the technology’s development.

Jacob Coxon, who has worked as a researcher at both companies, said in a post on X that he resigned out of concern that Anthropic and OpenAI are “gambling with our lives.” He said the people building AI “earnestly believe that it could kill us all by the end of the decade.”

“Do not underestimate the power of this technology,” Coxon wrote. “These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.”

Coxon’s post, which has been viewed more than 70 million times, reflects a longstanding debate in Silicon Valley about whether AI can be safely developed and controlled. As Anthropic and OpenAI barrel toward potentially historic IPOs while releasing increasingly advanced models, many researchers are calling for a coordinated slowdown. 

OpenAI’s chief scientist Jakub Pachocki published a blog post on Sunday and warned that no AI company has “solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer.” In the AI industry, alignment refers to the work by AI developers to ensure that the system behaves in accordance with human values and intentions.

“I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established,” Pachocki wrote. “And I believe that international coordination on future AI development needs to become a top priority for governments around the world.”

Coxon’s post on Tuesday also struck a chord with industry researchers who are worried about recursive self-improvement, or an AI system becoming capable of designing and developing its successor without human intervention. Recursive self-improvement is not yet possible, but companies, including Anthropic and OpenAI, have warned that it would make it easier for humans to lose control over those systems. 

“Neither company is acting responsibly,” Coxon wrote. “They are racing straight to self-improving superintelligence.”

Evan Hubinger, an alignment lead at Anthropic, echoed Coxon’s comments in a post on X late Tuesday.

“Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” Hubinger wrote. “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

While extreme, concerns about the potential for AI to cause human extinction or other catastrophic events are not new in AI research circles. In 2023, for instance, prominent AI researchers and executives, including OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei, signed a statement that said “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Some experts even use a shorthand, p(doom), to estimate the probability of dire outcomes that could stem from AI.

Anthropic’s Hubinger was also one of roughly 1,400 AI researchers who signed an open letter called “Pacing the Frontier” in July. The letter urged the U.S. government to develop the tools necessary to support an effort to “deliberately pace the frontier of automated AI development.”

Some members of Congress have taken steps to try and address AI’s rapid advancement in the months following, but there’s no clear consensus about how the technology should be regulated. 

In July, Rep. Jay Obernolte (R-Calif.) and Rep. Lori Trahan (D-Mass.) introduced a bill called the FRONTIER Act, which aims to establish a framework for governing the deployment of advanced AI models. And earlier this month, Sen. Bernie Sanders (I-Vt.) and Rep. Greg Casar (D-Texas), introduced a bill called the Ban Artificial Superintelligence Act, which would temporarily pause advanced AI development until the federal government establishes safety rules. Both bills have been met with mixed reception.

“Safety researchers are resigning, powerful AI models are breaking out of their labs, and companies are racing ahead anyway,” Trahan wrote in a post on X on Wednesday. “It’s past time for Congress to get off the sidelines and do its job.”

Lawmakers are also trying to navigate growing public backlash against AI data centers, the large facilities that house the hardware for training and running AI models. The pushback has grown so intense that the The National Republican Senatorial Committee, or NRSC, said last month that data centers have become a “sleeper issue” for the entire midterm election cycle, as CNBC previously reported. 

Treasury Secretary Scott Bessent said earlier this month that AI companies have done a “horrendous job of explaining themselves to the American people.” 

“They’re going to have to take some of the blame, and they are going to have to convince the American people that all the benefits will not accrue to a small group,” Bessent said, following the G20 meetings with finance ministers and central bankers in Asheville, North Carolina. “That’s what they hear from me.”