Anthropic Researcher Resigns Over AI Safety Concerns, Warns of Existential Risks

Jacob Coxon, a 27-year-old British researcher at Anthropic, has publicly resigned, expressing grave concerns about the rapid development of artificial intelligence and its potential to lead to human extinction. His departure, which occurred around September 8-9, highlights a growing unease among AI professionals regarding the manageability of advanced AI systems.

Coxon’s resignation is particularly alarming as it has garnered support from his colleagues at Anthropic. Evan Hubinger, the Alignment Science Lead, echoed Coxon’s fears on social media, estimating a greater than 10% chance of AI causing significant human fatalities within the next decade. He also noted that Anthropic currently lacks a clear strategy for aligning with superintelligent systems. Samuel Marks, another researcher, shared similar concerns, indicating that many senior employees at the company are worried about potential extinction-level risks.

Coxon criticized the language used in AI development, such as “crunchtime” and “endgame,” arguing that it frames the pursuit of superintelligence as an urgent competition rather than a significant risk that needs careful management. This resignation is not an isolated incident; it follows the departure of Mrinank Sharma, who led Anthropic’s Safeguards Research team, earlier in the year due to similar concerns about escalating dangers posed by AI.

The irony of the situation is notable, as Anthropic was founded by former OpenAI staff who aimed to prioritize safety in AI development. Now, their own safety researchers are leaving due to perceived negligence in addressing these critical issues. Coxon specifically pointed to international competition, particularly from Chinese AI labs, as a factor exacerbating the urgency to advance AI technologies without adequate safety measures.