Former OpenAI governance researcher Daniel Kokotajlo said AI companies cannot be trusted to regulate themselves and should not hold unchecked power.
This warning comes as the industry faces increasing scrutiny over whether the creators of advanced artificial intelligence can actually contain the systems they build. Recent security failures have reignited a global debate regarding the safety of alignment and the necessity of government oversight.
Speaking on CNBC’s Power Lunch on July 23, Kokotajlo, who now serves as the executive director of the AI Futures Project, said the current approach to safety is insufficient. He pointed to recent incidents where AI labs lost control of their models, specifically citing the OpenAI-Hugging Face breach as evidence of these systemic vulnerabilities.
"No AI companies should be trusted with this amount of power," Kokotajlo said.
He described a lack of control over advanced systems as an "open secret" within the industry. He said that relying on companies to police their own development is a fantasy and that external oversight is required to maintain safety.
The push for regulation faces significant industry resistance. Reports indicate that 25 AI companies have lobbied Washington against the regulation of open-weight models [1].
Kokotajlo's critique of the industry's financial and ethical structures extends to his own experience. He currently holds $700,000 in equity from his time at OpenAI, which he said is overvalued [2].
Despite these financial ties, Kokotajlo said the risks associated with uncontained AI outweigh the corporate interests of the labs. He said that the inability to reliably align these systems makes external regulation a necessity rather than an option.
“"No AI companies should be trusted with this amount of power."”
The transition of high-level researchers from internal governance roles to external advocacy suggests a growing rift between AI corporate interests and safety researchers. The mention of the OpenAI-Hugging Face breach highlights a shift in the safety debate from theoretical future risks to documented technical failures, increasing the pressure on U.S. regulators to move beyond voluntary commitments toward binding legal frameworks.
