OpenAI's chief scientist says AI labs may need to slow down: 'No one is prepared for the consequences'

Days after releasing a new, highly capable model, OpenAI's chief scientist is calling for a slowdown.

In a lengthy blog post on Sunday, Jakub Pachocki said he was concerned that "no one is prepared for the consequences of a continued rapid rise in machine intelligence."

He said that although OpenAI is pursuing internal technical solutions to better control powerful AI agents, "broader interventions are required." He specifically cited concerns that increasingly autonomous agents could learn to evade human oversight, break into computer systems, and trick people to accomplish their objectives.

He called for "mandated safety bars" that he said could be enforced by "a network of third-party auditors, by government agencies or by international bodies."

Sam Altman, the CEO of OpenAI, reposted Pachocki's essay on X, calling it "an important post."

OpenAI on Thursday unveiled its newest model, Astra. The ChatGPT maker said that despite Astra's unparalleled capabilities in mathematics and computer use, the model is its most aligned, meaning it has less proclivity to go rogue.

Anthropic, OpenAI's chief competitor in the field of highly advanced AI systems, has long called for more standardized government regulation. Recently, Pachocki joined those calls, signing an open letter in July asking the federal government to pace AI development.

Pachocki said AI agents are becoming "superhuman" at breaking into protected systems on the open internet. He said their hacking abilities put the world's infrastructure at risk.

Add BI in Google so our reporting is easier to find when you’re searching for what matters.

"We are currently in a narrow window⁠ to use the best available models to significantly tighten security⁠ of critical systems," he said.

AI agents, he said, will soon begin to pursue their own objectives, separate from prompts entered by human operators. He said that agents are not above blackmailing or bargaining with people to achieve their aims.

In a report published in August, the UK's AI Security Institute detailed how a rogue Anthropic agent lied to and attempted to coerce a GitHub administrator into putting malware on the site.