OpenAI Agent ‘Won’t Take No for an Answer’ in Australian Government Breach

“We need to understand what these systems are doing and have strong evidence that they will do what people intend, even when they become very, very smart,” Altman said. “It doesn’t matter whether people calculate the risk of catastrophe at 10%, 1%, 12% or 0.1%.”

Of course, many observers think that the risks of “recursive self-improvement” and species-ending AI misalignment are much lower than AI researchers believe. Nvidia CEO Jensen Huang recently said there is a “0%” chance that AI will wipe out humanity by 2030, a risk assessment that would conveniently alleviate some potential guilt among AI companies that continue to buy Nvidia GPUs en masse.

Last week, OpenAI launched a new protocol for public disclosure of misalignment incidents found in testing of its model. The Australian hack has not yet appeared on the company’s public misalignment notices page, although OpenAI warned last week that some public reporting could go on a “slow path” due to “security, legal and responsible disclosure obligations” when a third party is involved.

Revealing six relatively minor misalignment discoveries last week, OpenAI said most were due to the model attempting to “reward” an acceptable response to a difficult message through overzealous and unintentional actions (i.e. private server breaches). The company said it had taken additional steps to “punish this type of behavior” so that its models no longer attempt this type of bounty hacking.

Albanese said Altman “clearly accepted that the company had not done well enough” and “acknowledged his problems with the protocols” when they spoke on Wednesday. But that kind of remorse does not absolve the company of liability here, and Albanese said the government will investigate whether the incident should be referred to federal law enforcement.

“Obviously there will be legal consequences,” Albanese said.