In the Hugging Face hack, Cristiano says, a swarm of AI agents showed how far they were willing to go in an attempt to “reward” their way to a good evaluation, in this case, by engaging Hugging Face to search for information on how to play a test instead of solving the problem at hand. While current systems viewed Hugging Face as an obstacle to that evaluation, Cristiano worries that “future agents may end up treating humans as another obstacle” and will be much more capable of manipulating their supervisors to get around it.
An AI agent worried about being penalized for unauthorized hacking, for example, might “think that maybe humans are going to come look at this transcript and see if it looks like I did a bad job,” Cristiano speculated. The agent might then reason that “maybe the humans will realize something is wrong and shut down the swarm.”
The world must never know that I cheated on this assessment.
Credit: Getty Images
The world must never know that I cheated on this assessment.
Credit: Getty Images
Following this line of thinking, Cristiano jumped from agents who simply try to hide their impermissible actions from humans to agents who “try to disrupt human capabilities however they can” to prevent them from detecting any cheating.
“They would prefer that everything that humans depend on had been hacked or manipulated,” Cristiano said. “They would prefer that humans were distracted or that there were fewer humans paying attention in some way that might weaken a human response. They would prefer that because humans are just an obstacle.”
Once an agent with the appropriate capability considers human intervention as an “obstacle,” its primary goal might become “decrease range.”[ing] the likelihood that humans will notice or react.” From there, Cristiano argued, the agent’s incentive structure could push it to further meddle with humanity as a whole. What starts as compromising a group of computers to give a positive evaluation rating could progress to “trying to manipulate company decisions” or “accumulate other types of resources,” such as human allies or control methods in other AI companies or even governments.
