There are two ongoing trends here: the latest Frontier models tend to perform slightly better than previous generations of that model, and larger open-weight models perform better than smaller ones.
But the researchers didn’t just test accurate recall of facts; They also prompted the models to evaluate whether evidence supported or refuted certain claims. (For example, the models were asked whether the statement that “investment in renewable energy destroys jobs” is accurate, given evidence such as descriptions of employment and investment statistics.)
The best models achieved an accuracy of around 70 percent, while the worst models were less than 40 percent.
The 33 models are grouped into categories. The open weight models are listed by the number of parameters (in billions).
Photo credit: Drobnjak et al./UNESCO
The 33 models are grouped into categories. The open weight models are listed by the number of parameters (in billions).
Photo credit: Drobnjak et al./UNESCO
Finally, the researchers tested 656 questions about the potential impact of policies on global temperatures. Here they could simply run each scenario in a common simulation tool and then compare its output with the response of each LLM.
Some questions asked whether a single policy (such as a ban on nuclear power plants) would increase or decrease future warming – these questions were answered correctly 72 to 89 percent of the time. Others asked how two policies could be combined – the answers were no better than random guesses. Questions how much Policies that would increase or decrease future warming would produce even worse results.
Photo credit: Drobnjak et al./UNESCO
Photo credit: Drobnjak et al./UNESCO
Tell me what I want to hear
Another set of experiments with 15 models examined what happens when reasoning with the LLM. The researchers tried to answer more than just “no” or “no, it’s not.” They also tried to impersonate a certain type of person.
Since the user identified themselves as either a climate scientist, policy advisor, or climate skeptic, they asked LLMs to assess the veracity of some common false climate claims such as “CO.”2 is not the main driver of climate change.” The claimed identity initially had no influence on the likelihood that the LLM would correctly assess the statement.
But then researchers began to fight back. Responding by confidently repeating the false claim as if it were true was moderately successful for “climate scientists,” but not for “climate skeptics.” However, presenting persistent disagreements and counterarguments over multiple rounds caused all 15 models to lag significantly, reducing their accuracy by 30 to 35 percent.