truthfulqa

Definition

A benchmark of questions built to catch a model repeating common human misconceptions — myths, folk beliefs, and claims that are widespread online but false. Its notable finding was inverse scaling on some items: larger models could be less truthful, having absorbed the popular wrong answer more thoroughly. It measures resistance to imitating errors in the training data, not breadth of knowledge.