MIT Develops Improved Method to Detect Overconfident Language Models

Researchers at the Massachusetts Institute of Technology (MIT) have developed an advanced technique to address the long-standing issue of overconfidence in large language models (LLMs)—AI systems that generate human-like text but sometimes provide plausible yet incorrect answers.

Traditional methods for evaluating an LLM’s reliability often involve uncertainty quantification, where a prompt is submitted multiple times to gauge if the model consistently returns the same answer. This approach, however, only measures how internally confident an individual model is, not whether its confidence is justified. This can lead to users being misled by models that are confidently incorrect, a significant risk in high-stakes environments such as healthcare and finance.

To address this, a team led by Kimia Hamidieh, a graduate student in electrical engineering and computer science at MIT, devised a method that shifts the measuring focus. Instead of relying solely on a single model’s behavior, the new technique estimates a different type of uncertainty—epistemic uncertainty—by comparing the target model’s responses to those from a group of similar language models.

Epistemic uncertainty quantifies the risk associated with using the model itself, rather than just its internal variability. The MIT method involves assembling a small ensemble of LLMs trained by different organizations but sharing similar architecture or scale, and comparing their outputs for given prompts. A higher level of disagreement across these models signals greater epistemic uncertainty and a higher chance that even seemingly confident predictions may be incorrect.

Combining this ensemble-based measure with a model’s inherent self-consistency—also known as aleatoric uncertainty—the researchers created what they term a "total uncertainty" (TU) metric. The TU metric was tested on ten practical tasks, spanning question-answering, mathematical reasoning, translation, and summarization, showing consistent outperformance over traditional uncertainty metrics in identifying unreliable outputs.

“Our results show that relying solely on one model’s self-consistency is insufficient for trustworthy AI,” Hamidieh explained. “Evaluating cross-model disagreement offers a more robust way to flag confidently wrong predictions.” Co-authors include Veronika Thost of the MIT-IBM Watson AI Lab, Walter Gerych of Worcester Polytechnic Institute, Mikhail Yurochkin of MIT-IBM Watson AI Lab, and senior author Marzyeh Ghassemi of MIT.

A key advantage of the new method is that, by leveraging diverse models with different training backgrounds, it helps expose blind spots specific to individual models. The ensemble approach proved particularly effective for tasks with clearly defined correct answers, such as factual question-answering, though researchers note it is less reliable for open-ended problems where answers may legitimately vary.

Moreover, estimating total uncertainty often required fewer interactions (queries) with the models, which could reduce computational costs and lower the environmental impact associated with large-scale AI operations.

Looking ahead, the team plans to adapt and refine the approach for more open-ended tasks and to explore additional forms of uncertainty quantification. The work was funded in part by the MIT-IBM Watson AI Lab.

Reference: news.mit.edu{:target="_blank"}