MIT Researchers Advance AI Explainability with Concept Extraction

In high-risk environments such as medical diagnostics, trust in artificial intelligence (AI) systems is often contingent on the ability to explain how predictions are made. Addressing this, researchers from MIT and the Polytechnic University of Milan have introduced a novel method that enhances the explainability of computer vision models, drawing on concepts the models have already learned, rather than relying solely on predefined human concepts.

The research, led by Antonio De Santis during his time at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL), focuses on improving concept bottleneck models (CBMs). CBMs force a deep-learning system—such as a neural network—to make decisions based on specific, human-understandable concepts before offering a final prediction. For example, in medical imaging, a model might first identify features such as "clustered brown dots" or "variegated pigmentation" to assess whether a skin lesion is likely melanoma.

Historically, these intermediate concepts are established either by clinical experts or generated by large language models (LLMs), such as those employing transformer architectures. However, if the predefined concepts do not align well with a particular task, or if they are too broad, the resulting predictions may suffer from reduced accuracy and limited transparency. Moreover, models sometimes use unintended information not captured by the official concept set—a phenomenon known as information leakage.

To address these challenges, the MIT-led team developed a method that extracts from the model itself the most relevant conceptual features learned during training. This involves a sparse autoencoder, a deep-learning architecture that compresses and reconstructs key features, which identifies a minimal set of significant concepts. These are later labeled in plain language through a multimodal LLM, providing both human-readable descriptions and automatic annotations to the corresponding images.

The team restricts each prediction to just five extracted concepts, focusing explanations and minimizing the inclusion of extraneous or potentially biased information. The annotated data is then used to train a dedicated concept bottleneck module, which is integrated back into the original model. This process enables the system to explain its predictions using only those distilled concepts, improving both accuracy and interpretability compared to conventional CBMs.

In comparative evaluations, the new approach demonstrated superior accuracy and clarity across various tasks, such as distinguishing bird species and identifying dermatological conditions in medical images. The results suggest that using model-learned concepts reflects a more authentic picture of the AI’s reasoning process.

Nevertheless, the researchers acknowledge ongoing challenges. Black-box AI models that lack interpretability still surpass their explainable counterparts in performance, indicating a persistent trade-off between transparency and predictive power. To further mitigate issues like information leakage, the team is considering the introduction of additional concept bottleneck modules and expanding the approach with more advanced multimodal language models and larger datasets.

Notably, this work has broad implications for the development of responsible and trustworthy AI, especially in sectors where explainability is critical. Andreas Hotho, a professor at the University of Würzburg who was not involved in the study, notes that deriving explanations from a model’s internal mechanisms paves the way for more faithful and structured AI reasoning, potentially linking current systems to symbolic AI and knowledge graph methodologies.

The research received support from several European institutions, including the Progetto Rocca Doctoral Fellowship, the Italian Ministry of University and Research, Thales Alenia Space, and the European Union under the NextGenerationEU initiative.

Source: news.mit.edu.