10.3 Detecting Concepts | Interpretable Machine Learning

So far, we have encountered many methods to explain black box models through feature attribution. However, there are some limitations regarding the feature-based approach. First, features are not necessarily user-friendly in terms of interpretability. For example, the importance of a single pixel in an image usually does not convey much meaningful interpretation. Second, the expressiveness of a feature-based explanation is constrained by the number of features. The concept-based approach addresses both limitations mentioned above. A concept can be any abstraction, such as a color, an object, or even an idea. Given any user-defined concept, although a neural network might not be explicitly trained with the given concept, the concept-based approach detects that concept embedded within the latent space learned by the network. In other words, the concept-based approach can generate explanations that are not limited by the feature space of a neural network. In this chapter, we will primaril

29 Detecting Concepts – Interpretable Machine Learning Author: Fangzhou Li @ University of California, Davis So far, we have encountered many methods to explain black box models through feature attribution. However, there are some limitations regarding the feature-based approach. First, features are not necessarily user-friendly in terms of interpretability. For example, the importance of a single pixel in an image usually does not convey much meaningful interpretation. Second, the expressiveness of a feature-based explanation is constrained by the number of features. The concept-based approac

related reading

• The Building Blocks of Interpretabilitydistill.pub
• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• Faithful, Interpretable Model Explanations via Causal Abstraction | SAIL Blogai.stanford.edu
• Transformer Circuits Threadtransformer-circuits.pub
• Feature Visualizationdistill.pub
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• Neuronpedianeuronpedia.org
• 6 – Interpretability – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu
• Mapping the Mind of a Large Language Model \ Anthropicanthropic.com
• Painting With Concepts Using Diffusion Model Latentsgoodfire.ai
• [2512.15712] Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistantsarxiv.org