Explainable Concept Generation through Vision-Language Preference Learning
Concept based Explainable AI;Vision-Language Models;Reinforcement Learning
Yes
No ethics review needed.
1. While Tabel 4 shows the concept images generated by RLPO are more diverse, how do we know if they are faithfully reflecting the the same concept instead of overfitting the TCAV?
2. Why should the problem be designed as a sequential decision-making problem? 
3. Do people need to finetune one RLPO framework with DQN+Diffusion for each class? Would it be more efficiently and equally effective if they just use an untuned Diffusion model and prompt LLMs/VLMs to generate more detailed text descriptions for the target class given the seed prompt?
4. What is $t_\eta$ in Property 2?
1. Optimizing diffusion models to maximize TCAV scores sounds interesting and original to me.
2. The authors performed extensive experiments to demonstrate the effectiveness RLPO.
3. Most parts of the paper are clearly written and easy to follow.
4. The visual results presented are helpful in illustrating the advantages introduced by RLPO.
This paper focuses on the generation of concept images to explain black-box image classification models. It proposes a reinforcement learning-based preference optimization (RLPO) algorithm to fine-tune a diffusion model for generating images that can maximize the TCAV scores. It also proposes to use DQN to search appropriate actions from the seed prompts. Experiments show that the proposed approach can generate complex and abstract concepts aligning with the test class.
1. While diffusion models make the whole system more flexible than "retrieval methods" by generating new images that may not be presented in current datasets, the quality of such images is questionable. For example, in Figure 5, the generated image at timestep 30 looks unrealistic and is not closely aligned with the seed prompt (worse than the initial image).
2. Though Table 1 shows RL helps the model to select approprate seed prompts from the given set, I'm still concerned about the necessity of using DQN in the framework. VLM/LLM can be used to generate a small high-quality seed prompt set, and even rank them based on their potential importance (e.g., the experiment in this paper has a set with only 20 prompts). One can use the whole set and focus on the fine-tuning of the diffusion model. 
3. The efficiency seems to be a problem.
Yes
No ethics review needed.
Please see my comments on the weakness section.
The paper is well-written and easy to follow. The problem definition is novel and interesting.  The paper effectively reframes the generation of concept image sets as an image generation problem, providing a novel perspective that addresses the limitations of traditional concept-based explanation methods. This shift enhances the efficiency and effectiveness of generating meaningful concepts.
Through a series of well-designed experiments, the paper demonstrates the capability of the proposed method to generate complex and abstract concepts that align with the target class. This empirical evidence strengthens the paper's contributions.
Concept-based explanation methods typically require practitioners to guess and gather multiple candidate concept image sets, a process that can be imprecise and labor-intensive. To address this challenge, this paper redefines the creation of concept image sets as an image generation problem. To this end, in this work, the authors introduce an RL-based preference optimization algorithm that fine-tunes the vision-language generative model using approximate textual descriptions of concepts. They also conduct extensive sets of experiments to demonstrate that the proposed method effectively articulates complex and abstract concepts that align with the target class, which are often difficult to create manually.
While the authors demonstrate the effectiveness of their proposed method through both qualitative and quantitative analyses, several aspects of the framework are not thoroughly justified. For instance:

(a) The generated concept sets using SD+LORA are randomly divided into two groups. Why is "random" grouping considered optimal? Could this be problematic?

(b) The use of reinforcement learning (RL) is also not justified. Is it really necessary to implement an RL policy for the purpose mentioned in the paper? Why not simply calculate the TCAV score for each possible seed prompt kt, and then fine-tune the SD+LORA weights based on the seed prompt that yields the highest TCAV value?

(c) Incorporating each component of the proposed framework increases computational demands and time constraints. Consequently, such an analysis may not be feasible in real-time, as generating images with Stable Diffusion, training the DQN-based RL policy, and fine-tuning the Stable Diffusion model with preferences all require significant processing time. Is this complex pipeline truly necessary?

(d) I also struggle to see a practical significance for the proposed problem. Why is it important to generate a diverse set of concept images at all? It seems more crucial to generate concepts that directly explain the task at hand.

(e) Furthermore, why choose DQN? As we continuously update the LORA weights based on the TCAV preference score, the underlying RL environment becomes non-stationary. This means that the same action taken by an RL policy could lead to different reward values at different times. How can this issue be addressed?
Yes
No ethics review needed.
Same as weakness
The idea is very innovative, and the writing is clear. By using a generative model instead of traditional manual concept collection, it reduces human intervention and improves efficiency. RLPO can generate concepts at different levels of abstraction, offering a more detailed explanation of the DNN’s internal decision-making process.
Authors proposed a new algorithm, Reinforcement Learning-based Preference Optimization (RLPO), designed to generate high-level visual concepts that explain the decisions of DNNs. Unlike traditional concept-based explanation methods (such as TCAV), RLPO creates sets of concept images, eliminating the need for manual concept image collection, thus making the process of explaining DNN decisions more efficient and generalizable. RLPO fine-tunes a stable diffusion model with preference optimization to generate images that effectively explain neural network decisions.
In some sections (such as the algorithm explanation and mathematical proofs), the descriptions appear overly lengthy, which might hinder readers' understanding. The description of how RLPO is applied in sentiment analysis tasks is somewhat vague, and further detailing the specific steps of this experiment could be helpful. Certain terms (such as “concept generation” and “concept extraction”) are defined and used inconsistently throughout the paper, which could lead to confusion.
Yes
No ethics review needed.
Questions:
1. “As a specific application, we see what concepts are removed and added, as well as how the concept importance changes when we fine-tune ResNet50 model on ImageNet to improve accuracy” If we want to track the evolution of concepts during the training procedure, do we need to retrain the concepts every time we update the model, or is there any workaround which would help track the concepts during the finetuning process? Either way is fine, however might be good to clarify it in the paper or in the appendix. 
2. Figure  9 should have the x axis annotated ( I understand it represent concept identifier)?
3. One of the concerns might be the lack of systematic quantitative comparison with the retrieval-based explanations. Is there any possibility to compare the explanations numerically with the state-of-the-art prototypical retrieval-based explanation methods, such as, e.g. Craft (Fel et al, 2023)?
4. I think there is one important limitation of the proposed method which I would like the authors to cover in the paper: the explanations also, in a way, depend upon the inner working of the generative model. Imagine, for example, that the generative model has mode collapse and does not represent the whole set of patterns available to the model we explain. In this case, it would lead us to obtaining the best possible explanation amongst the suboptimal ones, which may lead to the explanations not representing some aspects of the model’s inner working. In a case when it is a safety-critical application, that mode collapse might represent some anomalous event, which may not be covered by the set of explanations achievable by the model and therefore would not allow us to understand the reasons behind the model. A variation of such problem might include the problem of explanation when the generator only offers poor quality of image output(for whatever reason, e.g. it does not cover some particular concept). To make it clear, I understand that there is also a counterpart to such limitation in a case of standard, retrieval-based, prototypical explanation: there might not be such a piece of data which would closely match the phenomenon. It is therefore, in one way or the other, a limitation of many post hoc explanation models. I wonder if the authors agree with such limitation, and in any case would like to ask to include  the discussion. 
5. In relation to that, perhaps not mandatory, but another idea of an experiment: how does the numerical performance of the algorithm, e.g. Figure 8, would compare for different image generators? Does the GAN model, which is prone to mode collapse, result in worse C-insertion metrics?
Novelty: the work builds upon existing methods in interpretability, such as concept based explanation, but contains a set of novel ideas including (1) the problem statement and the method for optimisation of concept set through reinforcement learning technique and LoRA (2) the idea to use such concept set generation for to produce explanations for different abstract layers

Clarity: the paper is clearly written and well presented. 

Motivation: The authors give a strong motivation for the proposed method: the retrieved data might not filly describe the concepts embedded into the model, and the optimisation of such concepts using generative models and RL can be a promising alternative

Significance: I think this work is significantly improving the available options for explaining visual (and potentially language) models and therefore I believe it is significant enough. 

Correctness: I checked the paper and I believe the claims and the maths are correct

Reproducibility: as far as I can see, taking into account both the paper and the code, the experiments look reproducible to me.
The paper proposes to treat the concept set creation as an optimisation problem. Through the means of deep reinforcement learning, it iteratively refines the concepts obtained using Stable Diffusion generative model so one can generate concepts instead of retrieving them in the existing data. They also introduce the concept generation with respect to an abstraction level. The authors also explore the ideas of analysing the concepts during fine-tuning and NLP problems.
Clarity: there are a few questions regarding the limitations of this work (see below in the questions section)
Yes
N/A
No ethics review needed.
Maybe I missed it, but is there any information on the training time of the proposed algorithm? If not, I think this could be valuable information to provide at least in the appendix, but mentioned in the main text. The authors mentioned limitations, but it would be good to have real numbers.

Whats the difference between the concept images in Figure 6 and 5? In figure 5 they look very abstract, whereas in figure 6 the are high-quality images.
The paper tackles an important topic, and I find the combination of previous approaches that the authors propose with their work intuitive and interesting.
This paper introduces an algorithm for generating concept images to post-hoc explain a model. It hereby focuses on the use of the TCAV score, an RL learning setting and stable diffusion.
Although i find the algorithmic approach interesting enough, I am unsure what the actual underlying goal is of the approach and how this is evaluated/evidenced in the experimental section. I.e., in the intro the authors mention "Therefore, it is importantto automatically find human-centric concepts that matter to the DNN’s decision-making process." So my understanding is that the ultimate goal is to provide concepts that are helpful for humans to understand the decision making processes, but without the need to provide extensive concept data precollection. But I am missing a clear experiment to evaluate the usefulness of the generated concept images. Table 2 seems to be hinting at something along this line, but the details are not clear to me from the text. IN any case this should be one of the key evaluations to perform. Especially, as the exemplary concept images look far too abstract for me to be able to understand what they should represent. Please point me towards the relevant sections in case this is missing. 

Overall, I am leaning towards accept, but would like the following points clarified first:
- What is the exact goal of the method?
- How do the experimental evaluations provide evidence for reaching this goal?

Minor: Overall, the paper could benefit from a grammar check, e.g., "also explains what type of features is the model focuses on." in line 426
Yes
No ethics review needed.
The most important question is: can the authors provide any experimental result that concretely shows that their method is able to explain model decisions? 

Some secondary questions are: 
1) Why RL at all? Why not just optimize every "seed prompt"?
2) How often do seed prompts converge to the same outputs? 
3) This work brings up the question, under what domain should we care about explaining model behavior? The generated images are OOD for the model. When do we want to explain OOD behavior for the model? Is this useful?
4) How does your method relate to adversarial attack methods?
5) How long does it take for your method to run?
(S1) The authors identify an interesting question: how to choose a probe set for extracting concepts from models? 
The proposed method considers an interesting direction of using generative methods to create images that may not be present in the dataset. 
(S2) The formulation of considering TCAV scores as preference scores is interesting. 
(S3) The authors make an effort to quantitatively assess the concept images generated by their method.
Some concept-based XAI methods, such as [1,2], require the creation of a concept-specific set of images to pass through the network. To eliminate the human-in-the-loop, the authors explore using a combination of reinforcement learning and generative models to create concept-specific image sets. The authors stated goals include:
 (1) producing concept sets that are beyond what human practitioners may be able to discover on their own. 
 (2) producing concepts at a variety of abstraction levels that they are able to control with a parameter. 
 (3) producing concept sets that demonstrate an increases in novelty, abstractness, diversity, generalizability (to non-vision domains) and actionability (utility).

The core of the method is to optimize a LORA-adapted stable diffusion model to produce concept sets that trigger the target model (model to be explained). The RL agent is used to search a set of text prompts for the diffusion model for the text prompts. 

[1] Kim, Been, et al. "Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)." International conference on machine learning. PMLR, 2018.
[2] Schut, Lisa, et al. "Bridging the human-ai knowledge gap: Concept discovery and transfer in alphazero." arXiv preprint arXiv:2310.16410 (2023).
[3] Hu, Edward J., et al. "Lora: Low-rank adaptation of large language models." arXiv preprint arXiv:2106.09685 (2021).
(W1) The authors stated goals for this work are misaligned with the goals of explainable AI in general. The  goal of XAI is to improve a user's understanding of a model as a primary goal. However, the authors focus on "abstractness" and "novelty" as the goals of their method. While the authors propose several experiments to measure abstractness and novelty, they fail to provide any convincing experiments on the utility of their method for explaining network behavior. For example, [1] provides a clear, well-motivated experiment, i.e. users are asked to predict model behavior when given an explanation.

(W2) Additionally, I find the term novelty to be misleading. The term novelty can be interpreted in at least two ways, for example, it could be interpreted as how distinct different discovered concepts are from each other. When measuring novelty in this manner, it seems that the author's method may not exhibit novelty, since in Line 288 they state, "it is possible for different actions to lead to the same explainable state." Instead, the authors measure novelty as the distance between concept images and images the test set. They use this metric to compare their work to prior work, resulting in the fairly trivial result that generated images are farther from the test set than methods that use the test set. 

(W3) The ablation study for choosing TCAV is unnecessary. The authors choose to use either a human, LLM, or TCAV scores from the model to make preference judgements. However, since the goal is to explain the model the other first two options are completely unnecessary. 

(W4) The authors claim to be able to generate concepts sets at different levels of "abstraction". However, whether this "abstraction" is due to SD or due to the target model is unclear. Since, the goal is to understand the target model, once again, I find this experiment to be insufficient to say anything interesting about the target model. 

(W5) Finally, the computational cost of this method is quite high and it is unclear if the results are worth the cost. 

In summary, the authors lack a key experiment measuring the utility of their method. Additionally, I find that some experiments that the authors conduct to have trivial results. 

There are several places with typos. 
L91 mode -> model
L534 the term NUT is introduced without being defined?

[1] Colin, Julien, et al. "What i cannot predict, i do not understand: A human-centered evaluation framework for explainability methods." Advances in neural information processing systems 35 (2022): 2832-2845.
A method to automatically articulate concepts to explain neural networks.
@inproceedings{
anonymous2024explainable,
title={Explainable Concept Generation through Vision-Language Preference Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=9fMNxWDZsP},
note={under review}
}
Concept-based explanations have become a popular choice for explaining deep neural networks post-hoc because, unlike most other explainable AI techniques, they can be used to test high-level visual "concepts" that are not directly related to feature attributes. For instance, the concept of "stripes" is important to classify an image as a zebra. Concept-based explanation methods, however, require practitioners to guess and collect multiple candidate concept image sets, which can often be imprecise and labor-intensive. Addressing this limitation, in this paper, we frame concept image set creation as an image generation problem. However, since naively using a generative model does not result in meaningful concepts, we devise a reinforcement learning-based preference optimization (RLPO) algorithm that fine-tunes the vision-language generative model from approximate textual descriptions of concepts. Through a series of experiments, we demonstrate the capability of our method to articulate complex and abstract concepts which aligns with the test class that are otherwise challenging to craft manually. In addition to showing the efficacy and reliability of our method, we show how our method can be used as a diagnostic tool for analyzing neural networks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Concept based Explainable AI Vision-Language Models Reinforcement Learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d16d178bc1d2966d9526da405f7da5057782bd0b.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Explainable Concept Generation through Vision-Language Preference Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission