DebugAgent: Efficient and Interpretable Error Slice Discovery for Comprehensive Model Debugging
slice discovery;interpretable error analysis;multimodal;robustness;model debug
Yes
No ethics review needed.
- Q1 - What is the naive approach to slice enumeration?
- Q2 - How is the tree-structured baseline approach different from DebugAgent's method?
- Q3 - How difficult is comparing tag generation time between DebugAgent and the SoTA? (At least HiBug).
- Q4 - Is it possible to identify slices from other SoTA tag generators and compute their performances with the same models?
- Q5 - Is it possible to provide the shape (number of instances, features, and unique labels) of the private dataset used for pose estimation or an alternative proxy public dataset?
## S1 - Organization
The paper is unambiguous, well-structured, and well-written.

## S2 - Relevance
The problem is interesting and relevant to the community.

## S3 - Approach
The approach is well-articulated, simple to understand, and easy to implement.

## S4 - Coverage
The slice discover problem comprises several challenges in computer vision. The paper effectively presents and addresses them, and the experiments demonstrate the potential of DebugAgent in all of them.
The article presented DebugAgent, an approach for error slice discovery for image classification, pose estimation, and
object detection. The approach consists of three main blocks: attribute and tag generation, error slice discovery, and
model repair. In the experiments, they evaluate the different components of their approach, such as the slice enumeration algorithm, to address the challenges of the combinatorial explosion during the slice exploration and conclude by comparing their model repair against HiBug, another automated approach. The work presents performance improvement upon the SOTA, although the time complexity comparison is done only between baseline versions of the author's method.
## W1 - Baselines (Main issue)
The work lacks experiments with a broader set of SOTA methods; the authors comment on the tags generated by two additional methods, Domino and HiBug, but do not use them for most of the quantitative comparison. The complexity analysis tests the technique against a not-detailed naive approach and a baseline version of one of DebugAgent's algorithms. DebugAgent shows marginal improvement over the HiBug method when testing the performance improvement in model repair, but the reader does not know how costly this method is. The other experiments show that DebugAgent can identify salient slices, but nothing is said about SOTA. The only discussion on the matter is that different methods employ varied workflows, making comparing them difficult, but the authors do not specify the challenges that need to be overcome in order to make the other methods comparable (Please see questions Q3 and Q4).

## W2 - Scalability
Using more instance types, for example, the other 7 classes of the KITTI dataset, in the experiments would better understand the method's generalization capabilities. The authors discuss the scalability regarding tag generation in the appendices, but in the main text, there is only a time analysis of the slice enumeration task. How does the tag generation task scale with the data load? (Please also refer to Q3) 

## W3 - Reproducibility
Including private datasets limits the reproducibility of the research; an indication of a public proxy dataset of a description of the pose estimation dataset's metadata would improve the reasoning of the comparison challenge (please see question Q5). There is also no specification for the baseline setups. Is it possible to have a detailed description of the naive approach and the baseline tree-structured method in the appendices? (Please see Q1 and Q2)

## W4 - Integration
A deeper ablation study would help understand the proposed method and how all algorithms are integrated. The ablation mentioned in Figure 5 can be considered just part of the time analysis of the slice enumeration task.
Yes
NA
No ethics review needed.
1. Table 1 presents the degradation of the model's performance in predicting erroneous segments when using label replacement and instruction-based approaches. Negative values in the table indicate a decrease in prediction accuracy compared to the baseline. Regarding the relationship between the data results and conclusions in Section 5.4, the section shows that the model's average performance on predicted erroneous segments is notably lower than its overall performance. This emphasizes the effectiveness of the DebugAgent approach in predicting additional erroneous segments. However, your concern is valid: while the paper introduces label replacement and instruction-based methods as supplements to the DebugAgent approach, Section 5.4 only validates the effectiveness of these internally proposed methods. The absence of comparative analyses and the negative results presented raise questions about the justification of the proposed methods' effectiveness in the absence of further comparisons.

2、Comparative analysis with existing methods for comprehensive erroneous slice discovery is indeed a crucial aspect in research. It appears that the experiment comparing the proposed methods with existing approaches for erroneous slice discovery is missing in the paper, yet such comparisons play a significant role in assessing the effectiveness and advancements of the proposed techniques.
Originality: The method for generating attributes and labels takes into account environmental factors and image quality metrics, enabling the capture of nuanced details that are highly pertinent to erroneous slice discovery. The slice enumeration approach effectively expedites the enumeration process.
Quality: Good. Further elucidation and interpretation of the experimental results, such as those in Section 5.4, is needed.
Clarity: Fair. However, some procedures still lack clarity, and the comparative experiments are not sufficiently comprehensive.
Significance: The approach has enhanced the consistency and precision in identifying erroneously sliced samples, while significantly bolstering the model's reparative capabilities.
An entirely automated closed-loop debugging framework, DebugAgent, has been introduced for error slicing discovery and model rectification. DebugAgent encompasses attribute and label generation, error slicing discovery, and model rectification. Besides, a structured generation process has been implemented to create comprehensive visual attributes. An efficient slicing enumeration algorithm has been developed leveraging the unique features of data slices to alleviate the issue of combinatorial explosion. This paper employs feature-based label replacement and instruction-based approaches to address errors invisible outside the validation set.
1、The formatting of Figures 1 and 4 lacks aesthetic appeal; for Tables 1 and 2, it is recommended to utilize triple-line tables.
2、In the experimental section, the efficacy of attribute and label generation, as well as slice enumeration, was individually assessed in Part V, without comprehensive comparisons to existing methodologies.
3、In the abstract, it is mentioned that DebugAgent visualizes the attributes of task generation through an interpretable and structured process. While this structured approach is referenced elsewhere in the text, its specific meaning in the context of utilizing GPT for attribute and label generation in Section 3.2 may not be entirely clear.
4、In Section 4.2, the mention of tag substitution and instruction-based method is aimed at predicting erroneous segments beyond the validation set. While Table 1 illustrates model degradation in predicting erroneous slices using these methods, Section 5.4 reveals that the model's average performance on these predicted erroneous slices significantly falls below its overall performance, emphasizing the efficacy of the DebugAgent approach in predicting additional erroneous segments. Your query highlights a potential inconsistency: the absence of testing the effectiveness of the DebugAgent approach in predicting additional erroneous segments, coupled with the fact that tag substitution and instruction-based method are proposed by your team, revealing a logical gap in the narrative.
5、Some data points are referenced in Section 5.3 without accompanying tables for display.
Yes
No ethics review needed.
1. Can the authors provide the prompts used in the method?

2. Were repeated experiments conducted in the experimental section to mitigate the impact of randomness on model repair?

3. Is it possible to verify whether the error slices from different clusters correspond to distinct bugs in the model?
1. The focus of the paper on error slice discovery is important, as understanding these error slices is crucial for comprehending the incorrect behavior of deep learning models. The paper presents an automated method aimed at addressing this issue of error slice discovery and model repair.

2. The proposed method's effectiveness was validated across multiple task scenarios (image classification, pose estimation, and object detection), demonstrating improvements in the quality of attribute and tag generation, as well as model repair effectiveness compared to previous methods.
This paper proposes a method called DebugAgent for the automated discovery of error slices in deep learning models and for model repair. The approach first uses GPT to generate attributes for the dataset and to assign tags to the data. Then, based on the generated attribute-tag pairs, it employs tree-structured enumeration for clustering and identifying error slices. Finally, the method utilizes these error slices to repair the model. To validate the effectiveness of DebugAgent, the authors compared it with previous methods in terms of the generated attributes and tags, clustering efficiency, and model repair effectiveness.
1. DebugAgent employs GPT in several parts of the algorithm, including attribute and tag generation, as well as instruction-based slice generation. However, the specific prompts used in these processes are not detailed in the paper. It would be beneficial to provide the prompts used in these steps to enhance the understanding of the method.

2. In the model repair experiment section, the improvements of DebugAgent over baseline methods are not significant. Given the randomness inherent in the training process of deep learning models, it would be beneficial to clarify whether these results are based on repeated experiments. If not, it is necessary to conduct repeated experiments to mitigate the effects of randomness.

3. DebugAgent identifies different error slices through clustering. It would be beneficial to verify whether the error slices from different clusters correspond to distinct model bugs. For instance, one could repair the model using the error slices from a specific cluster and then evaluate the model's performance on that cluster compared to its performance on other clusters to assess any differences.
DebugAgent automatically identify error slices in deep learning models through visual attribute generation and slice enumeration, showing improvements in attribute quality, enumeration speed, slice coherence, model repair
@inproceedings{
anonymous2024debugagent,
title={DebugAgent: Efficient and Interpretable Error Slice Discovery for Comprehensive Model Debugging},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=l30moNjSY9},
note={under review}
}
Despite the significant success of deep learning models in computer vision, they often exhibit systematic failures on specific data subsets, known as error slices. Identifying and mitigating these error slices is crucial to enhancing model robustness and reliability in real-world scenarios. In this paper, we introduce DebugAgent, an automated framework for error slice discovery and model repair. DebugAgent first generates task-specific visual attributes to highlight instances prone to errors through an interpretable and structured process. It then employs an efficient slice enumeration algorithm to systematically identify error slices, overcoming the combinatorial challenges that arise during slice exploration. Additionally, DebugAgent extends its capabilities by predicting error slices beyond the validation set, addressing a key limitation of prior approaches. Extensive experiments across multiple domains — including image classification, pose estimation, and object detection — show that DebugAgent not only improves the coherence and precision of identified error slices but also significantly enhances the model repair capabilities.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
slice discovery interpretable error analysis multimodal robustness model debug
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/1d72688b831426f60d4ff52985e0f49d5a9fc43b.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/923906b6570a410419aae6b6589c2ea520ef4f0d.zip
DebugAgent: Efficient and Interpretable Error Slice Discovery for Comprehensive Model Debugging
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission