Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors
weakly supervised affordance grounding;foundation model;pseudo label
Yes
No ethics review needed.
See weaknesses.
- The problem is important and well-motivated, as affordance grounding is crucial for robotic manipulation and human-object interaction understanding
- The proposed pseudo-labeling approach effectively leverages existing foundation models (VLpart, SAM) to provide supervision, addressing limitations of previous CAM-based methods
- The label refinement process using exocentric images is novel and well-designed, providing a clever way to improve initial pseudo labels
- The reasoning module helps generalize to unseen objects, which is crucial for practical applications
- The writing is clear and the method is well-explained with appropriate visualizations
This paper tackles weakly supervised affordance grounding (WSAG) by leveraging foundation models to generate pseudo labels, departing from previous CAM-based approaches. The authors propose a three-stage pipeline: (1) using VLpart and SAM to generate initial pseudo labels by mapping affordance-object pairs to part names, (2) refining these labels using human-object interaction cues from exocentric images, and (3) training an affordance grounding model with the refined pseudo labels. The method also includes cross-view feature alignment and a reasoning module to handle unseen objects. The approach shows significant improvements over existing WSAG methods
The choice of CLIP as the vision encoder could be better justified given previous work suggesting limitations (vs DINO, OWLViT, SAM). For example, the paper will be stronger with an ablation study of different visual encoders.
Yes
No ethics review needed.
1. Could you provide more details about failure cases and limitations of the proposed approach?
2. How sensitive is the method to the results of VFM? How well can the refine state correct possible errors by VLpart and SAM?
3.  How does the computational cost (training & inference) compare to existing CAM-based methods?
- Clear writing and organization.
- Well-motivated technical approach with clear problem formulation.
- This paper propose a novel approach that uses visual foundation models and part-level semantic priors for WSAG, unleashing the power of these models for affordance learning.
- Using human occlusion cues for label refinement, which is an innovative insight.
- Comprehensive experimental validation and thoughtful analysis of limitations in existing methods.
This paper addresses the task of weakly supervised affordance grounding (WSAG), where the goal is to identify affordance regions on objects using only image-level labels and human-object interaction images. 
The key contributions include:
- A novel pseudo-supervised training framework and pipeline that leverages visual foundation models to generate affordance heatmaps, mapping affordance classes to object parts.
- Three key enhancements to improve performance:
    - Label refinement using interaction cues
    - Fine-grained object feature alignment with exocentric images
    - Reasoning module for better generalization
- Extensive experiments demonstrating significant performance improvements over existing methods
- Could benefit from more analysis of failure cases.
- The label refinement stage using human occlusion cues may be problematic when interactions are ambiguous or when multiple affordances exist.
- The mapping from affordance to part names is ad-hoc and manually crafted, which limits the scalability to new affordance types and more complex objects.
Yes
No ethics review needed.
1. Aligning the features of an object from different views is a commonly used strategy for feature learning. How is this strategy related to pseudo label generation and refinement.
2. Some designs need more detailed ablation studies. E.g., how does the proposed fine-grained feature alignment process with SAM perform when compared with the previous work aligning the features directly. Is there any significant performance difference?
1. Training affordance grounding models with object labels is an interesting question.
2. Using off-the-shelf foundation models to generate affordance label is an interesting idea.
3. Experiments show promising results.
This paper propose a weakly supervised affordance grounding framework. It uses off-the-shelf foundation models to generate pseudo labels of object parts. To further improve the performance, a label refining strategy, a fine-grained feature alignment process, and a lightweight reasoning module are introduced. Experiments show promising results.
1. As shown in the ablation study table 2, the improvements of using all these three modules look marginal over using one module. It seems that the effectiveness of the three components are not significant.
2. In section 3.4, the authors propose to align the features of exo- and egocentric images after SAM segmentation while the existing methods directly align the features of the two images. However, there is no solid experiments to show the effectiveness of this design.
3. The framework refines the affordance labels with the need of the corresponding exocentric image which may not be available sometimes.
We propose a pseudo-label based method for weakly supervised affordance grounding, utilizing the semantic priors of vision foundation models.
@inproceedings{
anonymous2024weaklysupervised,
title={Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=0823rvTIhs},
note={under review}
}
In this work, we focus on the task of weakly supervised affordance grounding, where a model is trained to identify affordance regions on objects using human-object interaction images and egocentric object images without dense labels. 
Previous works are mostly built upon class activation maps, which are effective for semantic segmentation but may not be suitable for locating actions and functions. Leveraging recent advanced foundation models, we develop a supervised training pipeline based on pseudo labels. The pseudo labels are generated from an off-the-shelf part segmentation model, guided by a mapping from affordance to part names.
Furthermore, we introduce three key enhancements to the baseline model: a label refining stage, a fine-grained feature alignment process, and a lightweight reasoning module. These techniques harness the semantic knowledge of static objects embedded in off-the-shelf foundation models to improve affordance learning, effectively bridging the gap between objects and actions.
Extensive experiments demonstrate that the performance of the proposed model has achieved a breakthrough improvement over existing methods.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
weakly supervised affordance grounding foundation model pseudo label
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/8e86da678f19c14a55c806dd8360c86e09890140.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/6d4a8cc6f8e2fdea56ed70c51ffccd79c27af4ac.zip
Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission