RMP-SAM: Towards Real-Time Multi-Purpose Segment Anything
segment anything; real-time segmentation; multi-purpose model;
Yes
No ethics review needed.
1. Do you have the results for TopFormer in Table 3? Additionally, please bold the results in all comparison tables for clarity.
1. The model achieves multi-purpose segmentation through an efficient structure and unified training approach.
2. The paper is well-written and easy to follow.
3. The experiments on panoptic segmentation, interactive segmentation, and video segmentation are solid, comprehensive, and persuasive, effectively demonstrating the model's contribution.
The paper presents a real-time, versatile segmentation model capable of interactive segmentation, panoptic segmentation, and video instance segmentation. 
While retaining the SAM encoder-decoder structure, the model incorporates an efficient encoder and adapter to enhance performance.
In the decoder, RAP-SAM introduces a three-stage pipeline that leverages novel pooling-based dynamic convolutions to refine mask tokens. Following the decoder, two additional prompt adapters are implemented to improve interaction between visual prompts and segmentation tokens.
RAP-SAM demonstrates efficiency and generalizability across various segmentation benchmarks.
1. The paper lacks a detailed comparison with other SAM-like methods. A single COCO instance segmentation comparison in Table 4 is insufficient to substantiate claims of superiority over SAM. The results presented in Table 4 are not particularly outstanding. Additional experiments, such as on the SegAny task, with detailed metrics (AP for small, medium, large objects) on COCO instance segmentation, and evaluations with different object detectors, would strengthen the case.

2. Efficiency benchmarks are insufficiently detailed. For a model promoting efficiency, there should be a more comprehensive evaluation across different GPU platforms, such as the 3090 and V100, testing throughput and latency. Additionally, plotting latency versus performance compared to other SAM-like methods would provide a clearer visualization of the model's efficiency.
Yes
No ethics review needed.
See weakness.
1.Demonstrates impressive performance and inference speed.

2.Filling the gap in real-time multi-purpose segmentation.

3.The whole method is very simple and easy to understand.

4.Code is provided for easy reproduction by the reader.
This work addresses the need for real-time multi-purpose segmentation by introducing a novel setting that encompasses interactive, panoptic, and video instance segmentation, striving for a single end-to-end model capable of handling all tasks in real-time. The proposed Real-Time Multi-Purpose SAM (RMP-SAM) utilizes an efficient encoder and a decoupled adapter for prompt-driven decoding, along with innovative training strategies and adapter designs, demonstrating effectiveness and strong generalization across benchmarks and specific semantic tasks while achieving an optimal balance between accuracy and speed.
1.Based on existing technology development, the entire pipeline is not novel.

2.Differences with SAMv2 should be further clarified, especially in terms of claimed semantic labels?
Yes
No ethics review needed.
Please see above.
- RMP-SAM unifies interactive segmentation, panoptic segmentation, and video instance segmentation within a single model.
- RMP-SAM offers a good trade-off between speed and accuracy.
- Extensive experiments demonstrate the model's effectiveness.
This paper presents a real-time multi-purpose segmentation model called RMP-SAM. RMP-SAM handles various tasks such as interactive segmentation, panoptic segmentation, and video instance segmentation using a single model. To balance the accuracy and speed, RMP-SAM utilizes a lightweight encoder and a dynamic convolution-based decoder. RMP-SAM achieves fast inference while maintaining satisfactory performance.
- The authors do not provide detailed information for joint training. Joint training for multiple tasks can be complex. How do the authors train RMP-SAM for some potential problems, such as avoiding the model being dominated by a single task and performance degradation by conflicts between different tasks?

- This paper ignores some related methods, making it difficult to assess the model's performance relative to existing SOTA approaches. For example, some universal methods[1,2,3] obtain better results than RMP-SAM using ResNet50. The authors should make a comprehensive comparison with other methods. 

[1] Tube-Link: A flexible cross tube framework for universal video segmentation. CVPR 2023.

[2] Dvis: Decoupled video instance segmentation framework. CVPR 2023.

[3] Univs: Unified and universal video segmentation with prompts as queries. CVPR 2024.
Yes
No ethics review needed.
- What does the dot size in Fig. 1b indicate?
- The abstract says "generalization ability of these models across diverse scenarios", a learnable classifier with CLIP text embeddings is also used and “segment anything” is in the title. Is there a connection to open-vocabulary?
- Large models can perform many tasks, but are not real-time capable because of the large encoders, while real-time models are often specialized in only one task. The method presented here aims to combine the two things, i.e., "the first real-time multi-purpose segmentation model".
- Precise implementation details are given and the comparisons with the other methods appear to be fair.
- The method achieves good results in the trade-off between performance and speed across the various tasks and datasets.
- The ablation studies are useful and show interesting insights.
The authors explore a novel real-time segmentation setting called real-time multi-purpose segmentation. It contains three fundamental sub-tasks: interactive segmentation, panoptic segmentation, and video instance segmentation. In contrast to previous methods that use a separate design for each task, the authors use only a single end-to-end model to handle all these tasks in real time. To fulfill the real-time requirements and balance multitask learning, a new dynamic convolution-based method, Real-Time Multi-Purpose SAM (RMP-SAM), is introduced. They benchmark several strong baselines by extending existing work to support multi-purpose segmentation.
- Many architectural elements were adopted from other works, it is not clear to me if there are already similar architectures as proposed here, or where exactly is the innovation (except the jointly training).
- In the related work section, many works are cited and also compared at the task level, but I also miss a comparison at the architectural level.
- The tables, especially table 3, are difficult to read because nothing is in bold print and you have to search for the trade-off here. A plot like Fig. 1b would be more useful.
- The references to the appendix could be a little more precise and there is no reference to Table 2.
@inproceedings{
anonymous2024rmpsam,
title={{RMP}-{SAM}: Towards Real-Time Multi-Purpose Segment Anything},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=1pXzC30ry5},
note={under review}
}
Recent segmentation methods, which adopt large-scale data training and transformer architecture, aim to create one foundation model that can perform multiple tasks.
    However, most of these methods rely on heavy encoder and decoder frameworks, hindering their performance in real-time scenarios.
    To explore real-time segmentation, recent advancements primarily focus on semantic segmentation within specific environments, such as autonomous driving. However, they often overlook the generalization ability of these models across diverse scenarios.
    Therefore, to fill this gap, this work explores a novel real-time segmentation setting called real-time multi-purpose segmentation.
    It contains three fundamental sub-tasks: interactive segmentation, panoptic segmentation, and video instance segmentation. 
    Unlike previous methods, which use a specific design for each task, we aim to use only a single end-to-end model to accomplish all these tasks in real-time.
    To meet real-time requirements and balance multi-task learning, we present a novel dynamic convolution-based method, Real-Time Multi-Purpose SAM (RMP-SAM). 
    It contains an efficient encoder and an efficient decoupled adapter to perform prompt-driven decoding. 
    Moreover, we further explore different training strategies and one new adapter design to boost co-training performance further. 
    We benchmark several strong baselines by extending existing works to support our multi-purpose segmentation.
    Extensive experiments demonstrate that RMP-SAM is effective and generalizes well on proposed benchmarks and other specific semantic tasks. 
    Our implementation of RMP-SAM achieves the optimal balance between accuracy and speed for these tasks.
    Code and model will be available.
    %
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
segment anything; real-time segmentation; multi-purpose model;
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/30979426da6d204ba216f640a2788532112ac7fc.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
RMP-SAM: Towards Real-Time Multi-Purpose Segment Anything
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission