Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures
RWKV;Visual Perception;Linear Attention
Yes
No ethics review needed.
see weaknesses
1.The paper is well-written and easy to follow;

2.Extensive empirical evidence supports the effectiveness of the model, indicating its practical application potential;

3.The adaptation of RWKV for visual tasks goes beyond mere transfer, incorporating modifications that enhance its suitability for image processing.
The paper introduces a new network architecture, VISION-RWKV, a vision-adapted version of the RWKV network from the NLP community, which employs an RNN-based linear attention mechanism. The authors have made necessary adaptations to the RWKV to better suit visual tasks, recognizing the differences between vision and language processing tasks.
Major：Although the paper discusses RWKV and mamba as prominent RNN-based linear attention models transitioning from NLP to computer vision, it lacks a direct comparison with the mamba (vision) model. Such a comparison would be valuable for assessing the respective strengths and weaknesses of each model in the field of computer vision.

Minor：From an innovation perspective, the approach of adapting other domains' mature architectures to computer vision, similar to what was done post-transformer, appears somewhat incremental.
Yes
No ethics review needed.
Clarifying the practical benefit from their proposed approach and why not simply use vanilla ViTs considering the current gains are not that considerable when looking at Tables 2 and 4. Hence, the reason I am leaning towards a marginal reject but since the method seems to provide interesting direction and their results for detection seems sufficiently good I am not going lower.
- Comprehensive results across three tasks.
- Interesting direction to replace ViTs with other efficient techniques that provide on-par or better performance.
Authors propose an approach that adapts NLP models for RWKV to vision tasks, termed as V-RWKV. Their proposed method could be looked upon as a cost effective solution than ViTs. They propose Quad-directional Shift and change the causal attention to bidirectional one towards learning locl concepts that are more relevant in vision rather than simple text. They evaluate their approach on the image classification, object detection and semantic segmentation tasks.
- The gains in efficiency seem to be relatively minor, i.e., it is not an order of magnitude which still brings the question whether it is worth exploring these type of models to replace ViTs to begin with or not.

 For example, in Table 2 ViT-L vs. VRWKV-L 191.1 vs 189.5G FLOPS and parameters at 309.5M vs. 334.9M respectively. I think the reduction in FLOPS when scaling to Large variant seems to be around 3G. Similarly in Table 4, FLOPS 446.8 vs. 421.9G with the expense of an increase in the number of parameters. I am not expert in such type of methods focused on improving efficiency but I do not see the results are impressive enough to show the benefit from the V-RWKV design, especially that it is increasing the parameters.
Yes
No ethics review needed.
- Can you please clarify what you meant in line 527-528 by task fine-tuning using Q-shift manner?
- Novel contribution for quad-directional token shifting called Q-shift. This essentially increases the models range of semantic understanding
- Authors expanded causal RWKV to a bidirectional global RWKV. They modified the exponent in the RWKV attention that leads to transforming the absolution positional bias to relative bias.
- Well written paper with through experimentation including MAE pretraining
The authors describe a Vision-RWKV architecture that employs novel techniques like bi-directional RWKV, quad token shifting method Q-shift etc. These techniques helps Vision-RWKV  architecture surpass window-based ViTs and comparable to global attention ViTs along with lower FLOPs, lower GPU memory cost and faster processing as shown in Figure 1. Although the MAE finetuning is not as straightforward as typical finetuning on downstream task, overall this paper has a good contribution for the vision research community.
- The authors implemented a bidirectional shift operation that removed the vertical shift in Q-shift, thereby enabling for MAE pretraining. IMO this is a source of complexity. As a result, MAE finetuning needs to be done in Q-shift manner
@inproceedings{
anonymous2024visionrwkv,
title={Vision-{RWKV}: Efficient and Scalable Visual Perception with {RWKV}-Like Architectures},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=nGiGXLnKhl},
note={under review}
}
Transformers have revolutionized computer vision and natural language processing, but their high computational complexity limits their application in high-resolution image processing and long-context analysis. This paper introduces Vision-RWKV (VRWKV), a model that builds upon the RWKV architecture from the NLP field with key modifications tailored specifically for vision tasks. Similar to the Vision Transformer (ViT), our model demonstrates robust global processing capabilities, efficiently handles sparse inputs like masked images, and can scale up to accommodate both large-scale parameters and extensive datasets. Its distinctive advantage is its reduced spatial aggregation complexity, enabling seamless processing of high-resolution images without the need for window operations. Our evaluations demonstrate that VRWKV surpasses ViT's performance in image classification and has significantly faster speeds and lower memory usage processing high-resolution inputs. In dense prediction tasks, it outperforms window-based models, maintaining comparable speeds. These results highlight VRWKV's potential as a more efficient alternative for visual perception tasks. Code and models shall be available.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
RWKV Visual Perception Linear Attention
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/16014702b2d3d00bb32bb72aedbd378e9e55dc53.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/dd99aac6f499afa1fb546b8fb687363ec11850e2.zip
Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission