The Case for Gradual Structured Pruning in Image-based Deep Reinforcement Learning
gradual structured pruning;deep reinforcement learning;Procgen benchmark
Yes
No ethics review needed.
See weaknesses
1. The overall writing is clear.
2. Efficiency in the domain of RL is interesting.
The paper introduces a group-structured pruning framework for image-based deep reinforcement learning (DRL), specifically designed to maintain performance while reducing computational costs. The authors first demonstrate that the benefits of unstructured pruning diminish when making architectural improvements to neural networks. They then propose a structured pruning approach that removes entire network structures (channels/neurons) while accounting for inter-layer dependencies. The method is extensively evaluated on the Procgen Benchmark using PPO and DQN agents, showing comparable performance to unstructured pruning while achieving significant reductions in inference time.
1. The motivation is not clear. Although efficiency in the domain of RL is interesting, the author should include the difference between "pruning in image classification" and "pruning in reinforcement learning." This motivation should serve the algorithm.

2. There are some other references [a] to do gradual pruning. What is the difference? Why not cite this?

3. Some arguments are wrong. For example, in line 15, the authors claim, "unstructured pruning merely zeroes out individual weights, the resulting networks usually retain high computational demands despite sparsity". This is not correct. For example, Thinet [b] can achieve realistic acceleration. There are so many other filter pruning methods to achieve practical acceleration.

4. The performance is not good. In table 1, if the up arrow means "higher is better", the proposed method is not as good as others.

[a] H. Wang, C. Qin, Y. Zhang, and Y. Fu, “Neural pruning via growing regularization,” in Proc. Int. Conf. Learn. Represent., 2022
[b] https://github.com/Roll920/ThiNet
Yes
No ethics review needed.
1. Is their some comparsion between impala-CNN and impoola-CNN when both of them are processed by using gradual structured pruning? The authors may use the proposed pruning mehtod to both impala-CNN and impoola-CNN under different configurations to show the improvements introduced by impoola-CNN. 

2. Is their any experimental results to compare the performance of using different scoring functions?
1. The gradual structured pruning method is well-defined, and the experimental results show that is has good computation efficiency
This paper proposes a gradual structured pruning method for deep reinforcement learning. The authors firstly analyzes the drawbacks of the unstructured pruning, then introduces a improved impoola-CNN model to replace the previous impala-CNN model, and a gradual structured pruning for image-based deep reinforcement learning tasks. The experimental results show that the proposed method has compariable perfermances comparing with unstrutured pruning mehtod, and it has good robustness when intruducing nosies. Moreover, the proposed pruning method results in less latency times comparing with original dense model and unstructured pruned model.
1. The experimental results did not include the comparsion of impala-CNN and impoola-CNN, thus the advantages of the new network cannot be proved. The authors may consider to provide a direct comparison of performance between Impala-CNN and Impoola-CNN under the same pruning conditions across multiple environments, as well as the un-pruned editions. This would give clearer evidence for the claimed advantages of the new architecture.

2. The advantages of the scoring function are not proved. The reason that use ||w||_1 should be carefully analyzed. For instance, the authors may consider to compare L1-norm, L2-norm or other regularization methods to show the advantages of the proposed scoring function.
Yes
No ethics review needed.
N/A
- Exploring scaling laws for deep RL is an important challenge.
- New CNN architecture and structured pruning algorithm are proposed.
- Outperform both unstructured and naive structured pruning baselines.
Scaling neural networks has generally been ineffective for deep reinforcement learning, as larger networks do not necessarily improve performance. Recent work by Obando-Ceron et al. (2024) suggests that pruning a large network could overcome this limitation. By initially increasing the network size and applying pruning during training, they achieved performance gains beyond those possible with a standard dense network. However, their unstructured sparsity approach often fails to generalize well outside the original setting. This paper introduces an improved CNN architecture, called Impoola, along with a structured pruning algorithm that outperforms unstructured pruning in both generalization and computational efficiency.
**Clarification of objectives and contribution**

The primary goal of this paper—whether to build upon Obando-Ceron et al.'s (2024) approach for finding scalable architectures or to develop a structured pruning method specifically for deep RL—is somewhat unclear. If the objective is scaling, adding experiments that compare the proposed method with the original dense network and unstructured pruning across different scenarios would strengthen its claims. On the other hand, if structured pruning is the focus, revisiting the introduction and motivation to reflect this intent could help clarify the paper’s contributions in this area.

---
**Need for broader benchmark comparisons**

If this paper aims to address scaling issues in deep RL, it would benefit from broader comparisons with relevant methods, such as those presented in [1] and [2]. Including at least one of these as a baseline would provide a more comprehensive context for the paper’s approach. If these methods are excluded, providing a brief rationale would clarify the baseline selection choices for readers.

---
**Potential for alternative focus on structured pruning**

An alternative approach for the paper could be to frame it as a structured pruning study specifically for deep RL, rather than as an extension of scaling work. The proposed Impoola architecture appears specialized, so comparing it with existing structured pruning methods for deep RL, such as [3], would reinforce its relevance in this area. Additionally, if the technique is intended to be broadly applicable to CNNs, testing it on traditional CNN tasks (e.g., video classification) would better demonstrate its versatility. Including comparisons with CNN-focused pruning techniques, such as [4], would further highlight the method’s adaptability across different applications.



[1] Obando-Ceron et al. Mixtures of Experts Unlock Parameter Scaling for Deep RL. ICML 2024.\
[2] Sokar et al. Don't Flatten, Tokenize! Unlocking the Key to SoftMoE's Efficacy in Deep RL. arXiv 2024.\
[3] Su et al. Compressing Deep Reinforcement Learning Networks with a Dynamic Structured Pruning Method for Autonomous Driving. arXiv 2024.\
[4] He & Xiao. Structured Pruning for Deep Convolutional Neural Networks: A Survey. TPAMI 2023.
We demonstrate that group-structured pruning is the preferred neural network pruning method for image-based deep reinforcement learning compared to other pruning techniques.
@inproceedings{
anonymous2024the,
title={The Case for Gradual Structured Pruning in Image-based Deep Reinforcement Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=LTdtyzPQoZ},
note={under review}
}
Gradual unstructured pruning of large neural networks during training has been shown to enhance performance in image-based deep reinforcement learning. However, we demonstrate that these performance improvements often diminish when making slight enhancements to the neural network architecture. Additionally, as unstructured pruning merely zeroes out individual weights, the resulting networks usually retain high computational demands despite sparsity. In contrast, structured pruning removes entire structures, such as channels or neurons, directly reducing the computational cost. Given these findings, we propose a novel gradual structured pruning framework for image-based deep reinforcement learning that accounts for inter-layer dependencies to preserve the network's functional integrity. We evaluate the impact of our group-structured pruning method on generalization performance through extensive experiments on the Procgen Benchmark using PPO and DQN. Our results show that group-structured pruning maintains performance levels comparable to unstructured pruning while reducing inference time, positioning it as the preferred approach for real-world applications.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
gradual structured pruning deep reinforcement learning Procgen benchmark
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c99cb940bc73e2566e77e4dfdc0325625f53461f.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
The Case for Gradual Structured Pruning in Image-based Deep Reinforcement Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission