Boosting Multiple Views for pretrained-based Continual Learning
continual learning;ViT-pretrained continual learning
Yes
No ethics review needed.
Does some ablation study be conducted on the proposed modules, Prompt selection process and Self-improvement process?
(1)The proposed method BoostCL performs better than existing CL baselines, including Hide-prompt and ranpac.
(2)This work addressed the challenge when applied AdaBoost directly to CL.
(3)A self-improvement process, a simple but effective strategy is designed to help select prompts more accurately when inference.
This work considers the random projection (RP) strategy for pre-trained models in CL. Motivated by the benefits of RP in high-dimensional space, a multi-view strategy for an efficient classifier is further proposed, in which the principle of Adaboost is adapted to 
overcome inherent obstacles and applied for the first time in CL. In addition, a self-improvement process technique, although simple, also shows significant effectiveness in selecting proper taskspecific prompts. The experimental results demonstrate a positive impact of the proposed method in improving model quality while only applying to linear classifiers.
As shown in Table 1, the proposed method takes no advantage in anti-forgetting, FFM metric.
Yes
No ethics review needed.
See weakness
1.	The paper provides a novel theoretical analysis of the benefits of random projection in higher-dimensional spaces for continual learning.
2.	The authors conduct a comprehensive set of experiments to evaluate the performance of their proposed method, BoostCL, across various tasks and datasets.
3.	The paper demonstrates the practicality of the proposed method by applying it to real-world continual learning scenarios.
This paper presents BoostCL, a method for pretrained-based continual learning that leverages multiple views and random projection (RP) to enhance performance. The authors theoretically analyze the benefits of RP in higher-dimensional spaces and demonstrate its effectiveness through experimental results.
1. The paper mentions the motivation behind using random projection, but it lacks a comprehensive discussion on why this approach is superior to other potential methods for feature transformation.
2. While the paper proposes a novel method for continual learning, it would be helpful to see a more in-depth comparison with other recent methods in the literature. Specifically, how does BoostCL differ from and improve upon existing approaches?
3. The technical details of the proposed method are somewhat challenging to follow. The paper could benefit from a more clear and concise explanation of the algorithm and its components. 
4. Discussing the scalability of the proposed method to larger datasets or more complex models would be relevant.
5. The author needs to further analyze the consumption of the proposed method compared to other methods in terms of computational resources and training time.
6. The paper could benefit from a more organized and focused presentation of the material. The introduction could be more clearly tailored to motivate the problem and the proposed solution. Additionally, the paper includes several appendices that contain detailed proofs and additional experimental results. While these appendices provide valuable information, they could be more effectively integrated into the main text to enhance the clarity and readability of the paper.
Yes
No ethics review needed.
1. How to effectively choose the best projection dimension to make appropriate trade-off between performance and computational overhead.
2. How much gpu resource does the proposed method need, will the gpu memory usage increase rapidly by increasing projection dimension.
1. The manuscript provides theoretical analysis of Random Projection, demonstrating how RP improves feature separability in high-dimensional space. 
2. The manuscript is clear and well organized. The formulas are presented in an unambiguous manner.
This manuscript introduces "BoostCL", a novel approach to improve continual learning based on pre-trained models. The manuscript provides theoretical analysis to show that feature separability and model generalization ability benefit from random projection. Then a multi-view random projection is proposed to adapt AdaBoost for CL. A self-improvement process is also proposed for appropriately selecting prompts for each task sample. Empirical validation on multiple CL benchmarks is then conducted.
A substantive assessment of the weaknesses of the paper. Focus on constructive and actionable insights on how the work could improve towards its stated goals. Be specific, avoid generic remarks. For example, if you believe the contribution lacks novelty, provide references and an explanation as evidence; if you believe experiments are insufficient, explain why and exactly what is missing, etc.

1. Although RP and multi-view strategies are theoretically shown to improve feature separability, there is limited discussion on the theoretical upper bound of generalization error.
2. The manuscript does not provide theoretical guidance on choosing the optimal projection dimension in RP, which can greatly impact model performance for different tasks.
3. The proposed self-improvement process for prompt selection could increase the model’s inference complexity, but this potential trade-off is not fully analyzed, especially in scenarios with many tasks.
4. Minor spelling errors, like “sammple” in line 250.
5. More experiment regarding hyper-parameters like number of views, projection dimensions, and threshold settings for the voting mechanism should be conducted.
Yes
No ethics review needed.
Please refer to weaknesses.
1.	This paper provides theoretical insights into Random Projection (RP), reinforcing its foundation in an intuitive manner.

2.	The motivation for the proposed Multi-View Random Projection is clear and well-justified, demonstrating its potential to enhance performance.

3.	The paper is well-organized and easy to understand, addressing most of my queries within the main text or the appendix.
This paper provides some theoretical analysis on Random Projection (RP) which project features into a higher-dimensional space to improve their linear separability. Furthermore, this paper proposes a Multi-View Random Projection scheme to create a robust ensemble classifier motivated by AdaBoost. Then the proposed method is applied to prompt-based CL methods for a better performance. Experimental results demonstrate its efficacy in various continual learning settings.
**Major**

1.	The necessity of the huge views should be clarified. As mentioned in the paper, the subsequent atomic views aim to capture the complement knowledge of previous views. However, the final classifier only leverages a part of these views rather than all of them, which may omit some complement knowledge.

2.	The theoretical results are interesting, but the method is heuristic. For example, the design of the voting strategy and the self-improvement are quite complex. It seems that a lot of tricks are introduced directly to improve performance.

3.	The design of the self-improvement process is not intuitive, which may accumulate errors in multiple-step prediction. Specifically, if an incorrect prompt is chosen in the first step, the model may become biased towards predicting the class associated with that wrong prompt, potentially leading to accumulated errors and hindering improvements in prompt prediction accuracy.

4.	The experiments do not include comparisons with simpler ensemble methods, such as randomly sampling views or using all views for the ensemble. Such comparisons can provide more intuitive insights to demonstrate the effectiveness of the proposed method.

**Minor**

1.	There are too many notations which may limit the fluency and readability. For example, in line 177, there is no description for the notation $\gamma (s,g,\mathbb{R}^d)$ and no explanation for the symbolic $g$. So please simplify the notations as much as possible.
2.	Typo in Corollary 4.4 (line 250), “sammple” should be “sample”.
@inproceedings{
anonymous2024boosting,
title={Boosting Multiple Views for pretrained-based Continual Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=AZR4R3lw7y},
note={under review}
}
Recent research has shown that Random Projection (RP) can effectively improve
the performance of pre-trained models in Continual learning (CL). The authors
hypothesized that using RP to map features onto a higher-dimensional space can
make them more linearly separable. In this work, we theoretically analyze the
role of RP and present its benefits for improving the model’s generalization ability
in each task and facilitating CL overall. Additionally, we take this result to the
next level by proposing a Multi-View Random Projection scheme for a stronger
ensemble classifier. In particular, we train a set of linear experts, among which
diversity is encouraged based on the principle of AdaBoost, which was initially very
challenging to apply to CL. Moreover, we employ a task-based adaptive backbone
with distinct prompts dedicated to each task for better representation learning. To
properly select these task-specific components and mitigate potential feature shifts
caused by misprediction, we introduce a simple yet effective technique called the
self-improvement process. Experimentally, our method consistently outperforms
state-of-the-art baselines across a wide range of datasets.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
continual learning ViT-pretrained continual learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/5c6f6a556c9e6c1c90684426e7a8ede7ccd4fa26.pdf
transfer learning, meta learning, and lifelong learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/cc7cc773d82179bc1be1f0fa1d00961ed51a3c6e.pdf
Boosting Multiple Views for pretrained-based Continual Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission