ICL-TSVD: Bridging Theory and Practice in Continual Learning with Pre-trained Models
Continual Learning; Pretrained Models; Overparameterization; Generalization; Random Feature Models
Dear Reviewer EXjv,

Thank you for your comments. We are happy to read the strength that **The authors provided comprehensive theoretical and empirical results.**

**The question is whether experience-replay-based methods are compared. The answer is yes (this is implicit in the paper).**

**Details**. We compared our approach with the joint training method (that replays all training data, rather than just a small fraction). In some sense,  joint training serves as a performance upper bound for experience-replay-based methods.

Specifically, LC ($X_{1:T}$) refers to joint training of a linear classifier using vanilla ViT features $X_{1:T}$, while LC ($H_{1:T}$) is the same approach, but using the random ReLU features $H_{1:T}$. In Table 1, we show our method outperforms both approaches, without storing any of the samples or features $X_{1:T}$.

There could be other variants of joint training. For example, one could consider joint training of a two-layer ReLU network using ViT features $X_{1:T}$. We didn't show its performance in the paper, but we confirm that it is comparable to LC ($X_{1:T}$) and LC ($H_{1:T}$). We'd be happy to add such a result in the revision.

**Please**: Feel free to raise any further concerns, as we will be available to respond in the next few days. For example, if you are interested in seeing a specific kind of experience replay methods, let us know and we will compare them.

**Please**: Dear reviewer, as you've seen, we received divergent reviews. We would be extremely grateful if you could look further at the paper and update your thoughts.


Thank you very much,

Authors
Reply to Reviewer EXjv
Yes
No ethics review needed.
Did the authors compare with experience replay method?
The authors provided comprehensive theoretical and empirical results.
This paper proposes a method for continual learning. In this method, the singular values are continuously truncated. The authors claim that their method has theoretical guarantees as well as good practical performance.
A simple baseline in continual learning is experience replay. But I did not see the authors providing the results for experience-replay-based method.
Yes
No ethics review needed.
In the ICL framework of Peng et al. (2023), they are maintaining a minimum norm solution based on proof of Proposition 5 in that work where they are using the pseudoinverse. So I think the statements on top of page 3 might not be quite accurate. 


In lines 202-208 the cause of instability of RanPAC is attributed to the double descent phenomenon. I think it would be helpful to explain this more to the reader.


New notation that is used in lines 242-243 has not been introduced yet.

To judge the significance of the experimental results, it would be helpful if the reported numbers included also a measure of uncertainty.
The problem and setup is well motivated. The results are natural and the writing is clear. The experimental results are extensive and seem significant.
This paper proposes a new method to address instability issues of RanPAC. Part of the contribution is a diagnosis of the cause of the instability which is attributed to the condition number of the features exploding as the number of tasks increases. The proposed method truncates the smaller singular values of the features in an online or continual way, so that it doesn't have to store all of the previous tasks samples. Effectiveness of the new method is validated empirically.
It seems to me that the main contribution of this work is an algorithm for continual principal component regression, with an application to training with random features. Specially, the properties of the features that arise from pretraining, random projections and nonlinearity don't seem to matter much to the derivations theoretically. On the other hand, if I view the work mainly as an algorithm for continual principal component regression, then the guarantees given in theorems 1 and 2 are a black box, since the quantities $\gamma_t$ and $a_t$ which have an unpacked recursive form. It is not clear to me how they behave in general.
Yes
No ethics concerns.
No ethics review needed.
1. In Table 2, why the final accuracy of RanPAC on StanfordCars dataset is only 1.19%? Could you provide an explanation for it?

2. Figure 5 does not plot the training time for RanPAC when embedding dimension $E = 50\mathrm{K}, 100\mathrm{K}$, is this because the training process is too slow?

3. How about the running time of ICL-TSVD compared to other CL baselines, especially those that do not utilize SVD in their implementation?

4. For original definition of $\overline{\bf{W}}_t$ in Eq. (2) and practical implementation of $\widetilde{\bf{W}}_t$ in Eq. (4), do they close to each other? In other words, is there any theoretically guarantee showing that $\\|\overline{\bf{W}}_t - \widetilde{\bf{W}}_t\\|_F$ is small in general?

**Minor issues and typos**

- In Eq. (5) of Section 5, the denominator of $\gamma_t$ should be $\max\_{1\leq i\leq t-1}\mu\_{k_{i+1}}(\widetilde{\bf{B}}_i\widetilde{\bf{B}}_i^{\top})$ instead of $\max\_{1\leq i\leq t-1}\mu\_{k\_{i}+1}(\widetilde{\bf{B}}_i\widetilde{\bf{B}}_i^{\top})$.
1. The paper clearly demonstrates the contribution of the proposed approach and is easy to follow.

2. Extensive experiments across multiple datasets show that ICL-TSVD remains stable regarding hyperparameter selection (including regularization parameter $\lambda$ and truncation percentage $\zeta$) and consistently outperforms previous CL baselines, particularly RanPAC.

3. It provides theoretical guarantees for the proposed method ICL-TSVD by revealing a recurrence relation throughout the CL process, showing that it has small estimation and generalization errors when a suitable fraction of SVD factors are truncated.
This paper introduces a new algorithm, ICL-TSVD, designed to bridge the gap between theory and practice in continual learning (CL) with pre-trained models. It integrates an empirically strong approach RanPAC within the Ideal Continual Learner (ICL) framework. Specifically, ICL-TSVD lifts pre-trained features into a higher dimensional space, then addresses the ill-conditioning of these lifted features through continually SVD truncation, and solves the truncated form of an over-parameterized minimum-norm least-squares problem. Theoretically, it is shown that the method achieves small training and generalization error under certain conditions. Empirically, ICL-TSVD demonstrates stability across various hyperparameter settings and outperforms other CL baselines across multiple datasets.
1. The continual Implementation of $\overline{\bf{W}}_t$ involves the computation of $\widetilde{\bf{B}}_t$, which has the form of $\widetilde{\bf{B}}_t = [\widetilde{\bf{U}}\_{1:t-1}\widetilde{\bf{\Sigma}}\_{1:t-1}, \widetilde{\bf{H}}_t]$ when $t\geq 2$ according to Eq. (3). Appendix C.1 briefly mentions that it is empirically found continually updating all SVD factors $\widetilde{\bf{U}}\_{1:t}, \widetilde{\bf{\Sigma}}\_{1:t}$ and $\widetilde{\bf{V}}\_{1:t}$ lead to large test errors. However, it remains unclear why authors use $\widetilde{\bf{B}}_t = [\widetilde{\bf{U}}\_{1:t-1}\widetilde{\bf{\Sigma}}\_{1:t-1}, \widetilde{\bf{H}}_t]$ instead of $\widetilde{\bf{B}}_t = [\widetilde{\bf{U}}\_{1:t-1}\widetilde{\bf{\Sigma}}\_{1:t-1}\widetilde{\bf{V}}\_{1:t-1}, \widetilde{\bf{H}}_t]$ from a theoretical perspective. What are the difference and connection between these two implementations? Why just maintaining $\widetilde{\bf{U}}\_{1:t-1}$ and $\widetilde{\bf{\Sigma}}\_{1:t-1}$ can serve as a substitution for all SVD factors, what is the motivation for that?

2. In Section 6.2,  ICL-TSVD chooses embedding dimension $E = 10^5$ while RanPAC uses the default choice $E=10^4$. It is not so convincing that this is a fair comparison although the authors claim that ICL-TSVD’s implementation is more scalable and more efficient than RanPAC’s. Figure 8 and Table 13 in Appendix K indeed show that ICL-TSVD does not outperform RanPAC on about half of the datasets.

3. 
  - For Theorem 1, take Figures 3 and 4 for example, the truncation percentage is set to 25%. Then based on eigenvalues vs. eigenvalue index shown in Figure 1, $a_t$ should be at least in the order of $10^5 \sim 10^6$, and $\gamma_t$ is around or slightly larger than 1. In this case, the bound for the estimation error can be very large even if the unknown ground-truth $\\|\mathbf{W}_t^*\\|_F^2$ and noise $\\|\mathcal{E}\_{1:t}\\|^2$ are reasonably small, which means that the theoretical guarantee does not match with empirical results. And similar issue exists in Theorem 2 as well. 
  - Furthermore, without independent i.i.d. Gaussian vectors assumption on $\bf{h}$ and the columns of $\bf{H}\_{1:t}$ (which is unrealistic for practice), we may not guarantee that $\\|\Lambda - \frac{1}{M_t}\bf{H}\_{1:t}\bf{H}\_{1:t}^{\top}\\|$ converges to 0 as $M_t\rightarrow\infty$. 

4. Figure 5 shows the comparison of training times for varying embedding dimensions $E$ between ICL-TSVD and RanPAC. How about comparison with other CL baselines in terms of training time? Typically, the computational cost of SVD-type algorithms is large, is ICL-TSVD still faster than other CL methods under the current experimental settings?
Yes
No ethics review needed.
Other than what I wrote above,

1. I could not immediately understand why both RanPAC and the proposed method outperform the $LC(H_{1:T})$ benchmark (trained with the original cross entropy loss like explained in Appendix J). Even if the data matrices are ill-conditioned, early stopping (e.g., when performing a fixed number of steps) should create a remedying effect and help generalization. Is it possible that a better hyperparameter tuning is needed here? Did the authors check the training *and* test curves throughout the training of this competitor?

1. In Figure 4a, what is going on with the scaling of the x-axis? What was the maximal truncation percentage that you examined? I would expect seeing a sharp decrease at the for high values, but it is not apparent. Moreover, why can we truncate so much without losing performance even though the eigenvalues do not decrease so quickly (as seen from Figure 1a)?
1. The proposed method seems to empirically improve upon the existing RanPAC method (McDonnell et al., NeurIPS 2023).
1. The proposed method is easy to implement.
The paper tackles the problem of class-incremental learning (CIL).
To this end, the paper improves RanPAC (McDonnell et al., 2023), making it more numerically stable as a least-squares problem, by truncating small singular values from the SVD of the (pretrained) embeddings of the samples from *all* previous tasks. The truncation might also positively impact the generalization itself.
The authors link their method to a theoretical framework (ICL by Peng et al., ICML 2023) which derived sufficient conditions to preventing forgetting in linear models, and use this connection to derive some bounds on the training and generalization error of the proposed algorithm.
# Major concerns

1. **Proposed analysis feels contrived.**   
To me, the analytical connection between ICL (Peng et al., ICML 2023) and the proposed method feels overly strained.
    1. To begin with, the ICL formulations presented in Line 111 and 116, are purely theoretical (formalizing a sufficient condition to prevent forgetting), and *cannot* be considered continual learning approaches, since they require storing the data from all previous tasks.
    1. In Line 105 (Eq. 1), the raw feature matrices $X_t$ are randomly mapped to the "effective" matrices $H_t=\text{relu}(PX_t)$ using a random matrix $P\in\mathbb{R}^{E\times d}$ where $E\gg d$.
To facilitate the comparison to the ICL framework, in Line 113, the authors assume that $E$ is so large that it makes the *entire* task sequence feasible (jointly!). However, for most datasets, this would practically make the time and space requirements of the proposed method infeasible.
    1. Theorem 1 also implicitly requires such extreme overparameterization or that the data can already be explained linearly almost perfectly. Otherwise, $\Vert\mathcal{E}_{1:t}\Vert$ would be large, making the bound uninformative ($\frac{M_t-k_t}{M_t}$ is not small).  
However, if the data is linearly explained by some model $W^{*}_t$, then ICL trivially obtains 0 loss (since all previous data is stored and optimized simultaneously) and all that is left to analyze is the error stemming from the truncation (which implicitly assumes that the smaller singular values are very small, which might not be the case for arbitrary data).   
In my opinion, this is **not** a significant theoretical contribution.

1. **Unclear if accuracy improvements are significant.**  
 It seems like the results in the main Table 1 are of a single run per model-dataset combination. No standard deviation is reported. Given that most of the improvements upon RanPAC are somewhat small, it is unclear if these are indeed statistically significant.

---

## Summary of weaknesses

 Considering all of the above, I believe the paper constitutes a nice incremental improvement over RanPAC (using truncation instead of regularization), but I am not convinced that the analysis makes the paper valuable enough for ICLR.

-----

# Minor concerns

I do not expect any response on the following concerns but I recommend addressing them in any future revised version.

1. Many times (e.g., Line 160), the authors specify constant values like $\lambda = 10^{4}$ and $\lambda = 10^{-5}$  and refer to them as being very small or very large. However, I find these references problematic since the regularization strength in least-squares problems is meaningless without taking into account the scale of the data and the fitting error. Similarly, I would even go further and say that the main benefit from the proposed method is not that it gets rid of the smallest singular values, but that it remedies the *condition number* of the matrices. The smallest singular values can only be considered "small" when comparing to the largest singular values. An alternative method could have limit the condition number instead of using a fixed percentage $\zeta$.

1. I think Figure 1c would be better if it showed the training accuracy, since this is the origin of the problem there.

1. The time complexity analysis should appear immediately after the algorithm, rather than as a side note on page 9. Space complexity should also be specified. Moreover, at some point, given enough tasks, $M_t = k_{t-1} + m_t$ can become larger than $E$, which then makes the time complexity $E^3$ (notice that the $M_t - $E$ smallest singular values are zeros), just like in RanPAC.

1. In Line 130, the authors say "the theoretical justification for the strong empirical performance of RanPAC remains limited". I kindly disagree and claim that it is trivially justified, since according to the formula in Line 124, it just minimizes *all* the losses simultaneously.

1. To be slightly more precise, step 3 in Algorithm 1 should use the floor operator $\lfloor(1-\zeta)M_t\rfloor$.

1. The paper would be more readable if the dimensions are specified when matrices are first presented, e.g., the authors should specify the dimensions of ${\widetilde{U,\Sigma,B}_{1:t}}$ in Eq. (3).

1. The sentence in Line 319 (about the values of $\delta$) is unclear.

1. The message of Figure 6 is unclear.

1. In Theorem 1, I believe $\frac{M_t-k_t}{M_t}$ is simply $(1-\zeta)$.

1. In Theorem 1, specify the subscript for the norm used for $\mathcal{E}_{1:t}$.

1. The presentation is often overcomplicated and inconsistent. For instance, in line 105 (Eq. 1), a projection matrix $P$ is introduced, while most of the analysis throughout the paper, especially in Section 5, completely ignores that matrix and deals with arbitrary deterministic feature matrices $H_t$. Moreover, I would consider presenting the ICL details only after the algorithm itself, as a retrospect.
The paper proposes a simple method that delivers stable and strong performance with theoretical guarantees for continual learning with pre-trained models.
@inproceedings{
anonymous2024icltsvd,
title={{ICL}-{TSVD}: Bridging Theory and Practice in Continual Learning with Pre-trained Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=bqv7M0wc4x},
note={under review}
}
The goal of continual learning (CL) is to train a model that can solve multiple tasks presented sequentially. Recent CL approaches have achieved strong performance by leveraging large pre-trained models that generalize well to downstream tasks. However, such methods lack theoretical guarantees, making them prone to unexpected failures. Conversely, principled CL approaches often fail to achieve competitive performance. In this work, we bridge this gap between theory and practice by integrating an empirically strong approach (RanPAC) into a principled framework, Ideal Continual Learner (ICL), designed to prevent forgetting. Specifically, we lift pre-trained features into a higher dimensional space and formulate an over-parametrized minimum-norm least-squares problem. We find that the lifted features are highly ill-conditioned, potentially leading to large training errors (numerical instability) and increased generalization errors (double descent). We address these challenges by continually truncating the singular value decomposition (SVD) of the lifted features. Our approach, termed ICL-TSVD, is stable with respect to the choice of hyperparameters, can handle hundreds of tasks, and outperforms state-of-the-art CL methods on multiple datasets. Importantly, our method satisfies a recurrence relation throughout its continual learning process, which allows us to prove it maintains small training and generalization errors by appropriately truncating a fraction of SVD factors. This results in a stable continual learning method with strong empirical performance and theoretical guarantees.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Continual Learning; Pretrained Models; Overparameterization; Generalization; Random Feature Models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/fa008bc5d6fa227703917b672d69e05e981e57dc.pdf
transfer learning, meta learning, and lifelong learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
ICL-TSVD: Bridging Theory and Practice in Continual Learning with Pre-trained Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission