Features are fate: a theory of transfer learning in high-dimensional regression
transfer learning;deep linear networks;fine tuning;random matrix theory;high dimensional statistics
Yes
No ethics review needed.
Q1- Does the proposed theory shed some light into why some approaches (e.g., contrastive learning with self-supervised learning ) learn better and more transferable representation compared to others? 

Q2- Is it possible to use the proposed theory to improve the transfer learning capabilities of the models? For example, during the representation learning phase with the source data, we can add a regularizer that tries to maximize the the overlap in the feature space using a small amount of the target data. I would like to hear the authors thoughts on this and other ways to leverage this theory in practice.
1- Even thought the structure of the paper is atypical (e.g., short introduction with related work being a subsection), the paper is well-written and easy to follow.

2- Transfer learning is very important topic in DL and understanding when/why it works is crucial for the overall understanding of deep learning.

3- I did not read all the proofs in extreme details. I only checked the skeleton of the proofs and they seem correct and the results are reasonable. However, I am not update with the current advancements in transfer learning theory so i can not judge the novelty of the proofs/theoretical results. Hence so the low confidence in the score.
This paper addresses the gap in understanding how to adapt large pre-trained models to data-limited tasks by examining the theory behind task similarity in transfer learning. It challenges the conventional belief that similarity between source and target data distributions, such as through ϕ-divergences, predicts transfer success. Instead, it suggests a feature-centric approach where transfer learning is effective when the target task aligns well with the feature space of the pre-trained model. The authors demonstrate their theory using deep linear networks, providing insights into when transfer or fine-tuning outperforms training from scratch, especially with limited data.
1- Lack of the empirical results: the authors only consider a small example of a two layer neural networks. So it is hard to see if the conclusion/insights of their theory  can really translate to deep state-of-the-art models. 

2- The proposed theory does not offer any insights on the design of the representation learning approaches and why some work better than other (e.g., contrastive learning with self-supervised learning) which in my opinion is a very interesting question crucial to understand transfer learning.
Yes
No ethics review needed.
If I have a misconception regarding Theorem 2.2, can you please clarify your explanation? The entire paper revolves around this result.
The authors are correct that a theory of fine-tuning foundation models is lacking, and so work in this space is welcome. Indeed an explanatory theory for transfer/fine-tuning of nonlinear, deep networks would be a significant advance in the field.

The feature-centric viewpoint appears novel.
This paper aims to provide new theoretical insights into fine-tuning and transfer learning with large-scale foundation models. In particular, they aim to improve the theoretical basis of measuring the similarity between two machine learning tasks - the baseline task for which the foundation model was trainied and the target task for which we are fine-tuning/transfering. The authors argue that one should study this problem from the standpoint of data features. The authors claim to make progress and relate their results to feature learning dynamics.
I found it difficult to assess the contributions of the paper because of unclear explanations. In particular, Theorem 2.2 is crucial because it is claimed to motivate the work (“General Theoretical Setting”), but I cannot follow the argument. What I can best infer from the text is that the authors are saying the following:  If the source and target distributions are the same (or similar) but the features are different, then transferring will fail. This means that, even if the data is the same for both source and target, if a network f is the pretrained one, then we can find a "bad" network g that has worse performance. But this seems obvious, since one can pick a random network g that will with overwhelmingly high probability have worse performance than any trained network. My suspicion from this is that the authors’ results won't actually hold up in practice. But it is hard to tell because the set up is so unclear.

The claimed theory (that I could not exactly follow due to the unclarity above) is proved only for deep linear networks and shallow (e.g, 2 layer) nonlinear networks, which significantly reduces its potential explanatory potential in realistic deep learning settings.
Yes
No ethics review needed.
See weaknesses.
1. The results presented, especially Theorem 3.7, 3.8, and 3.9 are, to my knowledge, novel and relevant to the development of the theory of transfer learning in neural networks. Although in the linear setting, these results are non-trivial to obtain since the authors operate in the rich regime, which corresponds to a non-convex optimization problem from an optimization perspective (all the weight matrices move from initialization in the limit).

2. The paper is overall very well-written. The results are nicely presented, and the storyline is logically flawless. I especially appreciated how results from existing work are adequately presented, such that the novel theorems feel tied into the existing literature. Finally, the results are adequately discussed and put into context (e.g. Theorem 3.6).
The paper theoretically studies transfer learning for deep linear networks in the feature-learning "rich" regime, in the joint scaling of large number of fine-tuning samples $n$ and dimension (including width) $d$. The pertaining is performed on the population loss to simulate the fact that the source task has to be larger than the fine-tuning one.

The authors first argue that the model-agnostic task-similarity metrics (based solely on the source and target distributions) are insufficient to capture how good the transfer is and find counter examples where it can be the case. Then, they propose a new metric based on the discrepancy between the performances of the fine-tuned model and the trained-from-scratch model.

Given this metric, they compute the network performances in the asymptotic limit described above, both in the case of linear fine-tuning and "full" fine-tuning of all the parameters.
The main fundamental weaknesses of this paper in my opinion lie in how certain results are portrayed, which can be quite misleading. More concretely:

1. First of all, I think studying linear networks in the specific context of transfer learning is an inherent limitation that should be more clearly discussed. This is because the function class is linear no matter how many hidden layers. In fact, one could consider a simple linear regression model (for which the asymptotic analysis is known, as the authors correctly cite), and then perform linear evaluation on fresh new weights. This is how I interpret Theorem 3.5 and 3.6, where there is no dependence on the depth.

2. I find it a bit misleading to consider as misleading the dataset-based discrepancy metrics based on the “anomalous positive transfer”. In this case, as the authors state, the positive transferability comes purely from the double descent phenomenon and not from anything inherent positive feature of fine-tuning. In a sense, the benefits of pretraining come purely from the additional datapoints that the model is pre-trained on and that makes you avoid double descent. In this sense, I feel that the message stated in the abstract that model-agnostic similarity metrics are insufficient, while the proposed feature-centric metric is portrayed as the solution, is misleading: the proposed metric has precisely this flaw that does not take into account the double descent phenomenon. To give an example, when the source and target distributions are exactly the same, the metric would still present this anomaly. Thus, comparing with the trained-from-scratch model is not enough to evaluate transferability.

3.  The results in Section 3.8 are also incomplete. How does the measure of transferability $\mathcal{T}$ change? I appreciate the result that in general, the transfer is worse (and this is a nice result per se), but it would be nice to tie it to the previous section, i.e. comparing it to the trained-from-scratch model. I would imagine that the picture in Figure 1 (b) would be different, as regularization makes training from scratch in general a bit better by avoiding double descent. Again, in this case, I would imagine that the dataset-based discrepancy metrics correlate with transferability. More generally, I wonder how much of these conclusions would hold excluding the double descent phenomenon, or if what we are observing is entirely due to it. If this is the case, then advocating for a feature-centric view of transfer learning is misleading.

4. Figure C requires better formatting. 

Minor issues/questions:

1. What is M, line 161?

2. Dudley Metric not introduced

3. Assumption (22) should be explained in the main text, or at least referenced because initialization is not really discussed there.
Yes
No ethics review needed.
Can this framework also explain what happens in the case of gradient descent (rather than gradient flow)? As a follow up, what would be the effect of noise in stochastic gradient descent?

While I do understand and agree with Theorem 2.2, it seems to go against prior literature showing that these metrics correlate well with transfer. Would it be possible to empirically test the claim? More concretely, would it be possible to show an experiment where the source and target distributions are “far apart” with respect to the stated metrics, yet the transfer achieved is still positive?

Can these results also be extended to logistic regression in a classification setting?
The paper is very well written in general and the result is novel and interesting. The model is able to capture transfer learning both just by fine tuning on the last layer, and over the whole parameters. The authors validate empirically their results on regression tasks.
The paper studies an analytical model for transfer learning in deep linear networks. To begin, the authors prove that general wisdom surrounding feature learning, namely that being close in distribution between the source and the target domains, is not a necessary condition for transfer learning. The authors study deep linear networks trained using gradient flow and establish a measure of transfer, namely the transferability, which is the difference between the generalization error of the model trained from scratch and the one that has undergone transfer learning. The authors study the transferability phase plots as a function of source-target alignment and the data to parameter ratio for linear transfer, meaning just the readout layer is trained on the target task, and fine-tuning, where all the layers are trained on the target. Finally, the authors show empirically that their predictions hold in the case of a 2 layer relu network.
The paper is generally well written and understandable, and there aren’t really any major flaws that I could find. While one may consider the very simplified setting as a weakness, especially since in deep non-linear networks and Transformers the transfer learning behaviour might be different, I believe that such models would currently be too complicated to study. While I do not find any technical flaw with the paper, I do have several questions that I detailed below.
Yes
No ethics review needed.
1. Equations (1) and (2) use the same $\epsilon$. Does this imply that the source and target outputs for a given input $x$ share the same noise component?

2. In Equation (6), could the authors clarify the interpretation of $\Theta_i$?

3. In line 174, what is the meaning of the symbol $\leq_L$?

4. Theorems 3.4 and 3.5 are referenced from Yun et al. What is the motivation for re-proving these theorems, and why are they prominently highlighted in the main text?
This paper addresses the essential and meaningful topic of understanding transferability in transfer learning. The problem setup, including assumptions, is clearly and concisely presented.
This paper develops a theoretical framework to analyze transferability in high-dimensional regression tasks within the context of transfer learning. The authors adopt a feature-based approach, proposing that the similarity between source and target tasks is best understood through feature space overlap rather than distributional similarity. The paper establishes phase diagrams that quantify the conditions under which transfer learning outperforms training from scratch, particularly when feature space alignment is high and target data is limited. The analysis primarily focuses on deep linear networks but extends some insights to nonlinear networks.
The main theoretical results depend on assumptions like normally distributed inputs and linear source and target functions, which limit applicability. Nonetheless, those simplifications are reasonable for an initial exploration of this topic.
@inproceedings{
anonymous2024features,
title={Features are fate: a theory of transfer learning in high-dimensional regression},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Gc2qkiYUkh},
note={under review}
}
With the emergence of large-scale pre-trained neural networks, methods to adapt such "foundation" models to data-limited downstream tasks have become a necessity.
Fine-tuning, preference optimization, and transfer learning have all been successfully employed for these purposes when the target task closely resembles the source task, but a precise theoretical understanding of ``task similarity'' is still lacking. 
While conventional wisdom suggests that simple measures of similarity between source and target distributions, such as $\phi$-divergences or integral probability metrics, can directly predict the success of transfer, we prove the surprising fact that, in general, this is not the case.
We adopt, instead, a \emph{feature-centric} viewpoint on transfer learning and establish a number of theoretical results that demonstrate that when the target task is well represented by the feature space of the pre-trained model, transfer learning outperforms training from scratch.
We study deep linear networks as a minimal model of transfer learning in which we can analytically characterize the transferability phase diagram as a function of the target dataset size and the feature space overlap.
For this model, we establish rigorously that when the feature space overlap between the source and target tasks is sufficiently strong, both linear transfer and fine-tuning improve performance, especially in the low data limit. 
These results build on an emerging understanding of feature learning dynamics in deep linear networks, and we demonstrate numerically that the rigorous results we derive for the linear case also apply to nonlinear networks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
transfer learning deep linear networks fine tuning random matrix theory high dimensional statistics
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/daf03d4d24c05e41a42460761daa7b948ed828c3.pdf
learning theory
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Features are fate: a theory of transfer learning in high-dimensional regression
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission