Implicit Bias of Mirror Descent for Shallow Neural Networks in Univariate Regression
implicit bias;overparametrized neural network;mirror descent;univariate regression;lazy training
Yes
No ethics review needed.
* Given that the networks are in lazy regimes and similar to random feature models, is the setting related to this paper on random feature models with Lp penalties on the weights? https://arxiv.org/abs/2103.15996

* Do you have a sense of whether such mirror flow analyses extend to feature learning regimes, e.g. by switching to a mean-field parameterization? What would be the resulting implicit bias?
The paper is well written, and the obtained implicit bias results for scaled potential is quite interesting as it involves new norms that have not been used before in the context of neural networks, to my knowledge.
The paper studies the implicit bias of mirror flow on shallow univariate neural networks in the standard parameterization. The authors show that the networks are in the lazy regime (only the second layer weights move non-negligibly), and characterize the representation norm in two different regimes. In the "unscaled" regime, the mirror map plays a minor role, only through its Taylor expansion around 0, leading to an NTK-like implicit bias similar to GD. In the "scaled" regime, where the potentials are rescaled by width, a richer implicit bias arises involving the corresponding Bregman divergence, which is no longer a RKHS norm.
The significance is unclear: while the obtained norms are interesting, it is unclear if mirror flow is a relevant method in the study of neural networks. The lazy training regime suggests that it may be good to reframe the paper more generally outside of neural networks (which in practice are often in non-lazy regimes), for instance mirror descent on random features models, which could be of interest to the statistical learning and kernels community more broadly. The analysis is also somewhat incremental over existing implicit bias results.
Yes
No ethics review needed.
1) All the convex potentials bake in the initial parameter $\hat \theta := \theta_0$. I wonder how much of the implicit bias results are affected by the choice of $\hat \theta$. If $\hat \theta$ was arbitrary (or any point other than $\theta_0$), how do the lazy training results change (i.e. is it still true that the parameters do not move far from initialization)? Of course when the coordinate function $\phi(x) = \frac 1 2 x^2$ (i.e. GF), the choice of $\hat \theta$ does not matter but for $\phi(x) = x^2 + x^4$, the mirror flow update explicitly penalizes movement away from $\hat \theta$ and hence it less clear whether lazy training should occur. 

2) Why is the mirror flow update scaled by $\eta$? Do similar results hold if $\eta$ is dropped?

3) Is the restriction to the univariate case only needed for the variational characterization of the asymptotic solution? 
(i.e. can the univariate requirement be dropped for establishing lazy training results and risk minimization?)
Characterizing the asymptotic solution of mirror flow via a variational problem is interesting.
The paper establishes various implicit biases for wide shallow networks trained via mirror flow in the context of univariate regression which roughly fall into two categories: (i) parameters do not move far from initialization while asymptotically minimizing risk (ii) the asymptotic solution (asymptotic with respect to time) can be characterized via a variational problem.
The work only considers univariate models. 


minor comment:
Considering the work only deals with mirror flow, I believe the title should be remedied to "Implicit bias of Mirror Flow ...".
Yes
No ethics review needed.
In both Theorem 2 and Theorem 8:
“with high probability over the random parameter initialization,
there exist constants C_1, C_2 such that …”. It sounds like the constants C_1,C_2 may depend on the initialization (which also depends on n). Did you mean to write “there exist constants C_1, C_2 such that w.h.p. over the initialization …”?
The implicit bias of mirror descent/flow has been studied mostly for linear models. Here, the authors extend our understanding of mirror flow to the case of univariate two-layer networks in the lazy regime, and also give a more precise characterization of the implicit bias in function space for gradient flow in this setting. I think that the contribution is of interest to the implicit bias community, and continues two lines of work: the works on the implicit bias of mirror descent, and the works on the implicit bias of univariate shallow networks.

I believe that the contribution is significant and hence recommend acceptance.
The paper studies the implicit bias of mirror flow in univariate two-layer networks in the lazy regime. For a broad class of potential functions, they show that the implicit bias at the limit of infinite width is similar to the implicit bias of mirror flow. They also provide a characterization of this implicit bias in function space which generalized prior results. Moreover, they introduce scaled potential functions, where training is in the lazy regime but not in the kernel regime, and characterize its implicit bias.
It would be nice to extend Theorem 2 beyond Assumption 1, which restricts the class of potential functions covered here, and also to relax the many assumptions in Theorem 8. But I understand that the authors needed these restrictions for technical reasons.
Univariate wide shallow networks trained with mirror flow are biased towards smooth interpolants and scaled mirror potentials induce biases affecting the different magnitudes of the curvature.
@inproceedings{
anonymous2024implicit,
title={Implicit Bias of Mirror Descent for Shallow Neural Networks in Univariate Regression},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=IF0Q9KY3p2},
note={under review}
}
We examine the implicit bias of mirror flow in univariate least squares error regression with wide and shallow neural networks. For a broad class of potential functions, we show that mirror flow exhibits lazy training and has the same implicit bias as ordinary gradient flow when the network width tends to infinity. For ReLU networks, we characterize this bias through a variational problem in function space. Our analysis includes prior results for ordinary gradient flow as a special case and lifts limitations which required either an intractable adjustment of the training data or networks with skip connections. We further introduce *scaled potentials* and show that for these, mirror flow still exhibits lazy training but is not in the kernel regime. For networks with absolute value activations, we show that mirror flow with scaled potentials induces a rich class of biases, which generally cannot be captured by an RKHS norm. A takeaway is that whereas the parameter initialization determines how strongly the curvature of the learned function is penalized at different locations of the input space, the scaled potential determines how the different magnitudes of the curvature are penalized.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
implicit bias overparametrized neural network mirror descent univariate regression lazy training
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/50caf785d5a2cb53939e72101991cc760df1a5fb.pdf
optimization
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Implicit Bias of Mirror Descent for Shallow Neural Networks in Univariate Regression
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission