Multiscale Training of Convolutional Neural Networks
Multilevel Stochastic Gradient Descent;Multiscale Training;Mesh Free Convolutions
Yes
No ethics review needed.
1. All the results show mixed performance bettween the regular SGD and the two Multiscale-SGD. If you could train longer for the multiscale-SGD, could you improve all the performance comparing to SGD?
2. The MSE results are strange, with the two UNet have huge gap. What is the reason behind it?
This paper introduces an innovative approach to multiscale CNN training through Multiscale-SGD and Mesh-Free Convolutions (MFCs).
All the developments are backed by mathematically reasoning.
And the whole paper is logically organized.
This paper introduces an approach to improving the computational efficiency of CNNs through a multiscale training framework.
It first presents Multiscale Stochastic Gradient Descent (Multiscale-SGD), which leverages the Multilevel Monte Carlo method to reduce computational costs by approximating gradients across resolutions.
However, the authors identified that noisy or high-frequency inputs can lead to unbounded gradients across mesh resolutions, complicating the convergence process.
To address this, they propose Mesh-Free Convolutions (MFCs), which are independent of specific input scales and ensure consistent gradient convergence across resolutions, even with noisy inputs. This mesh-independence of MFCs overcomes the convergence limitations observed in standard CNNs during multiscale training.
1. The experimental results are really limited, and you should also compare the performance with the fixed computational budget.
2. The experimental comparison with existing multiscale or Fourier-based CNN methods is not presented
3. The mathematical foundation for Mesh-Free Convolutions, particularly the differential operator theory, could be challenging for readers less familiar with this domain. Adding a visual explanation or intuitive analogies could make the theory more accessible.
4. In the experiments, the network seems really shallow, could you train deeper networks, to see whether the methods works?
Yes
No ethics review needed.
__Q1.__ To illustrate the difference between conventional CNNs and MFC, would it be possible to compare feature maps between these two approaches?

__Q2.__ A more formal definition of the _work unit_ metric #WU might be beneficial. As I understand it, #WU measures how many evaluations of the highest resolution are required, with evaluations at lower resolutions being weighted by the corresponding “downsampling factor”. Is this correct? If so, should this not lead to fractional values of #WU?
I believe this paper provides several important contributions and extends the existing literature in different significant ways:

__S1.__  The authors propose new techniques for training CNNs within a multiscale resolution framework, which improve computational efficiency while maintaining test performance. This alone can be a valuable contribution, as shown by the “non-MFC” empirical results in Tables 3, 4, and 5. 

__S2.__ As far as I can tell, MFCs are an original and promising alternative to traditional convolutional layers (though some clarifications may be necessary; see W2 below) that can alleviate some of the shortcomings of traditional CNNs in multiscale approaches. Given their potential application across various high-resolution tasks, this can be a significant contribution, too.

__S3.__ The quality of the paper is high, with claims well-supported through mathematical derivations, proofs, and empirical evidence. In particular, the experimental results in Tables 3, 4, and 5 highlight that significant speed-ups might be feasible (as measured by #WU; however, see Q2 below).

For these reasons, I consider this a good paper.
Training convolutional neural networks (CNNs) on high-resolution images can be computationally demanding. One way to alleviate the computational burden is by including downsampled versions at different scales during training. This paper identifies shortcomings of standard CNNs in such multiscale approaches and introduces “Mesh-Free Convolutions” (MFCs) to address these limitations. The paper presents two novel multiscale training algorithms, “Multiscale-SGD” and “Full-Multiscale-SGD”, which estimate high-resolution parameters using coarser-resolution samples, thus avoiding costly training on high-resolution data. For such multiscale training schemes, gradients of conventional CNNs for noisy data might diverge with decreasing resolution compared to “smooth data”. Inspired by differential operators, MFCs offer a resolution-independent generalisation of standard convolutions. The empirical validation demonstrates that these techniques can improve training time and ensure consistent gradient behaviour, as shown for standard UNet and ResNet architectures and image blur estimation, image denoising, and image super-resolution tasks.
Some clarity aspects could be addressed to strengthen the paper. In more detail, I currently see the following weaknesses:

__W1.__ Section 3.1 on “Mesh-Free Convolutions” is relatively dense and challenging to follow. For instance, some notations typically used with parabolic PDEs, like indices denoting partial derivatives, should be more clearly introduced. More importantly, the connection between $u$, $v$, $\tilde{v}$, and $\mathcal{C}$ is not very clear and should be made more explicit.

__W2.__ Expanding on the previous point, the paper would benefit from a more intuitive explanation or illustration of the connection between standard convolutions and mesh-free convolutions. Currently, the connection is difficult to assess. In that regard, Figure 3 lacks clarity and might require additional explanation. 

__W3.__ _Computational considerations_: As the authors highlight in lines 334-338, the required Fast Fourier Transform dominates the overall computational cost. To provide the complete picture, additional consideration of running time and optimisation steps in the results in section 5 on “Experimental Results” would strengthen the contribution. This would allow a more comprehensive assessment of MFC's computational feasibility. 

Minor remarks:
- Line 231: “[…] without compromising on its efficiently.” should likely be “[…] on its _efficiency_.”
- Line 293: “[…] let $v = \mathcal{C}(\xi)u$ be the mesh-free convolution is parameterized by $\xi$“ needs to be rewriten.
- Line 296: Equation “(17a)” likely should just read “(17)”.

My rating is currently “borderline accept” due to the aspects raised in W1 to W3. However, I am willing to adjust my rating if some of these points and the questions below can be addressed.
Yes
No ethics review needed.
1) Why are there no experiments on CV prediction tasks like classification, segmentation, object detection, etc.? Are the proposed algorithms not expected to work well in these contexts?

2) Corresponding to the data augmentation comment above: It was not clear to me if you use data augmentations like RandomResizedCrop or similar in any of your experiments; do you? If not, would you be able to show what happens when these are added to the single-scale SGD training procedure?

3) Corresponding to the comment on presentation of results comment above: could you add (even approximate) FLOPs computations or raw timings to your experiments?

4) Are there situations in which it would be preferable to use the Multiscale-SGD algorithm that you proposed instead of the Full-Multiscale-SGD algorithm? If not, it would be better to focus more of the experiments on the Full-Multiscale algorithm, and leave the Multiscale algorithm to an ablation study.
1) The primary research questions in the paper are well-motivated.
2) The technical analysis seems to be sound, and the authors are able to propose interesting, non-obvious training algorithms from them (i.e., this goes far beyond "multiscale training is useful for learning convolutional kernels"). I also found the further step of mesh-free convolutions to be very interesting from a theoretical perspective.
This work begins with an error analysis of the the standard training procedure for convolutional networks when training at multiple resolutions, and (using some clever re-interpretations by varying mesh size and sample size) derive new multiscale training algorithms that require fewer network evaluations at the finest mesh size. They further present mesh-free convolutions that can be thought of as limits of progressively refined mesh-dependent convolutions. Finally, experimental results are presented to compare the training algorithms and network variants against deep learning standards like SGD and U-Nets/ResNets.
My major problems with the paper pertain to the experimental results, and fall into two main categories:

1) The experiments done are on extremely small networks and datasets. There are very standard collections of computer vision experiments that could be used to show the benefits of the proposed multiscale-SGD training algorithms (see, for example, the benchmarks used for experiments in any landmark CV papers from the past few years, like CLIP or Segment Anything). I understand that compute access could be a factor here, but many of these datasets are small enough that access to even a single GPU should be sufficient to perform the necessary experiments.

2) Unclear presentation of results. It's difficult to understand the relative compute requirements of different algorithms based on the work unit representation that is used in the experiments. It would be much clearer to either present the FLOPs required per iteration or backward pass, or best of all would be to simply show the amount of time taken to train the network in each case.

3) Non-competitive baselines. In practice, multiscale training is implicitly accounted for in most deep learning training procedures via data augmentations in the form of varying-sized crops of images (see the widely-used RandomResizedCrop transform in torchvision). To me, this is the actual baseline to which we should compare to, but it is not mentioned or compared against at all in your work.

Based on these, I find it difficult to be convinced by the presented experiments that the proposed multiscale algorithms would be better to use than standard SGD on usually-sized computer vision trianing tasks.
Yes
No ethics review needed.
See the above weaknesses
1.	The paper presents a good framework that integrates multiscale training with convolutional neural networks (CNNs).
2.	The writing of this paper is easy to follow.
3.	The experiments validate the effectiveness of the proposed methods, showcasing practical improvements of the proposed method.
4.	This paper offers thorough mathematical analysis, including proofs and lemmas.
The paper presents a training framework designed to optimize CNNs more efficiently by employing multiscale training strategies. This approach targets the significant computational challenges that arise when training CNNs at high resolutions, especially when handling noisy input data. By introducing Multiscale-SGD and the MFCs, the framework reduces computational costs and ensures stable gradient behavior across varying scales.
1.	A significant limitation is the evaluation performed with shallow CNNs, all models containing fewer than four layers. Given that practical applications often leverage deeper architectures like ResNet with over 18 layers, it raises concerns about the scalability and real-world applicability of the proposed multiscale framework. Demonstrating results with deeper networks could help validate its utility for larger and more complex tasks.
2.	In the context of modern architectures, transformers such as Vision Transformers (ViT) are becoming more prominent for various applications. The paper does not explore whether the proposed method could be adapted or extended to transformer-based architectures. This is particularly relevant as transformers may handle multiscale data differently from CNNs. Discussing or experimenting with how the proposed method might extend to these architectures would strengthen the paper's broader applicability
3.	While the paper presents experimental validation of the proposed method's effectiveness, the experiments are mostly focused on simpler tasks. The applicability of the method to more complex and challenging applications, like object detection, video understanding tasks, remains untested. 
4.	While the paper claims significant computational advantages with the multiscale approach, it lacks a thorough comparative analysis of computational costs with existing training methods. For instance, how this method compares in terms of hardware resource use, memory consumption, or training speed against some techniques like mixed-precision training is not clearly outlined.
5.	The paper could benefit from a deeper exploration and further discussion of how the training dynamics change with the proposed method compared to standard methods.
We propose a PDE-inspired multiscale framework to accelerate training of neural networks, and demonstrate it on several benchmarks.
@inproceedings{
anonymous2024multiscale,
title={Multiscale Training of Convolutional Neural Networks},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=n0YCAMVh8b},
note={under review}
}
Convolutional Neural Networks (CNNs) are the backbone of many deep learning methods, but optimizing them remains computationally expensive. To address this, we explore multiscale training frameworks and mathematically identify key challenges, particularly when dealing with noisy inputs. Our analysis reveals that in the presence of noise, the gradient of standard CNNs in multiscale training may fail to converge as the mesh-size approaches to $0$, undermining the optimization process. This insight drives the development of Mesh-Free Convolutions (MFCs), which are independent of input scale and avoid the pitfalls of traditional convolution kernels. We demonstrate that MFCs, with their robust gradient behavior, ensure convergence even with noisy inputs, enabling more efficient neural network optimization in multiscale settings. To validate their generality and effectiveness, we show that MFCs can seamlessly replace standard convolutional layers across diverse architectures and benchmarks, delivering substantial computational speedups without sacrificing performance.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multilevel Stochastic Gradient Descent Multiscale Training Mesh Free Convolutions
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/5fb632d5c68cf4fb866609c68454bc460eb576b6.pdf
optimization
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Multiscale Training of Convolutional Neural Networks
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission