Revisit Micro-batch Clipping: Adaptive Data Pruning via Gradient Manipulation
Micro-batch Clipping;Convergence Analysis
Yes
No ethics review needed.
See Weaknesses.
1. Novel Conceptualization: The paper innovatively frames micro-batch clipping as a data pruning technique, introducing the concept of "dragger" gradients to explain its mechanism. This provides new insights into why and when micro-batch clipping works.
2. Comprehensive Analysis: The paper conducts both theoretical convergence analysis and extensive empirical evaluation to support its hypotheses. The results demonstrate the effectiveness of micro-batch clipping across different model architectures and datasets.
This paper revisits micro-batch clipping, an optimization technique initially proposed for memory efficiency in differentially private stochastic gradient descent. The paper conceptualizes micro-batch clipping as a form of data pruning, proposing that certain "dragger" gradients impede model convergence. Through convergence analysis and empirical evaluation, the paper demonstrates that micro-batch clipping can asymptotically accelerate convergence but introduces a constant bias term. This bias term explains the existence of an optimal micro-batch size. The paper further validates the effectiveness of micro-batch clipping beyond speech models, showing promising results on vision and language tasks. However, limitations are identified when training data originates from multiple distinct domains.
1. Lack of interpretability or robustness. This paper does not discuss the potential impact of micro-batch clipping on the interpretability or robustness of the trained models, which are important considerations in many real-world applications.
2. Limited scope of experiments: The experiments are primarily conducted on speech recognition models and limited vision and language tasks. The generalizability of the findings to a broader range of models and datasets is unclear. More extensive experimentation across different domains and model architectures would strengthen the conclusions.
3. Concern of memory overhead: While the paper mentions that data sharding with a micro-batch size equal to the per-core batch size can mitigate memory overhead, this may not be feasible for larger models. The growing size of modern models necessitates alternative solutions to ensure computational efficiency.
4. Rigorous relationship between micro-batch size and performance: The paper suggests that there exists an optimal micro-batch size that minimizes the constant bias term in the convergence analysis. However, the relationship between the micro-batch size b and the factor c is not rigorously established. A tighter convergence bound and a more accurate relationship between b and c could better guide the choice of micro-batch sizes in practice.
Yes
N/A
No ethics review needed.
Please refer to the weakness section.
- Uncovering the mechanism behind the performance improvements brought by micro-batch clipping is both interesting and valuable.
- The paper provides a series of theoretical foundation to analyze this mechanism and support its conclusions.
- The theoretical findings in this paper can explain empirical results, such as the performance gains achieved through micro-batch clipping and why these improvements occur only at specific micro-batch sizes.
- Extensive experiments are conducted to explore the benefits of micro-batch clipping beyond ASR.
Micro-batch clipping has recently demonstrated potential in improving model performance in automatic speech recognition (ASR). This paper seeks to uncover the mechanism driving this enhancement. The authors assume that certain data samples ("draggers") may impede model convergence. Through theoretical analysis, they find that micro-batch clipping can accelerate convergence by diminishing the impact of these draggers, albeit introducing a constant bias dependent on micro-batch size. This insight further explains the observed phenomenon where improvements occur only at specific micro-batch sizes. Additional empirical experiments are conducted beyond ASR to investigate the effectiveness of micro-batch clipping.
- The Appendix of this paper is somewhat lost, and the full proofs of the theorems are thus invisible, requiring further verification.
- Assumption 4.4 posits that dragger gradients are orthogonal to the benign gradient subspace, which may be a restrictive assumption and not entirely realistic in practical scenarios.
- In Sec.1, the authors claim that micro-batch clipping can adaptively suppress samples that hinder convergence. However, it remains somewhat unclear in the theorem whether the effectiveness of micro-batch clipping directly stems from suppressing these draggers.
- It appears that as $\epsilon$ decreases, approaching zero and indicating a lower likelihood of draggers, the convergence rate bound in Thm 4.2 actually increases, which seems counterintuitive.
Yes
No ethics review needed.
- How are micro-batches grouped within a mini-batch? Is this process conducted randomly?

- Around line 114, the text states that mini-batch gradients are "summed," but isn't it more accurate to describe this as "averaged" or "aggregated," given that the mean is typically computed?
- The topic of theoretically analyzing the batch-size dependency issue of micro-batch clipping in training ASR models, which has shown empirical success, is both meaningful and interesting.

- Despite the mathematical rigor throughout the paper, the overall writing is easy to read and follow; in particular, both the content and proof of Theorem 4.1 are easy to understand.

- In the case of the ASR task, the paper presents promising empirical results that align well with the theoretical analysis.
This paper presents a theoretical analysis of why the micro-batch clipping method, which has been shown to improve performance in automatic speech recognition (ASR) tasks, is only effective for specific micro-batch sizes. The authors first categorize mini-batch samples into benign samples and dragger samples, followed by a convergence analysis. This analysis provides a explanation of the sweet spot for micro-batch size. Finally, the paper demonstrates that micro-batch clipping is also effective in both vision and speech tasks.
**Methodology**
- Section 4.3 is significantly underdeveloped and requires more extensive elaboration. Expanding on this section with a detailed explanation of the method would improve clarity and depth.
  - In line 279, the paper states, “The proof can be found in ??,” but the proof is missing, and there is no supplementary material to refer to.
  - The proofs for Lemma 4.3 and Lemma 4.4 are also missing. Furthermore, in line 304, "The proof is provided in Appendix ??." mentions that the proof is in the appendix, but no appendix is provided in the paper.
  -  I suggest that the authors include these proofs in an appendix or supplementary material and ensure that all references to proofs are properly linked.

-  The samples that slow down convergence, termed "draggers," seem to refer not to those with gradients perpendicular to the benign samples, but rather those with gradients in the opposite direction (i.e., cosine similarity around -1); in Figure 1, there appear to be cases where benign samples have negative cosine similarity. Are there no samples with reverse gradient directions?

- As noted in the paper's limitations, determining the exact micro-batch size $b$ that achieves the "sweet spot" is a challenging task. This poses a significant empirical limitation, and even within the paper, the optimal micro-batch size changes due to various factors:
  - The best micro-batch size varies by modality; for example, the authors use a size of 4 for the speech task, 32 for the vision task, and 16 for the language task.
  - Additionally, memory constraints may impede the selection of the optimal micro-batch size. For the vision task, a micro-batch size of 32 was used per core, but such size may be difficult to adjust due to memory limitations.
  - I suggest the authors discuss potential strategies or heuristics for selecting micro-batch sizes in practice, considering the observed 
variability and memory constraints.

- Along with batch size, one of the most critical hyperparameters in training neural networks is the learning rate. Prior research has explored the relationship between batch size and learning rate [1]. Is micro-batch clipping independent of mini-batch size, as opposed to micro-batch size? Further investigation into this relationship could be beneficial.

- While this paper analyzes micro-batch clipping in the context of SGD, in practice, SGD is rarely used for training neural networks. It is important to determine whether a similar theoretical analysis holds for modern optimizers like Adam.

- In Section 5.1, to fine-tune the model, the LibriSpeech dataset and handcrafted canaries were synthetically used as benign and dragger samples, respectively. However, this setup seems artificial. In a more realistic scenario where only the LibriSpeech dataset is used, how would dragger samples be detected in practice?  Including practical strategies for detecting draggers would add relevance to the findings.

**Experiments**
- In Tables 2 and 3, experiments were conducted with only batch sizes of 1, 4, and 512, which is insufficient. The authors should have included batch sizes such as 1, 4, 16, 64, 256, and 1024 to provide a more comprehensive evaluation. If these additional results are not feasible, explaining the rationale behind the specific batch sizes chosen would help clarify this decision.

- The analysis related to $|| \mu || \le C || g||$ is completely missing in Section 5.2. I suggest the authors include this analysis in the section or provide an explanation if it was intentionally omitted. 

- How does micro-batch clipping perform when training from scratch instead of fine-tuning? This could offer valuable insights into its effectiveness in different training scenarios.

- When conducting the language task experiments, reporting validation performance is inappropriate. It would have been better to test on a separate dataset. Additionally, the paper states that validation set information was not used at all, so how was hyperparameter optimization carried out? Clarifying this would improve transparency.

- The experimental results shown in Table 5 reveal little to no improvement in performance. It would be helpful to discuss possible reasons for this outcome and consider further refinements or additional evaluation metrics.

**Typo**
- The term "nominator" in line 395 should be corrected to "numerator."

- The equation $ C = \sqrt{\frac{2(1-\epsilon)\eta - (1 - \epsilon^2) L \eta^2}{2 \epsilon^2 L \eta^2}}$ in line 219 should be updated to $ C = \sqrt{\frac{2(1-\epsilon)\eta - (1 - \epsilon)^2 L \eta^2}{2 \epsilon^2 L \eta^2}}. $

- Additionally, the expression $ \frac{2(1-\epsilon)\sigma^2}{LB} $ in line 232 should be updated to $ \frac{(1-\epsilon)\sigma^2}{2LB}. $

[1] Huo et al. "Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling." AAAI 2021.
We revisit micro-batch clipping, a technique to improve memory efficiency in differentially private stochastic gradient descent, and find that it can also provide performance boost.
@inproceedings{
anonymous2024revisit,
title={Revisit Micro-batch Clipping: Adaptive Data Pruning via Gradient Manipulation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=pAkQhhn4vB},
note={under review}
}
Micro-batch clipping, a gradient clipping method, has recently shown potential in enhancing auto-speech recognition (ASR) model performance. However, the underlying mechanism behind this improvement remains mysterious, particularly the observation that only certain micro-batch sizes are beneficial. In this paper, we make the first attempt to explain this phenomenon. Inspired by recent data pruning research, we assume that specific training samples may impede model convergence during certain training phases. Under this assumption, the convergence analysis shows that micro-batch clipping can improve the convergence rate asymptotically at the cost of an additional constant bias that does not diminish with more training iterations. The bias is dependent on a few factors and can be minimized at specific micro-batch size, thereby elucidating the existence of the sweet-spot micro-batch size observed previously. We also verify the effectiveness of micro-batch clipping beyond speech models on vision and language models, and show promising performance gains in these domains. An exploration of potential limitations shows that micro-batch clipping is less effective when training data originates from multiple distinct domains.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Micro-batch Clipping Convergence Analysis
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f566b619f79c61d42b722e79bfeaefe8b00f797b.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Revisit Micro-batch Clipping: Adaptive Data Pruning via Gradient Manipulation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission