Training Over a Distribution of Hyperparameters for Enhanced Performance and Adaptability on Imbalanced Classification
class imbalance;hyperparameter tuning;ROC curves;machine learning
Yes
No ethics review needed.
1. Could the authors clarify how their approach extends beyond a straightforward application of LCT? Highlighting specific innovations or modifications would help in assessing the paper’s originality.
2. Could the authors provide more details on the FiLM implementation? 
3. The method relies on hyper-parameter inputs during inference. What guidelines or heuristics do the authors recommend for setting these parameters effectively?
4. The results in Table 4 indicate that without FiLM, LCT performs worse than the baseline. Could the authors provide an analysis of why FiLM is critical and why its absence degrades performance? Similarly, why does the method not consistently outperform VS-SAM?
1. The paper presents a novel integration of loss conditional training (LCT) within the context of imbalanced image classification. While LCT itself is an existing technique, its application in this specific domain brings new insights, particularly in handling varying hyper-parameters to improve model adaptability and performance on diverse datasets.
2. The research demonstrates rigorous experimental validation. By applying the proposed method across multiple binary image classification datasets, the paper effectively highlights its potential in improving baseline performances. The objective function is thoughtfully designed, focusing on minimizing classification loss over a hyper-parameter distribution, which is both innovative and practical.
3. The paper is generally well-structured and presents its contributions clearly. The explanation of the objective function and its rationale is straightforward, aiding readers in understanding the methodology.
4. Given the widespread challenges posed by imbalanced datasets in image classification, the method’s capability to flexibly adapt post-training, as well as its empirical success on standard benchmarks, underscores its relevance and significance in the field.
This paper is targeted at addressing the imbalanced image classification task. It reveals that the performance of existing methods trained using fixed hyper-parameters is sensitive to hyper-parameters, namely the highest precision is achieved with different hyper-parameter values at different recall. Targeted at learning a single model which can achieve consistently high performance across recall, it introduces the loss conditional training strategy which is originally proposed for image compression into image classification. This enables the hyper-parameters to flexibly tuned during inference. Extensive experiments on a few binary image classification datasets demonstrate the proposed method improves baseline methods focal loss and vector scaling loss consistently.
1. While the application of LCT to imbalanced classification is a reasonable extension, the core technique—LCT—is not novel. The contribution could be viewed as incremental, leveraging existing methodologies without substantial innovation.
2. The choice of baselines seems outdated. To substantiate the method's efficacy, comparisons with more recent state-of-the-art techniques should be included：
SuperDisco: Super-class discovery improves visual recognition for the long-tail, cvpr2023
Balanced product of calibrated experts for long-tailed recognition, cvpr2023
Constructing balance from imbalance for long-tailed image recognition, eccv2022
3. The paper lacks sufficient details on the FiLM (Feature-wise Linear Modulation) module, which is critical for reproducibility. A clearer explanation and inclusion of implementation specifics are necessary.
4. The requirement for hyper-parameter input during inference could pose practical challenges. Providing a guideline for optimal hyper-parameter settings would be beneficial for practitioners and researchers aiming to apply this method.
5. The method's inability to outperform VS-SAM in several settings is concerning. This raises questions about the generalizability and robustness of the approach. Furthermore, the observation from Table 4, where LCT without FiLM underperforms compared to the baseline, warrants a deeper analysis to understand the underlying reasons.
Yes
No ethics review needed.
Why was the model trained for 500 epochs on cifar (section 5.1)? That's excessive for CIFAR.

Why do you need gradient clipping? Is the loss unstable?
* The paper is well-written, and the motivation is clear.

* The method is well-motivated and learning on imbalanced dataset is important.
The paper presents a method for training models on imbalanced datasets by using a distribution over loss hyperparameter values instead of a single value. The hypothesis is that by conditioning the model on different values of hyper-parameters, the model leads to different functions and thus, using a distribution of hyper-parameters leads to a more robust model. The method is evaluated on CIFAR and APTOS datasets.
* Only binary classification is studied in the paper. It is not clear if the proposed method will work on multi-class classification or unsupervised learning. 
* The datasets used in the experiments are very small. Experiments on more large-scale datasets are required (e.g. ImageNet). 
*Only focal loss and VS loss are used as baselines. More extensive comparison against Sota methods for learning imbalanced datasets and long-tail learning. 
* Hyper-parameter search space for focal loss and VS loss is very small, making the comparison somewhat unfair. 
* It is not clear how pdf for sampling $\lambda$ is defined in section 4.3.
* Authors should run the method on different kinds of learning problems in different domains.
* Improvement on CIFAR and SIIM-ISIC datasets ARE very minor. 
* Standard deviation is not mentioned in the result table. How many different seeds were used for each section?
Yes
No ethics review needed.
- I am confused by LCT training part. My understanding is that LCT will point out the best hyper-parameter after training, and in inference we just use the best combination. But Sec. 4.3 claim we still need to evaluate with multiple values $\lambda$. So LCT is more similar to meta-learning, through its training paradigm, the model can generalize better?
- Given there is an underlying distribution over these hyper-parameters, could a VAE approach work as well?
The idea of hyper-parameter sensitivity in imbalanced learning is new.
This paper focuses on imbalanced classification. Authors focus on the hyper-parameter choices of imbalanced loss function and find no optimal solution for existing approach. Instead, the proposed method treating these hyper-parameters as a distribution and sample from it, combine with original data sample input as an additional input for training.
- How is SAM trained with LCT? Since $\lambda$ is associated with each data sample, but SAM is an optimizer?
- How is $P_\Lambda$ calculated? Is it a small network or it is just a pre-defined distribution?
- Line 248-253 is vague in details. How is FiLM compute $\mu$ and $\sigma$? Does it just take one $\lambda$ in and output 2 values? Because without any constraints, these two values are directly applied on activation for linear transformation.
Yes
No ethics review needed.
1. Why does the paper focus solely on binary classification? Could the proposed method be extended to multi-class classification? What about class-balanced scenario?

2. The paper uses Focal Loss and VS Loss as examples of successful applications of the LCT framework. Why were only these two loss functions chosen? Are there more generalizable insights on which types of losses are or are not applicable? 

4. Have the authors considered alternative methods for conditioning the model on lambda (lines 248-253)?
The paper extends the Loss Conditional Training framework to binary classification with class-imbalance setting. The extension is simple but seems useful.
The paper extends the Loss Conditional Training framework to address class imbalance issues, based on the observation that varying hyperparameters in the loss function results in different performance across evaluation metrics. The proposed method shows good performance and desirable properties on several datasets.
1. It is unclear how the distribution of lambda during training and the lambda values to use in inference are chosen. More discussion is needed. 

2. The proposed method seems a simple extension of existing method (Dosovitskiy & Djolonga, 2020).
Yes
No ethics review needed.
1. Scalability in Real-World Applications: How does your approach handle scalability when dealing with significantly larger datasets? For example, have you considered scenarios involving resource constraints, such as limited memory or computational power? If so, any insights on computational overhead, or how these challenges might be mitigated, would be valuable.

2. Generalizability Across Architectures and Domains: The empirical evaluation currently focuses on specific architectures and datasets. Can you comment on how generalizable the method might be for other architectures or domains? For example, would adjustments be needed for application in NLP or reinforcement learning, where data structures and model requirements differ?

3. Handling Noisy and Imbalanced Datasets: How does the proposed method perform with noisy or imbalanced datasets? Many real-world datasets are affected by such issues, and any insights into how your method might handle these scenarios would be useful, along with potential limitations or planned future work to address these cases.

4. Practical Implications of the Method: In practical applications, how would you suggest users balance between data selection efficiency and computational costs? Given the importance of efficiency in large-scale training, any practical guidelines for tuning the method to optimize this balance would enhance the usability of the approach.
1. Originality: This paper presents a well-defined approach to data selection for machine learning, addressing the under-explored issue of efficiently selecting training samples that can maximize model performance. This focus is both timely and impactful, as data-centric methodologies are gaining traction, particularly within the scope of resource-constrained learning environments.

2. Quality: The proposed method is thoroughly evaluated with multiple datasets and model configurations. The authors apply rigorous metrics and conduct comparative studies that reflect the effectiveness and robustness of their approach. The experimentation section is comprehensive, reinforcing the paper’s claims with evidence and displaying attention to experimental rigor.

3. Clarity: The paper is generally well-structured and clear in its articulation of goals, methodology, and findings. Methodological descriptions are sufficiently detailed, allowing readers to follow the step-by-step approach, particularly in defining and explaining the selection criteria for informative data points.

4. Significance: The contributions of this paper are significant to the machine learning and data selection community. By introducing a more nuanced approach to sample selection, this work not only adds value to the theoretical understanding of informative data selection but also offers practical implications for model training and efficiency.

In sum, this paper makes a meaningful contribution to data selection strategies in machine learning, combining innovative methodological elements with solid empirical evaluation and offering insights valuable to both researchers and practitioners.
Briey summarize the paper and its contributions. This is not the place to critique the paper;
the authors should generally agree with a well-written summary.

This paper introduces a new method called Informative Data Selection (IDS) designed to improve thorax disease classification by addressing challenges in Generative Data Augmentation (GDA). GDA uses generative models, such as diffusion models, to create synthetic training data that supplements real datasets, particularly useful in medical fields where high-quality annotated data is limited. However, synthetic data often introduces noise, which can degrade model performance. IDS tackles this by applying the Information Bottleneck (IB) principle to re-weight synthetic samples, emphasizing informative data to improve the classifier’s accuracy.
There are no major weaknesses. Equations and formulas in the article are not helpful in understanding the key ideas and contributions of the work.

Please refer to the questions in the next section.
@inproceedings{
anonymous2024training,
title={Training Over a Distribution of Hyperparameters for Enhanced Performance and Adaptability on Imbalanced Classification},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=UZQl0rbxj6},
note={under review}
}
Although binary classification is a well-studied problem, training reliable classifiers under severe class imbalance remains a challenge. Recent techniques mitigate the ill effects of imbalance on training by modifying the loss functions or optimization methods. We observe that different hyperparameter values on these loss functions perform better at different recall values. We propose to exploit this fact by training one model over a distribution of hyperparameter values--instead of a single value--via Loss Conditional Training (LCT). Experiments show that training over a distribution of hyperparameters not only approximates the performance of several models, but actually improves the overall performance of models on both CIFAR and real medical imaging applications such as melanoma and diabetic retinopathy detection. Furthermore, training models with LCT is more efficient because some hyperparameter tuning can be conducted after training to meet individual needs without needing to retrain from scratch. Code will be made available upon acceptance of this paper.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
class imbalance hyperparameter tuning ROC curves machine learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/948b4ff1321f3f3e547a2ad6ae62d1cd9a1f6676.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Training Over a Distribution of Hyperparameters for Enhanced Performance and Adaptability on Imbalanced Classification
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission