ConBatch-BAL: Batch Bayesian Active Learning under Budget Constraints
Bayesian Active Learning;Real-World Datasets;Batch Acquisition;Budget Constraints
Yes
No ethics review needed.
$\bullet$ 
Can you consider any more metrics for $c(x),$ such as articular types of cost metrics that might be relevant for other real-world active learning scenarios beyond geospatial constraints?

$\bullet$
Can you provide any criteria or discussion for deciding max budget? Please let me know if you consider the specific factor for real datasets.
The paper proposed an algorithm addressing the budget-constrained dynamic querying algorithm, which can be applied to the examined datasets. The analytic results show the trade-off between budget and informativeness (measures by mutual information). The appropriate strategy for reducing the labeling costs and gaining a large information under budget constraints is established and shows a potential to be applied in the real-world problem.
This paper considers an active learning strategy with budget constraints. The proposed algorithm is based on dynamic thresholding and batch acquisition. Two real-world datasets are considered in experiments, with high cost and having geospatial constraints, under various budget and cost scenarios. The results show that the developed ConBatch-BAL strategies can reduce active learning iterations and data acquisition costs in real-world settings. The performance was compared concerning an infinite budget.  Therefore, the main study focuses on budge-constrained active learning with real-world two-image datasets.
$\bullet$
The main issue is concerning $c(\bf x)$. It seems that there is no general $c(\bf x)$ to be applied for various datasets such as tablet datasets, conventional image datasets, and so on.  In the paper, only geospatial constraints are related to $c(\bf x)$. It weakens the contribution of the paper. When the image datasets can be considered, can the variance of each pixel be $c(\bf x)$? At least, the more general criteria for the $c(\bf x)$ can be required.  

$\bullet$
Second, the range of experiments can be problematic on datasets. Combined with the first issue, there are few datasets to apply to the proposed algorithm. In active learning, there are datasets such as images, natural language, tablet datasets, and so on. The coverage of the proposed algorithm should not be all over the domains. However, the applicability or potential to other domains is important.
Yes
No ethics review needed.
No. See weakness parts.
Diverse datasets and interesting applications. This paper's innovations lie in introducing dynamic thresholding for addressing varying annotation costs problems in AL. This problem definition is clear and worth studying. The experiments datasets are diverse, as they are not limited to typical vision datasets (CIFAR10, CIFAR100, tinyimageNet) but use aerial images.
This work introduces two Bayesian active learning strategies, called ConBatch-BAL, designed for batch acquisition under budget constraints. One strategy uses dynamic thresholding to redistribute the budget across the batch, while the other follows a greedy approach to select top-ranked samples within the budget. Both strategies leverage uncertainty metrics computed by Bayesian neural networks.

The study focuses on scenarios with high annotation costs and geospatial constraints, releasing two new datasets of geolocated aerial building images annotated with energy efficiency or typology classes. ConBatch-BAL strategies are benchmarked against a random acquisition baseline and demonstrate improvements, reducing active learning iterations and data acquisition costs, while outperforming the baseline in real-world settings.
1. The authors shall provide more hyperparameter tuning/ablation study on Section 4.1 on how to choose dynamic thresholding.

The initial threshold, c_th,1 = c_max / n_max, is determined by the total budget divided by the maximum allowed batch steps. This choice is intuitive, but it may not adapt well to scenarios with highly varying annotation costs or where uncertainty scores vary significantly from step to step. Can you conduct  an ablation study on the impact of different n_max values could help understand how sensitive the method is to this parameter and provide insights into optimal settings for datasets listed in the papers?

2. The threshold is updated by subtracting the cost of the previously selected sample and dividing by the remaining steps. This is straightforward but may lead to inefficient sampling if a high-cost sample is selected early on, reducing the budget for remaining samples.

Can the authors provide theoretical insights or guarantees about the convergence or stability of this threshold adjustment process could work? For instance, under certain assumptions about cost distribution or sample uncertainty, it might be possible to guarantee that the threshold remains within certain bounds or converges to a stable value?

3. In real-world scenarios, annotation costs and uncertainty values can be highly variable. It’s unclear if the dynamic thresholding method can robustly handle such variability or if it might be affected by extreme values (outliers). Can the authors provide experiments on highly extreme values (outliers) results?
Yes
No ethics review needed.
Please review the weaknesses mentioned above.
The paper addresses the practical challenge of batch Bayesian active learning under constraints on both batch size and cost. The two real-world datasets introduced offer valuable resources for evaluating active learning solutions with geolocated points.
This paper tackles the challenge of batch Bayesian active learning under budget constraints by introducing two novel heuristics: dynamic thresholding and greedy strategies. It also presents two new real-world datasets featuring geolocated aerial images of buildings. Experimental results show that these strategies outperform a random selection baseline across both real-world datasets and a geolocated MNIST dataset.
The paper would benefit from a clearer and more detailed description of the method and an improvement of the technical contributions. 

1. While the objective is to achieve high accuracy on the test set, the test set itself has not been utilized in the the solution. If the goal is to improve test set accuracy, why not consider incorporating mutual information with the classification of points in the test set, similar to the approach in the well-known work by Andreas Krause, Near-Optimal Sensor Placements in Gaussian Processes (2008)? In that study, the test set can be treated as the entire design space.

2. The main technical contribution, the two active learning strategies, is explained too briefly and lacks essential details. For instance, the description omits the acquisition function selecting each point in the batch sequentially (before showing it in the pseudocode), as well as the expression of mutual information using Dropout samples. Lines 185–188 raise several challenges, but there is no clear explanation of how these issues are addressed. Additionally, there are notation inconsistencies in Algorithms 1 and 2:

    + Algorithm 1: $\\mathbf{x}\_1^*, \\dots, \\mathbf{x}\_n^*$ and $n$ are used (lines 222, 231) before they are defined. Line 222 could be revised to "return $A\_{i-1}$" and Line 231 to "return $A\_{n\_{\\max}}$". Similar issues exist in Algorithm 2.

    + Line 225: argmax can be revised to argmax_x.

    + $c(x)$ depends on the previous selected point so the input to $c$ should include the previous selected point.

    + What is the computational complexity of the approach?

3. The heuristic employed is relatively straightforward: the algorithm divides the budget constraint by the batch size or aims to fully utilize the available budget with each sampled point.

4. The experiments have several weaknesses:

+ Why not plot accuracy over iterations, instead of only showing the number of iterations required to reach selected accuracy levels?
+ The experiment includes only a simple baseline (random selection), while other batch Bayesian active learning approaches could also be evaluated under an unlimited budget. For a finite budget, some selected points could simply be dropped from the batch to stay within budget.
+ There is no clear description of the training set, test set, initial training set (before active learning), or the number of Dropout samples.
+ The impact of batch size on algorithm performance is not explored.

Additional notation issues:
  
+ $y\_1,\\dots,y\_n$ is used to denote the outputs of $\\mathbf{x}\_1, \\dots, \\mathbf{x}\_n$, but in lines $172$ $y\_{c\_1}, \\dots, y\_{c\_k}$ refer to the $k$ classes. 
  
+ Line 318 states that the "cost is inversely proportional to the number of buildings," yet it also notes that "crowded areas are more expensive," which seems contradictory.
Yes
No ethics review needed.
It appears that the two strategies, which are the main contributions of this paper, can be easily extended to classical active learning algorithms, such as CoreSet or BADGE. Even in a Bayesian setting, average embeddings can still be used for active learning with CoreSet or BADGE. I believe the main technical challenge lies in addressing the combinatorial cost. The authors should avoid restricting their approach to Bayesian Neural Networks (BNNs) without offering additional benefits beyond BNN model accuracy. I am open to discussing this further in the future.
The authors clearly presented the two heuristics.
This paper introduces two Bayesian active learning heuristics—greedy acquisition and dynamic thresholding—for batch acquisition under budget or cost constraints. The authors conducted experiments on one artificial dataset derived from MNIST and two real-world datasets, demonstrating improved performance.
(1) Vague problem setting: The authors do not provide a clear objective or constraints in Section 3.1. For instance, when considering the examples in Figure 2(b), the cost of a batch depends on the order of the batches, which implies that in Equation 2, the constraints should be $c(x_1, ..., x_n) \leq c_{max}, n \leq n_{max}$. The combinatorial nature of the cost is overlooked in the problem setting.

(2) Limited applications: The proposed methods are applied to only two real-world datasets, both of which are aerial imagery, limiting the scope of the applications.

(3) Limited Bayesian Neural Network (BNN) models tested: The experiments focus solely on MC-Dropout, but other BNN models with superior performance—such as SGHMC [1], SG-MCMC [2], and cSG-MCMC [3]—should also be evaluated.


references:

[1] Chen, Tianqi, Emily Fox, and Carlos Guestrin. "Stochastic gradient hamiltonian monte carlo." In International conference on machine learning, pp. 1683-1691. PMLR, 2014.

[2] Welling, Max, and Yee W. Teh. "Bayesian learning via stochastic gradient Langevin dynamics." In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688. 2011.

[3] Zhang, Ruqi, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. "Cyclical stochastic gradient MCMC for Bayesian deep learning." arXiv preprint arXiv:1902.03932 (2019).

Minor suggestions:

(1) In line 134, use ${x_1, ..., x_n} \subset D_{pool}$.

(2) Ensure consistency in the use of "Batch-BALD" or "batch-BALD" (see line 149 vs. line 346).

(3) There is no need to use both "return" and "output" with the same content simultaneously in Algorithm 1 and Algorithm 2.
We introduce two Bayesian active learning strategies for batch acquisition under constraints and show that they can reduce the number of active learning iterations and costs required in real-world settings.
@inproceedings{
anonymous2024conbatchbal,
title={ConBatch-{BAL}: Batch Bayesian Active Learning under Budget Constraints},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=mmpVsDjVgn},
note={under review}
}
Varying annotation costs among data points and budget constraints can hinder the adoption of active learning strategies in real-world applications. This work introduces two Bayesian active learning strategies for batch acquisition under constraints (ConBatch-BAL), one based on dynamic thresholding and one following greedy acquisition. Both select samples using uncertainty metrics computed via Bayesian neural networks. The dynamic thresholding strategy redistributes the budget across the batch, while the greedy one selects the top-ranked sample at each step, limited by the remaining budget. Focusing on scenarios with costly data annotation and geospatial constraints, we also release two new real-world datasets containing geolocated aerial images of buildings, annotated with energy efficiency or typology classes. The ConBatch-BAL strategies are benchmarked against a random acquisition baseline on these datasets under various budget and cost scenarios. The results show that the developed ConBatch-BAL strategies can reduce active learning iterations and data acquisition costs in real-world settings, and even outperform the unconstrained baseline solutions.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Bayesian Active Learning Real-World Datasets Batch Acquisition Budget Constraints
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/afa95981e41d93f05eeace517c2625adf065b8bd.pdf
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/fabde8728f265c863a2011d2197744d957120d4d.zip
ConBatch-BAL: Batch Bayesian Active Learning under Budget Constraints
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission