BALSA: Benchmarking Active Learning Strategies for Autonomous laboratories
active learning;experimental design;AI for science
Yes
No ethics review needed.
See Weaknesses.
* The authors provide a broad set of tasks, from synthetic to complex real-world scenarios in biology and materials science. The authors explain clear difference from traditional optimization benchmarks.
* The authors introduce the landscape flatness metric, which can quantify the complexity of the objective landscape.
* The authors provide detailed experiments with 11 baseline methods.
This paper introduces a benchmark suite for evaluating active learning strategies in autonomous laboratories. The work includes synthetic benchmark functions, two real-world tasks (protein design and electron microscopy), and introduces a new metric called landscape flatness to characterize objective functions. The authors evaluate 11 baseline methods on synthetic tasks and 4 methods on real-world applications.
* The authors do not provide enough validation for the proposed landscape flatness metric. It would be better to have theoretical evidence to support the robustness of this metric across different tasks.
* For experiments, the authors do not explain different numbers of trials (5 trials for synthetic tasks, 3 trials for real-world tasks). Results in Table 1 show high variance across trials, but the authors do not discuss this variation.
* The authors highlight the scalability as a key contribution, but the paper's analysis of this aspect is limited. For example, there is no quantitative analysis of how computation time scales with dimensionality. And the maximum dimension is 100D.
* Others:
    * It is hard to understand the figures (e.g., Fig. 2, 6) due to small size, unclear labeling and limited context.
    * The introduction contains redundant information about self-driving labs; Minor typo errors and inconsistencies are present in the paper.
Yes
n/a
No ethics review needed.
* In active learning for molecular optimization (regardless of protein or materials), one of the most crucial components we seek to optimize is sample efficiency; however, it was not clear to me how this is assessed in this paper, even though the authors say this is what they are trying to assess. For instance, none of the presented results touch on sample efficiency (and, if they do, this was not clear). Can this be clarified?
* The figure and table captions are not informative, and could be improved for clarity. For instance, what are the values that are being shown in Table 1, is higher/lower better, and what are the bounds? Same for Table 2. What the values are could be made clear in the captions, without needing to go dig through the text (and, I could not always find what it was that was being shown in the tables).
* It is a good ambition to try and develop a suite of benchmarks that aim to ensure reproducibility of algorithmic performance across a wide variety of synthetic and real-world tasks.
* I like the idea of having the six functions as part of the benchmark (Ackley, Rastrigin, Rosenbrock, Griewank, Schwefel, and Michalewicz), but overall I think the benchmark in its current state does not seem general or rigorous enough to qualify for acceptance at ICLR.
* The plots themselves are very nice.
BALSA is a paper introducing new benchmarks for active learning in autonomous labs, both in the context of protein design and materials design. However, I get the feeling that the authors have tried to do too much, and I found the paper to be hard to read and poorly organized, though the ideas are certainly interesting. I think the paper could benefit from significant restructuring, and possibly even first perfecting the benchmark for protein design only, then extending to materials (or vice-versa); in its current state, it feels imperfect for both, and I did not feel that the paper adequately reviewed either existing protein design benchmarks or existing materials design benchmarks, making it hard to understand what is the value added here by BALSA. Datasets and benchmarks is perhaps the hardest track of papers in which to get accepted, as authors have to make things simple yet rigorous (i.e., foolproof) and transparent for other researchers to use, and I do not think BALSA in its current state meets this criteria.
* Unfortunately, the anonimized repository does not link to anything, which is a shame because it means the benchmarks introduced here are not reproducible.
* Not sure if AF2 is the most relevant model for evaluating the protein design tasks, as we are generally trying to push models outside distribution when designing new proteins, which is precisely the scenarios in which AF2 would fail. Furthermore, the materials benchmark proposed herein would only apply to crystalline materials, limiting its applicability.
* It is not fully clear to me what is the novelty of the benchmarks introduced here, relative to previous benchmarks.
* The paper could benefit from a more thorough proofreading, including of the formatting which was strange in places throughout the text and distracting. For instance, the way references were inserted in the text made little sense to me and it as hard to understnad what part of the sentence was being referenced.
Yes
No ethics review needed.
None.
1. Laboratory automation is a field gaining attention and beginning to be explored across various disciplines; however, a lack of suitable benchmark problems makes it challenging to accurately assess the effectiveness of heuristic methods proposed from different fields. Efforts to establish benchmark problems for active learning in laboratory automation are, therefore, very important to address this issue.

2. The no-free lunch theorem implies that no single method excels universally across all optimization problems; methods should be chosen based on the specific type of problem. I fully support this author's view, as well as the approach of classifying and analyzing problem types according to the landscape smoothness of optimization problems.
This paper aims to provide benchmark problems for active learning algorithms in automatically selecting experimental conditions in a Self-Driving Laboratory (laboratory automation). Laboratory automation is a field gaining attention across various disciplines, and establishing benchmarks for its core methodology, active learning methods, is essential. In this paper, benchmark problems using six artificial functions and two simulators are examined, and experiments are conducted on eleven types of active learning methods.
1. I do not believe that the problem settings, datasets, and methods discussed in this paper are sufficient for evaluating algorithms for laboratory automation. First, it seems necessary to verify, in some way, whether benchmark functions such as Ackley and Rosenbrock sufficiently cover the class of optimization problems in laboratory automation. Since these benchmark functions were designed to measure the effectiveness of traditional nonlinear optimization algorithms, using them directly may not be appropriate.

2. Since this paper aims to provide benchmark problems, properly evaluating its originality for Top-tier conferences such as ICLR is difficult. However, for example, the metric presented as a novel measure for landscape flatness in Equation (2) is quite naive and not particularly new. A more original perspective tailored specifically to the issues in laboratory automation would strengthen this paper.
We introduce BALSA: a benchmark tailored to evaluate search algorithms in autonomous laboratories within the active learning framework.
@inproceedings{
anonymous2024balsa,
title={{BALSA}: Benchmarking Active Learning Strategies for Autonomous laboratories},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=PHkUNcno9n},
note={under review}
}
Accelerating the scientific discoveries holds significant potential to address some of the most pressing challenges facing society, from mitigating climate change to combating public health crises, such as the growing resistance to existing antibiotics. The vast and complex nature of design parameter spaces makes identifying promising candidates both time-consuming and resource-intensive, rendering conventional exhaustive searches impractical. However, recent advancements in data-driven methods, particularly within the framework of "active learning," have led to more efficient strategies for scientific discovery. By iteratively identifying and labeling the most informative data points, these methods operate within a closed loop, guiding experiments or simulations to accelerate the identification of optimal candidates while reducing the demand of data labeling. Despite this progress, the lack of standardized benchmarks in this emerging field of autonomous scientific discovery impedes progress and limits its potential translational impact. To address this, we introduce BALSA: a comprehensive benchmark specifically designed for evaluating various search algorithms applied in autonomous laboratories within the active learning framework. BALSA offers a standardized evaluation protocol, provides a metric to characterize high-dimensional objective functions, and includes reference implementations of recent methodologies, with a focus on minimizing the data required to reach optimal results. It provides not only a suite of synthetic functions, but also real-world active learning tasks in biology and materials science — each presenting unique challenges for autonomous laboratory tasks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
active learning experimental design AI for science
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e27542ae0dc1f1eabd95cff318a61ec1b45be948.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
BALSA: Benchmarking Active Learning Strategies for Autonomous laboratories
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission