In summary, we provide a comprehensive discussion and benchmark of Bayesian optimization strategies to deal with the presence of a priori unknown parameter space constraints, a critical aspect of self-driving laboratory research. We discuss and benchmark several existing approaches and provide novel techniques in this regard, manifested as feasibility aware acquisition function which balance sampling of promising parameters with avoidance of parameters predicted to be infeasible. A variational Gaussian process classifier is used to efficiently learn the constraint function on-the-fly. All feasibilityaware acquisition functions are implemented in the Atlas BO library for facile use by the community in their self-driving laboratories. Extensive benchmarks are reported for both analytic optimization functions and on two real-world chemical design/discovery applications. We discuss best practices for choosing feasibility-aware acquisition functions in the context of SDL applications, and encourage researchers to strongly consider constraint violation inconvenience within their SDL setup before choosing a strategy. This work lays the foundation for future studies on Bayesian optimization in the presence of a priori unknown constraints and increases the practicality of available optimization software for autonomous science applications.