Gridsearchcv vs Bayesian optimization
Grid search is known to be worse than random search for optimizing hyperparameters [1], both in theory and in practice. Never use grid search unless you are optimizing one parameter only.
On the other hand, Bayesian optimization is stated to outperform random search on various problems, also for optimizing hyperparameters [2]. However, this does not take into account several things: the generalization capabilities of models that use those hyperparameters, the effort to use Bayesian optimization compared to the much simpler random search, and the possibility to use random search in parallel.
So in conclusion, my recommendation is: never use grid search, use random search if you just want to try a few hyperparameters and can try them in parallel (or if you want the hyperparameters to generalize to different problems), and use Bayesian optimization if you want the best results and are willing to use a more advanced method.
[1] Random Search for Hyper-Parameter Optimization, Bergstra & Bengio 2012.
[2] Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020, Turner et al. 2021.
Which one among Gridsearchcv and Bayesian optimization works better for optimizing hyper parameters?