It is tempting to think that with a single model metric and state-of-the-art ML optimization software, it should be possible to form an automatic pipeline wherein simulation data are fed in, and a single optimized MSM describing the kinetics and thermodynamics of the system comes out. However, many detailed questions need to be answered before such a pipeline is possible. First, do variational scores refer to the same relaxation mode across all possible combinations of hyperparameters? It is possible that with certain combinations of hyperparameters, the eigenvectors could describe different relaxation modes. It is therefore possible that the variational scores do not compare the same set of processes across different sets of hyperparameters. Second, the MSM lag time and number of scored processes interest will affect the variational scores -does this have any material effect on how we rank different sets of hyperparameters? Third, do we need variational scores to optimize models at all? Will model observables, such as the implied timescales suffice to optimize MSMs? Finally, does hyperparameter optimization work for MSMs compared to randomly sampling hyperparameters? Here, we use a common method (Bayesian optimization with tree Parzen estimators) for optimizing machine learning models to find optimal hyperparameters. The remainder of this work is structured as follows. In the next section, we cover the necessary theory to understand MSMs and Bayesian optimization of hyperparameters; this is followed by the methods; and results analysis. The paper is concluded with some recommendations based on our findings.