Does this hyperparameter optimisation approach yield the optimal hyperparameters? Say I have a ML model which is not very costly to train. It has around say 5 hyperparameters. 

One way to select best hyperparameters would be to keep all the other hyperparamaters fixed and train the model by changing only one hyperparameter within a certain range. For the sake of mathematical convenience, we assume for the hyperparameter $h^1$, keeping all other hyperparameters fixed to their initial values, the model performs best when  $h^1_{low} &lt; h^1 &lt; h^1_{high}$ (which we found out by running the model on a huge range of $h^1$). Now we, fix $h^1$ to one of the best values and tune $h^2$ the same way, where $h^1$ is chosen and the rest of the hyperparameters are again fixed on their initial values.

My question is: Does this method find the best hyperparameter choices for the model? I know if the hyperparameters are independent, then this definitely does find the best solution, but in a general case, what is the general theory around this? (NOTE: I am not asking about the problem of choosing hyperparamaters, but I am asking about the aforementioned approach of choosing hyperparameters)
After you've computed $h^{1}_{optimal}$ the only thing you can be sure is that this is the best (assuming constrained case) value of $h^1$ (with respect to some model performance metric) given your initial values for $h^2, ..., h^n$. If you change a bit any of $h^2, ..., h^n$ you're no longer certain that the value $h^1$ you found is the optimal one. So yes, the key here is the assumption about the independence.