Is the search space of Hyperparameters Continuous or Discrete?
Continuous means only you have continuous variables. It can be convex or concave. It might not even be differentiable. Gradient descent only applies to differentiable convex problems (or convex approximations of concave problems).

Example:


learning rate of SGD is continuous hyper parameter.
NN optimizer (SGD, Adam, RMSprop) is discrete parameter


From my perspective: right now, rarely we have resources to perform cross validation on a number of deep models. Most of the hyper parameters are tuned using intuition, a bit of math and tuned in isolation.

If you can afford it:


grid search to initialize solutions
if your'e unhappy with the results:


run some hill climber on space, like simulated annealing
run simple GA, there'll be problems because of mixed space (some params are continuous, some are discrete), you'll need custom operators