Bounds for MLP hyperparameter search I'm trying to optimize a neural network architecture for a particular problem, but there just seems to be so many hyperparameters that I'm concerned that there are much better options that I'm not exploring (e.g. I might be getting trapped in a local minima for hyperparams).

Essentially, I'd like some standard bounds for hyperparameter search. Ideally, if a person were to see the breadth of explored parameters, they might be reasonably certain to try another class of machine learning models. In particular, I am looking for concrete advice on at least

What size neurons to include in the search


The number of hidden layers, and the width of the layers 
The
optimizer and learning rates 
Activation functions and loss functions


Thank you so much!
In designing an MLP architecture, we can restrict ourselves to 4-8 layers with 8-128 (power of 2) neurons per layer. In addition, we can assume recommended ReLU activations with He normal weight initialization and Adam or SGD with Nesterov momentum optimizers (see ipython notebook for comparison). Your loss function will depend on the problem: cross-entropy for classification and MSE for regression. 

In order to avoid local optima in parameter search, it's better to use random search or bayesian optimization. For additional answers, see this post.