title: In what order should we tune hyperparameters in Neural Networks?
body: <p>I have a quite simple ANN using Tensorflow and AdamOptimizer for a regression problem and I am now at the point to tune all the hyperparameters. </p>
  
  <p>For now, I saw many different hyperparameters that I have to tune : </p>
  
  <ul>
  <li>Learning rate : initial learning rate, learning rate decay</li>
  <li>The AdamOptimizer needs 4 arguments (learning-rate, beta1, beta2, epsilon) so we need to tune them - at least epsilon</li>
  <li>batch-size </li>
  <li>nb of iterations</li>
  <li>Lambda L2-regularization parameter</li>
  <li>Number of neurons, number of layers </li>
  <li>what kind of activation function for the hidden layers, for the output layer </li>
  <li>dropout parameter</li>
  </ul>
  
  <p>I have 2 questions :</p>
  
  <p>1) Do you see any other hyperparameter I might have forgotten ? </p>
  
  <p>2) For now, my tuning is quite "manual" and I am not sure I am not doing everything in a proper way. 
  Is there a special order to tune the parameters ? E.g learning rate first, then batch size, then ... 
  I am not sure that all these parameters are independent - in fact, I am quite sure that some of them are not. Which ones are clearly independent and which ones are clearly not independent ? Should we then tune them together ? 
  Is there any paper or article which talks about properly tuning all the parameters in a special order ? </p>
  
  <p>EDIT : 
  Here are the graphs I got for different initial learning rates, batch sizes and regularization parameters. The purple curve is completely weird for me... Because the cost decreases like way slowly that the others, but it got stuck at a lower accuracy rate. Is it possible that the model is stuck in a local minimum ? </p>
  
  <p><a href="http://i.stack.imgur.com/C0IcD.png" rel="noreferrer">Accuracy</a></p>
  
  <p><a href="http://i.stack.imgur.com/OId4A.png" rel="noreferrer">Cost</a></p>
  
  <p>For the learning rate, I used the decay : 
  LR(t) = LRI/sqrt(epoch) </p>
  
  <p>Thanks for your help !
  Paul </p>