title: With neural networks, should the learning rate be in some way proportional to hidden layer sizes? Should they affect each other?
body: <p>My neural network is normal feed-forward and back prop. Has 10 outputs, which should be a vector where one of the output is 1, and the rest 0. So something like [0,0,0,0,1,0,0,0,0]. So an output I would expect is something like this:</p>
  
  <pre><code>[ 0.21332215,0.13782996,0.13548511,0.09321094,0.16769843,0.20333131, 0.06613014,0.10699013,0.10622562,0.09809167]
  </code></pre>
  
  <p>and ideally once trained, this:</p>
  
  <pre><code>[ 0.21332215,0.13782996,0.93548511 ,0.09321094 ,**0.9**676984,0.20333131, 0.06613014,0.1069901,0.10622562, 0.09809167]
  </code></pre>
  
  <p>When I have 30 neurons on the hidden layer, and a learning rate of > 0.1 but &lt; 1, i get these results. However, when i have 100 neurons on hidden, and have a learning rate of 0.01, i get results like this:</p>
  
  <pre><code>[  1.75289110e-05,1.16433042e-04 ,2.83848791e-01,4.47291309e-02, 1.63011592e-01,8.12974408e-05 , 1.06284533e-03 , 2.95174797e-02, 7.54112632e-05, 1.33177529e-03]
  </code></pre>
  
  <p>Why is this? Is this what over-learning looks like?</p>
  
  <p>Then, when I change the learning rate to 0.0001 with 100 neurons on hidden, it get normal results again.</p>
  
  <p>So my question is: how should the learning rate affect the hidden layer count? Should bigger hidden layers mean lower learning rates?</p>