What are the challenges of using gradient descent on hyper parameters λ and η to find out their optimum values? A question from chapter 3 of Michael Nielsen's [Neural Networks and Deep Learning]:


  It's tempting to use gradient descent to try to learn good values for hyper-parameters such as the regularization parameter $\lambda$ and the learning rate $\eta$. Can you think of an obstacle to using gradient descent to determine values of $\lambda$ and $\eta$? 




What are the challenges of using gradient descent on hyper parameters λ and  η to find out their optimum values?



C is the L2 regularized cost function.

n is the length of the training data

w is the weight

The Learning rule for the weights will be:





I am having trouble understanding how λ and η will affect the cost function.
I know that we should decrease η as we get closer to the Cost function minima.
I don't have a clue how to optimize λ.
I think that the values of the hyper parameters will change for every epoch so there can't be constant optimal values.
The cost function $C(w,\lambda)$ is not a function of learning rate $\eta$. You can't compute the gradient of $\eta$.

$\lambda$ is part of the cost function. You can indeed compute its gradient and apply SGD on $\lambda$. However, when you do that, $\lambda$ will keep decreasing because its optimal value is obviously $-\infty$.

Note that it is possible to optimize hyperparameters like $\eta$ and $\lambda$ with another neural net, but that is a different story.