title: How to disable momentum in specific layer in tensorflow?
body: <p>I have a neural network with three hidden layers, which can be trained with "pure" gradient descent or with some more sophisticated techniques. I also noticed that in the my problem momentum-based optimization methods (adam, adadelta, momentum) works much better.</p>
  
  <p>Now to the interesting part. By design, I want to disable momentum in the first layer of the NN. That means, I want to update weights with Adam in second and third layers, but use simple gradient descent in first layer.</p>
  
  <p>Of course, I can always write my own optimizer: calculate gradients with <code>tf.gradients(loss, tf.trainable_variables())</code> and then just do the momentum trick myself. But it would be nice to have option to use special optimizer parameters in every layer. Have anybody heard about the way to do such thing?</p>