title: Slowdown on tensorflow convolutional network with custom parameter update
body: <p>I'm trying to implement a custom parameter update on a convolutional network, but every mini batch executed gets slower and slower.</p>
  
  <p>I realize that there's no need to go through this trouble with a fixed learning rate, but I plan to update this later.</p>
  
  <p>I call this in a loop where the feed_dict is the mini_batch.</p>
  
  <pre><code>sess.run(layered_optimizer(cost,.1,1),feed_dict = feed_dict)   
  </code></pre>
  
  <p>where</p>
  
  <pre><code>def layered_optimizer(cost,base_rate, rate_multiplier):
  
      gradients = tf.gradients(cost, [*weights, *biases])
      print(gradients)
  
      #update parameters based on gradients: var = var - gradient * base_rate * multiplier
      for i in range(len(weights)-1):
          weights[i].assign(tf.subtract(weights[i], tf.multiply(gradients[i], base_rate * rate_multiplier)))
          biases[i].assign(tf.subtract(biases[i], tf.multiply(gradients[len(weights)+i], base_rate * rate_multiplier)))
  
  
  
      return(cost)
  </code></pre>
  
  <p>I'm not sure if this is has to do with the problem, but after trying to run the code a second time I get the following errors and have to restart.</p>
  
  <p>could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
  error retrieving driver version: Unimplemented: kernel reported driver version not implemented on Windows
  could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
  Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo(), &amp;algorithms) </p>