title: What is the best way to implement weight constraints in TensorFlow?
body: <p>Suppose we have weights</p>
  
  <pre><code>x = tf.Variable(np.random.random((5,10)))
  cost = ...
  </code></pre>
  
  <p>And we use the GD optimizer:</p>
  
  <pre><code>upds = tf.train.GradientDescentOptimizer(lr).minimize(cost)
  session.run(upds)
  </code></pre>
  
  <p>How can we implement for example non-negativity on weights?</p>
  
  <p>I tried clipping them:</p>
  
  <pre><code>upds = tf.train.GradientDescentOptimizer(lr).minimize(cost)
  session.run(upds)
  session.run(tf.assign(x, tf.clip_by_value(x, 0, np.infty)))
  </code></pre>
  
  <p>But this slows down my training by a factor of 50.</p>
  
  <p>Does anybody know a good way to implement such constraints on the weights in TensorFlow?</p>
  
  <p>P.S.: in the equivalent Theano algorithm, I had</p>
  
  <pre><code>T.clip(x, 0, np.infty)
  </code></pre>
  
  <p>and it ran smoothly.</p>