title: Theano gradient calculation creates float64
body: <p>I have some standard NN code on Theano with two separate compiled functions.  One that calculates the cost and one that calculates the cost with AdaGrad updates.</p>
  
  <p>For GPU speed, I'm trying to keep everything <code>float32</code>.  Problem is that I'm getting a warning that the gradient calculation is creating a <code>float64</code>.  In particular, for the following line of code.</p>
  
  <pre><code>gradients = TT.grad(lossFn, self.params)
  paramUpdates = [(param, param - TT.inv(TT.sqrt(sumGrad)) * grad) for param, grad, sumGrad in zip(self.params, gradients, self.gradSums)]
  </code></pre>
  
  <p>If I comment out the gradient calculation and replace the second line with a placeholder, everything is fine.  Obviously this is a junk update but it helps pinpoint the problem.</p>
  
  <pre><code>paramUpdates = [(self.params[0], self.params[0])]
  </code></pre>
  
  <p>For reference, this is the loss function:</p>
  
  <pre><code>self.loss = TT.mean(TT.sqr(self.logisticLayer.yHat - TT.cast(self.inputs[1:-1, :], floatX)), acc_dtype=floatX, dtype=floatX)
  </code></pre>
  
  <p>The cost only compiled function is:</p>
  
  <pre><code>    self._calculateCost = theano.function(
          inputs=[self.inputs],
          outputs=self.loss,
      )
  </code></pre>
  
  <p>whereas the cost + update function is:</p>
  
  <pre><code>    self._calculateCostAndUpdate = theano.function(
          inputs=[self.inputs],
          outputs=self.loss,
          updates=updates
      )
  </code></pre>
  
  <p><strong>UPDATE</strong></p>
  
  <p>I pinned down the problem to a LSTM layer.  Here is minimal code</p>
  
  <pre><code>class TestLSTMLayer(object):
      def __init__(self, inputs, outputSize, dropout=0.9, inputSize=None, adagradInit=1, forgetGateBias=3, srng=None):
          self.h0 = theano.shared(np.random.randn(outputSize).astype(floatX))
  
          self.params = [self.h0]
  
          def _recurrence(hBelow):
              return hBelow
  
          hOutputs, _ = theano.scan(
              fn=_recurrence,
              outputs_info=self.h0,
              n_steps=inputs.shape[0]
          )
          self.hOutputs = hOutputs
  
      def getUpdates(self):
          gradients = TT.grad(TT.sum(self.hOutputs), self.params)
          paramUpdates = [(self.params[0], self.params[0])]
          return paramUpdates
  </code></pre>
  
  <p>The problem occurs when I call <code>getUpdates()</code> to get gradients (here it is a junk update for example).  I've found that commenting out <code>gradients = ...</code> or replacing the <code>scan()</code> with something that repeats <code>h0</code> to the required shape fixes things. </p>
  
  <p><strong>UPDATE 2</strong></p>
  
  <p>Debugging reveals a mess of <code>int64</code> in the <code>scan</code> function.</p>
  
  <pre><code> |Subtensor{int64::} [@B] &lt;TensorType(float32, matrix)&gt; ''
     |for{cpu,scan_fn} [@C] &lt;TensorType(float32, matrix)&gt; ''
     | |Subtensor{int64} [@D] &lt;TensorType(int64, scalar)&gt; ''
     | | |Shape [@E] &lt;TensorType(int64, vector)&gt; ''
     | | | |&lt;TensorType(float32, matrix)&gt; [@F] &lt;TensorType(float32, matrix)&gt;
     | | |Constant{0} [@G] &lt;int64&gt;
     | |IncSubtensor{Set;:int64:} [@H] &lt;TensorType(float32, matrix)&gt; ''
     |   |Alloc [@I] &lt;TensorType(float32, matrix)&gt; ''
     |   | |TensorConstant{0.0} [@J] &lt;TensorType(float32, scalar)&gt;
     |   | |Elemwise{add,no_inplace} [@K] &lt;TensorType(int64, scalar)&gt; ''
     |   | | |Subtensor{int64} [@D] &lt;TensorType(int64, scalar)&gt; ''
     |   | | |Subtensor{int64} [@L] &lt;TensorType(int64, scalar)&gt; ''
     |   | |   |Shape [@M] &lt;TensorType(int64, vector)&gt; ''
     |   | |   | |Rebroadcast{0} [@N] &lt;TensorType(float32, matrix)&gt; ''
     |   | |   |   |DimShuffle{x,0} [@O] &lt;TensorType(float32, row)&gt; ''
     |   | |   |     |&lt;TensorType(float32, vector)&gt; [@P] &lt;TensorType(float32, vector)&gt;
     |   | |   |Constant{0} [@Q] &lt;int64&gt;
     |   | |Subtensor{int64} [@R] &lt;TensorType(int64, scalar)&gt; ''
     |   |   |Shape [@S] &lt;TensorType(int64, vector)&gt; ''
     |   |   | |Rebroadcast{0} [@N] &lt;TensorType(float32, matrix)&gt; ''
     |   |   |Constant{1} [@T] &lt;int64&gt;
     |   |Rebroadcast{0} [@N] &lt;TensorType(float32, matrix)&gt; ''
     |   |ScalarFromTensor [@U] &lt;int64&gt; ''
     |     |Subtensor{int64} [@L] &lt;TensorType(int64, scalar)&gt; ''
     |Constant{1} [@V] &lt;int64&gt;
  </code></pre>