title: Deconvolutional autoencoder in theano
body: <p>I'm new to theano and trying to use the examples <a href="http://deeplearning.net/tutorial/lenet.html" rel="nofollow">convolutional network</a> and <a href="http://deeplearning.net/tutorial/dA.html" rel="nofollow">denoising autoencoder</a> to make a denoising convolutional network. I am currently struggling with how to make W', the reverse weights. <a href="http://people.idsia.ch/~masci/papers/2011_icann.pdf" rel="nofollow">In this paper</a> they use tied weights for W' that are flipped in both dimensions.</p>
  
  <p>I'm currently working on a 1d signal, so my image shape is (batch_size, 1, 1, 1000) and filter/W size is (num_kernels, 1, 1, 10)  for example. The output of the convolution is then (batch_size, num_kernels, 1, 991).
  Since I want to W' to be just the flipped in 2 dimensions (or 1d in my case), I'm tempted to do this</p>
  
  <pre><code>w_value = numpy_rng.uniform(low=-W_bound, high=W_bound, size=filter_shape)
  self.W = theano.shared(np.asarray((w_value), dtype=theano.config.floatX), borrow=True)
  self.W_prime = T.repeat(self.W[:, :, :, ::-1], num_kernels, axis=1)
  </code></pre>
  
  <p>where I reverse flip it in the relevant dimension and repeat those weights so that they are the same dimension as the feature maps from the hidden layer.</p>
  
  <p>With this setup, do I only have to get the gradients for W to update or should W_prime also be a part of the grad computation?</p>
  
  <p>When I do it like this, the MSE drops a lot after the first minibatch and then stops changing. Using cross entropy gives NaN from the first iteration. I don't know if that is related to this issue or if it's one of many other potential bugs I have in my code.</p>