title: How to find why a RBM does not work correctly?
body: <p>I'm trying to implement a RBM and I'm testing it on MNIST dataset. However, it does not seems to converge. </p>
  
  <p>I've 28x28 visible units and 100 hidden units. I'm using mini-batches of size 50. For each epoch, I traverse the whole dataset. I've a learning rate of 0.01 and a momentum of 0.5. The weights are randomly generated based on a Gaussian distribution of mean 0.0 and stdev of 0.01. The visible and hidden biases are initialized to 0. I'm using a logistic sigmoid function as activation. </p>
  
  <p>After each epoch, I compute the average reconstruction error of all mini-batches, here are the errors I get: </p>
  
  <pre><code>epoch 0: Reconstruction error average: 0.0481795
  epoch 1: Reconstruction error average: 0.0350295
  epoch 2: Reconstruction error average: 0.0324191
  epoch 3: Reconstruction error average: 0.0309714
  epoch 4: Reconstruction error average: 0.0300068
  </code></pre>
  
  <p>I plotted the histograms of the weights to check (left to right: hiddens, weights, visibles. top: weights, bottom: updates):  </p>
  
  <p><strong>Histogram of the weights after epoch 3</strong>
  <a href="http://baptiste-wicht.com/static/finals/histogram_epoch_3.png" rel="nofollow noreferrer">Histogram of the weights after epoch 3 http://baptiste-wicht.com/static/finals/histogram_epoch_3.png</a></p>
  
  <p><strong>Histogram of the weights after epoch 4</strong>
  <a href="http://baptiste-wicht.com/static/finals/histogram_epoch_4.png" rel="nofollow noreferrer">Histogram of the weights after epoch 4 http://baptiste-wicht.com/static/finals/histogram_epoch_4.png</a></p>
  
  <p>but, except for the hidden biases that seem a bit weird, the remaining seems OK. </p>
  
  <p>I also tried to plot the hidden weights: </p>
  
  <p><strong>Weights after epoch 3</strong></p>
  
  <p><a href="http://baptiste-wicht.com/static/finals/hiddens_weights_epoch_3.png" rel="nofollow noreferrer">Weights after epoch 3 http://baptiste-wicht.com/static/finals/hiddens_weights_epoch_3.png</a></p>
  
  <p><strong>Weights after epoch 4</strong></p>
  
  <p><a href="http://baptiste-wicht.com/static/finals/hiddens_weights_epoch_4.png" rel="nofollow noreferrer">Weights after epoch 4 http://baptiste-wicht.com/static/finals/hiddens_weights_epoch_4.png</a></p>
  
  <p>(they are plotted in two colors using that function: </p>
  
  <pre><code>static_cast&lt;size_t&gt;(value &gt; 0 ? (static_cast&lt;size_t&gt;(value * 255.0) &lt;&lt; 8) : (static_cast&lt;size_t&gt;(-value * 255.)0) &lt;&lt; 16) &lt;&lt; " ";
  </code></pre>
  
  <p>)</p>
  
  <p>And here, they do not make sense at all...</p>
  
  <p>If I go further, the reconstruction error falls a bit more, but do no go further than 0.025. Even if I change the momentum after sometime, it goes higher and then goes down a bit but not interestingly. Moreover, the weights do no make more sense after more epochs. In most example implementations I've seen, the weights were making some sense after iterating through the complete data set two or three times. </p>
  
  <p>I've also tried to reconstruct an image from the visible units, but the results seems almost random. </p>
  
  <p>What could I do to check what goes wrong in my implementation ? Should the weights be within some range ? Does something seems really strange in the data ?</p>
  
  <p>Complete code: <a href="https://github.com/wichtounet/dbn/blob/master/include/rbm.hpp" rel="nofollow noreferrer">https://github.com/wichtounet/dbn/blob/master/include/rbm.hpp</a></p>