title: Why the loss of Variational Autoencoder in many implementations have opposite sign from paper?
body: <p>I think I understand the paper of Auto-Encoding Variational Bayes. And I am reading some tensorflow codes implementing this paper. But I don't understand their loss function in those codes. Since lots of codes are written in same way, probably I am wrong. </p>
  
  <p>The problem is like this. The following equation is from <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer">AEVB paper</a>. 
  <a href="https://i.stack.imgur.com/uiVpp.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uiVpp.png" alt="enter image description here"></a>
  The loss function is like this equation. This equation can be divided into two: Regularization term and Reconstruction term. Therefore, it becomes</p>
  
  <pre><code>Loss_function = Regularization_term + Reconstruction_term
  </code></pre>
  
  <p>However, lots of codes implement this Regularization term in a negative sign, like</p>
  
  <pre><code>Loss_function = -Regularization_term + Reconstruction_term 
  </code></pre>
  
  <p>For example, in <a href="https://github.com/tegg89/VAE-Tensorflow/blob/master/model.py" rel="nofollow noreferrer">this code</a>, 79th line shows Regularization term as</p>
  
  <pre><code>KLD = -.5 * tf.reduce_sum(1. + enc_logsd - tf.pow(enc_mu, 2) - tf.exp(enc_logsd), reduction_indices=1)
  </code></pre>
  
  <p>And then, it just adds to Reconstruction term. </p>
  
  <pre><code>loss = tf.reduce_mean(KLD + BCE)
  </code></pre>
  
  <p>I don't understand. The sign of KLD is opposite from equation from the paper. There are lots of codes like this. I think I am wrong but I don't know what is wrong. Can you explain why it should be like this?</p>
  
  <p>Reference codes: <a href="https://github.com/ikostrikov/TensorFlow-VAE-GAN-DRAW/blob/master/vae.py" rel="nofollow noreferrer">code1</a>, <a href="https://github.com/hwalsuklee/tensorflow-mnist-VAE/blob/master/vae.py" rel="nofollow noreferrer">code2</a>, <a href="https://jmetzen.github.io/2015-11-27/vae.html" rel="nofollow noreferrer">code3</a>    </p>