# What are the advantages of contrastive divergence vs the gradient of the quadratic difference between the original data and the reconstructed data?

In this example I have a RBM with a visible layer and a hidden layer. The original data is "data", the values of the hidden neurons is "hid", the values of the visible neurons calculated from "hid" is "vis", and then the value for the hidden neurons calculated from "vis"is "hid2".

I calculate the positive_association and the negative_association by:

pos_associations = T.dot(data.T, hid)
neg_associations = T.dot(vis.T, hid2)


where T.dot represents the usual matrix calculation. I then update the weights using contrastive divergence (CD)

w = w + (training_rate)*(pos_associations - neg_associations)


like described in this post

I do get a good reconstruction (vis looks like data) and I can see features representing the weights. However, it takes a while and many epochs.

If instead of using CD I simply calculate the quadratic differnce between data and vis

cost = T.sum((data - vis)**2).mean()


and I update the weights using the gradient of cost, I get a faster approximation of the data. So my question is, what is the advantage of using CD?

It is natural that RBM requires some time to converge, but if you see that the reconstruction is decent, then your algorithm works well.

I don't understand the idea of optimizing the model by your objective function: cost = T.sum((data - vis)**2).mean(). I think you just cannot train RBM (or any other stochastic model) that way because vis is a nonlinear function of the data. If you omit sampling of the hidden vector you might use this formula and end up having model similar to the feed forward network that learns an identity transformation. It fact it can be used as a feature extractor.

Also notice that your second network no longer will be a proper generative model, because its learning rule doesn't follow the gradient of RBM's cost function. Hence it won't properly model either an energy function or a joint probability P(v, h) of your visible and hidden vector.

You may also be interested in reading this:

Although it is convenient, the reconstruction error is actually a very poor measure of the progress of learning. It is not the function that CD learning is approximately optimizing, [...] and it systematically confounds two different quantities that are changing during the learning. The frst is the difference between the empirical distribution of the training data and the equilibrium distribution of the RBM. The second is the mixing rate of the alternating Gibbs Markov chain. If the mixing rate is very low, the reconstruction error will be very small even when the distributions of the data and the model are very different. As the weights increase the mixing rate falls, so decreases in reconstruction error do not necessarily mean that the model is improving and, conversely, small increases do not necessarily mean the model is getting worse.

So what is the cost function that RBM tries to optimize? good description can be found here $$-\frac{\partial log(p(x))}{\partial \theta }=\frac{\partial F(x)}{\partial \theta}-\sum_{\tilde{x}}p(\tilde{x})\frac{\partial F(\tilde{x})}{\partial \theta}$$ On the left hand side we have minus log probability of the data which we must minimize in order to increase the probability of the data. On the right hand side we have two terms. The function $F(x)$ coresponds to the free energy of the network given the data $x$ and current weights $\theta$ so minimization of this function increases the probability of our data. On the other hand we must take into account the normalizing factor $Z$- its role is to give correct probability distribution of the input vectors and this sum in the equation corresponds to the normalizing factor. In other words we must maximize the free energy (decrease probability) over all set of inputs proportionally to its current probability $p(\tilde{x})$. If we omit to maximize this sum our model won't converge to the optimal solution. Contrastive Divergence is only crude approximation of this objective funtion but it works very well in practice. I recommend that you watch this film by Geoffrey Hinton

• Thank you @padura. So what is exactly the function optimized by the RBM? – user Jan 23 '16 at 9:48
• @user I edited my answer and changed my recommendation to this lecture: youtube.com/watch?v=AyzOUbkUf3M – padura Jan 23 '16 at 16:45

RBM maximizes probability of visible units $p(\mathbf{v})$, defined by the model, over all training data. It is equivalent to minimizing the KL-divergence between the model distribution and the (empirical) data distribution (more details here).

Maximum likelihood learning typically is performed by gradient descent. There are analytical expressions for the gradients of log-likelihood w.r.t. model weights, but they are intractable for regular-sized RBMs because of the exponential number of terms in the respective sums. Therefore the gradients themselves are approximated by MCMC-based algorithm called Contrastive Divergence (or its variants).
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 1, "mathjax_display_tex": 1, "mathjax_asciimath": 1, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "math_score": 0.8788125514984131, "perplexity": 549.466024501572}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "warc_path": "s3://commoncrawl/crawl-data/CC-MAIN-2020-45/segments/1603107882581.13/warc/CC-MAIN-20201024110118-20201024140118-00063.warc.gz"}