KL Divergence of Normal and Laplace isn't Implemented in TensorFlow Probability and PyTorch
In both TensorFlow Probability (v0.4.0) and PyTorch (v0.4.1) the KL Divergence of the Normal distribution (tfp, PyTorch) and the Laplace distribution (tfp, PyTorch) isn't implemented resulting in a NotImplementedError error being thrown.

&gt;&gt;&gt; import tensorflow as tf
&gt;&gt;&gt; import tensorflow_probability as tfp
&gt;&gt;&gt; tfd = tfp.distributions
&gt;&gt;&gt; import torch
&gt;&gt;&gt;
&gt;&gt;&gt; tf.__version__
'1.11.0'
&gt;&gt;&gt; tfp.__version__
'0.4.0'
&gt;&gt;&gt; torch.__version__
'0.4.1'
&gt;&gt;&gt; 
&gt;&gt;&gt; p = tfd.Normal(loc=0., scale=1.)
&gt;&gt;&gt; q = tfd.Laplace(loc=0., scale=1.)
&gt;&gt;&gt; tfd.kl_divergence(p, q)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/root/miniconda/envs/example/lib/python3.6/site-packages/tensorflow/python/ops/distributions/kullback_leibler.py", line 95, in kl_divergence
    % (type(distribution_a).__name__, type(distribution_b).__name__))
NotImplementedError: No KL(distribution_a || distribution_b) registered for distribution_a type Normal and distribution_b type Laplace
&gt;&gt;&gt; 
&gt;&gt;&gt; a = torch.distributions.normal.Normal(loc=0., scale=1.)
&gt;&gt;&gt; b = torch.distributions.laplace.Laplace(loc=0., scale=1.)
&gt;&gt;&gt; torch.distributions.kl.kl_divergence(a,b)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/root/miniconda/envs/example/lib/python3.6/site-packages/torch/distributions/kl.py", line 161, in kl_divergence
    raise NotImplementedError
NotImplementedError


I assume as this is missing from both of these libraries that there is some good reason for this and that the user would be expected to implement it themselves with tfp.distributions.RegisterKL in TensorFlow Probability and torch.distributions.kl.register_kl in PyTorch.

Is this the correct assumption? If so, can someone explain why the KL Divergence wouldn't be implemented for given distribution classes? I think I am missing something very basic about this.

If my assumption is wrong, can someone explain how to properly have TensorFlow and PyTorch implement these operations?

For additional reference, using for this example an older version of TensorFlow that works with Edward,

pip install tensorflow==1.7
pip install edward


In this minimal example above, I'm trying to implement the equivalent of the following edward toy example code in tfp (or in torch).

import tensorflow as tf
import edward as ed

p = ed.models.Normal(loc=0., scale=1.)
s = tf.Variable(1.)
q = ed.models.Laplace(loc=0., scale=s)
inference = ed.KLqp({p: q})
inference.run(n_iter=5000)


IIRC, Edward's KLqp switches tries to use the Analytic form, and if not switches to using the sample KL.

For TFP, and I think PyTorch, kl_divergence only works for distributions 
registered, and unlike Edward only computes the analytic KL. As you mention, these aren't implemented in TFP, and I would say that's more of because the common cases (such as KL(MultivariateNormal || MultivariateNormal) have been implemented.

To register the KL divergence, you would do something like: https://github.com/tensorflow/probability/blob/07878168731e0f6d3d0e7c878bdfd5780c16c8d4/tensorflow_probability/python/distributions/gamma.py#L275. (It would be great if you could file a PR at https://github.com/tensorflow/probability!). 

If it turns out that there isn't a suitable analytic form of this (off the top of my head, I don't know if there is one), then one can form the sample KL and do optimization with that. That can be done explicitly in TFP (by sampling and computing the sample KL. Also please file a PR if you would like this to be done more automatically as well. This is something some of us on TFP are interested in.

It would be interesting to see for what cases analytic KL's can be automated. For instance, if q and p come from the same exponential family, then there is a nice form for the KL divergence in terms of sufficient statistics and the normalizer. But for KL's that are across exponential families (or even not exponential families), I'm not aware of results on classes of distributions where you can calculate the KL within the class semi-automatically.
