title: GPflow classification: interpretation of posterior variance
body: <p>In the <a href="https://gpflow.readthedocs.io/en/latest/notebooks/multiclass.html" rel="nofollow noreferrer">tutorial on multiclass classification on the GPflow website</a>, a Sparse Variational Gaussian Process (SVGP) is used on a 1D toy example. As is the case for <a href="https://gpflow.readthedocs.io/en/master/model_options.html" rel="nofollow noreferrer">all other GPflow models</a>, the SVGP model has a method <code>predict_y(self, Xnew)</code> which returns the mean and variance of held-out data at the points <code>Xnew</code>. </p>
  
  <p>From the tutorial it is clear that the first argument that is unpacked from <code>predict_y</code> is the posterior predictive probability of each of the three classes (cells <code>[7]</code> and <code>[8]</code>), shown as the colored lines in the second panel of the plot below. However, the authors do not elaborate on the second argument that can be unpacked from <code>predict_y</code>, which are the variances of the predictions. In a regression setting its interpretation is clear to me, as the posterior predictive distribution in this case would be a Gaussian. </p>
  
  <hr>
  
  <p><em>But I fail to understand what could be the interpretation here. Especially, I would like to know how this measure could be used to construct error bars denoting uncertainty around class predictions for any new data point.</em></p>
  
  <hr>
  
  <p>I altered the code of the tutorial slightly to add an additional panel to the plot below: the third panel shows in black the maximal standard deviation (square root of the obtained variance from <code>predict_y</code>). It clearly is a good measure for uncertainty and it is probably also no coincidence that the highest possible value is 0.5, but I could not find how it is calculated and what it represents.</p>
  
  <p><a href="https://i.stack.imgur.com/pUhol.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pUhol.png" alt="Black line: highest variance of three"></a></p>
  
  <p>Complete notebook with all code <a href="https://gpflow.readthedocs.io/en/latest/notebooks/multiclass.html" rel="nofollow noreferrer">here</a>.</p>
  
  <pre><code>def plot(m):
  f = plt.figure(figsize=(12,8))
  a1 = f.add_axes([0.05, 0.05, 0.9, 0.5])
  av = f.add_axes([0.05, 0.6, 0.9, 0.1])
  a2 = f.add_axes([0.05, 0.75, 0.9, 0.1])
  a3 = f.add_axes([0.05, 0.9, 0.9, 0.1])
  
  
  xx = np.linspace(m.X.read_value().min()-0.3, m.X.read_value().max()+0.3, 200).reshape(-1,1)
  mu, var = m.predict_f(xx)
  mu, var = mu.copy(), var.copy()
  p, v = m.predict_y(xx)
  
  a3.set_xticks([])
  a3.set_yticks([])
  
  av.set_xticks([])
  
  lty = ['-', '--', ':']
  
  for i in range(m.likelihood.num_classes):
      x = m.X.read_value()[m.Y.read_value().flatten()==i]
      points, = a3.plot(x, x*0, '.')
      color=points.get_color()
      a1.fill_between(xx[:,0], mu[:,i] + 2*np.sqrt(var[:,i]), mu[:,i] - 2*np.sqrt(var[:,i]), alpha = 0.2)
      a1.plot(xx, mu[:,i], color=color, lw=2)
  
      a2.plot(xx, p[:,i], '-', color=color, lw=2)
  
  av.plot(xx, np.sqrt(np.max(v[:,:], axis = 1)), c = "black", lw=2)
  
  for ax in [a1, av, a2, a3]:
    ax.set_xlim(xx.min(), xx.max())
  
  a2.set_ylim(-0.1, 1.1)
  a2.set_yticks([0, 1])
  a2.set_xticks([])
  
  plot(m)
  </code></pre>