<a href="https://colab.research.google.com/github/mella30/Deep-Learning-with-Tensorflow-2/blob/main/Course3-Probabilistic_Deep_Learning_with_Tensorflow2/week1_distributions.ipynb" target="_parent"></a>


```python
import tensorflow as tf
import tensorflow_probability as tfp
tfd = tfp.distributions

print("TF version:", tf.__version__)
print("TFP version:", tfp.__version__)
```

    TF version: 2.6.0
    TFP version: 0.13.0



```python
s# Additional imports and setting fixed random seed to have reproducibility

import matplotlib.pyplot as plt
import numpy as np
tf.random.set_seed(123)
```

## Coding Tutorials
#### 1. [Univariate Distributions](#univariate_distributions)
#### 2. [Multivariate Distributions](#multivariate_distributions)
#### 3. [The Independent Distribution](#the_independent_distribution)
#### 4. [Sampling and log probs](#sampling_and_log_probs)
#### 5. [Trainable Distributions](#trainable_distributions)


```python
# Create a normal distribution from Tensorflow Distributions

normal = tfd.Normal(loc=0, scale=1)
normal
```




    <tfp.distributions.Normal 'Normal' batch_shape=[] event_shape=[] dtype=float32>




```python
# Sample from the chosen distribution...

normal.sample()
```




    <tf.Tensor: shape=(), dtype=float32, numpy=-0.8980837>




```python
# ... or sample multiple times

normal.sample(10)
```




    <tf.Tensor: shape=(10,), dtype=float32, numpy=
    array([ 0.33875433,  0.3449861 , -0.6605785 , -0.2854994 ,  0.43852386,
            0.8288566 , -0.53591555, -0.5353483 , -1.0324248 , -2.942705  ],
          dtype=float32)>




```python
# Obtain value of probability's density

normal.prob(0)
```




    <tf.Tensor: shape=(), dtype=float32, numpy=0.3989423>




```python
# Obtain value of logprobability

normal.log_prob(0)
```




    <tf.Tensor: shape=(), dtype=float32, numpy=-0.9189385>




```python
# Verify that this really is the log of the probability

np.log(normal.prob(0))
```




    -0.9189385




```python
# Plot a histogram, approximating the density

plt.hist(normal.sample(10000), bins=50, density=True)
plt.show()
```


```python
# Do the same for the exponential distribution

exponential = tfd.Exponential(rate=1)
plt.hist(exponential.sample(10000), bins=50, density=True)
plt.show()
```


```python
# Sample as before

exponential.sample(10)
```




    <tf.Tensor: shape=(10,), dtype=float32, numpy=
    array([0.31796157, 0.9595855 , 0.6190708 , 1.3738598 , 1.6796894 ,
           0.40142855, 1.6830153 , 1.78942   , 0.38126466, 0.55283946],
          dtype=float32)>




```python
# Create a Bernoulli distribution (discrete)

bernoulli = tfd.Bernoulli(probs=0.8)
bernoulli.sample(20)
```




    <tf.Tensor: shape=(20,), dtype=int32, numpy=
    array([0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1],
          dtype=int32)>



#### A word of caution on discrete distributions

TensorFlow will evaluate the probability mass function of a Bernoulli at points which are not zero or one. This can get some unexpected results if you're not careful. To show this, let's pass in four points to `.prob`, two of which 0.5 and minus one will give odd-looking results as a probability mass function is only defined at zero and one. To find out what's going on, let's take a closer look at how the probability mass function is evaluated. We'll make our own function called `my_bernoulli`. What we're doing here is evaluating a Bernoulli by taking the probability of the outcome being equal to one to the power
of k and multiplying that by the probability of the outcome being equal to zero to the power of one minus k. If we pass in the same values into our function,
we see that the results are the same. 


```python
# Calculate Bernoulli prob and see that 0.5 and -1 do not give the correct probability!

for k in [0,0.5,1,-1]:
    print('prob result {} for k = {} '.format(bernoulli.prob(k), k))
```

    prob result 0.19999998807907104 for k = 0 
    prob result 0.3999999761581421 for k = 0.5 
    prob result 0.800000011920929 for k = 1 
    prob result 0.04999999701976776 for k = -1 



```python
# Replicate the scores to see what is occurring under the hood

def my_bernoulli(p_success, k):
    return np.power(p_success,k)*np.power((1-p_success),(1-k))
```


```python
# Evaluate it as before

for k in [0,0.5,1,-1]:
    print('prob result {} for k = {} '.format(bernoulli.prob(k), k))
```

    prob result 0.19999998807907104 for k = 0 
    prob result 0.3999999761581421 for k = 0.5 
    prob result 0.800000011920929 for k = 1 
    prob result 0.04999999701976776 for k = -1 


#### Work with batch distributions


```python
# Create a batched Bernoulli distribution

bernoulli_batch = tfd.Bernoulli(probs=[0.1, 0.25, 0.5, 0.75, 0.9])
bernoulli_batch
```




    <tfp.distributions.Bernoulli 'Bernoulli' batch_shape=[5] event_shape=[] dtype=int32>




```python
# Sample from it, noting the shape (5x5)

bernoulli_batch.sample(5)
```




    <tf.Tensor: shape=(5, 5), dtype=int32, numpy=
    array([[0, 0, 0, 1, 1],
           [0, 1, 0, 1, 1],
           [0, 0, 1, 1, 1],
           [0, 0, 0, 1, 1],
           [0, 0, 0, 1, 1]], dtype=int32)>




```python
# Use a batch shape with higher rank

probs = [[[0.5, 0.5], 
          [0.8, 0.3], 
          [0.25, 0.75]]]
bernoulli_batch_2D = tfd.Bernoulli(probs=probs)
bernoulli_batch_2D
```




    <tfp.distributions.Bernoulli 'Bernoulli' batch_shape=[1, 3, 2] event_shape=[] dtype=int32>




```python
# Sample from this batch of distributions

bernoulli_batch_2D.sample(5)
```




    <tf.Tensor: shape=(5, 1, 3, 2), dtype=int32, numpy=
    array([[[[0, 0],
             [1, 1],
             [0, 1]]],
    
    
           [[[0, 0],
             [1, 0],
             [0, 1]]],
    
    
           [[[1, 0],
             [1, 1],
             [0, 1]]],
    
    
           [[[1, 0],
             [0, 0],
             [0, 1]]],
    
    
           [[[0, 1],
             [1, 1],
             [1, 0]]]], dtype=int32)>




```python
# Determine probabilities from this batch distribution

bernoulli_batch_2D.prob([[[1, 0], 
                         [0, 0], 
                         [1, 1]]])
```




    <tf.Tensor: shape=(1, 3, 2), dtype=float32, numpy=
    array([[[0.5       , 0.5       ],
            [0.19999999, 0.7       ],
            [0.25      , 0.75      ]]], dtype=float32)>



***
<a id='multivariate_distributions'></a>
## Multivariate Distributions


#### Basic multivariate distributions


```python
# Define 2D multivariate Gaussian with diagonal covariance matrix

normal_diag = tfd.MultivariateNormalDiag(loc=[0,1], scale_diag=[1,2])
normal_diag
```




    <tfp.distributions.MultivariateNormalDiag 'MultivariateNormalDiag' batch_shape=[] event_shape=[2] dtype=float32>




```python
# Sample from it
# shape is nr of samples x number of random variables (event shape)

normal_diag.sample(10)
```




    <tf.Tensor: shape=(10, 2), dtype=float32, numpy=
    array([[-0.48958328,  2.889566  ],
           [-0.50546694,  0.9930791 ],
           [ 1.300877  ,  1.770391  ],
           [-2.9889584 ,  2.2526274 ],
           [-0.8891398 ,  1.9469274 ],
           [-1.5844265 , -1.2503903 ],
           [-1.0179803 ,  1.302659  ],
           [-0.09586073,  1.3923955 ],
           [-0.3122441 ,  2.9065297 ],
           [ 0.39839435,  0.40829754]], dtype=float32)>




```python
# Make a plot

plt_sample = normal_diag.sample(10000)
plt.scatter(plt_sample[:, 0], plt_sample[:, 1], marker='.', alpha=0.05)
plt.axis('equal')
plt.show()
```

#### Batches of multivariate distributions


```python
# Create three "batches" of multivariate normals

normal_diag_batch = tfd.MultivariateNormalDiag(loc=[[0,0], [0,0], [0,0]],
                                               scale_diag=[[1,2], [2,1], [2,2]])
normal_diag_batch
```




    <tfp.distributions.MultivariateNormalDiag 'MultivariateNormalDiag' batch_shape=[3] event_shape=[2] dtype=float32>




```python
# Sample from it
# shape is number of samples x batch size x event shape (number of random variables)

samples = normal_diag_batch.sample(5)
samples
```




    <tf.Tensor: shape=(5, 3, 2), dtype=float32, numpy=
    array([[[ 0.29961362,  1.1648543 ],
            [ 0.9561108 , -0.05747294],
            [-1.854802  ,  2.1274722 ]],
    
           [[ 0.1547592 , -1.3092546 ],
            [-1.1497984 , -0.03509409],
            [-3.043224  , -1.449043  ]],
    
           [[-1.300846  , -1.8879085 ],
            [ 1.4802053 , -0.5850459 ],
            [-1.841367  , -5.250524  ]],
    
           [[-0.9080883 ,  0.5145316 ],
            [-0.0360918 ,  1.5906875 ],
            [ 1.4850668 ,  1.9667028 ]],
    
           [[-0.6103264 ,  1.2452672 ],
            [-0.18394378,  1.2608575 ],
            [-1.1042159 ,  4.270896  ]]], dtype=float32)>




```python
# Compute log probs
# shape is nr of samples x batch size 

normal_diag_batch.log_prob(samples)
```




    <tf.Tensor: shape=(5, 3), dtype=float32, numpy=
    array([[-2.7455192, -2.6469443, -4.219975 ],
           [-2.757268 , -2.6968946, -4.6442885],
           [-3.8226492, -2.9760394, -7.094001 ],
           [-2.9764292, -3.7963305, -3.9833393],
           [-2.9111097, -3.3301344, -5.656652 ]], dtype=float32)>




```python
# Create a sample for a plot -- notice the shape
# shape is number of samples x batch size x event shape

plt_sample_batch = normal_diag_batch.sample(1000)
plt_sample_batch.shape

```




    TensorShape([1000, 3, 2])




```python
# Plot samples from the batched multivariate Gaussian

fig, axs = (plt.subplots(1, 3, sharex=True, sharey=True, figsize=(10, 3)))
titles = ['cov_diag=[1, 2]','cov_diag=[2, 1]', 'cov_diag=[2, 2]']

for i, (ax, title) in enumerate(zip(axs,titles)):
    samples = plt_sample_batch[:,i,:] #take the ith batch [samples x event_shape]
    ax.scatter(samples[:, 0], samples[:, 1], marker='.', alpha=0.05)
    ax.set_title(title)
plt.show()
```

***
<a id='the_independent_distribution'></a>
## The Independent Distribution


```python
# Start by defining a batch of two univariate Gaussians, 
# then combine them into a bivariate Gaussian with independent components 
# (the Independent class allows us to turn batches of univariate distributions into multivariate one)

locs = [-1., 1]  # means
scales = [0.5, 1]  # covs
batch_of_normals = tfd.Normal(loc=locs, scale=scales)
```


```python
# Univariate density functions

import seaborn as sns

t = np.linspace(-4, 4, 10000)
densities = batch_of_normals.prob(np.repeat(t[:, np.newaxis], 2, axis=1)) # each column is a vector of densities for one distn

sns.lineplot(x=t, y=densities[:, 0], label='loc={}, scale={}'.format(locs[0], scales[0]))
sns.lineplot(x=t, y=densities[:, 1], label='loc={}, scale={}'.format(locs[1], scales[1]))
plt.ylabel('Probability density')
plt.xlabel('Value')
plt.legend()
plt.show()
```


```python
# Check their batch_shape and event_shape

batch_of_normals
```




    <tfp.distributions.Normal 'Normal' batch_shape=[2] event_shape=[] dtype=float32>




```python
# Use Independent to convert the batch shape to the event shape
# (turn batches of separate distributions into a single joint density) 
# reinterpreted_batch_ndims = how many of the rightmost batch dimensions will be regarded as events in the new distribution. 

bivariate_normal_from_Independent = tfd.Independent(batch_of_normals, reinterpreted_batch_ndims=1)
```


```python
# Note that dimension from batch_shape has shifted to event_shape

bivariate_normal_from_Independent
```




    <tfp.distributions.Independent 'IndependentNormal' batch_shape=[] event_shape=[2] dtype=float32>




```python
# Create a plot showing joint density contours and marginal density functions
# joint distribution in the middle and marginal distributions on the edges 

samples = bivariate_normal_from_Independent.sample(10000)
x1 = samples[:, 0]
x2 = samples[:, 1]
sns.jointplot(x=x1, y=x2, kind="kde", space=0, color='b', xlim=[-4, 4], ylim=[-4, 4])
```


```python
# Use MultivariateNormalDiag to create the equivalent distribution
# Note that diagonal covariance matrix => no correlation => independence (for the multivariate normal distribution)

bivariate_normal_from_Multivariate = tfd.MultivariateNormalDiag(loc=locs, scale_diag=scales)
```


```python
# Plot the joint density function of bivariate_normal_from_Independent
# Refer back to bivariate_normal_from_Independent to show that the plot is the same
# Summarise how Independent has been used

samples = bivariate_normal_from_Multivariate.sample(10000)
x1 = samples[:, 0]
x2 = samples[:, 1]
sns.jointplot(x=x1, y=x2, kind="kde", space=0, color='b', xlim=[-4, 4], ylim=[-4, 4])
```

#### Shifting batch dimensions to event dimensions using 
`reinterpreted_batch_ndims`


```python
# Demonstrate use of reinterpreted_batch_ndims
# By default all batch dims except the first are transferred to event dims

# 3 gaussians each with 2 dims
loc_grid = [[-100., -100.],
            [100., 100.],
            [0., 0.]]
scale_grid = [[1., 10.],
              [1., 10.],
              [1., 1.]]

normals_batch_3by2_event_1 = tfd.Normal(loc=loc_grid, scale=scale_grid)
```


```python
# Highlight batch_shape
# by default, independent will reinterpret all dimensions, except the first, as events 

normals_batch_3by2_event_1
```




    <tfp.distributions.Normal 'Normal' batch_shape=[3, 2] event_shape=[] dtype=float32>




```python
# We now have a batch of 3 bivariate normal distributions,
# each parametrised by a column of our original parameter grid

normals_batch_2_event_2 = tfd.Independent(normals_batch_3by2_event_1)
normals_batch_2_event_2
```




    <tfp.distributions.Independent 'IndependentNormal' batch_shape=[3] event_shape=[2] dtype=float32>




```python
# Evaluate log_prob

normals_batch_2_event_2.log_prob(value=[[-10., 10.],
                                        [100., 100.],
                                        [1., 1.]])
```




    <tf.Tensor: shape=(3,), dtype=float32, numpy=array([-4.1146406e+03, -4.1404624e+00, -2.8378770e+00], dtype=float32)>




```python
# Can reinterpret _all_ batch dimensions as event dimensions

normals_batch_1_event_2by3 = tfd.Independent(normals_batch_3by2_event_1, reinterpreted_batch_ndims=2)
normals_batch_1_event_2by3
```




    <tfp.distributions.Independent 'IndependentNormal' batch_shape=[] event_shape=[3, 2] dtype=float32>




```python
# Take log_probs 

normals_batch_1_event_2by3.log_prob(value=[[-10., 10.],
                                        [100., 100.],
                                        [1., 1.]])
```




    <tf.Tensor: shape=(), dtype=float32, numpy=-4121.619>



#### Using `Independent` to build a Naive Bayes classifier

#### Introduction to `newsgroups` data set

In this tutorial, just load the dataset, fetch train/test splits, probably choose a subset of the data.

Construct the class conditional feature distribution (with Independent, using the Naive Bayes assumption) and sample from it.

We can just use the ML estimates for parameters, in later tutorials we will learn them.


```python
# Convenience function for retrieving the 20 newsgroups data set

# Usenet was a forerunner to modern internet forums
# Users could post and read articles
# Newsgroup corresponded to a topic
# Example topics in this data set: IBM computer hardware, baseball
# Our objective is to use an article's contents to predict its newsgroup,
# a 20-class classification problem.

# 18000 newsgroups, posts on 20 topics
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer
```


```python
# Get the train data

newsgroup_data = fetch_20newsgroups(data_home='20_Newsgroup_Data/', subset='train')
```


```python
# More information about the data set

print(newsgroup_data['DESCR'])
```

    .. _20newsgroups_dataset:
    
    The 20 newsgroups text dataset
    ------------------------------
    
    The 20 newsgroups dataset comprises around 18000 newsgroups posts on
    20 topics split in two subsets: one for training (or development)
    and the other one for testing (or for performance evaluation). The split
    between the train and test set is based upon a messages posted before
    and after a specific date.
    
    This module contains two loaders. The first one,
    :func:`sklearn.datasets.fetch_20newsgroups`,
    returns a list of the raw texts that can be fed to text feature
    extractors such as :class:`sklearn.feature_extraction.text.CountVectorizer`
    with custom parameters so as to extract feature vectors.
    The second one, :func:`sklearn.datasets.fetch_20newsgroups_vectorized`,
    returns ready-to-use features, i.e., it is not necessary to use a feature
    extractor.
    
    **Data Set Characteristics:**
    
        =================   ==========
        Classes                     20
        Samples total            18846
        Dimensionality               1
        Features                  text
        =================   ==========
    
    Usage
    ~~~~~
    
    The :func:`sklearn.datasets.fetch_20newsgroups` function is a data
    fetching / caching functions that downloads the data archive from
    the original `20 newsgroups website`_, extracts the archive contents
    in the ``~/scikit_learn_data/20news_home`` folder and calls the
    :func:`sklearn.datasets.load_files` on either the training or
    testing set folder, or both of them::
    
      >>> from sklearn.datasets import fetch_20newsgroups
      >>> newsgroups_train = fetch_20newsgroups(subset='train')
    
      >>> from pprint import pprint
      >>> pprint(list(newsgroups_train.target_names))
      ['alt.atheism',
       'comp.graphics',
       'comp.os.ms-windows.misc',
       'comp.sys.ibm.pc.hardware',
       'comp.sys.mac.hardware',
       'comp.windows.x',
       'misc.forsale',
       'rec.autos',
       'rec.motorcycles',
       'rec.sport.baseball',
       'rec.sport.hockey',
       'sci.crypt',
       'sci.electronics',
       'sci.med',
       'sci.space',
       'soc.religion.christian',
       'talk.politics.guns',
       'talk.politics.mideast',
       'talk.politics.misc',
       'talk.religion.misc']
    
    The real data lies in the ``filenames`` and ``target`` attributes. The target
    attribute is the integer index of the category::
    
      >>> newsgroups_train.filenames.shape
      (11314,)
      >>> newsgroups_train.target.shape
      (11314,)
      >>> newsgroups_train.target[:10]
      array([ 7,  4,  4,  1, 14, 16, 13,  3,  2,  4])
    
    It is possible to load only a sub-selection of the categories by passing the
    list of the categories to load to the
    :func:`sklearn.datasets.fetch_20newsgroups` function::
    
      >>> cats = ['alt.atheism', 'sci.space']
      >>> newsgroups_train = fetch_20newsgroups(subset='train', categories=cats)
    
      >>> list(newsgroups_train.target_names)
      ['alt.atheism', 'sci.space']
      >>> newsgroups_train.filenames.shape
      (1073,)
      >>> newsgroups_train.target.shape
      (1073,)
      >>> newsgroups_train.target[:10]
      array([0, 1, 1, 1, 0, 1, 1, 0, 0, 0])
    
    Converting text to vectors
    ~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    In order to feed predictive or clustering models with the text data,
    one first need to turn the text into vectors of numerical values suitable
    for statistical analysis. This can be achieved with the utilities of the
    ``sklearn.feature_extraction.text`` as demonstrated in the following
    example that extract `TF-IDF`_ vectors of unigram tokens
    from a subset of 20news::
    
      >>> from sklearn.feature_extraction.text import TfidfVectorizer
      >>> categories = ['alt.atheism', 'talk.religion.misc',
      ...               'comp.graphics', 'sci.space']
      >>> newsgroups_train = fetch_20newsgroups(subset='train',
      ...                                       categories=categories)
      >>> vectorizer = TfidfVectorizer()
      >>> vectors = vectorizer.fit_transform(newsgroups_train.data)
      >>> vectors.shape
      (2034, 34118)
    
    The extracted TF-IDF vectors are very sparse, with an average of 159 non-zero
    components by sample in a more than 30000-dimensional space
    (less than .5% non-zero features)::
    
      >>> vectors.nnz / float(vectors.shape[0])
      159.01327...
    
    :func:`sklearn.datasets.fetch_20newsgroups_vectorized` is a function which 
    returns ready-to-use token counts features instead of file names.
    
    .. _`20 newsgroups website`: http://people.csail.mit.edu/jrennie/20Newsgroups/
    .. _`TF-IDF`: https://en.wikipedia.org/wiki/Tf-idf
    
    
    Filtering text for more realistic training
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    It is easy for a classifier to overfit on particular things that appear in the
    20 Newsgroups data, such as newsgroup headers. Many classifiers achieve very
    high F-scores, but their results would not generalize to other documents that
    aren't from this window of time.
    
    For example, let's look at the results of a multinomial Naive Bayes classifier,
    which is fast to train and achieves a decent F-score::
    
      >>> from sklearn.naive_bayes import MultinomialNB
      >>> from sklearn import metrics
      >>> newsgroups_test = fetch_20newsgroups(subset='test',
      ...                                      categories=categories)
      >>> vectors_test = vectorizer.transform(newsgroups_test.data)
      >>> clf = MultinomialNB(alpha=.01)
      >>> clf.fit(vectors, newsgroups_train.target)
      MultinomialNB(alpha=0.01, class_prior=None, fit_prior=True)
    
      >>> pred = clf.predict(vectors_test)
      >>> metrics.f1_score(newsgroups_test.target, pred, average='macro')
      0.88213...
    
    (The example :ref:`sphx_glr_auto_examples_text_plot_document_classification_20newsgroups.py` shuffles
    the training and test data, instead of segmenting by time, and in that case
    multinomial Naive Bayes gets a much higher F-score of 0.88. Are you suspicious
    yet of what's going on inside this classifier?)
    
    Let's take a look at what the most informative features are:
    
      >>> import numpy as np
      >>> def show_top10(classifier, vectorizer, categories):
      ...     feature_names = np.asarray(vectorizer.get_feature_names())
      ...     for i, category in enumerate(categories):
      ...         top10 = np.argsort(classifier.coef_[i])[-10:]
      ...         print("%s: %s" % (category, " ".join(feature_names[top10])))
      ...
      >>> show_top10(clf, vectorizer, newsgroups_train.target_names)
      alt.atheism: edu it and in you that is of to the
      comp.graphics: edu in graphics it is for and of to the
      sci.space: edu it that is in and space to of the
      talk.religion.misc: not it you in is that and to of the
    
    
    You can now see many things that these features have overfit to:
    
    - Almost every group is distinguished by whether headers such as
      ``NNTP-Posting-Host:`` and ``Distribution:`` appear more or less often.
    - Another significant feature involves whether the sender is affiliated with
      a university, as indicated either by their headers or their signature.
    - The word "article" is a significant feature, based on how often people quote
      previous posts like this: "In article [article ID], [name] <[e-mail address]>
      wrote:"
    - Other features match the names and e-mail addresses of particular people who
      were posting at the time.
    
    With such an abundance of clues that distinguish newsgroups, the classifiers
    barely have to identify topics from text at all, and they all perform at the
    same high level.
    
    For this reason, the functions that load 20 Newsgroups data provide a
    parameter called **remove**, telling it what kinds of information to strip out
    of each file. **remove** should be a tuple containing any subset of
    ``('headers', 'footers', 'quotes')``, telling it to remove headers, signature
    blocks, and quotation blocks respectively.
    
      >>> newsgroups_test = fetch_20newsgroups(subset='test',
      ...                                      remove=('headers', 'footers', 'quotes'),
      ...                                      categories=categories)
      >>> vectors_test = vectorizer.transform(newsgroups_test.data)
      >>> pred = clf.predict(vectors_test)
      >>> metrics.f1_score(pred, newsgroups_test.target, average='macro')
      0.77310...
    
    This classifier lost over a lot of its F-score, just because we removed
    metadata that has little to do with topic classification.
    It loses even more if we also strip this metadata from the training data:
    
      >>> newsgroups_train = fetch_20newsgroups(subset='train',
      ...                                       remove=('headers', 'footers', 'quotes'),
      ...                                       categories=categories)
      >>> vectors = vectorizer.fit_transform(newsgroups_train.data)
      >>> clf = MultinomialNB(alpha=.01)
      >>> clf.fit(vectors, newsgroups_train.target)
      MultinomialNB(alpha=0.01, class_prior=None, fit_prior=True)
    
      >>> vectors_test = vectorizer.transform(newsgroups_test.data)
      >>> pred = clf.predict(vectors_test)
      >>> metrics.f1_score(newsgroups_test.target, pred, average='macro')
      0.76995...
    
    Some other classifiers cope better with this harder version of the task. Try
    running :ref:`sphx_glr_auto_examples_model_selection_grid_search_text_feature_extraction.py` with and without
    the ``--filter`` option to compare the results.
    
    .. topic:: Recommendation
    
      When evaluating text classifiers on the 20 Newsgroups data, you
      should strip newsgroup-related metadata. In scikit-learn, you can do this by
      setting ``remove=('headers', 'footers', 'quotes')``. The F-score will be
      lower because it is more realistic.
    
    .. topic:: Examples
    
       * :ref:`sphx_glr_auto_examples_model_selection_grid_search_text_feature_extraction.py`
    
       * :ref:`sphx_glr_auto_examples_text_plot_document_classification_20newsgroups.py`
    



```python
# Example article

print(newsgroup_data['data'][0])
```

    From: lerxst@wam.umd.edu (where's my thing)
    Subject: WHAT car is this!?
    Nntp-Posting-Host: rac3.wam.umd.edu
    Organization: University of Maryland, College Park
    Lines: 15
    
     I was wondering if anyone out there could enlighten me on this car I saw
    the other day. It was a 2-door sports car, looked to be from the late 60s/
    early 70s. It was called a Bricklin. The doors were really small. In addition,
    the front bumper was separate from the rest of the body. This is 
    all I know. If anyone can tellme a model name, engine specs, years
    of production, where this car is made, history, or whatever info you
    have on this funky looking car, please e-mail.
    
    Thanks,
    - IL
       ---- brought to you by your neighborhood Lerxst ----
    
    
    
    
    



```python
# Associated label

newsgroup_data['target'][0]
```




    7




```python
# Name of label

newsgroup_data['target_names'][7]
```




    'rec.autos'




```python
# Preprocessing boilerplate

n_documents = len(newsgroup_data['data'])

count_vectorizer = CountVectorizer(input='content', binary=True,
                                   max_df=0.25, min_df=1.01/n_documents) # ignore common words, words that appear once
binary_bag_of_words = count_vectorizer.fit_transform(newsgroup_data['data']) # input is a list of strings

```


```python
# Check shape

binary_bag_of_words.shape
```




    (11314, 56365)




```python
# Check that the fit has been successful

count_vectorizer.inverse_transform(binary_bag_of_words[0,:])
```




    [array(['lerxst', 'wam', 'umd', 'where', 'thing', 'car', 'rac3',
            'maryland', 'college', 'park', '15', 'wondering', 'anyone',
            'could', 'enlighten', 'saw', 'day', 'door', 'sports', 'looked',
            'late', '60s', 'early', '70s', 'called', 'bricklin', 'doors',
            'were', 'really', 'small', 'addition', 'front', 'bumper',
            'separate', 'rest', 'body', 'tellme', 'model', 'name', 'engine',
            'specs', 'years', 'production', 'made', 'history', 'whatever',
            'info', 'funky', 'looking', 'please', 'mail', 'thanks', 'il',
            'brought', 'neighborhood'], dtype='<U80')]




```python
# Dict that will be useful later 

inv_vocabulary = {value:key for key, value in count_vectorizer.vocabulary_.items()}
```

#### A Naive Bayes classifier for `newsgroup`

Each feature vector $x$ is a list of indicators for whether a word appears in the article. $x_i$ is 1 if the $i$th word appears, and 0 otherwise. `inv_vocabulary` matches word indices $i$ to words.

Each label $y$ is a value in $0, 1, \ldots, 19$.

The parts of a naive Bayes classifier for this problem can be summarised as:  


- A probability distribution for the feature vector by class, $p(x|y = j)$ for each $j = 0, 1, \ldots, 19$. These probability distributions are assumed to have independent components: we can factorize the joint probability as a product of marginal probabilities
\begin{equation}
    p(x|y = j) = \prod_{i=1}^d p(x_i|y = j)
\end{equation}
These marginal probability distributions are Bernoulli distributions, each of which has a single parameter $\theta_{ji} := p(x_i = 1|y = j)$. This parameter is the probability of observing word $i$ in an article of class $j$. 

- We will use the Laplace smoothed maximum likelihood estimate to compute these parameters. Laplace smoothing involves adding small counts to every feature for each class. Else, if a feature did not appear in the training set of a class, but then we observed it in our test data the log probability would be undefined.

- A collection of class prior probabilities $p(y = j)$. These will be set by computing the class base rates in the training set.  


- A function for computing the probability of class membership via Bayes' theorem:  

\begin{equation}
    p(y = j|x) = \frac{p(x|y = j)p(y = j)}{p(x)}
\end{equation}


```python
# Compute the parameter estimates (adjusted fraction of documents in class that contain word)

n_classes = newsgroup_data['target'].max() + 1
y = newsgroup_data['target']
n_words = binary_bag_of_words.shape[1]

alpha = 1e-6 # parameters for Laplace smoothing

theta = np.zeros([n_classes, n_words]) # stores parameter values - prob. word given class
for c_k in range(n_classes): # 0, 1, ..., 19
    class_mask = (y == c_k)
    N = class_mask.sum() # number of articles in class
    theta[c_k, :] = (binary_bag_of_words[class_mask, :].sum(axis=0) + alpha)/(N + alpha*2)
```


```python
# Check whether the most probable word in each class is reasonable

most_probable_word_ix = theta.argmax(axis=1) # most probable word for each class

for j, ix in enumerate(most_probable_word_ix):
    print('Most probable word in class {} is "{}".'.format(newsgroup_data['target_names'][j],
                                                           inv_vocabulary[ix]))
```

    Most probable word in class alt.atheism is "people".
    Most probable word in class comp.graphics is "graphics".
    Most probable word in class comp.os.ms-windows.misc is "windows".
    Most probable word in class comp.sys.ibm.pc.hardware is "thanks".
    Most probable word in class comp.sys.mac.hardware is "mac".
    Most probable word in class comp.windows.x is "window".
    Most probable word in class misc.forsale is "sale".
    Most probable word in class rec.autos is "car".
    Most probable word in class rec.motorcycles is "dod".
    Most probable word in class rec.sport.baseball is "he".
    Most probable word in class rec.sport.hockey is "ca".
    Most probable word in class sci.crypt is "clipper".
    Most probable word in class sci.electronics is "use".
    Most probable word in class sci.med is "reply".
    Most probable word in class sci.space is "space".
    Most probable word in class soc.religion.christian is "god".
    Most probable word in class talk.politics.guns is "people".
    Most probable word in class talk.politics.mideast is "people".
    Most probable word in class talk.politics.misc is "people".
    Most probable word in class talk.religion.misc is "he".



```python
# Define a distribution for each class (20 distributions for 20 classes)

batch_of_bernoullis = tfd.Bernoulli(probs=theta)
p_x_given_y = tfd.Independent(batch_of_bernoullis, reinterpreted_batch_ndims=1)
p_x_given_y
```




    <tfp.distributions.Independent 'IndependentBernoulli' batch_shape=[20] event_shape=[56365] dtype=int32>




```python
# Take a sample of words from each class

n_samples = 10
sample = p_x_given_y.sample(n_samples)
sample.shape  # nr of samples x nr of classes x vocab size
```




    TensorShape([10, 20, 56365])




```python
# Choose a class

chosen_class = 15
newsgroup_data['target_names'][chosen_class]
```




    'soc.religion.christian'




```python
# Indicators for words that appear in the sample

class_sample = sample[:, chosen_class, :]
class_sample
```




    <tf.Tensor: shape=(10, 56365), dtype=int32, numpy=
    array([[1, 0, 0, ..., 0, 0, 0],
           [0, 0, 0, ..., 0, 0, 0],
           [0, 0, 0, ..., 0, 0, 0],
           ...,
           [0, 0, 0, ..., 0, 0, 0],
           [0, 0, 0, ..., 0, 0, 0],
           [0, 0, 0, ..., 0, 0, 0]], dtype=int32)>




```python
# Perform inverse transform to test quality of fit

count_vectorizer.inverse_transform(class_sample)
```




    [array(['00', '03', '05', '14', '1989', '2a', 'aaron', 'absolutely',
            'accepted', 'actually', 'alastair', 'alle', 'ancient', 'apart',
            'ask', 'association', 'bass', 'bbn', 'because', 'becoming', 'been',
            'before', 'being', 'believe', 'bible', 'big', 'bind', 'bit',
            'brainwashed', 'called', 'came', 'check', 'christian',
            'christians', 'cited', 'commandments', 'committed',
            'contradictory', 'conveys', 'cooperate', 'csd', 'cup', 'death',
            'define', 'definitely', 'describing', 'did', 'die', 'discovering',
            'discussion', 'dispute', 'disputes', 'does', 'either', 'emphasis',
            'end', 'even', 'every', 'evidence', 'excellent', 'explicit',
            'eyes', 'following', 'found', 'generally', 'god', 'going', 'greek',
            'groups', 'happen', 'happening', 'he', 'help', 'holds',
            'hypothesis', 'insists', 'intended', 'introductory', 'jews', 'joy',
            'let', 'lifestyle', 'live', 'losing', 'lukewarm', 'maintains',
            'making', 'man', 'matter', 'may', 'met', 'moral', 'movie', 'never',
            'new', 'non', 'now', 'okay', 'over', 'overseas', 'passages',
            'past', 'people', 'person', 'pity', 'plurality', 'praying',
            'probably', 'promised', 'question', 'radio', 'rapacity', 'reason',
            'refrain', 'related', 'reliability', 'remind', 'reread',
            'responding', 'right', 'rutgers', 'salt', 'same', 'service', 'sil',
            'society', 'specifically', 'spiritual', 'ssd', 'stephen', 'steven',
            'stronger', 'such', 'take', 'talking', 'temporal', 'testimony',
            'tests', 'their', 'them', 'then', 'these', 'thing', 'those',
            'through', 'thrust', 'timing', 'tom', 'too', 'towards', 'true',
            'trust', 'two', 'urbana', 'use', 'uu', 'viewed', 'way', 'why',
            'woman', 'worship', 'write'], dtype='<U80'),
     array(['18669', '19', '1950', '1993', '22102', 'aaron', 'above',
            'actions', 'address', 'again', 'always', 'atheists', 'athos',
            'atoned', 'authority', 'bear', 'been', 'being', 'bill', 'board',
            'born', 'breastplate', 'causes', 'changed', 'choices', 'christ',
            'church', 'claim', 'clare', 'college', 'comment', 'conspiracy',
            'context', 'continues', 'correct', 'course', 'cu', 'definitely',
            'desire', 'desired', 'deviations', 'discover', 'dogmatism', 'down',
            'dreams', 'eternal', 'europe', 'evensont', 'fear', 'fi', 'find',
            'finds', 'fisher', 'flourished', 'formal', 'found', 'fully',
            'geneva', 'gmi', 'god', 'good', 'gospel', 'governments', 'greater',
            'gregory', 'ground', 'guess', 'heard', 'heaven', 'hence',
            'historical', 'holy', 'hope', 'james', 'jayne', 'jesus', 'junior',
            'knowlege', 'koresh', 'later', 'lewis', 'lifetime', 'lot', 'lpa',
            'maintain', 'man', 'many', 'marriage', 'maybe', 'mmh', 'modern',
            'moon', 'mosque', 'most', 'must', 'myself', 'nature',
            'nebuchadnezzar', 'need', 'now', 'okay', 'once', 'pastor',
            'people', 'personal', 'personally', 'phrase', 'please', 'points',
            'posted', 'prism', 'programs', 'promoting', 'question', 'rarely',
            'really', 'rebellion', 'received', 'rest', 'return', 'revelation',
            'right', 'said', 'same', 'saved', 'seconds', 'sentence', 'show',
            'sins', 'sometimes', 'standing', 'stores', 'straws', 'stupid',
            'suggest', 'suspect', 'talk', 'teachings', 'tell', 'thee', 'these',
            'things', 'think', 'third', 'thou', 'time', 'together', 'told',
            'too', 'try', 'unbelievable', 'understanding', 'uni', 'using',
            'various', 'vaxc', 'very', 'weight', 'why', 'without', 'world',
            'york'], dtype='<U80'),
     array(['09', '19', '1993', '23', 'allows', 'amount', 'answer', 'anyone',
            'apr', 'assembled', 'attribute', 'back', 'basic', 'because',
            'been', 'biblical', 'brimstone', 'canon', 'carolina', 'case',
            'christians', 'cristo', 'dead', 'denial', 'difficulty', 'drew',
            'ecn', 'encounter', 'encyclopedia', 'entire', 'faith', 'finger',
            'form', 'found', 'further', 'gather', 'general', 'gives', 'guards',
            'he', 'here', 'hidden', 'him', 'himself', 'his', 'honest', 'humor',
            'immediately', 'important', 'inherit', 'introduction', 'jude',
            'known', 'labs', 'letter', 'log', 'longer', 'made', 'magazine',
            'marry', 'meaning', 'member', 'messiahs', 'might', 'nearly',
            'need', 'needs', 'negative', 'never', 'normally', 'now', 'order',
            'parts', 'people', 'period', 'persecuted', 'person', 'praise',
            'probably', 'problem', 'processes', 'prophet', 'publicly',
            'reason', 'received', 'recommend', 'recommended', 'redemption',
            'request', 'right', 'satanic', 'seen', 'send', 'sending',
            'severly', 'simply', 'small', 'someday', 'sometimes', 'statement',
            'statements', 'student', 'supposed', 'teneis', 'than', 'their',
            'them', 'these', 'things', 'those', 'through', 'times', 'tiring',
            'together', 'tone', 'treat', 'under', 'valid', 'veiled', 'were',
            'whatever', 'whereby', 'while', 'why', 'woman', 'work'],
           dtype='<U80'),
     array(['05', '213', '25', '30602', '59', '66', 'accurate', 'acs',
            'albrecht', 'arrogance', 'ask', 'association', 'awfully', 'bad',
            'being', 'below', 'boredom', 'both', 'bps', 'brimstone', 'ca',
            'caused', 'christ', 'christian', 'church', 'clh', 'collosions',
            'commands', 'conclusion', 'coupled', 'cross', 'crossan', 'darius',
            'david', 'dead', 'decided', 'discard', 'discussion', 'effect',
            'embrase', 'enough', 'ere', 'eternal', 'excellent', 'far', 'fold',
            'free', 'general', 'going', 'got', 'gov', 'grace', 'greatest',
            'he', 'hence', 'him', 'his', 'important', 'impression',
            'indescribably', 'indications', 'internal', 'interpretation',
            'invited', 'jemurray', 'jews', 'jh', 'judge', 'leads', 'lies',
            'life', 'lot', 'lots', 'many', 'mark', 'mary', 'maybe', 'means',
            'mind', 'modern', 'moses', 'moslem', 'most', 'northwestern',
            'nothing', 'order', 'part', 'pogrom', 'point', 'process', 'proof',
            'punish', 'put', 'rather', 'react', 'reading', 'real', 'reasoning',
            'resurrection', 'right', 'rolfe', 'rsteele', 'saying',
            'scriptures', 'should', 'society', 'store', 'suffer', 'suggest',
            'tell', 'thanks', 'then', 'things', 'think', 'threatened',
            'through', 'throughout', 'tibet', 'trying', 'turned', 'videotapes',
            'way', 'ways', 'while', 'whole', 'withheld', 'work'], dtype='<U80'),
     array(['01', '02', '0358', '11', '19', '21', '28445', '64', '8029',
            '9760', 'also', 'amusing', 'analogy', 'angry', 'appears', 'aren',
            'assembled', 'attracts', 'because', 'been', 'being', 'believe',
            'bgsu', 'body', 'born', 'box', 'breakup', 'buddhist', 'carnegie',
            'century', 'checked', 'claim', 'claiming', 'come', 'condemnation',
            'courageously', 'detail', 'didn', 'died', 'diety', 'different',
            'does', 'drawing', 'easter', 'ever', 'every', 'excuse', 'exists',
            'expectation', 'explicitly', 'fabrication', 'fall', 'fiat', 'find',
            'gave', 'general', 'gifts', 'god', 'great', 'group', 'groups',
            'he', 'hear', 'heaven', 'held', 'help', 'him', 'history',
            'interpretation', 'jesus', 'jimmy', 'lake', 'last', 'least',
            'legislate', 'let', 'lift', 'male', 'mary', 'might', 'mussack',
            'natural', 'night', 'older', 'opinions', 'organizational', 'oulu',
            'our', 'over', 'people', 'provided', 'ran', 'religions',
            'removing', 'representative', 'rest', 'rev', 'riches', 'road',
            'rushing', 'sake', 'saving', 'see', 'send', 'sin', 'solar',
            'something', 'spelling', 'sps', 'suggest', 'sure', 'surrounding',
            'textbooks', 'think', 'through', 'tolerate', 'too', 'transfer',
            'true', 'truths', 'uchicago', 'undergraduate', 'upenn', 'usually',
            'utterly', 'version', 'vic', 'view', 'welch', 'whereas', 'without',
            'wrong', 'yes'], dtype='<U80'),
     array(['01', '02', '09', '16', '41', '72', 'able', 'accepting', 'ahmadi',
            'also', 'anderson', 'answer', 'appearances', 'assuming',
            'atheists', 'au', 'away', 'band', 'being', 'believe', 'billh',
            'body', 'changes', 'chanted', 'children', 'christians', 'clh',
            'college', 'compuserve', 'conclusion', 'convert', 'coordinated',
            'could', 'courage', 'crying', 'debated', 'deficiencies',
            'definition', 'doctrine', 'earliest', 'entered', 'escape',
            'eternal', 'events', 'exist', 'form', 'former', 'forth',
            'freshman', 'grace', 'greek', 'had', 'hardly', 'heard', 'here',
            'home', 'immediately', 'impossibility', 'impression',
            'invitations', 'involving', 'isn', 'its', 'itself', 'jesus',
            'jewish', 'jews', 'johnson', 'kingdom', 'lady', 'least', 'los',
            'many', 'mark', 'materialists', 'may', 'motive', 'murray',
            'nature', 'new', 'next', 'non', 'nothing', 'often', 'ogden',
            'once', 'opinion', 'order', 'part', 'particular', 'parts', 'paul',
            'personal', 'poster', 'principle', 'product', 'promise', 'public',
            'ready', 'reasonable', 'refrain', 'region', 'religion', 'remember',
            'rooted', 'say', 'sciences', 'second', 'seeing', 'site', 'society',
            'songs', 'spirit', 'steve', 'studying', 'such', 'sure',
            'teachings', 'their', 'these', 'think', 'thomas', 'throughout',
            'tone', 'trojan', 'trying', 'unnecessary', 'us', 'useful',
            'validity', 'verbal', 'violate', 'war', 'well', 'were',
            'wittgenstein', 'wrongly', 'wrote', 'york', 'yours'], dtype='<U80'),
     array(['01', '0358', '13', '20', '44', '54', 'account', 'accounts',
            'acts', 'actually', 'again', 'am', 'angels', 'assumption',
            'authority', 'awarded', 'basis', 'because', 'been', 'behavior',
            'believers', 'bible', 'bulls', 'called', 'cc', 'certain', 'christ',
            'christian', 'church', 'claims', 'cloak', 'clock', 'colorado',
            'commendation', 'con', 'crusades', 'cult', 'current', 'dayton',
            'dear', 'demanded', 'disagreement', 'division', 'double', 'each',
            'effects', 'either', 'equally', 'etc', 'ever', 'everything',
            'expression', 'fact', 'family', 'festival', 'follows', 'fraser',
            'fsu', 'fundamental', 'general', 'granted', 'greatest', 'greeks',
            'having', 'hebrew', 'heresey', 'him', 'hot', 'however', 'iu',
            'jesus', 'kingdom', 'later', 'leads', 'lifetime', 'listened',
            'liturgical', 'look', 'looks', 'marry', 'model', 'most', 'moved',
            'much', 'natural', 'newsletter', 'non', 'now', 'occasion', 'off',
            'operates', 'optimistic', 'pain', 'participants', 'penetration',
            'people', 'point', 'presents', 'primarily', 'probably', 'question',
            'quo', 'rampant', 'realize', 'relevant', 'religion', 'said',
            'salvation', 'same', 'selfish', 'series', 'serious', 'service',
            'sexes', 'should', 'soc', 'someone', 'st', 'stick', 'such',
            'suggested', 'talks', 'taught', 'terrible', 'then', 'thing',
            'think', 'thought', 'toledo', 'topic', 'trinity', 'true',
            'trunking_fixed', 'trying', 'two', 'uniwa', 'unix', 'via',
            'wanted', 'wasn', 'way', 'whole', 'wished', 'worse'], dtype='<U80'),
     array(['03', '13', '19', '1993', '24', '30', '592682', 'absolute',
            'abstinence', 'ac', 'again', 'animals', 'apparently', 'asked',
            'assumptions', 'basics', 'bed', 'begin', 'being', 'believe',
            'blame', 'box', 'brings', 'came', 'cannot', 'carried', 'chris',
            'christendom', 'christian', 'christians', 'claim', 'completely',
            'computer', 'confirmed', 'confusion', 'consecration', 'could',
            'covers', 'created', 'defined', 'deny', 'descent', 'determination',
            'dies', 'disciples', 'done', 'early', 'engaged', 'esteem', 'even',
            'ever', 'falsely', 'fellowship', 'form', 'further', 'general',
            'gentile', 'give', 'god', 'happen', 'he', 'held', 'himself', 'his',
            'imagine', 'interpretation', 'ishtar', 'john', 'judge', 'judged',
            'knowing', 'led', 'let', 'life', 'ling', 'little', 'make',
            'marched', 'michael', 'moderate', 'moreover', 'mormon', 'never',
            'new', 'nice', 'nyx', 'often', 'old', 'order', 'organisms',
            'people', 'peter', 'pihatie', 'pihko', 'point', 'president',
            'proved', 'punishable', 'reading', 'recent', 'resurection', 'same',
            'see', 'seperated', 'several', 'share', 'should', 'show', 'sig',
            'smoldering', 'spoke', 'stood', 'styles', 'such', 'systems',
            'talking', 'than', 'then', 'therefore', 'toward', 'traditional',
            'training', 'tull', 'us', 'uth', 'very', 'wage', 'weak', 'woman',
            'world'], dtype='<U80'),
     array(['10', '16', '20', '5of5', '62', '68', '706', 'account', 'acs',
            'ages', 'also', 'am', 'amazing', 'among', 'anyone', 'apostles',
            'approaching', 'athena', 'bears', 'because', 'biblical',
            'blessings', 'body', 'book', 'ca', 'call', 'called', 'capable',
            'cease', 'center', 'charge', 'china', 'christians', 'civil',
            'claims', 'cleanse', 'company', 'day', 'dead', 'decided', 'dept',
            'did', 'disobedience', 'disproves', 'due', 'duty', 'eternal',
            'evangelicals', 'everything', 'evidence', 'fairly', 'faith',
            'find', 'fire', 'forms', 'front', 'fundamentalist', 'gets', 'god',
            'got', 'gov', 'gut', 'haya', 'he', 'hell', 'his', 'incarnate',
            'indeed', 'infonode', 'instead', 'involving', 'limit', 'literal',
            'longer', 'mankind', 'meantime', 'might', 'minds', 'near', 'never',
            'new', 'news', 'nobody', 'normally', 'now', 'objects', 'offer',
            'okay', 'openly', 'opinion', 'own', 'people', 'percieved',
            'perfect', 'phone', 'pointed', 'position', 'possible', 'post',
            'presence', 'probably', 'problem', 'question', 'randy', 'read',
            'refusal', 'regards', 'remarked', 'sad', 'same', 'say', 'science',
            'scientist', 'seminary', 'sense', 'share', 'sight', 'signs',
            'silence', 'sin', 'since', 'slave', 'somebody', 'statements',
            'states', 'sunday', 'suspend', 'tell', 'than', 'them', 'though',
            'through', 'times', 'troi', 'unisa', 'urge', 'us', 'utterly',
            'very', 'weekly', 'well', 'without', 'woman', 'working', 'written'],
           dtype='<U80'),
     array(['06', '100', '111', '16', '17', '90', 'absure', 'actually',
            'africa', 'african', 'ago', 'ah', 'allowed', 'along', 'anyone',
            'anything', 'apr', 'areas', 'associate', 'associated', 'bask',
            'because', 'before', 'behind', 'being', 'better', 'children',
            'christ', 'claim', 'clean', 'clear', 'collection', 'come',
            'coming', 'contorting', 'creator', 'cruel', 'defy', 'difficult',
            'digress', 'discipline', 'earthly', 'else', 'enacted',
            'enlightenment', 'everyone', 'example', 'favor', 'fell', 'firstly',
            'fortunately', 'general', 'gippsland', 'gives', 'goat', 'group',
            'he', 'heard', 'helps', 'himself', 'hinduism', 'insist',
            'interventionists', 'involved', 'jeezus', 'judgment', 'kind',
            'kulikauskas', 'larsen', 'lois', 'lost', 'many', 'medicine', 'met',
            'mine', 'moment', 'much', 'must', 'newness', 'nietzsche', 'non',
            'nothing', 'opinion', 'original', 'otherwise', 'paganism',
            'particular', 'peter', 'proof', 'provoking', 'punched', 'really',
            'refers', 'relativism', 'religions', 'revelation', 'rutgers',
            'sackcloth', 'safety', 'say', 'seemed', 'seems', 'separation',
            'sept', 'shanti', 'since', 'specifically', 'strongly', 'takes',
            'than', 'theological', 'therefore', 'thing', 'things', 'tolerate',
            'trinity', 'trying', 'unbiblical', 'uncertainty', 'until', 'us',
            'uw', 'very', 'wbt', 'were', 'whose', 'witness', 'words', 'works',
            'world', 'wouldn', 'year'], dtype='<U80')]



***
<a id='sampling_and_log_probs'></a>
## Sampling and log probs


```python
# Make Multivariate Distribution

normal_distributions = tfd.MultivariateNormalDiag(loc=[[0.5, 1], [0.1, 0], [0, 0.2]],
                                 scale_diag=[[2, 3], [1, 3], [4, 4]])
normal_distributions
```




    <tfp.distributions.MultivariateNormalDiag 'MultivariateNormalDiag' batch_shape=[3] event_shape=[2] dtype=float32>




```python
# Sample

normal_distributions.sample(5)
```




    <tf.Tensor: shape=(5, 3, 2), dtype=float32, numpy=
    array([[[-1.2961674 , -3.4648843 ],
            [-1.7259144 , -2.356639  ],
            [-1.7776723 ,  0.9847862 ]],
    
           [[ 0.8520931 ,  3.0043614 ],
            [-1.4252505 ,  4.269158  ],
            [ 2.541176  ,  0.38246575]],
    
           [[ 0.0661459 ,  0.64083046],
            [-1.6084905 , -3.1842263 ],
            [ 2.0696645 , -1.9636502 ]],
    
           [[ 0.6721385 ,  2.5767398 ],
            [-1.1781933 , -0.02150659],
            [-1.5799453 , -3.6013174 ]],
    
           [[-1.9095645 , -0.96390367],
            [ 0.1170806 , -1.9153602 ],
            [ 0.15956835,  4.828182  ]]], dtype=float32)>




```python
# Multivariate Normal batched Distribution
# We are broadcasting batch shapes of `loc` and `scal_diag` 
# against each other

loc = [[[0.3, 1.5, 1.], [0.2, 0.4, 2.8]],
        [[2., 2.3, 8], [1.4, 1, 1.3]]]
scale_diag = [0.4, 1., 0.7]
normal_distributions = tfd.MultivariateNormalDiag(loc=loc, scale_diag=scale_diag)
normal_distributions
```




    <tfp.distributions.MultivariateNormalDiag 'MultivariateNormalDiag' batch_shape=[2, 2] event_shape=[3] dtype=float32>




```python
# Use independent to move part of the batch shape

ind_normal_distributions = tfd.Independent(normal_distributions, reinterpreted_batch_ndims=1)
ind_normal_distributions
```




    <tfp.distributions.Independent 'IndependentMultivariateNormalDiag' batch_shape=[2] event_shape=[2, 3] dtype=float32>




```python
# Draw some samples

samples = ind_normal_distributions.sample(5)
samples.shape
```




    TensorShape([5, 2, 2, 3])




```python
# `[B, E]` shaped input

inp = tf.random.uniform((2, 2, 3))
ind_normal_distributions.log_prob(inp)
```




    <tf.Tensor: shape=(2,), dtype=float32, numpy=array([-10.683702, -70.85609 ], dtype=float32)>




```python
# `[E]` shaped input (broadcasting over batch size)

inp = tf.random.uniform((2, 3))
ind_normal_distributions.log_prob(inp)
```




    <tf.Tensor: shape=(2,), dtype=float32, numpy=array([ -9.249443, -75.61279 ], dtype=float32)>




```python
#`[S, B, E]` shaped input (broadcasting over samples)

inp = tf.random.uniform((9, 2, 2, 3))
ind_normal_distributions.log_prob(inp)
```




    <tf.Tensor: shape=(9, 2), dtype=float32, numpy=
    array([[-12.21836 , -70.394684],
           [ -7.264469, -85.92087 ],
           [ -7.722307, -77.85683 ],
           [-10.986505, -77.435074],
           [-11.681156, -67.48659 ],
           [ -9.741246, -68.502914],
           [ -9.467726, -76.27688 ],
           [-11.667308, -75.85168 ],
           [-10.043331, -82.41718 ]], dtype=float32)>




```python
# `[S, b, e]` shaped input, where [b, e] is broadcastable over [B, E]

inp = tf.random.uniform((5, 1, 2, 1))
ind_normal_distributions.log_prob(inp)
```

    WARNING: Nested component "MultivariateNormalDiag_scale_matvec_linear_operator" in composition "MultivariateNormalDiag_chain_of_MultivariateNormalDiag_shift_of_MultivariateNormalDiag_scale_matvec_linear_operator" operates on inputs with increased degrees of freedom. This may result in an incorrect log_det_jacobian.





    <tf.Tensor: shape=(5, 2), dtype=float32, numpy=
    array([[ -9.131046, -70.296005],
           [-10.460278, -83.73175 ],
           [-11.657809, -80.00502 ],
           [-11.055916, -67.27801 ],
           [-10.54241 , -82.90212 ]], dtype=float32)>



#### Naive Bayes example

Lets now use what we have learned and continue the Naive Bayes classifier we were building last tutorial.


```python
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import BernoulliNB
from sklearn.metrics import f1_score
```


```python
# Making a function get_data which:
#   1) Fetches the 20 newsgroup dataset
#   2) Performs a word count on the articles and binarizes the result
#   3) Returns the data as a numpy matrix with the labels

def get_data(categories):
    
    newsgroups_train_data = fetch_20newsgroups(data_home='20_Newsgroup_Data/',
                                               subset='train', categories=categories)
    newsgroups_test_data = fetch_20newsgroups(data_home='20_Newsgroup_Data/',
                                              subset='test', categories=categories)

    n_documents = len(newsgroups_train_data['data'])
    count_vectorizer = CountVectorizer(input='content', binary=True,max_df=0.25, min_df=1.01/n_documents)
    
    train_binary_bag_of_words = count_vectorizer.fit_transform(newsgroups_train_data['data'])
    test_binary_bag_of_words = count_vectorizer.transform(newsgroups_test_data['data']) 

    return (train_binary_bag_of_words.todense(), newsgroups_train_data['target']),  (test_binary_bag_of_words.todense(), newsgroups_test_data['target'])
```


```python
# Defining a function to conduct Laplace smoothing. This adds a base level of probability for a given feature
# to occur in every class.

def laplace_smoothing(labels, binary_data, n_classes):
    # Compute the parameter estimates (adjusted fraction of documents in class that contain word)
    n_words = binary_data.shape[1]
    alpha = 1 # parameters for Laplace smoothing
    theta = np.zeros([n_classes, n_words]) # stores parameter values - prob. word given class
    for c_k in range(n_classes): # 0, 1, ..., 19
        class_mask = (labels == c_k)
        N = class_mask.sum() # number of articles in class
        theta[c_k, :] = (binary_data[class_mask, :].sum(axis=0) + alpha)/(N + alpha*2)

    return theta
```


```python
# Getting a subset of the 20 newsgroup dataset

categories = ['alt.atheism', 'talk.religion.misc', 'comp.graphics', 'sci.space']

(train_data, train_labels), (test_data, test_labels) = get_data(categories=categories)
smoothed_counts = laplace_smoothing(labels=train_labels, binary_data=train_data, n_classes=len(categories))
```

To now make our NB classifier we need to build three functions:
* Compute the class priors
* Build our class conditional distributions
* Put it all together and classify our data


```python
# Function which computes the prior probability of every class based on frequency of occurence in 
# the dataset

def class_priors(n_classes, labels):
    counts = np.zeros(n_classes)
    for c_k in range(n_classes):
        counts[c_k] = np.sum(np.where(labels==c_k, 1, 0))
    priors = counts / np.sum(counts)
    print('The class priors are {}'.format(priors))
    return priors
```


```python
# Run the function 

priors = class_priors(n_classes=len(categories), labels=train_labels)
```

    The class priors are [0.2359882  0.28711898 0.29154376 0.18534907]



```python
# Now we will do a function that given the feature occurence counts returns a Bernoulli distribution of 
# batch_shape=number of classes and event_shape=number of features.

def make_distribution(probs):
  batch_of_bernoullis = tfd.Bernoulli(probs=probs)  # shape (n_classes, n_words)
  dist = tfd.Independent(batch_of_bernoullis, reinterpreted_batch_ndims=1)
  return dist

tf_dist = make_distribution(smoothed_counts)
```


```python
# The final function predict_sample which given the distribution, a test sample, and the class priors:
#   1) Computes the class conditional probabilities given the sample
#   2) Forms the joint likelihood
#   3) Normalises the joint likelihood and returns the log prob

def predict_sample(dist, sample, priors):
  cond_probs = dist.log_prob(sample)
  joint_likelihood = tf.add(tf.cast(np.log(priors), tf.float32), cond_probs)
  # this has to be normalized to result in probabilities
  # easiest way is to stay in log space
  norm_factor = tf.math.reduce_logsumexp(joint_likelihood, axis=-1, keepdims=True)
  log_prob = joint_likelihood - norm_factor
  return log_prob

```

#### Computing log_probs


```python
# Predicting one example from our test data

log_probs = predict_sample(tf_dist, test_data[0], priors)
log_probs
```




    <tf.Tensor: shape=(4,), dtype=float32, numpy=
    array([-6.17363739e+01, -1.52587891e-05, -1.16197815e+01, -6.33276672e+01],
          dtype=float32)>




```python
# Loop over our test data and classify.

probabilities = []
for sample, label in zip(test_data, test_labels):
    probabilities.append(tf.exp(predict_sample(tf_dist, sample, priors)))

probabilities = np.asarray(probabilities)
predicted_classes = np.argmax(probabilities, axis =-1)
print('f1 ', f1_score(test_labels, predicted_classes, average='macro'))
```

    f1  0.7848499112849504



```python
# Make a Bernoulli Naive Bayes classifier using sklearn with another level of alpha smoothing. 

clf = BernoulliNB(alpha=0.1)
clf.fit(train_data, train_labels)
pred = clf.predict(test_data)
print('f1 from sklean ', f1_score(test_labels, pred, average='macro'))
```

    f1 from sklean  0.8496961488207146


***
<a id='trainable_distributions'></a>
## Trainable Distributions


```python
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import BernoulliNB
from sklearn.metrics import f1_score
```


```python
# Define an exponential distribution

exponential = tfd.Exponential(rate=0.3, name='exp')
```


```python
# Plot

plt.hist(exponential.sample(5000), bins=100, density=True)
plt.show()
```


```python
# Define an exponential distribution with a trainable rate parameter

exp_train = tfd.Exponential(rate=tf.Variable(1., name='rate'), name='exp_train')
```


```python
# Define the negative log likelihood

def nll(x_train, dist):
  # mean log probability of getting the training data from the distribution with its current parameters 
  return -tf.reduce_mean(dist.log_prob(x_train))
```


```python
# Define a function to compute the loss and gradients

@tf.function
def get_loss_and_grads(x_train, dist):
  with tf.GradientTape() as tape:
    # watch the distribution's rate as trainable (optimizable) variable 
    tape.watch(dist.trainable_variables)
    loss = nll(x_train, dist)
    grads = tape.gradient(loss, dist.trainable_variables)
  return loss, grads
```


```python
# Optimize

def exponential_dist_optimisation(data, distribution):

    # Keep results for plotting
    train_loss_results = []
    train_rate_results = []
    
    optimizer = tf.keras.optimizers.SGD(learning_rate=0.05)

    num_steps = 10

    for i in range(num_steps):
        loss, grads = get_loss_and_grads(data, distribution)
        # apply optimized gradient to trainable distribution rate
        optimizer.apply_gradients(zip(grads, distribution.trainable_variables))

        rate_value = distribution.rate.value()
        train_loss_results.append(loss)
        train_rate_results.append(rate_value)

        print("Step {:03d}: Loss: {:.3f}: Rate: {:.3f}".format(i, loss, rate_value))
        
    return train_loss_results, train_rate_results
```


```python
# Get some data and train

sampled_data = exponential.sample(5000)  # GT
train_loss_results, train_rate_results = exponential_dist_optimisation(data=sampled_data, distribution=exp_train)
```

    Step 000: Loss: 3.387: Rate: 0.881
    Step 001: Loss: 3.110: Rate: 0.768
    Step 002: Loss: 2.865: Rate: 0.664
    Step 003: Loss: 2.658: Rate: 0.570
    Step 004: Loss: 2.492: Rate: 0.488
    Step 005: Loss: 2.371: Rate: 0.421
    Step 006: Loss: 2.291: Rate: 0.371
    Step 007: Loss: 2.248: Rate: 0.336
    Step 008: Loss: 2.229: Rate: 0.316
    Step 009: Loss: 2.222: Rate: 0.305



```python
# Predicted value for the rate parameter

pred_value = exp_train.rate.numpy()
exact_value = exponential.rate.numpy()

print("Exact rate: ", exact_value)
print("Pred rate:  ", pred_value)
```

    Exact rate:  0.3
    Pred rate:   0.3046331



```python
# Plot to see the convergence of the estimated and true parameters

tensor_exact_value = tf.constant(exact_value, shape=[len(train_rate_results)])

fig, axes = plt.subplots(2, sharex=True, figsize=(12, 8))
fig.suptitle('Convergence')

axes[0].set_ylabel("Loss", fontsize=14)
axes[0].plot(train_loss_results)

axes[1].set_ylabel("Rate", fontsize=14)
axes[1].set_xlabel("Epoch", fontsize=14)
axes[1].plot(train_rate_results, label='trainable rate variable')
axes[1].plot(tensor_exact_value, label='exact rate')
axes[1].legend()
plt.show()
```


```python
# Making a function get_data which:
#   1) Fetches the 20 newsgroup dataset
#   2) Performs a word count on the articles and binarizes the result
#   3) Returns the data as a numpy matrix with the labels

def get_data(categories):

    newsgroups_train_data = fetch_20newsgroups(data_home='20_Newsgroup_Data/',
                                               subset='train', categories=categories)
    newsgroups_test_data = fetch_20newsgroups(data_home='20_Newsgroup_Data/',
                                              subset='test', categories=categories)

    n_documents = len(newsgroups_train_data['data'])
    count_vectorizer = CountVectorizer(input='content', binary=True,max_df=0.25, min_df=1.01/n_documents) 
    train_binary_bag_of_words = count_vectorizer.fit_transform(newsgroups_train_data['data']) 
    test_binary_bag_of_words = count_vectorizer.transform(newsgroups_test_data['data']) 

    return (train_binary_bag_of_words.todense(), newsgroups_train_data['target']),  (test_binary_bag_of_words.todense(), newsgroups_test_data['target'])

```


```python
# Defining a function to conduct laplace smoothing. This adds a base level of probability for a given feature
# to occur in every class.

def laplace_smoothing(labels, binary_data, n_classes):
    # Compute the parameter estimates (adjusted fraction of documents in class that contain word)
    n_words = binary_data.shape[1]
    alpha = 1 # parameters for Laplace smoothing
    theta = np.zeros([n_classes, n_words]) # stores parameter values - prob. word given class
    for c_k in range(n_classes): # 0, 1, ..., 19
        class_mask = (labels == c_k)
        N = class_mask.sum() # number of articles in class
        theta[c_k, :] = (binary_data[class_mask, :].sum(axis=0) + alpha)/(N + alpha*2)

    return theta
```


```python
# Function which computes the prior probability of every class based on frequency of occurence in 
# the dataset

def class_priors(n_classes, labels):
    counts = np.zeros(n_classes)
    for c_k in range(n_classes):
        counts[c_k] = np.sum(np.where(labels==c_k, 1, 0))
    priors = counts / np.sum(counts)
    print('The class priors are {}'.format(priors))
    return priors
```


```python
# Define function to get loss and grads

@tf.function
def get_loss_and_grads(x_train, dist):
  with tf.GradientTape() as tape:
    # watch the distribution's rate as trainable (optimizable) variable 
    tape.watch(dist.trainable_variables)
    loss = nll(x_train, dist)
    grads = tape.gradient(loss, dist.trainable_variables)
  return loss, grads
```


```python
# Now we learn the distribution using gradient tape

def make_distribution_withGT(data, labels, nb_classes):

    class_data = []
    train_vars = []
    distributions = []
    for c in range(nb_classes):
        train_vars.append(tf.Variable(initial_value=np.random.uniform(low=0.01, high =0.1, size=data.shape[-1])))
        distributions.append(tfd.Bernoulli(probs=train_vars[c]))
        class_mask = (labels == c)
        class_data.append(data[class_mask, :])

    for c_num in range(0,nb_classes):
        optimizer = tf.keras.optimizers.Adam()
        print('\n%-------------------%')
        print('Class ', c_num)
        print('%-------------------%')

        for i in range(0,100):  
          loss, grads = get_loss_and_grads(class_data[c_num], distributions[c_num])
          if(i%10==0):
            print("iter: {} loss: {}".format(i, loss))
          optimizer.apply_gradients(zip(grads, distributions[c_num].trainable_variables))
          # to get probabilities, the values have to be clipped
          eta = 1e-3
          clipped_probs = tf.clip_by_value(distributions[c_num].trainable_variables,
                                           clip_value_min=eta, clip_value_max=1)
          train_vars[c_num] = tf.squeeze(clipped_probs)       

    dist = tfd.Bernoulli(probs=train_vars)
    dist = tfd.Independent(dist,reinterpreted_batch_ndims=1)

    print(dist)

    return dist
```


```python
# Make the same Naive Bayes classifier we did last tutorial

categories = ['alt.atheism', 'talk.religion.misc', 'comp.graphics', 'sci.space']

(train_data, train_labels), (test_data, test_labels) = get_data(categories)

smoothed_counts = laplace_smoothing(labels=train_labels, binary_data=train_data, n_classes=len(categories))

priors = class_priors(n_classes=len(categories), labels=train_labels)
tf_dist = make_distribution(smoothed_counts)
```

    The class priors are [0.2359882  0.28711898 0.29154376 0.18534907]



```python
# Now train the distributions with gradient tape

GT_dist = make_distribution_withGT(data=train_data, labels=train_labels, nb_classes=4)
```

    
    %-------------------%
    Class  0
    %-------------------%
    iter: 0 loss: 0.07854119614389216
    iter: 10 loss: 0.06917998353150522
    iter: 20 loss: 0.06045106131063867
    iter: 30 loss: 0.05237378490540406
    iter: 40 loss: 0.044903314751887344
    iter: 50 loss: 0.03798992334946074
    iter: 60 loss: 0.03159033976746148
    iter: 70 loss: 0.025670404483880378
    iter: 80 loss: 0.02019417897334616
    iter: 90 loss: 0.01510219393151303
    
    %-------------------%
    Class  1
    %-------------------%
    iter: 0 loss: 0.07138453914297938
    iter: 10 loss: 0.06206154942546076
    iter: 20 loss: 0.0532934402166033
    iter: 30 loss: 0.04514443296537837
    iter: 40 loss: 0.037582781845049054
    iter: 50 loss: 0.030587127251298904
    iter: 60 loss: 0.02412468895748303
    iter: 70 loss: 0.018168529229501167
    iter: 80 loss: 0.012663131918426954
    iter: 90 loss: 0.007559713880148002
    
    %-------------------%
    Class  2
    %-------------------%
    iter: 0 loss: 0.07837568714981533
    iter: 10 loss: 0.06931577525772248
    iter: 20 loss: 0.06097502482341613
    iter: 30 loss: 0.05337264072228233
    iter: 40 loss: 0.046461185789963194
    iter: 50 loss: 0.04020043866568832
    iter: 60 loss: 0.03454797451079808
    iter: 70 loss: 0.029448241349536775
    iter: 80 loss: 0.02485642060230939
    iter: 90 loss: 0.02071159540040243
    
    %-------------------%
    Class  3
    %-------------------%
    iter: 0 loss: 0.0798414407992633
    iter: 10 loss: 0.07058749213552236
    iter: 20 loss: 0.06196382407201262
    iter: 30 loss: 0.05397472421575236
    iter: 40 loss: 0.04657014514841682
    iter: 50 loss: 0.039710932675118854
    iter: 60 loss: 0.03335621558676585
    iter: 70 loss: 0.027463418983289443
    iter: 80 loss: 0.021982574028941958
    iter: 90 loss: 0.016860185241183483
    tfp.distributions.Independent("IndependentBernoulli", batch_shape=[4], event_shape=[17495], dtype=int32)



```python
# Compare the two results

for dist in [GT_dist,tf_dist]:
    probabilities = []
    for sample, label in zip(test_data, test_labels):
        probabilities.append(predict_sample(dist, sample, priors))

    probabilities = np.asarray(probabilities)
    predicted_classes = np.argmax(probabilities, axis =-1)
    print('f1 ', f1_score(test_labels, predicted_classes, average='macro'))
```

    f1  0.8311797344967313
    f1  0.7848499112849504


9f7a8a3cba4bd8cc52f3c110b3b9ef31a00317c5
ipynb
Jupyter Notebook
Course3-Probabilistic_Deep_Learning_with_Tensorflow2/week1_distributions.ipynb
mella30/Probabilistic-Deep-Learning-with-TensorFlow-2
e9748316547d7f433632f4735990306d6e15da72
MIT
Course3-Probabilistic_Deep_Learning_with_Tensorflow2/week1_distributions.ipynb
mella30/Probabilistic-Deep-Learning-with-TensorFlow-2
e9748316547d7f433632f4735990306d6e15da72
MIT
Course3-Probabilistic_Deep_Learning_with_Tensorflow2/week1_distributions.ipynb
mella30/Probabilistic-Deep-Learning-with-TensorFlow-2
e9748316547d7f433632f4735990306d6e15da72
MIT
Qwen/Qwen-72B
1. YES
2. YES