*Sebastian Raschka*  
last modified: 03/31/2014

<hr>
I am really looking forward to your comments and suggestions to improve and extend this tutorial! Just send me a quick note   
via Twitter: [@rasbt](https://twitter.com/rasbt)  
or Email: [bluewoodtree@gmail.com](mailto:bluewoodtree@gmail.com)
<hr>

### Problem Category
- Statistical Pattern Recognition   
- Supervised Learning  
- Parametric Learning  
- Bayes Decision Theory  
- Univariate data  
- 2-class problem
- equal variances
- equal priors
- Gaussian model (2 parameters)
- No Risk function
<hr>

<p><a name="sections"></a>
<br></p>

# Sections


<p>&#8226; <a href="#given">Given information</a><br>
&#8226; <a href="#deriving_db">Deriving the decision boundary</a><br>
&#8226; <a href="#plotting_db">Plotting the class conditional densities, posterior probabilities, and decision boundary</a><br>
&#8226; <a href="#classify_rand">Classifying some random example data</a><br>
&#8226; <a href="#emp_err">Calculating the empirical error rate</a><br>

  

  
  



<hr>

<p><a name="given"></a>
<br></p>

## Given information:

[<a href="#sections">back to top</a>] <br>


####model: continuous univariate normal (Gaussian) model for the class-conditional densities


$ p(x | \omega_j) \sim N(\mu|\sigma^2) $

$ p(x | \omega_j) \sim \frac{1}{\sqrt{2\pi\sigma^2}} \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu}{\sigma}\bigg)^2 \bigg] } $


####Prior probabilities:

$ P(\omega_1) = P(\omega_1) = 0.5 $

#### Variances of the sample distributions

$ \sigma_1^2 = \sigma_2^2 = 1 $

#### Means of the sample distributions

$ \mu_1 = 4, \quad \mu_2 = 10 $


<br>

<p><a name="deriving_db"></a>
<br></p>

## Deriving the decision boundary
[<a href="#sections">back to top</a>] <br>
### Bayes' Rule:


$ P(\omega_j|x) = \frac{p(x|\omega_j) * P(\omega_j)}{p(x)} $


###Bayes' Decision Rule:

Decide $ \omega_1 $ if $ P(\omega_1|x) > P(\omega_2|x) $ else decide $ \omega_2 $.
<br>


\begin{equation}
\Rightarrow \frac{p(x|\omega_1) * P(\omega_1)}{p(x)} > \frac{p(x|\omega_2) * P(\omega_2)}{p(x)}
\end{equation} 

We can drop $ p(x) $ since it is just a scale factor.


$ \Rightarrow P(x|\omega_1) * P(\omega_1) > p(x|\omega_2) * P(\omega_2) $

$ \Rightarrow \frac{p(x|\omega_1)}{p(x|\omega_2)} > \frac{P(\omega_2)}{P(\omega_1)} $

$ \Rightarrow \frac{p(x|\omega_1)}{p(x|\omega_2)} > \frac{0.5}{0.5} $

$ \Rightarrow \frac{p(x|\omega_1)}{p(x|\omega_2)} > 1 $

$ \Rightarrow \frac{1}{\sqrt{2\pi\sigma_1^2}} \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu_1}{\sigma_1}\bigg)^2 \bigg] } > \frac{1}{\sqrt{2\pi\sigma_2^2}} \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu_2}{\sigma_2}\bigg)^2 \bigg] } $


Since we have equal variances, we can drop the first term completely.




$ Rightarrow \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu_1}{\sigma_1}\bigg)^2 \bigg] } >  \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu_2}{\sigma_2}\bigg)^2 \bigg] } \quad\quad \bigg| \;ln, \quad \mu_1 = 4, \quad \mu_2 = 10, \quad \sigma=1 $

$ \Rightarrow -\frac{1}{2} (x-4)^2  > -\frac{1}{2} (x-10)^2  \quad \bigg| \; \times(-2) $

$ \Rightarrow (x-4)^2  < (x-10)^2 $

$ \Rightarrow x^2 - 8x + 16 < x^2 - 20x + 100 $

$ \Rightarrow 12x < 84 $

$ \Rightarrow x < 7 $


<p><a name="plotting_db"></a>
<br></p>

## Plotting the class conditional densities, posterior probabilities, and decision boundary

[<a href="#sections">back to top</a>] <br>


```python
%pylab inline

import numpy as np
from matplotlib import pyplot as plt

def pdf(x, mu, sigma):
    """
    Calculates the normal distribution's probability density 
    function (PDF).  
        
    """
    term1 = 1.0 / ( math.sqrt(2*np.pi) * sigma )
    term2 = np.exp( -0.5 * ( (x-mu)/sigma )**2 )
    return term1 * term2

# generating some sample data
x = np.arange(0, 100, 0.05)

# probability density functions
pdf1 = pdf(x, mu=4, sigma=1)
pdf2 = pdf(x, mu=10, sigma=1)

# Class conditional densities (likelihoods)
plt.plot(x, pdf1)
plt.plot(x, pdf2)
plt.title('Class conditional densities (likelihoods)')
plt.ylabel('p(x)')
plt.xlabel('random variable x')
plt.legend(['p(x|w_1) ~ N(4,1)', 'p(x|w_2) ~ N(10,1)'], loc='upper right')
plt.ylim([0,0.5])
plt.xlim([0,20])
plt.show()
```


```python
def posterior(likelihood, prior):
    """
    Calculates the posterior probability (after Bayes Rule) without
    the scale factor p(x) (=evidence).  
        
    """
    return likelihood * prior

# probability density functions
posterior1 = posterior(pdf(x, mu=4, sigma=1), 0.5)
posterior2 = posterior(pdf(x, mu=10, sigma=1), 0.5)

# Class conditional densities (likelihoods)
plt.plot(x, posterior1)
plt.plot(x, posterior2)
plt.title('Posterior Probabilities w. Decision Boundary')
plt.ylabel('P(w)')
plt.xlabel('random variable x')
plt.legend(['P(w_1|x)', 'p(w_2|X)'], loc='upper right')
plt.ylim([0,0.5])
plt.xlim([0,20])
plt.axvline(7, color='r', alpha=0.8, linestyle=':', linewidth=2)
plt.annotate('R1', xy=(4, 0.3), xytext=(4, 0.3))
plt.annotate('R2', xy=(10, 0.3), xytext=(10, 0.3))
plt.show()
```

<p><a name="classify_rand"></a>
<br></p>

## Classifying some random example data

[<a href="#sections">back to top</a>] <br>

#### Note on generating univariate random data from a Normal Distribution

We can generate random samples drawn from a Normal distribution via the `np.random.randn()` function. Its default is a standard Normal distribution with $ \mu = 0 $ and $ \sigma^2 = 1 $. In order to draw random data from $ N(\mu, \sigma^2) $, we use   
`sigma * np.random.randn(...) + mu`


```python
# Parameters
mu_1 = 4
mu_2 = 10
sigma_1_sqr = 1
sigma_2_sqr = 1

# Generating 10 random samples drawn from a Normal Distribution for class 1 & 2
x1_samples = sigma_1_sqr**0.5 * np.random.randn(10) + mu_1
x2_samples = sigma_1_sqr**0.5 * np.random.randn(10) + mu_2
y = [0 for i in range(10)]

# Plotting sample data with a decision boundary

plt.scatter(x1_samples, y, marker='o', color='green', s=40, alpha=0.5)
plt.scatter(x2_samples, y, marker='^', color='blue', s=40, alpha=0.5)
plt.title('Classifying random example data from 2 classes')
plt.ylabel('P(x)')
plt.xlabel('random variable x')
plt.legend(['w_1', 'w_2'], loc='upper right')
plt.ylim([-0.1,0.1])
plt.xlim([0,20])
plt.axvline(7, color='r', alpha=0.8, linestyle=':', linewidth=2)
plt.annotate('R1', xy=(4, 0.05), xytext=(4, 0.05))
plt.annotate('R2', xy=(10, 0.05), xytext=(10, 0.05))
plt.show()
```

<p><a name="emp_err"></a>
<br></p>

## Calculating the empirical error rate

[<a href="#sections">back to top</a>] <br>


```python
w1_as_w2, w2_as_w1 = 0, 0
for x1,x2 in zip(x1_samples, x2_samples):
    if x1 >= 7:
        w1_as_w2 += 1
    if x2 < 7:
        w2_as_w1 += 1
        
emp_err =  (w1_as_w2 + w2_as_w1) / float(len(x1_samples) + len(x2_samples))
    
print('Empirical Error: {}%'.format(emp_err * 100))

```

    Empirical Error: 0.0%



```python

```


```python

```


```python

```


```python
test complete; Gopal
```

773047a42ee0208a1342da403b65b087a35b4bee
ipynb
Jupyter Notebook
tests/others/1_stat_superv_parametric.ipynb
gopala-kr/ds-notebooks
bc35430ecdd851f2ceab8f2437eec4d77cb59423
MIT
2019-05-10T09:16:23.000Z
2019-05-10T09:16:23.000Z
tests/others/1_stat_superv_parametric.ipynb
gopala-kr/ds-notebooks
bc35430ecdd851f2ceab8f2437eec4d77cb59423
MIT
tests/others/1_stat_superv_parametric.ipynb
gopala-kr/ds-notebooks
bc35430ecdd851f2ceab8f2437eec4d77cb59423
MIT
2019-10-14T07:30:18.000Z
2019-10-14T07:30:18.000Z
Qwen/Qwen-72B
1. YES
2. YES